2024-03-04T09:53:23.153 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T09:53:23.153 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900 --verbose --name vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi --owner scheduled_vshankar@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} -- /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/orig.config.yaml 2024-03-04T09:53:23.201 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T09:53:23.300 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900 branch: wip-vshankar-testing-20240304.042522 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} email: vshankar@redhat.com first_in_suite: false job_id: '7580900' kernel: client: sha1: distro flavor: default kdb: true sha1: 786939134d7eee4dc7a805245d3f595929af37d7 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240304.042522 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - Error loading MDS rank - missing journal object - Error recovering journal - error decoding table object - failed to read JournalPointer - Corrupt directory entry - Corrupt fnode header - corrupt sessionmap header - Corrupt dentry - Scrub error on inode - Metadata damage detected - MDS_READ_ONLY - force file system read-only - with standby daemon mds - MDS abort because newly corrupt dentry sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240304.042522 sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 owner: scheduled_vshankar@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3119 sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 sleep_before_teardown: 0 subset: 270/1024 suite: fs suite_branch: wip-vshankar-testing-20240304.042522 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBA7XaVI3qo8JUjP6Cs8BrbgZKbZvhNnSuf1DkTRSJ1Mvc+9SLi404KEQCk/cU3tb4mMjV3DbVDSXaavcQb4C3k= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDrBdhyWDFx2GvNg6H8LTIqPp5KPcNiv2djc6oiZNzNgrfV3Tc5cSISLGWsTtOPG0rdezwu3vs+c12rnYDTUIG4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_damage teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_08:26:39 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T09:53:23.300 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa; will attempt to use it 2024-03-04T09:53:23.301 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks 2024-03-04T09:53:23.301 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T09:53:23.303 INFO:teuthology.task.internal:Checking packages... 2024-03-04T09:53:23.327 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '717ce3cce8d0166e4e577faa8004e6f9cb4128c0' 2024-03-04T09:53:23.327 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T09:53:23.327 INFO:teuthology.packaging:ref: None 2024-03-04T09:53:23.327 INFO:teuthology.packaging:tag: None 2024-03-04T09:53:23.327 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240304.042522 2024-03-04T09:53:23.327 INFO:teuthology.packaging:sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T09:53:23.327 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240304.042522 2024-03-04T09:53:23.549 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1845.g717ce3cc 2024-03-04T09:53:23.550 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T09:53:23.556 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T09:53:23.556 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T09:53:23.576 INFO:teuthology.task.internal:Saving configuration 2024-03-04T09:53:23.593 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T09:53:23.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T09:53:23.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 09:48:08.685538', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBA7XaVI3qo8JUjP6Cs8BrbgZKbZvhNnSuf1DkTRSJ1Mvc+9SLi404KEQCk/cU3tb4mMjV3DbVDSXaavcQb4C3k='} 2024-03-04T09:53:23.657 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 09:48:08.684168', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDrBdhyWDFx2GvNg6H8LTIqPp5KPcNiv2djc6oiZNzNgrfV3Tc5cSISLGWsTtOPG0rdezwu3vs+c12rnYDTUIG4='} 2024-03-04T09:53:23.657 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T09:53:23.663 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-04T09:53:23.663 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-04T09:53:23.663 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T09:53:23.758 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa1ba141160>, signals=[15]) 2024-03-04T09:53:23.758 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T09:53:23.764 INFO:teuthology.task.internal:Opening connections... 2024-03-04T09:53:23.764 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T09:53:23.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:53:23.863 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T09:53:23.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:53:23.961 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T09:53:23.967 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-03-04T09:53:23.984 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-03-04T09:53:23.985 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="9" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="9" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el9" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:LOGO="fedora-logo-icon" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T09:53:24.041 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T09:53:24.042 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T09:53:24.042 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-03-04T09:53:24.073 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-03-04T09:53:24.092 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-03-04T09:53:24.092 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-03-04T09:53:24.148 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-03-04T09:53:24.149 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T09:53:24.149 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-03-04T09:53:24.149 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T09:53:24.149 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T09:53:24.149 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T09:53:24.149 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-03-04T09:53:24.189 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T09:53:24.199 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T09:53:24.204 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T09:53:24.204 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-03-04T09:53:24.207 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-03-04T09:53:24.223 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T09:53:24.229 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T09:53:24.229 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-03-04T09:53:24.263 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-03-04T09:53:24.381 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T09:53:24.454 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T09:53:24.468 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '786939134d7eee4dc7a805245d3f595929af37d7'} 2024-03-04T09:53:24.468 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-03-04T09:53:24.468 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-03-04T09:53:24.468 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T09:53:24.469 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T09:53:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:53:24.485 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-03-04T09:53:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:53:24.486 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-04T09:53:24.542 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-361.el9.x86_64 2024-03-04T09:53:24.542 INFO:teuthology.task.kernel:Running kernel on smithi099: 5.14.0-361.el9.x86_64 2024-03-04T09:53:24.543 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-03-04T09:53:24.544 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2024-03-04T09:53:24.544 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2024-03-04T09:53:24.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-03-04T09:53:27.233 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.7 MB 00:01 2024-03-04T09:53:27.866 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - AppStream 4.1 MB/s | 6.7 MB 00:01 2024-03-04T09:53:28.608 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-04T09:53:29.232 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-04T09:53:29.382 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-04T09:53:30.725 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.0 MB 00:01 2024-03-04T09:53:33.191 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 19 MB 00:02 2024-03-04T09:53:38.033 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.1 MB 00:01 2024-03-04T09:53:38.214 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-03-04T09:53:40.342 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-03-04T09:53:41.918 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-03-04T09:53:42.190 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-04T09:53:43.649 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2024-03-04T09:53:45.267 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-03-04T09:53:46.767 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-04T09:53:49.008 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T09:53:49.008 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T09:53:49.109 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-04T09:53:49.121 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-04T09:53:49.121 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-04T09:53:49.121 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-04T09:53:49.121 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-04T09:53:49.121 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-03-04T09:53:49.122 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:53:49.123 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 100 M 2024-03-04T09:53:49.123 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 123 M 2024-03-04T09:53:49.123 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-04T09:53:49.460 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 20 MB/s | 6.0 MB 00:00 2024-03-04T09:53:49.803 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 33 MB/s | 21 MB 00:00 2024-03-04T09:53:50.133 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-04T09:53:50.336 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 34 MB/s | 39 MB 00:01 2024-03-04T09:53:50.636 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 28 MB/s | 33 MB 00:01 2024-03-04T09:53:50.636 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-04T09:53:50.637 INFO:teuthology.orchestra.run.smithi162.stdout:Total 66 MB/s | 100 MB 00:01 2024-03-04T09:53:51.518 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-04T09:53:51.902 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-04T09:53:51.902 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-04T09:53:52.397 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T09:53:52.398 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T09:53:52.482 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-04T09:53:52.489 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-04T09:53:52.489 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-03-04T09:53:52.489 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-04T09:53:52.489 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-04T09:53:52.489 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout:Install 4 Packages 2024-03-04T09:53:52.490 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:53:52.491 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 100 M 2024-03-04T09:53:52.491 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 123 M 2024-03-04T09:53:52.491 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-04T09:53:53.289 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-04T09:53:53.289 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-04T09:53:54.292 INFO:teuthology.orchestra.run.smithi099.stdout:(1/4): kernel-core-5.14.0-427.el9.x86_64.rpm 12 MB/s | 21 MB 00:01 2024-03-04T09:53:54.368 INFO:teuthology.orchestra.run.smithi099.stdout:(2/4): kernel-5.14.0-427.el9.x86_64.rpm 3.3 MB/s | 6.0 MB 00:01 2024-03-04T09:53:55.742 INFO:teuthology.orchestra.run.smithi099.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 23 MB/s | 33 MB 00:01 2024-03-04T09:53:56.010 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-04T09:53:56.109 INFO:teuthology.orchestra.run.smithi099.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 11 MB/s | 39 MB 00:03 2024-03-04T09:53:56.113 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-04T09:53:56.113 INFO:teuthology.orchestra.run.smithi099.stdout:Total 27 MB/s | 100 MB 00:03 2024-03-04T09:53:56.659 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T09:53:56.793 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T09:53:56.993 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-04T09:53:57.362 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-04T09:53:57.362 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-04T09:53:58.692 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-04T09:53:58.692 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-04T09:53:59.576 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T09:53:59.753 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T09:54:01.682 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-04T09:54:02.320 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T09:54:02.469 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T09:54:04.482 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T09:54:04.612 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:05.264 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T09:54:05.449 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T09:54:09.375 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:10.183 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T09:54:10.392 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:15.321 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:38.706 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:38.707 INFO:teuthology.orchestra.run.smithi162.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T09:54:38.707 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:54:38.791 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:39.208 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:39.208 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T09:54:39.208 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T09:54:39.209 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T09:54:39.632 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:54:39.633 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-04T09:54:40.158 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T09:54:40.341 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T09:54:40.342 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T09:54:40.342 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T09:54:40.342 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T09:54:40.342 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T09:54:40.342 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2024-03-04T09:54:40.342 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T09:54:40.343 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T09:54:40.464 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T09:54:40.464 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T09:54:40.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-03-04T09:54:40.510 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T09:54:40.511 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T09:54:40.512 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-03-04T09:54:40.512 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T09:54:40.512 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T09:54:40.513 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T09:54:40.513 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-427.el9.x86_64 2024-03-04T09:54:40.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T09:54:41.108 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-03-04T09:54:42.495 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T09:54:42.521 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-03-04T09:54:42.523 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-04T09:54:42.538 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.1QTQ3ezqvk 2024-03-04T09:54:42.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1QTQ3ezqvk 2024-03-04T09:54:42.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.1QTQ3ezqvk 2024-03-04T09:54:42.745 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.1QTQ3ezqvk is 6KB 2024-03-04T09:54:42.767 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.1QTQ3ezqvk 2024-03-04T09:54:42.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T09:54:42.850 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T09:54:42.851 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T09:54:42.851 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T09:54:42.851 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T09:54:42.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T09:54:43.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-03-04T09:54:44.682 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:44.682 INFO:teuthology.orchestra.run.smithi099.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T09:54:44.682 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:54:44.766 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:45.215 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:45.215 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T09:54:45.215 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T09:54:45.215 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T09:54:45.630 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T09:54:45.631 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:54:45.631 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-04T09:54:46.147 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T09:54:46.297 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T09:54:46.298 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T09:54:46.298 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T09:54:46.298 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T09:54:46.298 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T09:54:46.298 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-04T09:54:46.298 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T09:54:46.299 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T09:54:46.419 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T09:54:46.420 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T09:54:46.420 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2024-03-04T09:54:46.453 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.06 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 68.el9 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Group : Unspecified 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 8264254 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : builder@centos.org 2024-03-04T09:54:46.454 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T09:54:46.455 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T09:54:46.456 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T09:54:46.456 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 5.14.0-427.el9.x86_64 2024-03-04T09:54:46.456 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T09:54:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2024-03-04T09:54:48.547 INFO:teuthology.orchestra.run.smithi099.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T09:54:48.587 INFO:teuthology.orchestra.run.smithi099.stderr:done 2024-03-04T09:54:48.589 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-03-04T09:54:48.644 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.iN4dNcNXiy 2024-03-04T09:54:48.645 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iN4dNcNXiy 2024-03-04T09:54:48.723 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.iN4dNcNXiy 2024-03-04T09:54:48.862 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.iN4dNcNXiy is 6KB 2024-03-04T09:54:48.914 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.iN4dNcNXiy 2024-03-04T09:54:48.928 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T09:54:48.993 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T09:54:48.993 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T09:54:48.993 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T09:54:48.993 INFO:teuthology.orchestra.run.smithi099.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T09:54:48.994 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T09:54:49.145 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2024-03-04T09:55:13.022 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-04T09:55:13.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-04T09:55:13.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:55:19.152 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-04T09:55:19.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-04T09:55:19.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:55:47.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-04T09:55:53.788 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-03-04T09:55:56.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-04T09:55:56.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:56:02.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-04T09:56:02.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:56:15.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-04T09:56:21.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-03-04T09:56:27.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-04T09:56:27.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:56:27.815 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-04T09:56:28.354 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-04T09:56:28.354 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.14.0-427.el9.x86_64"... 2024-03-04T09:56:28.355 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-04T09:56:28.368 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-427.el9.x86_64 2024-03-04T09:56:28.369 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T09:56:28.369 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T09:56:28.369 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T09:56:29.370 DEBUG:teuthology.parallel:result is None 2024-03-04T09:56:33.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-04T09:56:33.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T09:56:33.540 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-03-04T09:56:33.934 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-04T09:56:33.934 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-04T09:56:33.934 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-03-04T09:56:33.948 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-427.el9.x86_64 2024-03-04T09:56:33.948 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T09:56:33.948 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T09:56:33.948 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T09:56:34.949 DEBUG:teuthology.parallel:result is None 2024-03-04T09:56:34.949 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T09:56:34.957 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T09:56:34.957 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T09:56:34.960 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T09:56:34.987 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T09:56:34.995 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T09:56:35.004 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T09:56:35.005 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T09:56:35.014 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T09:56:35.185 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T09:56:35.191 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T09:56:35.191 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T09:56:35.193 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T09:56:35.270 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:56:35.316 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:56:35.333 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:56:35.374 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T09:56:35.376 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T09:56:35.383 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T09:56:35.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T09:56:35.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T09:56:35.430 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T09:56:35.439 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T09:56:35.439 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T09:56:35.453 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T09:56:35.479 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:56:35.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:56:35.595 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:56:35.685 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:56:35.719 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T09:56:35.719 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T09:56:35.786 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:56:35.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T09:56:35.878 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:56:35.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T09:56:36.002 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T09:56:36.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T09:56:36.071 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-03-04T09:56:36.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-03-04T09:56:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T09:56:36.302 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T09:56:36.709 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T09:56:36.723 INFO:teuthology.task.internal:Starting timer... 2024-03-04T09:56:36.723 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T09:56:36.734 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T09:56:36.743 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-04T09:56:36.744 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-03-04T09:56:36.810 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-03-04T09:56:36.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-03-04T09:56:36.869 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-03-04T09:56:36.871 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T09:56:36.872 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-03-04T09:56:36.895 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-03-04T09:56:36.895 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-03-04T09:56:36.911 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-03-04T09:56:36.911 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-03-04T09:56:36.912 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T09:56:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:56:36.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T09:56:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T09:56:36.989 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T09:56:36.989 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-03-04T09:56:37.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-03-04T09:56:37.088 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T09:56:37.103 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T09:56:37.126 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T09:56:37.128 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T10:00:51.495 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-03-04T10:00:51.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-04T10:00:51.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T10:00:51.649 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-03-04T10:00:51.727 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-03-04T10:00:51.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-04T10:00:51.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T10:00:51.831 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-04T10:00:51.913 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-04T10:00:51.913 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T10:00:52.074 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T10:00:52.074 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T10:00:52.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T10:00:52.079 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T10:00:52.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T10:00:52.106 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T10:00:52.110 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T10:00:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T10:00:52.126 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T10:00:52.152 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-03-04T10:00:52.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-03-04T10:00:52.165 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-03-04T10:00:52.170 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-03-04T10:00:52.182 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T10:00:52.185 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T10:00:52.198 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T10:00:52.202 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T10:00:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-03-04T10:00:52.254 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-04T10:00:52.256 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T10:00:52.256 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-03-04T10:00:52.256 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.256 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.256 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.256 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.258 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T10:00:52.258 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-04T10:00:52.258 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.258 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.258 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.258 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:00:52.258 INFO:teuthology.run_tasks:Running task install... 2024-03-04T10:00:52.281 DEBUG:teuthology.task.install:project ceph 2024-03-04T10:00:52.281 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'}} 2024-03-04T10:00:52.281 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'} 2024-03-04T10:00:52.281 INFO:teuthology.task.install:Using flavor: default 2024-03-04T10:00:52.290 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T10:00:52.290 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-04T10:00:52.290 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'tag': None, 'wait_for_package': False} 2024-03-04T10:00:52.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T10:00:52.293 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'tag': None, 'wait_for_package': False} 2024-03-04T10:00:52.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T10:00:52.537 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/ 2024-03-04T10:00:52.538 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1845.g717ce3cc 2024-03-04T10:00:52.555 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/ 2024-03-04T10:00:52.556 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1845.g717ce3cc 2024-03-04T10:00:52.679 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T10:00:52.680 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:00:52.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T10:00:52.709 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T10:00:52.709 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T10:00:52.746 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T10:00:52.746 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:00:52.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T10:00:52.778 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T10:00:52.779 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T10:00:52.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T10:00:52.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T10:00:52.864 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T10:00:52.930 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2024-03-04T10:00:52.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2024-03-04T10:00:52.934 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T10:00:53.002 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-03-04T10:00:53.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-03-04T10:00:53.181 INFO:teuthology.orchestra.run.smithi099.stdout:74 files removed 2024-03-04T10:00:53.209 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T10:00:53.252 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-03-04T10:00:53.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T10:00:53.854 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 250 kB/s | 82 kB 00:00 2024-03-04T10:00:53.937 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 242 kB/s | 82 kB 00:00 2024-03-04T10:00:54.221 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-04T10:00:54.263 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-04T10:00:54.614 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-04T10:00:54.647 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-04T10:00:55.016 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-04T10:00:55.090 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-04T10:00:56.624 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-04T10:00:56.625 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-04T10:00:58.423 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-03-04T10:00:58.531 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-03-04T10:01:00.541 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-04T10:01:03.206 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 19 MB 00:03 2024-03-04T10:01:05.600 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-04T10:01:07.276 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2024-03-04T10:01:07.749 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-04T10:01:09.321 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-03-04T10:01:09.550 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-04T10:01:11.434 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-03-04T10:01:14.185 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-04T10:01:16.300 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-04T10:01:16.755 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T10:01:16.756 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T10:01:16.768 INFO:teuthology.orchestra.run.smithi099.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T10:01:16.769 INFO:teuthology.orchestra.run.smithi099.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T10:01:16.770 INFO:teuthology.orchestra.run.smithi099.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T10:01:16.770 INFO:teuthology.orchestra.run.smithi099.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T10:01:16.772 INFO:teuthology.orchestra.run.smithi099.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T10:01:16.775 INFO:teuthology.orchestra.run.smithi099.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T10:01:16.776 INFO:teuthology.orchestra.run.smithi099.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T10:01:16.858 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-03-04T10:01:16.878 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-03-04T10:01:16.878 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 6.6 k 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.3 M 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 888 k 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 145 k 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 1.5 M 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 138 k 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 1.8 M 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 7.4 M 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 49 k 2024-03-04T10:01:16.879 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 9.8 M 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 47 M 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 264 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 749 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 220 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-shell x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 27 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-top noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 18 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 31 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 745 k 2024-03-04T10:01:16.880 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 127 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 162 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 320 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 298 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 99 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 85 k 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T10:01:16.881 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 170 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T10:01:16.882 INFO:teuthology.orchestra.run.smithi099.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.4 M 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T10:01:16.883 INFO:teuthology.orchestra.run.smithi099.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 20 M 2024-03-04T10:01:16.884 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 31 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 2.1 M 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 248 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 4.7 M 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 17 M 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 16 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 25 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T10:01:16.885 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 163 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 502 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.1 M 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T10:01:16.886 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T10:01:16.887 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 45 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 131 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T10:01:16.888 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T10:01:16.889 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T10:01:16.890 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T10:01:16.891 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T10:01:16.892 INFO:teuthology.orchestra.run.smithi099.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout:Installing weak dependencies: 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================================ 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout:Install 150 Packages 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 26 Packages 2024-03-04T10:01:16.893 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:01:16.894 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 223 M 2024-03-04T10:01:16.894 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-03-04T10:01:17.678 INFO:teuthology.orchestra.run.smithi099.stdout:(1/176): ceph-19.0.0-1845.g717ce3cc.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-04T10:01:17.995 INFO:teuthology.orchestra.run.smithi099.stdout:(2/176): ceph-fuse-19.0.0-1845.g717ce3cc.el9.x8 2.7 MB/s | 888 kB 00:00 2024-03-04T10:01:18.062 INFO:teuthology.orchestra.run.smithi099.stdout:(3/176): ceph-immutable-object-cache-19.0.0-184 2.1 MB/s | 145 kB 00:00 2024-03-04T10:01:18.437 INFO:teuthology.orchestra.run.smithi099.stdout:(4/176): ceph-mds-19.0.0-1845.g717ce3cc.el9.x86 5.7 MB/s | 2.1 MB 00:00 2024-03-04T10:01:18.612 INFO:teuthology.orchestra.run.smithi099.stdout:(5/176): ceph-base-19.0.0-1845.g717ce3cc.el9.x8 4.8 MB/s | 5.3 MB 00:01 2024-03-04T10:01:18.696 INFO:teuthology.orchestra.run.smithi099.stdout:(6/176): ceph-mgr-19.0.0-1845.g717ce3cc.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-03-04T10:01:18.856 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T10:01:18.857 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T10:01:18.871 INFO:teuthology.orchestra.run.smithi162.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T10:01:18.872 INFO:teuthology.orchestra.run.smithi162.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T10:01:18.873 INFO:teuthology.orchestra.run.smithi162.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T10:01:18.874 INFO:teuthology.orchestra.run.smithi162.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T10:01:18.876 INFO:teuthology.orchestra.run.smithi162.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T10:01:18.880 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T10:01:18.880 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T10:01:18.964 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-04T10:01:18.976 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-04T10:01:18.976 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-04T10:01:18.976 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 6.6 k 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.3 M 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 888 k 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 145 k 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 1.5 M 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 138 k 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 1.8 M 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 7.4 M 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 49 k 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 9.8 M 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 47 M 2024-03-04T10:01:18.977 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 264 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 749 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 220 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 27 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 18 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 31 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 745 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 127 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T10:01:18.978 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 162 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 320 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 298 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 99 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 85 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 170 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T10:01:18.979 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T10:01:18.980 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.4 M 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T10:01:18.981 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 20 M 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 31 k 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 2.1 M 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 248 k 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 4.7 M 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 17 M 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 16 k 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 25 k 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T10:01:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 163 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T10:01:18.983 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 502 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.1 M 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T10:01:18.984 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 45 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 131 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T10:01:18.985 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T10:01:18.986 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T10:01:18.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T10:01:18.988 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T10:01:18.989 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout:Install 150 Packages 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 26 Packages 2024-03-04T10:01:18.990 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:01:18.991 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 223 M 2024-03-04T10:01:18.991 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-04T10:01:19.154 INFO:teuthology.orchestra.run.smithi099.stdout:(7/176): ceph-mon-19.0.0-1845.g717ce3cc.el9.x86 8.7 MB/s | 4.7 MB 00:00 2024-03-04T10:01:20.221 INFO:teuthology.orchestra.run.smithi099.stdout:(8/176): ceph-radosgw-19.0.0-1845.g717ce3cc.el9 9.2 MB/s | 9.8 MB 00:01 2024-03-04T10:01:20.343 INFO:teuthology.orchestra.run.smithi162.stdout:(1/176): ceph-19.0.0-1845.g717ce3cc.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-04T10:01:20.413 INFO:teuthology.orchestra.run.smithi099.stdout:(9/176): ceph-common-19.0.0-1845.g717ce3cc.el9. 6.9 MB/s | 20 MB 00:02 2024-03-04T10:01:20.438 INFO:teuthology.orchestra.run.smithi099.stdout:(10/176): ceph-selinux-19.0.0-1845.g717ce3cc.el 116 kB/s | 25 kB 00:00 2024-03-04T10:01:20.640 INFO:teuthology.orchestra.run.smithi099.stdout:(11/176): ceph-osd-19.0.0-1845.g717ce3cc.el9.x8 8.7 MB/s | 17 MB 00:01 2024-03-04T10:01:20.664 INFO:teuthology.orchestra.run.smithi099.stdout:(12/176): cephfs-mirror-19.0.0-1845.g717ce3cc.e 976 kB/s | 220 kB 00:00 2024-03-04T10:01:20.707 INFO:teuthology.orchestra.run.smithi099.stdout:(13/176): cephfs-shell-19.0.0-1845.g717ce3cc.el 406 kB/s | 27 kB 00:00 2024-03-04T10:01:20.731 INFO:teuthology.orchestra.run.smithi099.stdout:(14/176): libcephfs-devel-19.0.0-1845.g717ce3cc 465 kB/s | 31 kB 00:00 2024-03-04T10:01:20.759 INFO:teuthology.orchestra.run.smithi162.stdout:(2/176): ceph-fuse-19.0.0-1845.g717ce3cc.el9.x8 2.1 MB/s | 888 kB 00:00 2024-03-04T10:01:20.799 INFO:teuthology.orchestra.run.smithi099.stdout:(15/176): libcephfs2-19.0.0-1845.g717ce3cc.el9. 7.9 MB/s | 745 kB 00:00 2024-03-04T10:01:20.829 INFO:teuthology.orchestra.run.smithi099.stdout:(16/176): libcephsqlite-19.0.0-1845.g717ce3cc.e 1.7 MB/s | 163 kB 00:00 2024-03-04T10:01:20.866 INFO:teuthology.orchestra.run.smithi099.stdout:(17/176): librados-devel-19.0.0-1845.g717ce3cc. 1.8 MB/s | 127 kB 00:00 2024-03-04T10:01:20.892 INFO:teuthology.orchestra.run.smithi162.stdout:(3/176): ceph-immutable-object-cache-19.0.0-184 1.1 MB/s | 145 kB 00:00 2024-03-04T10:01:20.910 INFO:teuthology.orchestra.run.smithi099.stdout:(18/176): libradosstriper1-19.0.0-1845.g717ce3c 6.5 MB/s | 502 kB 00:00 2024-03-04T10:01:21.000 INFO:teuthology.orchestra.run.smithi099.stdout:(19/176): python3-ceph-argparse-19.0.0-1845.g71 451 kB/s | 45 kB 00:00 2024-03-04T10:01:21.100 INFO:teuthology.orchestra.run.smithi099.stdout:(20/176): python3-ceph-common-19.0.0-1845.g717c 1.3 MB/s | 131 kB 00:00 2024-03-04T10:01:21.208 INFO:teuthology.orchestra.run.smithi099.stdout:(21/176): python3-cephfs-19.0.0-1845.g717ce3cc. 1.5 MB/s | 162 kB 00:00 2024-03-04T10:01:21.321 INFO:teuthology.orchestra.run.smithi099.stdout:(22/176): librgw2-19.0.0-1845.g717ce3cc.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-04T10:01:21.342 INFO:teuthology.orchestra.run.smithi099.stdout:(23/176): python3-rados-19.0.0-1845.g717ce3cc.e 2.3 MB/s | 320 kB 00:00 2024-03-04T10:01:21.384 INFO:teuthology.orchestra.run.smithi099.stdout:(24/176): python3-rbd-19.0.0-1845.g717ce3cc.el9 4.4 MB/s | 298 kB 00:00 2024-03-04T10:01:21.409 INFO:teuthology.orchestra.run.smithi099.stdout:(25/176): python3-rgw-19.0.0-1845.g717ce3cc.el9 1.4 MB/s | 99 kB 00:00 2024-03-04T10:01:21.452 INFO:teuthology.orchestra.run.smithi099.stdout:(26/176): rbd-fuse-19.0.0-1845.g717ce3cc.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-04T10:01:21.543 INFO:teuthology.orchestra.run.smithi099.stdout:(27/176): rbd-nbd-19.0.0-1845.g717ce3cc.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-04T10:01:21.643 INFO:teuthology.orchestra.run.smithi099.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1845.g 315 kB/s | 31 kB 00:00 2024-03-04T10:01:21.702 INFO:teuthology.orchestra.run.smithi099.stdout:(29/176): rbd-mirror-19.0.0-1845.g717ce3cc.el9. 11 MB/s | 3.1 MB 00:00 2024-03-04T10:01:21.728 INFO:teuthology.orchestra.run.smithi099.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1845.g717ce3c 1.6 MB/s | 138 kB 00:00 2024-03-04T10:01:21.953 INFO:teuthology.orchestra.run.smithi099.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1845.g717ce 7.1 MB/s | 1.8 MB 00:00 2024-03-04T10:01:22.061 INFO:teuthology.orchestra.run.smithi099.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1845.g71 2.2 MB/s | 248 kB 00:00 2024-03-04T10:01:22.128 INFO:teuthology.orchestra.run.smithi099.stdout:(33/176): ceph-mgr-rook-19.0.0-1845.g717ce3cc.e 743 kB/s | 49 kB 00:00 2024-03-04T10:01:22.203 INFO:teuthology.orchestra.run.smithi099.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1845.g7 220 kB/s | 16 kB 00:00 2024-03-04T10:01:22.260 INFO:teuthology.orchestra.run.smithi162.stdout:(4/176): ceph-mds-19.0.0-1845.g717ce3cc.el9.x86 1.5 MB/s | 2.1 MB 00:01 2024-03-04T10:01:22.320 INFO:teuthology.orchestra.run.smithi099.stdout:(35/176): ceph-volume-19.0.0-1845.g717ce3cc.el9 2.2 MB/s | 264 kB 00:00 2024-03-04T10:01:22.554 INFO:teuthology.orchestra.run.smithi099.stdout:(36/176): cephadm-19.0.0-1845.g717ce3cc.el9.noa 3.1 MB/s | 749 kB 00:00 2024-03-04T10:01:22.679 INFO:teuthology.orchestra.run.smithi099.stdout:(37/176): ceph-mgr-diskprediction-local-19.0.0- 7.8 MB/s | 7.4 MB 00:00 2024-03-04T10:01:22.705 INFO:teuthology.orchestra.run.smithi099.stdout:(38/176): cephfs-top-19.0.0-1845.g717ce3cc.el9. 121 kB/s | 18 kB 00:00 2024-03-04T10:01:22.930 INFO:teuthology.orchestra.run.smithi099.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-03-04T10:01:22.943 INFO:teuthology.orchestra.run.smithi162.stdout:(5/176): ceph-mgr-19.0.0-1845.g717ce3cc.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-03-04T10:01:22.988 INFO:teuthology.orchestra.run.smithi099.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 920 kB 00:00 2024-03-04T10:01:23.014 INFO:teuthology.orchestra.run.smithi099.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 106 kB 00:00 2024-03-04T10:01:23.039 INFO:teuthology.orchestra.run.smithi099.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 407 kB/s | 20 kB 00:00 2024-03-04T10:01:23.064 INFO:teuthology.orchestra.run.smithi099.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-04T10:01:23.090 INFO:teuthology.orchestra.run.smithi099.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-04T10:01:23.207 INFO:teuthology.orchestra.run.smithi099.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-04T10:01:23.232 INFO:teuthology.orchestra.run.smithi099.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 121 kB/s | 17 kB 00:00 2024-03-04T10:01:23.258 INFO:teuthology.orchestra.run.smithi099.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-04T10:01:23.274 INFO:teuthology.orchestra.run.smithi099.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-04T10:01:23.293 INFO:teuthology.orchestra.run.smithi162.stdout:(6/176): ceph-base-19.0.0-1845.g717ce3cc.el9.x8 1.6 MB/s | 5.3 MB 00:03 2024-03-04T10:01:23.300 INFO:teuthology.orchestra.run.smithi099.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-04T10:01:23.333 INFO:teuthology.orchestra.run.smithi099.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-04T10:01:23.359 INFO:teuthology.orchestra.run.smithi099.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-04T10:01:23.376 INFO:teuthology.orchestra.run.smithi099.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 597 kB/s | 24 kB 00:00 2024-03-04T10:01:23.409 INFO:teuthology.orchestra.run.smithi099.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 919 kB/s | 46 kB 00:00 2024-03-04T10:01:23.435 INFO:teuthology.orchestra.run.smithi099.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-04T10:01:23.460 INFO:teuthology.orchestra.run.smithi099.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-04T10:01:23.477 INFO:teuthology.orchestra.run.smithi099.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 816 kB/s | 34 kB 00:00 2024-03-04T10:01:23.619 INFO:teuthology.orchestra.run.smithi099.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-04T10:01:23.677 INFO:teuthology.orchestra.run.smithi099.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-03-04T10:01:23.719 INFO:teuthology.orchestra.run.smithi099.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-03-04T10:01:23.745 INFO:teuthology.orchestra.run.smithi099.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-04T10:01:23.770 INFO:teuthology.orchestra.run.smithi099.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-04T10:01:23.904 INFO:teuthology.orchestra.run.smithi099.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-04T10:01:23.937 INFO:teuthology.orchestra.run.smithi099.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-03-04T10:01:23.963 INFO:teuthology.orchestra.run.smithi099.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-04T10:01:23.969 INFO:teuthology.orchestra.run.smithi162.stdout:(7/176): ceph-common-19.0.0-1845.g717ce3cc.el9. 4.9 MB/s | 20 MB 00:04 2024-03-04T10:01:24.088 INFO:teuthology.orchestra.run.smithi099.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 38 MB/s | 5.8 MB 00:00 2024-03-04T10:01:24.113 INFO:teuthology.orchestra.run.smithi099.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 227 kB 00:00 2024-03-04T10:01:24.139 INFO:teuthology.orchestra.run.smithi099.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 490 kB/s | 24 kB 00:00 2024-03-04T10:01:24.164 INFO:teuthology.orchestra.run.smithi099.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-04T10:01:24.189 INFO:teuthology.orchestra.run.smithi099.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-04T10:01:24.214 INFO:teuthology.orchestra.run.smithi099.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-04T10:01:24.315 INFO:teuthology.orchestra.run.smithi099.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-04T10:01:24.340 INFO:teuthology.orchestra.run.smithi099.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-04T10:01:24.366 INFO:teuthology.orchestra.run.smithi099.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-04T10:01:24.391 INFO:teuthology.orchestra.run.smithi099.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-04T10:01:24.441 INFO:teuthology.orchestra.run.smithi099.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 548 kB/s | 41 kB 00:00 2024-03-04T10:01:24.466 INFO:teuthology.orchestra.run.smithi099.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-04T10:01:24.492 INFO:teuthology.orchestra.run.smithi099.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-04T10:01:24.502 INFO:teuthology.orchestra.run.smithi162.stdout:(8/176): ceph-mon-19.0.0-1845.g717ce3cc.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-04T10:01:24.517 INFO:teuthology.orchestra.run.smithi099.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-04T10:01:24.542 INFO:teuthology.orchestra.run.smithi099.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-04T10:01:24.563 INFO:teuthology.orchestra.run.smithi162.stdout:(9/176): ceph-selinux-19.0.0-1845.g717ce3cc.el9 418 kB/s | 25 kB 00:00 2024-03-04T10:01:24.570 INFO:teuthology.orchestra.run.smithi099.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-04T10:01:24.693 INFO:teuthology.orchestra.run.smithi099.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 315 kB/s | 39 kB 00:00 2024-03-04T10:01:24.927 INFO:teuthology.orchestra.run.smithi099.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-04T10:01:24.952 INFO:teuthology.orchestra.run.smithi099.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-04T10:01:24.977 INFO:teuthology.orchestra.run.smithi099.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-04T10:01:24.994 INFO:teuthology.orchestra.run.smithi099.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 813 kB/s | 34 kB 00:00 2024-03-04T10:01:25.019 INFO:teuthology.orchestra.run.smithi099.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-04T10:01:25.030 INFO:teuthology.orchestra.run.smithi162.stdout:(10/176): ceph-radosgw-19.0.0-1845.g717ce3cc.el 9.2 MB/s | 9.8 MB 00:01 2024-03-04T10:01:25.045 INFO:teuthology.orchestra.run.smithi099.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-04T10:01:25.070 INFO:teuthology.orchestra.run.smithi099.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-04T10:01:25.080 INFO:teuthology.orchestra.run.smithi162.stdout:(11/176): cephfs-mirror-19.0.0-1845.g717ce3cc.e 4.3 MB/s | 220 kB 00:00 2024-03-04T10:01:25.096 INFO:teuthology.orchestra.run.smithi099.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-04T10:01:25.120 INFO:teuthology.orchestra.run.smithi099.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-04T10:01:25.130 INFO:teuthology.orchestra.run.smithi162.stdout:(12/176): cephfs-shell-19.0.0-1845.g717ce3cc.el 544 kB/s | 27 kB 00:00 2024-03-04T10:01:25.146 INFO:teuthology.orchestra.run.smithi099.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-04T10:01:25.171 INFO:teuthology.orchestra.run.smithi099.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-04T10:01:25.196 INFO:teuthology.orchestra.run.smithi099.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-04T10:01:25.197 INFO:teuthology.orchestra.run.smithi162.stdout:(13/176): libcephfs-devel-19.0.0-1845.g717ce3cc 469 kB/s | 31 kB 00:00 2024-03-04T10:01:25.230 INFO:teuthology.orchestra.run.smithi099.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-04T10:01:25.255 INFO:teuthology.orchestra.run.smithi099.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-03-04T10:01:25.272 INFO:teuthology.orchestra.run.smithi099.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-04T10:01:25.289 INFO:teuthology.orchestra.run.smithi162.stdout:(14/176): libcephfs2-19.0.0-1845.g717ce3cc.el9. 8.0 MB/s | 745 kB 00:00 2024-03-04T10:01:25.297 INFO:teuthology.orchestra.run.smithi099.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-04T10:01:25.331 INFO:teuthology.orchestra.run.smithi099.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-04T10:01:25.339 INFO:teuthology.orchestra.run.smithi162.stdout:(15/176): libcephsqlite-19.0.0-1845.g717ce3cc.e 3.2 MB/s | 163 kB 00:00 2024-03-04T10:01:25.357 INFO:teuthology.orchestra.run.smithi099.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-04T10:01:25.389 INFO:teuthology.orchestra.run.smithi162.stdout:(16/176): librados-devel-19.0.0-1845.g717ce3cc. 2.5 MB/s | 127 kB 00:00 2024-03-04T10:01:25.390 INFO:teuthology.orchestra.run.smithi099.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.1 MB/s | 124 kB 00:00 2024-03-04T10:01:25.416 INFO:teuthology.orchestra.run.smithi099.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 1.9 MB/s | 113 kB 00:00 2024-03-04T10:01:25.441 INFO:teuthology.orchestra.run.smithi099.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-04T10:01:25.473 INFO:teuthology.orchestra.run.smithi162.stdout:(17/176): libradosstriper1-19.0.0-1845.g717ce3c 5.9 MB/s | 502 kB 00:00 2024-03-04T10:01:25.875 INFO:teuthology.orchestra.run.smithi099.stdout:(103/176): ceph-test-19.0.0-1845.g717ce3cc.el9. 8.7 MB/s | 47 MB 00:05 2024-03-04T10:01:25.890 INFO:teuthology.orchestra.run.smithi162.stdout:(18/176): librgw2-19.0.0-1845.g717ce3cc.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-04T10:01:25.932 INFO:teuthology.orchestra.run.smithi162.stdout:(19/176): python3-ceph-argparse-19.0.0-1845.g71 1.1 MB/s | 45 kB 00:00 2024-03-04T10:01:25.942 INFO:teuthology.orchestra.run.smithi099.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 45 kB/s | 22 kB 00:00 2024-03-04T10:01:25.982 INFO:teuthology.orchestra.run.smithi162.stdout:(20/176): python3-ceph-common-19.0.0-1845.g717c 2.6 MB/s | 131 kB 00:00 2024-03-04T10:01:26.036 INFO:teuthology.orchestra.run.smithi162.stdout:(21/176): python3-cephfs-19.0.0-1845.g717ce3cc. 3.2 MB/s | 162 kB 00:00 2024-03-04T10:01:26.084 INFO:teuthology.orchestra.run.smithi099.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 368 kB/s | 245 kB 00:00 2024-03-04T10:01:26.133 INFO:teuthology.orchestra.run.smithi162.stdout:(22/176): python3-rados-19.0.0-1845.g717ce3cc.e 5.4 MB/s | 320 kB 00:00 2024-03-04T10:01:26.134 INFO:teuthology.orchestra.run.smithi099.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-03-04T10:01:26.150 INFO:teuthology.orchestra.run.smithi162.stdout:(23/176): python3-rbd-19.0.0-1845.g717ce3cc.el9 5.0 MB/s | 298 kB 00:00 2024-03-04T10:01:26.191 INFO:teuthology.orchestra.run.smithi162.stdout:(24/176): python3-rgw-19.0.0-1845.g717ce3cc.el9 2.3 MB/s | 99 kB 00:00 2024-03-04T10:01:26.242 INFO:teuthology.orchestra.run.smithi162.stdout:(25/176): rbd-fuse-19.0.0-1845.g717ce3cc.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-04T10:01:26.376 INFO:teuthology.orchestra.run.smithi099.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-04T10:01:26.517 INFO:teuthology.orchestra.run.smithi162.stdout:(26/176): rbd-mirror-19.0.0-1845.g717ce3cc.el9. 11 MB/s | 3.1 MB 00:00 2024-03-04T10:01:26.567 INFO:teuthology.orchestra.run.smithi162.stdout:(27/176): rbd-nbd-19.0.0-1845.g717ce3cc.el9.x86 3.3 MB/s | 170 kB 00:00 2024-03-04T10:01:26.578 INFO:teuthology.orchestra.run.smithi099.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 865 kB/s | 548 kB 00:00 2024-03-04T10:01:26.610 INFO:teuthology.orchestra.run.smithi099.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 588 kB/s | 309 kB 00:00 2024-03-04T10:01:26.618 INFO:teuthology.orchestra.run.smithi162.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1845.g 631 kB/s | 31 kB 00:00 2024-03-04T10:01:26.668 INFO:teuthology.orchestra.run.smithi162.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1845.g717ce3c 2.7 MB/s | 138 kB 00:00 2024-03-04T10:01:26.835 INFO:teuthology.orchestra.run.smithi099.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 100 kB/s | 26 kB 00:00 2024-03-04T10:01:26.852 INFO:teuthology.orchestra.run.smithi162.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1845.g717ce 9.7 MB/s | 1.8 MB 00:00 2024-03-04T10:01:26.952 INFO:teuthology.orchestra.run.smithi099.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-04T10:01:27.202 INFO:teuthology.orchestra.run.smithi099.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 184 kB/s | 67 kB 00:00 2024-03-04T10:01:27.552 INFO:teuthology.orchestra.run.smithi162.stdout:(31/176): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-04T10:01:27.610 INFO:teuthology.orchestra.run.smithi162.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1845.g71 4.2 MB/s | 248 kB 00:00 2024-03-04T10:01:27.660 INFO:teuthology.orchestra.run.smithi162.stdout:(33/176): ceph-mgr-rook-19.0.0-1845.g717ce3cc.e 989 kB/s | 49 kB 00:00 2024-03-04T10:01:27.711 INFO:teuthology.orchestra.run.smithi162.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1845.g7 329 kB/s | 16 kB 00:00 2024-03-04T10:01:27.769 INFO:teuthology.orchestra.run.smithi162.stdout:(35/176): ceph-volume-19.0.0-1845.g717ce3cc.el9 4.4 MB/s | 264 kB 00:00 2024-03-04T10:01:27.870 INFO:teuthology.orchestra.run.smithi162.stdout:(36/176): cephadm-19.0.0-1845.g717ce3cc.el9.noa 7.3 MB/s | 749 kB 00:00 2024-03-04T10:01:27.920 INFO:teuthology.orchestra.run.smithi162.stdout:(37/176): cephfs-top-19.0.0-1845.g717ce3cc.el9. 365 kB/s | 18 kB 00:00 2024-03-04T10:01:27.977 INFO:teuthology.orchestra.run.smithi099.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 147 kB/s | 151 kB 00:01 2024-03-04T10:01:28.064 INFO:teuthology.orchestra.run.smithi099.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 976 kB/s | 838 kB 00:00 2024-03-04T10:01:28.079 INFO:teuthology.orchestra.run.smithi162.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-04T10:01:28.137 INFO:teuthology.orchestra.run.smithi162.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 920 kB 00:00 2024-03-04T10:01:28.179 INFO:teuthology.orchestra.run.smithi162.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-04T10:01:28.213 INFO:teuthology.orchestra.run.smithi162.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 611 kB/s | 20 kB 00:00 2024-03-04T10:01:28.255 INFO:teuthology.orchestra.run.smithi162.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-04T10:01:28.300 INFO:teuthology.orchestra.run.smithi162.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 759 kB/s | 32 kB 00:00 2024-03-04T10:01:28.422 INFO:teuthology.orchestra.run.smithi162.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-04T10:01:28.444 INFO:teuthology.orchestra.run.smithi099.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 113 kB/s | 43 kB 00:00 2024-03-04T10:01:28.536 INFO:teuthology.orchestra.run.smithi099.stdout:(116/176): libarrow-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 4.4 MB 00:02 2024-03-04T10:01:28.589 INFO:teuthology.orchestra.run.smithi162.stdout:(45/176): ceph-osd-19.0.0-1845.g717ce3cc.el9.x8 3.2 MB/s | 17 MB 00:05 2024-03-04T10:01:28.614 INFO:teuthology.orchestra.run.smithi162.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 89 kB/s | 17 kB 00:00 2024-03-04T10:01:28.648 INFO:teuthology.orchestra.run.smithi162.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 173 kB/s | 10 kB 00:00 2024-03-04T10:01:28.682 INFO:teuthology.orchestra.run.smithi162.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-04T10:01:28.698 INFO:teuthology.orchestra.run.smithi162.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-04T10:01:28.732 INFO:teuthology.orchestra.run.smithi162.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-04T10:01:28.787 INFO:teuthology.orchestra.run.smithi099.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 94 kB/s | 32 kB 00:00 2024-03-04T10:01:28.812 INFO:teuthology.orchestra.run.smithi099.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 54 kB/s | 15 kB 00:00 2024-03-04T10:01:28.849 INFO:teuthology.orchestra.run.smithi162.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 578 kB 00:00 2024-03-04T10:01:28.875 INFO:teuthology.orchestra.run.smithi162.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 173 kB/s | 24 kB 00:00 2024-03-04T10:01:28.900 INFO:teuthology.orchestra.run.smithi162.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-04T10:01:28.933 INFO:teuthology.orchestra.run.smithi162.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-04T10:01:28.959 INFO:teuthology.orchestra.run.smithi162.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-04T10:01:28.992 INFO:teuthology.orchestra.run.smithi162.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-04T10:01:29.159 INFO:teuthology.orchestra.run.smithi162.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-04T10:01:29.218 INFO:teuthology.orchestra.run.smithi162.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-04T10:01:29.245 INFO:teuthology.orchestra.run.smithi099.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 375 kB/s | 172 kB 00:00 2024-03-04T10:01:29.260 INFO:teuthology.orchestra.run.smithi162.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-03-04T10:01:29.293 INFO:teuthology.orchestra.run.smithi162.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-04T10:01:29.304 INFO:teuthology.orchestra.run.smithi099.stdout:(120/176): python3-asyncssh-2.13.2-4.el9.noarch 413 kB/s | 548 kB 00:01 2024-03-04T10:01:29.327 INFO:teuthology.orchestra.run.smithi162.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-04T10:01:29.362 INFO:teuthology.orchestra.run.smithi099.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 651 kB/s | 358 kB 00:00 2024-03-04T10:01:29.527 INFO:teuthology.orchestra.run.smithi162.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 24 MB/s | 5.7 MB 00:00 2024-03-04T10:01:29.561 INFO:teuthology.orchestra.run.smithi162.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-04T10:01:29.586 INFO:teuthology.orchestra.run.smithi162.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-04T10:01:29.605 INFO:teuthology.orchestra.run.smithi099.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 117 kB/s | 35 kB 00:00 2024-03-04T10:01:29.828 INFO:teuthology.orchestra.run.smithi162.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-04T10:01:29.862 INFO:teuthology.orchestra.run.smithi162.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 823 kB/s | 227 kB 00:00 2024-03-04T10:01:29.887 INFO:teuthology.orchestra.run.smithi162.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 419 kB/s | 24 kB 00:00 2024-03-04T10:01:29.929 INFO:teuthology.orchestra.run.smithi162.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 271 kB/s | 18 kB 00:00 2024-03-04T10:01:29.955 INFO:teuthology.orchestra.run.smithi162.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-04T10:01:29.963 INFO:teuthology.orchestra.run.smithi099.stdout:(123/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 423 kB/s | 303 kB 00:00 2024-03-04T10:01:29.988 INFO:teuthology.orchestra.run.smithi162.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 542 kB/s | 32 kB 00:00 2024-03-04T10:01:30.188 INFO:teuthology.orchestra.run.smithi162.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-04T10:01:30.205 INFO:teuthology.orchestra.run.smithi162.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-04T10:01:30.221 INFO:teuthology.orchestra.run.smithi099.stdout:(124/176): python3-google-auth-2.28.0-1.el9.noa 236 kB/s | 203 kB 00:00 2024-03-04T10:01:30.231 INFO:teuthology.orchestra.run.smithi162.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-04T10:01:30.256 INFO:teuthology.orchestra.run.smithi162.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-04T10:01:30.281 INFO:teuthology.orchestra.run.smithi162.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-04T10:01:30.306 INFO:teuthology.orchestra.run.smithi162.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-04T10:01:30.323 INFO:teuthology.orchestra.run.smithi162.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-04T10:01:30.365 INFO:teuthology.orchestra.run.smithi162.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-04T10:01:30.391 INFO:teuthology.orchestra.run.smithi162.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 706 kB/s | 47 kB 00:00 2024-03-04T10:01:30.405 INFO:teuthology.orchestra.run.smithi099.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-03-04T10:01:30.424 INFO:teuthology.orchestra.run.smithi162.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 732 kB/s | 43 kB 00:00 2024-03-04T10:01:30.681 INFO:teuthology.orchestra.run.smithi099.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 64 kB/s | 18 kB 00:00 2024-03-04T10:01:30.789 INFO:teuthology.orchestra.run.smithi099.stdout:(127/176): python3-grpcio-1.46.7-10.el9.x86_64. 1.7 MB/s | 2.0 MB 00:01 2024-03-04T10:01:30.814 INFO:teuthology.orchestra.run.smithi099.stdout:(128/176): python3-grpcio-tools-1.46.7-10.el9.x 170 kB/s | 144 kB 00:00 2024-03-04T10:01:30.899 INFO:teuthology.orchestra.run.smithi162.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 81 kB/s | 39 kB 00:00 2024-03-04T10:01:30.981 INFO:teuthology.orchestra.run.smithi099.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 77 kB/s | 23 kB 00:00 2024-03-04T10:01:31.065 INFO:teuthology.orchestra.run.smithi099.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 71 kB/s | 19 kB 00:00 2024-03-04T10:01:31.090 INFO:teuthology.orchestra.run.smithi099.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 71 kB/s | 20 kB 00:00 2024-03-04T10:01:31.416 INFO:teuthology.orchestra.run.smithi099.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-03-04T10:01:31.443 INFO:teuthology.orchestra.run.smithi099.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 225 kB/s | 79 kB 00:00 2024-03-04T10:01:31.586 INFO:teuthology.orchestra.run.smithi162.stdout:(82/176): ceph-test-19.0.0-1845.g717ce3cc.el9.x 6.8 MB/s | 47 MB 00:07 2024-03-04T10:01:31.734 INFO:teuthology.orchestra.run.smithi162.stdout:(83/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-03-04T10:01:31.763 INFO:teuthology.orchestra.run.smithi162.stdout:(84/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 48 kB/s | 41 kB 00:00 2024-03-04T10:01:31.784 INFO:teuthology.orchestra.run.smithi099.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 161 kB/s | 58 kB 00:00 2024-03-04T10:01:31.785 INFO:teuthology.orchestra.run.smithi162.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 215 kB 00:00 2024-03-04T10:01:31.812 INFO:teuthology.orchestra.run.smithi162.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 683 kB/s | 34 kB 00:00 2024-03-04T10:01:31.836 INFO:teuthology.orchestra.run.smithi162.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 582 kB/s | 43 kB 00:00 2024-03-04T10:01:31.861 INFO:teuthology.orchestra.run.smithi162.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-04T10:01:31.885 INFO:teuthology.orchestra.run.smithi162.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 756 kB/s | 56 kB 00:00 2024-03-04T10:01:31.910 INFO:teuthology.orchestra.run.smithi162.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-04T10:01:31.935 INFO:teuthology.orchestra.run.smithi162.stdout:(91/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-03-04T10:01:31.963 INFO:teuthology.orchestra.run.smithi162.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 583 kB/s | 44 kB 00:00 2024-03-04T10:01:31.974 INFO:teuthology.orchestra.run.smithi099.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 511 kB/s | 272 kB 00:00 2024-03-04T10:01:31.986 INFO:teuthology.orchestra.run.smithi162.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-04T10:01:32.011 INFO:teuthology.orchestra.run.smithi162.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 3.7 MB/s | 186 kB 00:00 2024-03-04T10:01:32.034 INFO:teuthology.orchestra.run.smithi099.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 63 kB/s | 16 kB 00:00 2024-03-04T10:01:32.070 INFO:teuthology.orchestra.run.smithi162.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 802 kB 00:00 2024-03-04T10:01:32.104 INFO:teuthology.orchestra.run.smithi162.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 194 kB 00:00 2024-03-04T10:01:32.129 INFO:teuthology.orchestra.run.smithi162.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-04T10:01:32.154 INFO:teuthology.orchestra.run.smithi162.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-04T10:01:32.179 INFO:teuthology.orchestra.run.smithi162.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-04T10:01:32.192 INFO:teuthology.orchestra.run.smithi099.stdout:(137/176): python3-kubernetes-26.1.0-2.el9.noar 864 kB/s | 1.0 MB 00:01 2024-03-04T10:01:32.238 INFO:teuthology.orchestra.run.smithi162.stdout:(100/176): python3-cryptography-36.0.1-4.el9.x8 11 MB/s | 1.1 MB 00:00 2024-03-04T10:01:32.263 INFO:teuthology.orchestra.run.smithi162.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.1 MB/s | 124 kB 00:00 2024-03-04T10:01:32.288 INFO:teuthology.orchestra.run.smithi162.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.0 MB/s | 113 kB 00:00 2024-03-04T10:01:32.293 INFO:teuthology.orchestra.run.smithi099.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 109 kB/s | 28 kB 00:00 2024-03-04T10:01:32.314 INFO:teuthology.orchestra.run.smithi162.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.4 MB/s | 187 kB 00:00 2024-03-04T10:01:32.317 INFO:teuthology.orchestra.run.smithi099.stdout:(139/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 262 kB/s | 90 kB 00:00 2024-03-04T10:01:32.460 INFO:teuthology.orchestra.run.smithi099.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-03-04T10:01:32.497 INFO:teuthology.orchestra.run.smithi162.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 107 kB/s | 22 kB 00:00 2024-03-04T10:01:32.668 INFO:teuthology.orchestra.run.smithi099.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2024-03-04T10:01:32.718 INFO:teuthology.orchestra.run.smithi099.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 138 kB/s | 36 kB 00:00 2024-03-04T10:01:32.743 INFO:teuthology.orchestra.run.smithi099.stdout:(143/176): python3-routes-2.5.1-5.el9.noarch.rp 418 kB/s | 188 kB 00:00 2024-03-04T10:01:32.914 INFO:teuthology.orchestra.run.smithi162.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 393 kB/s | 245 kB 00:00 2024-03-04T10:01:33.019 INFO:teuthology.orchestra.run.smithi099.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 145 kB/s | 51 kB 00:00 2024-03-04T10:01:33.086 INFO:teuthology.orchestra.run.smithi099.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 262 kB/s | 90 kB 00:00 2024-03-04T10:01:33.236 INFO:teuthology.orchestra.run.smithi099.stdout:(146/176): python3-webob-1.8.7-6.el9.noarch.rpm 444 kB/s | 230 kB 00:00 2024-03-04T10:01:33.256 INFO:teuthology.orchestra.run.smithi162.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 723 kB/s | 548 kB 00:00 2024-03-04T10:01:33.345 INFO:teuthology.orchestra.run.smithi099.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 77 kB/s | 20 kB 00:00 2024-03-04T10:01:33.364 INFO:teuthology.orchestra.run.smithi162.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-03-04T10:01:33.589 INFO:teuthology.orchestra.run.smithi162.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 457 kB/s | 309 kB 00:00 2024-03-04T10:01:33.698 INFO:teuthology.orchestra.run.smithi162.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-04T10:01:33.790 INFO:teuthology.orchestra.run.smithi162.stdout:(110/176): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-04T10:01:33.815 INFO:teuthology.orchestra.run.smithi162.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-04T10:01:33.966 INFO:teuthology.orchestra.run.smithi162.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 384 kB/s | 67 kB 00:00 2024-03-04T10:01:33.991 INFO:teuthology.orchestra.run.smithi162.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 861 kB/s | 151 kB 00:00 2024-03-04T10:01:34.053 INFO:teuthology.orchestra.run.smithi099.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 413 kB/s | 427 kB 00:01 2024-03-04T10:01:34.083 INFO:teuthology.orchestra.run.smithi162.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-04T10:01:34.170 INFO:teuthology.orchestra.run.smithi099.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 205 kB/s | 192 kB 00:00 2024-03-04T10:01:34.191 INFO:teuthology.orchestra.run.smithi162.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-03-04T10:01:34.216 INFO:teuthology.orchestra.run.smithi162.stdout:(116/176): python3-asyncssh-2.13.2-4.el9.noarch 2.4 MB/s | 548 kB 00:00 2024-03-04T10:01:34.300 INFO:teuthology.orchestra.run.smithi162.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-04T10:01:34.326 INFO:teuthology.orchestra.run.smithi162.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-04T10:01:34.370 INFO:teuthology.orchestra.run.smithi099.stdout:(150/176): librados2-19.0.0-1845.g717ce3cc.el9. 11 MB/s | 3.4 MB 00:00 2024-03-04T10:01:34.404 INFO:teuthology.orchestra.run.smithi099.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-03-04T10:01:34.409 INFO:teuthology.orchestra.run.smithi162.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-04T10:01:34.443 INFO:teuthology.orchestra.run.smithi162.stdout:(120/176): protobuf-compiler-3.14.0-13.el9.x86_ 405 kB/s | 863 kB 00:02 2024-03-04T10:01:34.468 INFO:teuthology.orchestra.run.smithi162.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-03-04T10:01:34.488 INFO:teuthology.orchestra.run.smithi099.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 27 MB/s | 2.3 MB 00:00 2024-03-04T10:01:34.518 INFO:teuthology.orchestra.run.smithi162.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-04T10:01:34.563 INFO:teuthology.orchestra.run.smithi099.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-04T10:01:34.585 INFO:teuthology.orchestra.run.smithi162.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-03-04T10:01:34.596 INFO:teuthology.orchestra.run.smithi099.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-04T10:01:34.639 INFO:teuthology.orchestra.run.smithi099.stdout:(155/176): thrift-0.15.0-2.el9.x86_64.rpm 1.2 MB/s | 1.6 MB 00:01 2024-03-04T10:01:34.664 INFO:teuthology.orchestra.run.smithi099.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 3.6 MB/s | 247 kB 00:00 2024-03-04T10:01:34.669 INFO:teuthology.orchestra.run.smithi162.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-04T10:01:34.681 INFO:teuthology.orchestra.run.smithi099.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2024-03-04T10:01:34.694 INFO:teuthology.orchestra.run.smithi162.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-04T10:01:34.706 INFO:teuthology.orchestra.run.smithi099.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 849 kB/s | 35 kB 00:00 2024-03-04T10:01:34.723 INFO:teuthology.orchestra.run.smithi099.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-04T10:01:34.748 INFO:teuthology.orchestra.run.smithi099.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-04T10:01:34.778 INFO:teuthology.orchestra.run.smithi162.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-04T10:01:34.782 INFO:teuthology.orchestra.run.smithi099.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-04T10:01:34.803 INFO:teuthology.orchestra.run.smithi162.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-03-04T10:01:34.807 INFO:teuthology.orchestra.run.smithi099.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 7.4 MB/s | 433 kB 00:00 2024-03-04T10:01:34.832 INFO:teuthology.orchestra.run.smithi099.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-03-04T10:01:34.858 INFO:teuthology.orchestra.run.smithi099.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-04T10:01:34.883 INFO:teuthology.orchestra.run.smithi099.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-03-04T10:01:34.887 INFO:teuthology.orchestra.run.smithi162.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-04T10:01:34.908 INFO:teuthology.orchestra.run.smithi099.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-04T10:01:34.912 INFO:teuthology.orchestra.run.smithi162.stdout:(129/176): python3-colorama-0.4.4-12.el9.noarch 75 kB/s | 35 kB 00:00 2024-03-04T10:01:34.925 INFO:teuthology.orchestra.run.smithi099.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-04T10:01:34.937 INFO:teuthology.orchestra.run.smithi162.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 145 kB/s | 19 kB 00:00 2024-03-04T10:01:34.942 INFO:teuthology.orchestra.run.smithi099.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 831 kB/s | 28 kB 00:00 2024-03-04T10:01:34.996 INFO:teuthology.orchestra.run.smithi162.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-04T10:01:35.001 INFO:teuthology.orchestra.run.smithi099.stdout:(169/176): librbd1-19.0.0-1845.g717ce3cc.el9.x8 3.8 MB/s | 3.1 MB 00:00 2024-03-04T10:01:35.043 INFO:teuthology.orchestra.run.smithi099.stdout:(170/176): openssl-3.0.7-27.el9.x86_64.rpm 9.9 MB/s | 1.2 MB 00:00 2024-03-04T10:01:35.093 INFO:teuthology.orchestra.run.smithi099.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-04T10:01:35.104 INFO:teuthology.orchestra.run.smithi162.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2024-03-04T10:01:35.130 INFO:teuthology.orchestra.run.smithi162.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 241 kB/s | 46 kB 00:00 2024-03-04T10:01:35.135 INFO:teuthology.orchestra.run.smithi099.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-04T10:01:35.160 INFO:teuthology.orchestra.run.smithi099.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 3.8 MB/s | 438 kB 00:00 2024-03-04T10:01:35.164 INFO:teuthology.orchestra.run.smithi162.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-03-04T10:01:35.185 INFO:teuthology.orchestra.run.smithi099.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 338 kB/s | 30 kB 00:00 2024-03-04T10:01:35.211 INFO:teuthology.orchestra.run.smithi099.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-04T10:01:35.213 INFO:teuthology.orchestra.run.smithi162.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-03-04T10:01:35.261 INFO:teuthology.orchestra.run.smithi099.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-04T10:01:35.267 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-03-04T10:01:35.267 INFO:teuthology.orchestra.run.smithi099.stdout:Total 12 MB/s | 223 MB 00:18 2024-03-04T10:01:35.272 INFO:teuthology.orchestra.run.smithi162.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-04T10:01:35.315 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-03-04T10:01:35.323 INFO:teuthology.orchestra.run.smithi162.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 828 kB/s | 90 kB 00:00 2024-03-04T10:01:35.381 INFO:teuthology.orchestra.run.smithi162.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-03-04T10:01:35.431 INFO:teuthology.orchestra.run.smithi162.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-03-04T10:01:35.457 INFO:teuthology.orchestra.run.smithi162.stdout:(140/176): python3-pecan-1.4.2-3.el9.noarch.rpm 834 kB/s | 272 kB 00:00 2024-03-04T10:01:35.499 INFO:teuthology.orchestra.run.smithi162.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-04T10:01:35.532 INFO:teuthology.orchestra.run.smithi162.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-04T10:01:35.566 INFO:teuthology.orchestra.run.smithi162.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-03-04T10:01:35.608 INFO:teuthology.orchestra.run.smithi162.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-03-04T10:01:35.631 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-03-04T10:01:35.631 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-03-04T10:01:35.641 INFO:teuthology.orchestra.run.smithi162.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-04T10:01:35.692 INFO:teuthology.orchestra.run.smithi162.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 715 kB/s | 90 kB 00:00 2024-03-04T10:01:35.742 INFO:teuthology.orchestra.run.smithi162.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-04T10:01:35.759 INFO:teuthology.orchestra.run.smithi162.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-03-04T10:01:35.834 INFO:teuthology.orchestra.run.smithi162.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-04T10:01:35.942 INFO:teuthology.orchestra.run.smithi162.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-04T10:01:35.984 INFO:teuthology.orchestra.run.smithi162.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-04T10:01:36.118 INFO:teuthology.orchestra.run.smithi162.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 17 MB/s | 2.3 MB 00:00 2024-03-04T10:01:36.194 INFO:teuthology.orchestra.run.smithi162.stdout:(153/176): librados2-19.0.0-1845.g717ce3cc.el9. 7.8 MB/s | 3.4 MB 00:00 2024-03-04T10:01:36.235 INFO:teuthology.orchestra.run.smithi162.stdout:(154/176): NetworkManager-libnm-1.46.0-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-03-04T10:01:36.261 INFO:teuthology.orchestra.run.smithi162.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 641 kB/s | 41 kB 00:00 2024-03-04T10:01:36.287 INFO:teuthology.orchestra.run.smithi162.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-04T10:01:36.311 INFO:teuthology.orchestra.run.smithi162.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 69 kB 00:00 2024-03-04T10:01:36.336 INFO:teuthology.orchestra.run.smithi162.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 704 kB/s | 35 kB 00:00 2024-03-04T10:01:36.362 INFO:teuthology.orchestra.run.smithi162.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 191 kB 00:00 2024-03-04T10:01:36.387 INFO:teuthology.orchestra.run.smithi162.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-04T10:01:36.429 INFO:teuthology.orchestra.run.smithi162.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 12 MB/s | 802 kB 00:00 2024-03-04T10:01:36.454 INFO:teuthology.orchestra.run.smithi162.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 6.4 MB/s | 433 kB 00:00 2024-03-04T10:01:36.479 INFO:teuthology.orchestra.run.smithi162.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-03-04T10:01:36.505 INFO:teuthology.orchestra.run.smithi162.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-04T10:01:36.530 INFO:teuthology.orchestra.run.smithi162.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-03-04T10:01:36.555 INFO:teuthology.orchestra.run.smithi162.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-04T10:01:36.581 INFO:teuthology.orchestra.run.smithi162.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-03-04T10:01:36.614 INFO:teuthology.orchestra.run.smithi162.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 473 kB/s | 28 kB 00:00 2024-03-04T10:01:36.673 INFO:teuthology.orchestra.run.smithi162.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-04T10:01:36.732 INFO:teuthology.orchestra.run.smithi162.stdout:(170/176): librbd1-19.0.0-1845.g717ce3cc.el9.x8 3.5 MB/s | 3.1 MB 00:00 2024-03-04T10:01:36.799 INFO:teuthology.orchestra.run.smithi162.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-04T10:01:36.882 INFO:teuthology.orchestra.run.smithi162.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-04T10:01:36.907 INFO:teuthology.orchestra.run.smithi162.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 2.5 MB/s | 438 kB 00:00 2024-03-04T10:01:36.933 INFO:teuthology.orchestra.run.smithi162.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 228 kB/s | 30 kB 00:00 2024-03-04T10:01:36.958 INFO:teuthology.orchestra.run.smithi162.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-04T10:01:37.016 INFO:teuthology.orchestra.run.smithi162.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-04T10:01:37.023 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-04T10:01:37.023 INFO:teuthology.orchestra.run.smithi162.stdout:Total 12 MB/s | 223 MB 00:18 2024-03-04T10:01:37.070 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-04T10:01:37.387 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-04T10:01:37.387 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-04T10:01:37.574 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-03-04T10:01:37.574 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-03-04T10:01:39.503 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-04T10:01:39.503 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-04T10:01:44.589 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-03-04T10:01:44.707 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T10:01:44.742 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T10:01:44.796 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T10:01:44.825 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T10:01:44.875 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T10:01:44.913 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T10:01:44.967 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T10:01:45.001 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T10:01:45.026 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T10:01:45.078 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T10:01:45.403 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T10:01:45.455 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T10:01:45.577 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T10:01:45.632 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T10:01:45.723 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T10:01:45.758 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T10:01:45.912 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T10:01:45.966 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T10:01:45.994 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T10:01:46.098 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T10:01:46.121 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T10:01:46.186 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T10:01:46.225 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T10:01:46.248 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T10:01:46.262 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T10:01:46.385 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-04T10:01:46.464 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T10:01:46.512 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T10:01:46.519 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T10:01:46.546 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T10:01:46.552 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T10:01:46.635 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T10:01:46.742 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T10:01:46.851 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T10:01:46.896 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T10:01:46.914 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T10:01:46.954 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T10:01:46.955 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T10:01:47.009 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T10:01:47.057 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T10:01:47.081 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T10:01:47.111 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T10:01:47.129 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T10:01:47.155 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T10:01:47.368 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T10:01:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T10:01:47.507 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T10:01:47.525 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T10:01:47.611 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T10:01:47.621 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T10:01:47.652 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T10:01:47.666 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T10:01:47.719 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T10:01:47.742 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T10:01:47.774 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T10:01:47.774 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T10:01:47.805 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T10:01:47.848 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T10:01:47.896 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T10:01:47.932 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T10:01:47.960 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T10:01:47.983 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T10:01:48.004 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T10:01:48.011 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T10:01:48.047 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T10:01:48.080 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T10:01:48.108 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T10:01:48.113 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T10:01:48.137 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T10:01:48.193 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T10:01:48.205 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T10:01:48.246 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 39/202 2024-03-04T10:01:48.266 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T10:01:48.289 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T10:01:48.289 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 40/202 2024-03-04T10:01:48.304 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T10:01:48.318 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 41/202 2024-03-04T10:01:48.387 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T10:01:48.432 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T10:01:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T10:01:48.505 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T10:01:48.529 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T10:01:48.578 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T10:01:48.640 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T10:01:48.701 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T10:01:48.896 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T10:01:48.937 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T10:01:48.998 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T10:01:49.114 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T10:01:49.166 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T10:01:49.230 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T10:01:49.304 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T10:01:49.379 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T10:01:49.401 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T10:01:49.449 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T10:01:49.458 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T10:01:49.493 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T10:01:49.504 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T10:01:49.542 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 52/202 2024-03-04T10:01:49.575 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T10:01:49.585 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T10:01:49.623 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T10:01:49.680 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T10:01:49.691 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T10:01:49.745 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T10:01:49.750 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T10:01:49.776 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T10:01:49.820 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T10:01:49.860 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T10:01:49.924 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T10:01:49.967 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T10:01:50.003 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T10:01:50.010 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T10:01:50.043 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T10:01:50.071 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T10:01:50.140 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T10:01:50.171 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T10:01:50.176 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 39/202 2024-03-04T10:01:50.219 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 40/202 2024-03-04T10:01:50.248 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 41/202 2024-03-04T10:01:50.285 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T10:01:50.325 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T10:01:50.371 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T10:01:50.453 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T10:01:50.465 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T10:01:50.586 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T10:01:50.606 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T10:01:50.640 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T10:01:50.659 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T10:01:50.712 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T10:01:50.785 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T10:01:50.816 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T10:01:50.853 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T10:01:50.899 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T10:01:50.943 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T10:01:50.986 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T10:01:51.030 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T10:01:51.163 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T10:01:51.167 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T10:01:51.209 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T10:01:51.239 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T10:01:51.252 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T10:01:51.281 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T10:01:51.336 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T10:01:51.376 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T10:01:51.381 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T10:01:51.389 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T10:01:51.414 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T10:01:51.427 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T10:01:51.464 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T10:01:51.465 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 52/202 2024-03-04T10:01:51.496 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T10:01:51.498 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T10:01:51.521 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T10:01:51.599 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T10:01:51.606 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T10:01:51.666 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T10:01:51.723 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T10:01:51.857 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T10:01:51.904 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T10:01:51.923 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T10:01:51.952 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T10:01:52.008 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T10:01:52.095 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T10:01:52.175 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T10:01:52.195 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T10:01:52.220 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T10:01:52.252 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T10:01:52.351 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T10:01:52.382 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T10:01:52.407 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T10:01:52.423 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T10:01:52.456 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T10:01:52.461 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T10:01:52.488 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T10:01:52.523 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T10:01:52.534 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T10:01:52.554 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T10:01:52.566 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T10:01:52.583 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T10:01:52.604 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T10:01:52.615 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T10:01:52.649 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T10:01:52.655 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T10:01:52.686 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T10:01:52.701 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T10:01:52.728 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T10:01:52.736 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T10:01:52.778 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T10:01:52.804 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T10:01:52.827 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T10:01:52.861 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T10:01:52.902 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T10:01:52.912 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T10:01:52.953 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T10:01:52.969 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T10:01:52.999 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T10:01:53.042 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T10:01:53.043 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T10:01:53.123 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T10:01:53.135 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T10:01:53.174 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T10:01:53.175 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T10:01:53.207 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T10:01:53.215 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T10:01:53.240 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T10:01:53.254 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T10:01:53.265 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T10:01:53.351 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T10:01:53.351 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T10:01:53.400 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T10:01:53.483 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T10:01:53.581 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T10:01:53.624 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T10:01:53.672 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T10:01:53.729 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T10:01:53.918 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T10:01:53.961 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T10:01:54.091 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T10:01:54.123 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T10:01:54.135 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T10:01:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T10:01:54.180 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T10:01:54.208 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T10:01:54.214 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T10:01:54.243 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T10:01:54.269 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T10:01:54.274 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T10:01:54.303 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T10:01:54.337 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T10:01:54.350 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T10:01:54.376 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T10:01:54.389 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T10:01:54.407 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T10:01:54.437 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T10:01:54.514 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T10:01:54.556 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T10:01:54.566 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T10:01:54.590 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T10:01:54.595 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T10:01:54.631 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T10:01:54.666 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T10:01:54.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T10:01:54.754 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T10:01:54.845 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T10:01:54.889 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T10:01:54.928 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T10:01:54.976 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T10:01:55.073 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T10:01:55.121 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T10:01:55.313 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T10:01:55.417 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T10:01:55.851 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T10:01:55.902 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T10:01:55.936 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T10:01:55.985 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T10:01:56.072 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T10:01:56.112 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T10:01:56.289 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T10:01:56.325 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T10:01:56.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T10:01:56.642 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T10:01:56.731 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T10:01:56.758 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T10:01:57.046 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T10:01:57.070 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T10:01:57.097 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T10:01:57.157 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T10:01:57.188 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T10:01:57.218 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 126/202 2024-03-04T10:01:57.254 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T10:01:57.710 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T10:01:57.772 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T10:01:57.813 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T10:01:57.843 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 130/202 2024-03-04T10:01:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T10:01:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T10:01:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T10:01:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T10:01:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T10:01:58.910 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T10:01:58.943 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 126/202 2024-03-04T10:01:58.980 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T10:01:59.438 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T10:01:59.466 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T10:01:59.537 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T10:01:59.568 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 130/202 2024-03-04T10:01:59.955 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T10:01:59.989 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T10:02:00.540 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T10:02:00.579 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T10:02:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T10:02:00.668 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T10:02:00.668 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T10:02:00.668 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:00.710 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T10:02:01.797 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T10:02:01.832 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T10:02:02.377 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T10:02:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T10:02:02.510 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T10:02:02.510 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T10:02:02.510 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T10:02:02.510 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:02.545 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T10:02:12.325 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T10:02:12.325 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2024-03-04T10:02:12.325 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2024-03-04T10:02:12.325 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2024-03-04T10:02:12.325 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2024-03-04T10:02:12.326 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2024-03-04T10:02:12.326 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /root 2024-03-04T10:02:12.326 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2024-03-04T10:02:12.326 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:12.488 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T10:02:12.504 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T10:02:12.504 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:12.504 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T10:02:12.504 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:13.927 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-03-04T10:02:13.928 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:13.991 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T10:02:14.016 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T10:02:14.016 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:14.016 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T10:02:14.016 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T10:02:14.016 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T10:02:14.016 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:14.103 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T10:02:14.119 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T10:02:14.119 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:14.120 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T10:02:14.120 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:14.217 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T10:02:14.239 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T10:02:14.240 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:14.240 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T10:02:14.240 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T10:02:14.240 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T10:02:14.240 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:14.684 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T10:02:14.708 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T10:02:14.708 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:14.708 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T10:02:14.708 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T10:02:14.708 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T10:02:14.709 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:14.794 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T10:02:15.161 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T10:02:15.193 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T10:02:15.215 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 140/202 2024-03-04T10:02:15.267 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 141/202 2024-03-04T10:02:15.290 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T10:02:15.524 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T10:02:15.558 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T10:02:15.615 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T10:02:15.640 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T10:02:15.640 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:15.640 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T10:02:15.640 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T10:02:15.641 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T10:02:15.641 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:15.849 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T10:02:15.871 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T10:02:15.872 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:15.872 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T10:02:15.872 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T10:02:15.872 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T10:02:15.872 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:16.316 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T10:02:16.339 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T10:02:16.339 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:16.339 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T10:02:16.339 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T10:02:16.339 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T10:02:16.339 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:16.435 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T10:02:16.528 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T10:02:16.575 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T10:02:16.675 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T10:02:16.847 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 145/202 2024-03-04T10:02:16.869 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T10:02:16.900 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T10:02:16.901 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T10:02:16.924 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T10:02:16.924 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:16.924 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T10:02:16.925 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T10:02:16.925 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T10:02:16.925 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:16.931 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 140/202 2024-03-04T10:02:16.969 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T10:02:16.980 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T10:02:16.991 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 141/202 2024-03-04T10:02:17.015 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T10:02:17.033 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 148/202 2024-03-04T10:02:17.243 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T10:02:17.274 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T10:02:17.896 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T10:02:17.929 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T10:02:17.952 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T10:02:17.952 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:17.952 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T10:02:17.952 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T10:02:17.953 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T10:02:17.953 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:17.997 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T10:02:18.019 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T10:02:18.019 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:18.019 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T10:02:18.019 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:18.064 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T10:02:18.086 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T10:02:18.086 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:18.086 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T10:02:18.086 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T10:02:18.086 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T10:02:18.087 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:18.238 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T10:02:18.274 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T10:02:18.373 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T10:02:18.398 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T10:02:18.420 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T10:02:18.420 INFO:teuthology.orchestra.run.smithi099.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:18.420 INFO:teuthology.orchestra.run.smithi099.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T10:02:18.420 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T10:02:18.420 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T10:02:18.421 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:18.555 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 145/202 2024-03-04T10:02:18.585 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T10:02:18.609 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T10:02:18.609 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:18.609 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T10:02:18.609 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T10:02:18.609 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T10:02:18.609 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:18.653 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T10:02:18.663 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T10:02:18.716 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 148/202 2024-03-04T10:02:19.577 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T10:02:19.604 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T10:02:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T10:02:19.627 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:19.628 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T10:02:19.628 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T10:02:19.628 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T10:02:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:19.672 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T10:02:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T10:02:19.694 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:19.694 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T10:02:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T10:02:19.770 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T10:02:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T10:02:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T10:02:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T10:02:19.770 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:20.090 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T10:02:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T10:02:20.113 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T10:02:20.113 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T10:02:20.113 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T10:02:20.113 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T10:02:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:22.852 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 154/202 2024-03-04T10:02:22.893 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 155/202 2024-03-04T10:02:22.923 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 156/202 2024-03-04T10:02:22.951 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T10:02:23.131 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 158/202 2024-03-04T10:02:23.250 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T10:02:23.307 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 160/202 2024-03-04T10:02:23.357 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T10:02:23.406 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T10:02:23.464 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T10:02:23.517 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T10:02:23.560 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T10:02:23.583 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 166/202 2024-03-04T10:02:23.614 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T10:02:23.691 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T10:02:23.750 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T10:02:23.786 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T10:02:24.431 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T10:02:24.467 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T10:02:24.486 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T10:02:24.589 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 154/202 2024-03-04T10:02:24.636 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 155/202 2024-03-04T10:02:24.667 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 156/202 2024-03-04T10:02:24.694 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T10:02:24.874 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 158/202 2024-03-04T10:02:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T10:02:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 160/202 2024-03-04T10:02:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T10:02:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T10:02:25.208 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T10:02:25.253 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T10:02:25.297 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T10:02:25.302 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 172/202 2024-03-04T10:02:25.328 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 166/202 2024-03-04T10:02:25.357 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T10:02:25.361 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T10:02:25.405 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T10:02:25.423 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T10:02:25.490 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T10:02:25.491 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T10:02:25.525 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T10:02:25.534 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T10:02:25.550 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T10:02:25.575 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T10:02:25.602 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T10:02:25.602 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T10:02:25.678 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T10:02:25.706 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T10:02:25.739 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T10:02:25.739 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T10:02:25.838 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T10:02:25.864 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T10:02:25.906 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T10:02:25.931 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T10:02:25.959 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T10:02:25.990 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T10:02:26.025 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T10:02:26.071 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T10:02:26.103 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T10:02:26.128 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T10:02:26.148 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T10:02:26.154 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T10:02:26.179 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T10:02:26.187 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T10:02:26.204 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T10:02:26.207 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T10:02:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T10:02:26.211 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T10:02:26.390 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T10:02:26.391 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T10:02:26.465 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T10:02:26.497 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T10:02:26.524 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T10:02:26.548 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T10:02:26.574 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T10:02:26.641 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T10:02:27.026 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 172/202 2024-03-04T10:02:27.100 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T10:02:27.142 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T10:02:27.232 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T10:02:27.262 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T10:02:27.287 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T10:02:27.313 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T10:02:27.340 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T10:02:27.340 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T10:02:27.424 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T10:02:27.452 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T10:02:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T10:02:27.482 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T10:02:27.576 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T10:02:27.602 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T10:02:27.636 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T10:02:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T10:02:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T10:02:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T10:02:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T10:02:27.812 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T10:02:27.842 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T10:02:27.867 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T10:02:27.893 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T10:02:27.918 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T10:02:27.944 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T10:02:27.950 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T10:02:27.950 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T10:02:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T10:02:28.130 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T10:02:28.221 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T10:02:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T10:02:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T10:02:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T10:02:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T10:02:28.400 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T10:02:31.629 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 1/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 3/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 4/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 5/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 6/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 7/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 8/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 9/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 10/202 2024-03-04T10:02:31.630 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 14/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 15/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 17/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 18/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 19/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 20/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 21/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 22/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 23/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 24/202 2024-03-04T10:02:31.631 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 25/202 2024-03-04T10:02:31.632 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 26/202 2024-03-04T10:02:31.632 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 27/202 2024-03-04T10:02:31.632 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 28/202 2024-03-04T10:02:31.632 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 29/202 2024-03-04T10:02:31.632 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 30/202 2024-03-04T10:02:31.651 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 31/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 32/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 33/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 34/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 35/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 36/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 37/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 38/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 39/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T10:02:31.652 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T10:02:31.653 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T10:02:31.654 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T10:02:31.655 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T10:02:31.656 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T10:02:31.657 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T10:02:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T10:02:31.660 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T10:02:31.661 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 151/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T10:02:31.662 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T10:02:31.663 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T10:02:31.664 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T10:02:31.665 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T10:02:33.471 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T10:02:33.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 1/202 2024-03-04T10:02:33.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2/202 2024-03-04T10:02:33.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 3/202 2024-03-04T10:02:33.472 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 4/202 2024-03-04T10:02:33.472 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 5/202 2024-03-04T10:02:33.472 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 6/202 2024-03-04T10:02:33.472 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 7/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 8/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 9/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 10/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 14/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 15/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T10:02:33.473 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 17/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 18/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 19/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 20/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 21/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 22/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 23/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 24/202 2024-03-04T10:02:33.474 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 25/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 26/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 27/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 28/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 29/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 30/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 31/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 32/202 2024-03-04T10:02:33.475 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 33/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 34/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 35/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 36/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 37/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 38/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 39/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T10:02:33.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T10:02:33.477 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T10:02:33.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T10:02:33.479 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T10:02:33.480 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T10:02:33.481 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T10:02:33.482 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T10:02:33.483 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T10:02:33.484 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T10:02:33.485 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 151/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T10:02:33.486 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T10:02:33.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T10:02:33.488 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T10:02:33.489 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T10:02:34.368 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T10:02:34.368 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:34.368 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2024-03-04T10:02:34.368 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:34.368 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:34.368 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.369 INFO:teuthology.orchestra.run.smithi099.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.370 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.371 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T10:02:34.372 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: jq-1.6-16.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T10:02:34.373 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T10:02:34.374 INFO:teuthology.orchestra.run.smithi099.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T10:02:34.375 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T10:02:34.376 INFO:teuthology.orchestra.run.smithi099.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T10:02:34.377 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T10:02:34.378 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T10:02:34.379 INFO:teuthology.orchestra.run.smithi099.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T10:02:34.380 INFO:teuthology.orchestra.run.smithi099.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T10:02:34.381 INFO:teuthology.orchestra.run.smithi099.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T10:02:34.381 INFO:teuthology.orchestra.run.smithi099.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T10:02:34.381 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T10:02:34.381 INFO:teuthology.orchestra.run.smithi099.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T10:02:34.381 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:34.381 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-03-04T10:02:35.007 DEBUG:teuthology.parallel:result is None 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T10:02:36.295 INFO:teuthology.orchestra.run.smithi162.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.296 INFO:teuthology.orchestra.run.smithi162.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T10:02:36.297 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.298 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.299 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.300 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.301 INFO:teuthology.orchestra.run.smithi162.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T10:02:36.302 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T10:02:36.303 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T10:02:36.304 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T10:02:36.305 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.306 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T10:02:36.307 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T10:02:36.308 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T10:02:36.309 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T10:02:36.309 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:36.309 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-04T10:02:36.928 DEBUG:teuthology.parallel:result is None 2024-03-04T10:02:36.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T10:02:37.163 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T10:02:37.189 INFO:teuthology.orchestra.run.smithi099.stdout:19.0.0-1845.g717ce3cc.el9 2024-03-04T10:02:37.190 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1845.g717ce3cc.el9 2024-03-04T10:02:37.190 INFO:teuthology.task.install:The correct ceph version 19.0.0-1845.g717ce3cc is installed. 2024-03-04T10:02:37.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T10:02:37.426 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T10:02:37.452 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-1845.g717ce3cc.el9 2024-03-04T10:02:37.452 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1845.g717ce3cc.el9 2024-03-04T10:02:37.452 INFO:teuthology.task.install:The correct ceph version 19.0.0-1845.g717ce3cc is installed. 2024-03-04T10:02:37.454 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T10:02:37.454 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:37.454 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T10:02:37.490 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:37.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T10:02:37.525 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T10:02:37.525 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:37.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T10:02:37.560 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T10:02:37.630 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:37.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T10:02:37.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T10:02:37.730 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T10:02:37.730 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:37.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T10:02:37.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T10:02:37.827 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:37.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T10:02:37.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T10:02:37.928 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T10:02:37.929 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:37.929 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T10:02:37.956 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T10:02:38.024 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:38.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T10:02:38.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T10:02:38.126 INFO:teuthology.run_tasks:Running task ceph... 2024-03-04T10:02:38.247 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-04T10:02:38.248 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2024-03-04T10:02:38.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /var/log/ceph 2024-03-04T10:02:38.280 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-04T10:02:38.281 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T10:02:38.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-04T10:02:38.351 INFO:tasks.ceph:Creating extra log directories... 2024-03-04T10:02:38.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T10:02:38.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-04T10:02:38.427 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-04T10:02:38.427 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-04T10:02:38.427 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-04T10:02:38.427 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:38.428 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-04T10:02:38.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:38.591 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:38.618 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:38.619 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-04T10:02:38.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:38.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-04T10:02:38.801 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-04T10:02:38.801 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-04T10:02:38.802 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900', 'branch': 'wip-vshankar-testing-20240304.042522', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7580900', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '786939134d7eee4dc7a805245d3f595929af37d7'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20240304.042522'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20240304.042522', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3119, 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'sleep_before_teardown': 0, 'subset': '270/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20240304.042522', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi099.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBA7XaVI3qo8JUjP6Cs8BrbgZKbZvhNnSuf1DkTRSJ1Mvc+9SLi404KEQCk/cU3tb4mMjV3DbVDSXaavcQb4C3k=', 'smithi162.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDrBdhyWDFx2GvNg6H8LTIqPp5KPcNiv2djc6oiZNzNgrfV3Tc5cSISLGWsTtOPG0rdezwu3vs+c12rnYDTUIG4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '786939134d7eee4dc7a805245d3f595929af37d7'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'Error loading MDS rank', 'missing journal object', 'Error recovering journal', 'error decoding table object', 'failed to read JournalPointer', 'Corrupt directory entry', 'Corrupt fnode header', 'corrupt sessionmap header', 'Corrupt dentry', 'Scrub error on inode', 'Metadata damage detected', 'MDS_READ_ONLY', 'force file system read-only', 'with standby daemon mds', 'MDS abort because newly corrupt dentry'], 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_damage']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-03-04_08:26:39', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-04T10:02:38.802 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-04T10:02:38.805 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-04T10:02:38.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T10:02:38.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/run/ceph 2024-03-04T10:02:38.919 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:38.919 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T10:02:38.940 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T10:02:38.940 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-03-04T10:02:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T10:02:38.994 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:38.994 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T10:02:38.995 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:38.995 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:38.995 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-04 10:02:26.990922805 +0000 2024-03-04T10:02:38.995 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-04 10:00:28.298186996 +0000 2024-03-04T10:02:38.995 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-04 10:00:28.298186996 +0000 2024-03-04T10:02:38.995 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-04 10:00:28.298186996 +0000 2024-03-04T10:02:38.995 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T10:02:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-04T10:02:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-04T10:02:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000271867 s, 1.9 MB/s 2024-03-04T10:02:39.061 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T10:02:39.117 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-03-04T10:02:39.172 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T10:02:39.172 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-04 10:02:26.991922781 +0000 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-04 10:00:28.640178697 +0000 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-04 10:00:28.640178697 +0000 2024-03-04T10:02:39.173 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-04 10:00:28.640178697 +0000 2024-03-04T10:02:39.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T10:02:39.238 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-04T10:02:39.238 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-04T10:02:39.238 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.00019698 s, 2.6 MB/s 2024-03-04T10:02:39.239 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T10:02:39.295 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-03-04T10:02:39.350 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-04 10:02:26.991922781 +0000 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-04 10:00:28.984170350 +0000 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-04 10:00:28.984170350 +0000 2024-03-04T10:02:39.351 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-04 10:00:28.984170350 +0000 2024-03-04T10:02:39.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T10:02:39.416 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-04T10:02:39.416 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-04T10:02:39.416 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000224649 s, 2.3 MB/s 2024-03-04T10:02:39.417 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T10:02:39.473 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-03-04T10:02:39.529 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T10:02:39.529 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:39.529 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-04T10:02:39.529 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:39.529 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:39.529 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-04 10:02:26.991922781 +0000 2024-03-04T10:02:39.530 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-04 10:00:29.312162391 +0000 2024-03-04T10:02:39.530 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-04 10:00:29.312162391 +0000 2024-03-04T10:02:39.530 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-04 10:00:29.312162391 +0000 2024-03-04T10:02:39.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T10:02:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-03-04T10:02:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-03-04T10:02:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000177809 s, 2.9 MB/s 2024-03-04T10:02:39.596 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T10:02:39.652 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:39.652 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:39.652 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T10:02:39.671 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T10:02:39.671 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-03-04T10:02:39.729 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-04 10:02:28.717487541 +0000 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-04 10:00:28.300648851 +0000 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-04 10:00:28.300648851 +0000 2024-03-04T10:02:39.730 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-04 10:00:28.300648851 +0000 2024-03-04T10:02:39.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T10:02:39.799 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-04T10:02:39.799 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-04T10:02:39.799 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000283532 s, 1.8 MB/s 2024-03-04T10:02:39.801 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T10:02:39.859 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-03-04T10:02:39.915 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T10:02:39.915 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-04 10:02:28.718487521 +0000 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-04 10:00:28.651641522 +0000 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-04 10:00:28.651641522 +0000 2024-03-04T10:02:39.916 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-04 10:00:28.651641522 +0000 2024-03-04T10:02:39.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T10:02:39.983 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-04T10:02:39.984 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-04T10:02:39.984 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000280362 s, 1.8 MB/s 2024-03-04T10:02:39.985 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T10:02:40.041 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-03-04T10:02:40.096 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-04 10:02:28.718487521 +0000 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-04 10:00:28.997634296 +0000 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-04 10:00:28.997634296 +0000 2024-03-04T10:02:40.097 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-04 10:00:28.997634296 +0000 2024-03-04T10:02:40.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T10:02:40.163 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-04T10:02:40.163 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-04T10:02:40.164 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000200951 s, 2.5 MB/s 2024-03-04T10:02:40.165 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T10:02:40.221 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-04 10:02:28.718487521 +0000 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-04 10:00:29.320627551 +0000 2024-03-04T10:02:40.276 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-04 10:00:29.320627551 +0000 2024-03-04T10:02:40.277 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-04 10:00:29.320627551 +0000 2024-03-04T10:02:40.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T10:02:40.342 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-04T10:02:40.342 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-04T10:02:40.342 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000259408 s, 2.0 MB/s 2024-03-04T10:02:40.343 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T10:02:40.400 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:40.400 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-04T10:02:40.400 INFO:tasks.ceph:Generating config... 2024-03-04T10:02:40.401 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-04T10:02:40.401 INFO:tasks.ceph:[client] debug client = 20 2024-03-04T10:02:40.402 INFO:tasks.ceph:[client] debug ms = 1 2024-03-04T10:02:40.402 INFO:tasks.ceph:[client] pid file = 2024-03-04T10:02:40.402 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-04T10:02:40.402 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-04T10:02:40.402 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-04T10:02:40.402 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-04T10:02:40.403 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-04T10:02:40.404 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-04T10:02:40.404 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-04T10:02:40.405 INFO:tasks.ceph:Setting up mon.a... 2024-03-04T10:02:40.405 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-04T10:02:40.464 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.keyring 2024-03-04T10:02:40.467 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-04T10:02:40.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T10:02:40.541 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.99'), ('mon.b', '172.21.15.162'), ('mon.c', '[v2:172.21.15.162:3301,v1:172.21.15.162:6790]')] 2024-03-04T10:02:40.542 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.99,172.21.15.162,[v2:172.21.15.162:3301,v1:172.21.15.162:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-04T10:02:40.542 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:40.543 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T10:02:40.597 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.99 --add b 172.21.15.162 --addv c '[v2:172.21.15.162:3301,v1:172.21.15.162:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:40.683 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = quincy 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:fsid 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2024-03-04T10:02:40.683474+0000 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:created 2024-03-04T10:02:40.683474+0000 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 17 (quincy) 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 1 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:2: [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] mon.c 2024-03-04T10:02:40.684 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-04T10:02:40.685 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-04T10:02:40.701 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47... 2024-03-04T10:02:40.702 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T10:02:40.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-04T10:02:40.790 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2024-03-04T10:02:40.791 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2024-03-04T10:02:40.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.792 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2024-03-04T10:02:40.792 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2024-03-04T10:02:40.792 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.792 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2024-03-04T10:02:40.793 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2024-03-04T10:02:40.794 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = "172.21.15.99,172.21.15.162,[v2:172.21.15.162:3301,v1:172.21.15.162:6790]" 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on skipped message = False 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd mclock profile = high_recovery_ops 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2024-03-04T10:02:40.795 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug trim objects = true 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.796 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore block size = 96636764160 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore fsck on mount = True 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluefs = 20 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluestore = 20 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 20 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: debug rocksdb = 10 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd full ratio = 0.9 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd nearfull ratio = 0.8 2024-03-04T10:02:40.797 INFO:teuthology.orchestra.run.smithi099.stdout: osd failsafe full ratio = 0.95 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: osd objectstore = bluestore 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: osd op complaint time = 180 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2024-03-04T10:02:40.798 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T10:02:40.799 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: # 1m isn't quite enough 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: mon_down_mkfs_grace = 2m 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_filestore_osds = false 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: mon op complaint time = 120 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2024-03-04T10:02:40.800 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: client mount timeout = 600 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: debug client = 20 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = "" 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: rados mon op timeout = 900 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout: rados osd op timeout = 900 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.b] 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.c] 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.a] 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.c] 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.b] 2024-03-04T10:02:40.801 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.d] 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout:[mds] 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: debug mds = 20 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: debug mds balancer = 20 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: mds debug frag = True 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: mds debug scatterstat = True 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: mds op complaint time = 180 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: mds verify scatter = True 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: osd op complaint time = 180 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: rados mon op timeout = 900 2024-03-04T10:02:40.802 INFO:teuthology.orchestra.run.smithi099.stdout: rados osd op timeout = 900 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout:[global] 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: chdir = "" 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: auth supported = cephx 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: filestore xattr use omap = true 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: mon clock drift allowed = 1.000 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: osd crush chooseleaf type = 0 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: auth debug = true 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on old message = true 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on bug = true 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: mon pg warn max object skew = 0 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: # disable pg_autoscaler by default for new pools 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default size = 2 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow primary affinity = true 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow pg remap = true 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on legacy crush tunables = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on crush straw calc version zero = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on no sortbitwise = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on osd down out interval zero = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on too few osds = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: mon_allow_pool_size_one = true 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: osd default data pool replay window = 5 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: mon allow pool delete = true 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: mon cluster log file level = debug 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: debug asserts on shutdown = true 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: mon health detail to clog = false 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: mon host = "172.21.15.99,172.21.15.162,[v2:172.21.15.162:3301,v1:172.21.15.162:6790]" 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on skipped message = False 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: fsid = 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout:[osd] 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: osd journal size = 100 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub load threshold = 5.0 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub max interval = 600 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock profile = high_recovery_ops 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd recover clone overlap = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd recovery max chunk = 1048576 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug shutdown = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug op order = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify stray on activate = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug trim objects = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd open classes on start = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug pg log writeout = true 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: osd deep scrub update digest min age = 30 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: osd map max advance = 10 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: journal zero on create = true 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: filestore ondisk finisher threads = 3 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: filestore apply finisher threads = 3 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: bdev debug aio = true 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug misdirected ops = true 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore block size = 96636764160 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore fsck on mount = True 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluefs = 20 2024-03-04T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluestore = 20 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: debug osd = 20 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: debug rocksdb = 10 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd backfillfull_ratio = 0.85 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd full ratio = 0.9 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd nearfull ratio = 0.8 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: osd failsafe full ratio = 0.95 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: osd objectstore = bluestore 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: osd op complaint time = 180 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr] 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: debug mgr = 20 2024-03-04T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mgr/telemetry/nag = false 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout:[mon] 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: debug paxos = 20 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon data avail warn = 5 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon mgr mkfs grace = 240 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd reporter subtree level = osd 2024-03-04T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd prime pg temp = true 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: auth mon ticket ttl = 660 # 11m 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: auth service ticket ttl = 240 # 4m 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: # don't complain about insecure global_id in the test suite 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: # 1m isn't quite enough 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: mon_down_mkfs_grace = 2m 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_filestore_osds = false 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: mon op complaint time = 120 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout:[client] 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: rgw cache enabled = true 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable ops log = true 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable usage log = true 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: client mount timeout = 600 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: debug client = 20 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: pid file = "" 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: rados mon op timeout = 900 2024-03-04T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: rados osd op timeout = 900 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.a] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.b] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.c] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.a] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.c] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.b] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.d] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout:[mds] 2024-03-04T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: debug mds = 20 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: debug mds balancer = 20 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: mds debug frag = True 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: mds debug scatterstat = True 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: mds op complaint time = 180 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: mds verify scatter = True 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: osd op complaint time = 180 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: rados mon op timeout = 900 2024-03-04T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: rados osd op timeout = 900 2024-03-04T10:02:40.816 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-04T10:02:40.816 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-04T10:02:40.911 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-04T10:02:40.911 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:40.911 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-04T10:02:40.925 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:40.926 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-04T10:02:40.980 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:02:40.980 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:40.980 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-04T10:02:40.980 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T10:02:41.059 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:02:41.060 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:41.114 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:02:41.115 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:41.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-04T10:02:41.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-04T10:02:41.156 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:02:41.156 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:41.211 INFO:tasks.ceph:Setting up mon nodes... 2024-03-04T10:02:41.211 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-04T10:02:41.212 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-04T10:02:41.266 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-04T10:02:41.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-04T10:02:41.338 INFO:teuthology.orchestra.run.smithi162.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-04T10:02:41.341 INFO:tasks.ceph:Setting up mds nodes... 2024-03-04T10:02:41.341 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-04T10:02:41.395 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-04T10:02:41.398 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-04T10:02:41.424 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-04T10:02:41.519 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-04T10:02:41.521 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-04T10:02:41.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-04T10:02:41.603 INFO:teuthology.orchestra.run.smithi162.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-04T10:02:41.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-04T10:02:41.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-04T10:02:41.727 INFO:teuthology.orchestra.run.smithi162.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-04T10:02:41.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-04T10:02:41.757 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-04T10:02:41.757 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T10:02:41.799 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-04T10:02:41.814 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-04T10:02:41.897 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-04T10:02:41.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T10:02:41.954 INFO:teuthology.orchestra.run.smithi162.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-04T10:02:41.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-04T10:02:42.053 INFO:teuthology.orchestra.run.smithi162.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-04T10:02:42.068 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-04T10:02:42.068 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-04T10:02:42.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-04T10:02:42.095 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:42.095 INFO:tasks.ceph:role: osd.0 2024-03-04T10:02:42.095 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:02:42.095 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:42.170 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:42.171 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:42.171 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:42.171 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:42.678 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-04T10:02:42.679 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-04T10:02:42.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-04T10:02:42.816 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-04T10:02:42.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-04T10:02:42.918 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:42.919 INFO:tasks.ceph:role: osd.1 2024-03-04T10:02:42.919 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:02:42.919 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T10:02:42.992 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:42.993 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:43.442 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-04T10:02:43.444 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-04T10:02:43.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-04T10:02:43.540 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-04T10:02:43.571 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-04T10:02:43.638 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:43.638 INFO:tasks.ceph:role: osd.2 2024-03-04T10:02:43.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:02:43.638 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T10:02:43.713 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:43.715 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:43.715 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:43.715 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:43.715 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:43.715 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:43.715 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:43.716 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:43.716 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:43.716 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:44.199 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-04T10:02:44.200 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-04T10:02:44.201 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-04T10:02:44.327 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-04T10:02:44.362 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-04T10:02:44.434 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:44.434 INFO:tasks.ceph:role: osd.3 2024-03-04T10:02:44.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:02:44.434 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:44.511 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:45.009 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2024-03-04T10:02:45.011 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2024-03-04T10:02:45.011 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-04T10:02:45.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-04T10:02:45.183 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:45.280+0000 7f59066bf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-04T10:02:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:45.280+0000 7f59066bf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-04T10:02:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:45.280+0000 7f59066bf640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-04T10:02:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:45.280+0000 7f59066bf640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-04T10:02:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:45.280+0000 7f59066bf640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-04T10:02:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:45.281+0000 7f59066bf640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-04T10:02:48.141 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-04T10:02:48.173 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:48.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:48.264+0000 7fc037d28640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-04T10:02:48.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:48.264+0000 7fc037d28640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-04T10:02:48.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:48.264+0000 7fc037d28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-04T10:02:48.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:48.264+0000 7fc037d28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-04T10:02:48.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:48.264+0000 7fc037d28640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-04T10:02:48.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:48.265+0000 7fc037d28640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-04T10:02:51.111 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-04T10:02:51.142 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:51.233+0000 7faca2821640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-04T10:02:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:51.233+0000 7faca2821640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-04T10:02:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:51.233+0000 7faca2821640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-04T10:02:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:51.233+0000 7faca2821640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-04T10:02:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:51.233+0000 7faca2821640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-04T10:02:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:51.234+0000 7faca2821640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-04T10:02:54.090 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-04T10:02:54.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:02:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:54.211+0000 7f23a80d5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-04T10:02:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:54.212+0000 7f23a80d5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-04T10:02:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:54.212+0000 7f23a80d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-04T10:02:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:54.212+0000 7f23a80d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-04T10:02:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:54.212+0000 7f23a80d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-04T10:02:54.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:02:54.212+0000 7f23a80d5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-04T10:02:57.061 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-04T10:02:57.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-04T10:02:57.123 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:57.124 INFO:tasks.ceph:role: osd.4 2024-03-04T10:02:57.124 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:02:57.124 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:57.204 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:57.205 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:57.718 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-04T10:02:57.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-04T10:02:57.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-04T10:02:57.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-04T10:02:57.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-04T10:02:57.947 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:57.947 INFO:tasks.ceph:role: osd.5 2024-03-04T10:02:57.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:02:57.947 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-04T10:02:58.022 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:58.023 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:58.477 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-04T10:02:58.479 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-04T10:02:58.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-04T10:02:58.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-04T10:02:58.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-04T10:02:58.669 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:58.669 INFO:tasks.ceph:role: osd.6 2024-03-04T10:02:58.669 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:02:58.669 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-04T10:02:58.743 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:58.743 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:58.744 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:58.744 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:58.744 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:02:59.230 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-04T10:02:59.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-04T10:02:59.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-04T10:02:59.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-04T10:02:59.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-04T10:02:59.446 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-04T10:02:59.446 INFO:tasks.ceph:role: osd.7 2024-03-04T10:02:59.446 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:02:59.446 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-04T10:02:59.521 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-04T10:03:00.038 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-04T10:03:00.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-04T10:03:00.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-04T10:03:00.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-04T10:03:00.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:03:00.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:00.328+0000 7fd2f88b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-04T10:03:00.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:00.328+0000 7fd2f88b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-04T10:03:00.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:00.328+0000 7fd2f88b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-04T10:03:00.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:00.328+0000 7fd2f88b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-04T10:03:00.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:00.328+0000 7fd2f88b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-04T10:03:00.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:00.329+0000 7fd2f88b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-04T10:03:03.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-04T10:03:03.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:03:03.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:03.298+0000 7f7467868640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-04T10:03:03.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:03.299+0000 7f7467868640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-04T10:03:03.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:03.299+0000 7f7467868640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-04T10:03:03.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:03.299+0000 7f7467868640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-04T10:03:03.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:03.299+0000 7f7467868640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-04T10:03:03.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:03.299+0000 7f7467868640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-04T10:03:06.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-04T10:03:06.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:03:06.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:06.258+0000 7fa895a0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-04T10:03:06.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:06.258+0000 7fa895a0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-04T10:03:06.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:06.259+0000 7fa895a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-04T10:03:06.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:06.259+0000 7fa895a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-04T10:03:06.260 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:06.259+0000 7fa895a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-04T10:03:06.261 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:06.259+0000 7fa895a0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-04T10:03:08.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:08.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:09.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-04T10:03:09.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:03:09.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:09.224+0000 7effe741b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-04T10:03:09.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:09.224+0000 7effe741b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-04T10:03:09.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:09.224+0000 7effe741b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-04T10:03:09.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:09.224+0000 7effe741b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-04T10:03:09.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:09.224+0000 7effe741b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-04T10:03:09.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:09.225+0000 7effe741b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-04T10:03:12.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-04T10:03:12.124 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-04T10:03:12.125 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-04T10:03:12.156 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-04T10:03:12.228 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.228 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-04T10:03:12.298 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.298 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-04T10:03:12.367 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-04T10:03:12.435 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-04T10:03:12.504 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:12.504 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-04T10:03:12.572 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-04T10:03:12.604 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-04T10:03:12.675 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-04T10:03:12.745 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-04T10:03:12.814 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-04T10:03:12.883 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-04T10:03:12.951 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:12.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-04T10:03:13.020 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:13.020 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-04T10:03:13.038 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:13.038 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-04T10:03:13.096 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:13.096 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-04T10:03:13.112 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:13.112 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-04T10:03:13.168 INFO:tasks.ceph:Adding keys to all mons... 2024-03-04T10:03:13.168 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-04T10:03:13.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-04T10:03:13.211 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.y] 2024-03-04T10:03:13.211 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.a] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.c] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.3] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA== 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2024-03-04T10:03:13.212 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw== 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.b] 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw== 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout:[mds.d] 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw== 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.4] 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw== 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.5] 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg== 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.6] 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg== 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.7] 2024-03-04T10:03:13.213 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA== 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout:[client.2] 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg== 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA== 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout:[client.3] 2024-03-04T10:03:13.214 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ== 2024-03-04T10:03:13.234 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.y] 2024-03-04T10:03:13.234 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ== 2024-03-04T10:03:13.234 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.a] 2024-03-04T10:03:13.234 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg== 2024-03-04T10:03:13.234 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.c] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg== 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.0] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg== 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.1] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ== 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.2] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw== 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.3] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA== 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.x] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw== 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.b] 2024-03-04T10:03:13.235 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[mds.d] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.4] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.5] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.6] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.7] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout:[client.2] 2024-03-04T10:03:13.236 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg== 2024-03-04T10:03:13.237 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-03-04T10:03:13.237 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA== 2024-03-04T10:03:13.237 INFO:teuthology.orchestra.run.smithi162.stdout:[client.3] 2024-03-04T10:03:13.237 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ== 2024-03-04T10:03:13.238 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T10:03:13.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T10:03:13.330 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.388 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.445 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.554 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.608 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.663 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T10:03:13.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-04T10:03:13.717 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-04T10:03:13.835 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.890 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.946 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:13.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:14.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:14.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-04T10:03:14.056 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.112 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.170 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.228 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-04T10:03:14.285 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-04T10:03:14.285 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-04T10:03:14.312 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T10:03:14.409 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-04T10:03:14.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-04T10:03:14.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T10:03:14.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-04T10:03:14.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-04T10:03:14.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-04T10:03:14.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-04T10:03:14.777 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:03:14.780 DEBUG:teuthology.orchestra.run.smithi162:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-04T10:03:14.832 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-04T10:03:14.832 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-04T10:03:14.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-04T10:03:14.834 INFO:tasks.ceph.mon.a:Started 2024-03-04T10:03:14.835 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-04T10:03:14.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-04T10:03:14.874 INFO:tasks.ceph.mon.b:Started 2024-03-04T10:03:14.875 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-04T10:03:14.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-04T10:03:14.877 INFO:tasks.ceph.mon.c:Started 2024-03-04T10:03:14.877 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-04T10:03:14.877 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-04T10:03:14.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-04T10:03:14.879 INFO:tasks.ceph.mgr.y:Started 2024-03-04T10:03:14.880 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-04T10:03:14.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-04T10:03:14.882 INFO:tasks.ceph.mgr.x:Started 2024-03-04T10:03:14.882 INFO:tasks.ceph:Setting crush tunables to default 2024-03-04T10:03:14.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd crush tunables default 2024-03-04T10:03:14.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.981+0000 7f7a22830640 1 Processor -- start 2024-03-04T10:03:14.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.981+0000 7f7a22830640 1 -- start start 2024-03-04T10:03:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a22830640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a1c161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a22830640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1c0d2400 con 0x7f7a1c0b9410 2024-03-04T10:03:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a22830640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1c0d26e0 con 0x7f7a1c0b5700 2024-03-04T10:03:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a22830640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0d29c0 con 0x7f7a1c0d34a0 2024-03-04T10:03:14.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a20dcd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7a1c0b9410 0x7f7a1c162120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55182/0 (socket says 172.21.15.99:55182) 2024-03-04T10:03:14.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a20dcd640 1 -- 172.21.15.99:0/2151505644 learned_addr learned my addr 172.21.15.99:0/2151505644 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:14.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a1b7fe640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a1c161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:14.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a1b7fe640 1 -- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 msgr2=0x7f7a1c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-04T10:03:14.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a1b7fe640 1 -- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 msgr2=0x7f7a1c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:03:14.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a1b7fe640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a1c161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:03:14.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:14.982+0000 7f7a1b7fe640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a1c161be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T10:03:15.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105604225 0 0) 0x7f7a1c0d2400 con 0x7f7a1c0b9410 2024-03-04T10:03:15.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04003520 con 0x7f7a1c0b9410 2024-03-04T10:03:15.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 242924387 0 0) 0x7f7a04003520 con 0x7f7a1c0b9410 2024-03-04T10:03:15.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 msgr2=0x7f7a1c161be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-04T10:03:15.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a1c161be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 >> v1:172.21.15.162:6789/0 conn(0x7f7a1c0b5700 legacy=0x7f7a1c0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.181+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c0d2ca0 con 0x7f7a1c0b9410 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7a140019f0 con 0x7f7a1c0b9410 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 >> v1:172.21.15.99:6789/0 conn(0x7f7a1c0b9410 legacy=0x7f7a1c162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a040039e0 0x7f7a04003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a04004800 0x7f7a04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0d29c0 con 0x7f7a040039e0 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a040054f0 con 0x7f7a04004800 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0400bf00 con 0x7f7a1c0d34a0 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a040039e0 0x7f7a04003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1b7fe640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a04004800 0x7f7a04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a20dcd640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1bfff640 1 -- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 msgr2=0x7f7a0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0d34a0 0x7f7a0400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1bfff640 1 -- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a04004800 msgr2=0x7f7a04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a04004800 0x7f7a04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.182+0000 7f7a1bfff640 1 -- 172.21.15.99:0/2151505644 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a04003520 con 0x7f7a040039e0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.183+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a040039e0 0x7f7a04003e00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a10001750 tx=0x7f7a04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3689d6954ca742f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.183+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10001370 con 0x7f7a040039e0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.183+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a100052d0 con 0x7f7a040039e0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.183+0000 7f7a1affd640 1 -- 172.21.15.99:0/2151505644 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10003830 con 0x7f7a040039e0 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a22830640 1 -- 172.21.15.99:0/2151505644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a040039e0 msgr2=0x7f7a04003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a22830640 1 --2- 172.21.15.99:0/2151505644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a040039e0 0x7f7a04003e00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a10001750 tx=0x7f7a04005830 comp rx=0 tx=0).stop 2024-03-04T10:03:15.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a20dcd640 1 -- 172.21.15.99:0/2151505644 reap_dead start 2024-03-04T10:03:15.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a22830640 1 -- 172.21.15.99:0/2151505644 shutdown_connections 2024-03-04T10:03:15.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a22830640 1 -- 172.21.15.99:0/2151505644 >> 172.21.15.99:0/2151505644 conn(0x7f7a1c0d0490 msgr2=0x7f7a1c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:15.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a22830640 1 -- 172.21.15.99:0/2151505644 shutdown_connections 2024-03-04T10:03:15.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.184+0000 7f7a22830640 1 -- 172.21.15.99:0/2151505644 wait complete. 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.185+0000 7f7a22830640 1 Processor -- start 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.185+0000 7f7a22830640 1 -- start start 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a22830640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 0x7f7a1c15e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a22830640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0b7390 0x7f7a1c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a22830640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a1c0b7810 0x7f7a1c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a22830640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0b5c60 con 0x7f7a1c0b7810 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a22830640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0b5ae0 con 0x7f7a040039e0 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a22830640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0b5de0 con 0x7f7a1c0b7390 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 0x7f7a1c15e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a20dcd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a1c0b7810 0x7f7a1c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a1b7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0b7390 0x7f7a1c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a1bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 0x7f7a1c15e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:37264/0 (socket says 172.21.15.99:37264) 2024-03-04T10:03:15.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.186+0000 7f7a1bfff640 1 -- 172.21.15.99:0/3505269540 learned_addr learned my addr 172.21.15.99:0/3505269540 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:15.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a1bfff640 1 -- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0b7390 msgr2=0x7f7a1c0ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0b7390 0x7f7a1c0ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a1bfff640 1 -- 172.21.15.99:0/3505269540 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a1c0b7810 msgr2=0x7f7a1c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a1c0b7810 0x7f7a1c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a1bfff640 1 -- 172.21.15.99:0/3505269540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c15e6b0 con 0x7f7a040039e0 2024-03-04T10:03:15.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a20dcd640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a1c0b7810 0x7f7a1c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:15.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.187+0000 7f7a1bfff640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 0x7f7a1c15e170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7a10008fd0 tx=0x7f7a10001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10041070 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10005480 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c1f37c0 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a10001370 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c1f3d80 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7a10044070 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.191+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7a10046790 con 0x7f7a040039e0 2024-03-04T10:03:15.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.192+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c1f3540 con 0x7f7a040039e0 2024-03-04T10:03:15.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.199+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7a1c1f3540 con 0x7f7a040039e0 2024-03-04T10:03:15.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.318+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f7a1c0d1d50 con 0x7f7a040039e0 2024-03-04T10:03:15.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.321+0000 7f7a18ff9640 1 -- 172.21.15.99:0/3505269540 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f7a10003e10 con 0x7f7a040039e0 2024-03-04T10:03:15.323 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 msgr2=0x7f7a1c15e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 0x7f7a1c15e170 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7a10008fd0 tx=0x7f7a10001120 comp rx=0 tx=0).stop 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 shutdown_connections 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a1c0b7810 0x7f7a1c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a1c0b7390 0x7f7a1c0ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 --2- 172.21.15.99:0/3505269540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a040039e0 0x7f7a1c15e170 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 >> 172.21.15.99:0/3505269540 conn(0x7f7a1c0d0490 msgr2=0x7f7a1c15f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 shutdown_connections 2024-03-04T10:03:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:15.322+0000 7f7a22830640 1 -- 172.21.15.99:0/3505269540 wait complete. 2024-03-04T10:03:15.335 INFO:tasks.ceph:check_enable_crimson: False 2024-03-04T10:03:15.335 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-04T10:03:15.335 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:15.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-04T10:03:15.361 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:15.362 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-04T10:03:15.428 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:15.428 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-04T10:03:15.496 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:15.496 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-04T10:03:15.564 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:15.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-04T10:03:15.593 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:15.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-04T10:03:15.660 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:15.659+0000 7f780b63a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T10:03:15.661 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:15.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-04T10:03:15.716 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:15.715+0000 7f780b63a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T10:03:15.728 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:03:15.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-04T10:03:15.729 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:15.727+0000 7f713ce4a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T10:03:15.783 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:15.781+0000 7f713ce4a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T10:03:15.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 3873f63d-4fe8-4fd9-8e74-98a9d964f0fc 0 2024-03-04T10:03:15.936 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.935+0000 7fba4ceab640 1 Processor -- start 2024-03-04T10:03:15.937 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.935+0000 7fba4ceab640 1 -- start start 2024-03-04T10:03:15.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4ceab640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba4806ff60 0x7fba480b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.938 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4ceab640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fba480cd820 0x7fba4815a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4ceab640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba48070340 con 0x7fba480cf190 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4ceab640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba480cc050 con 0x7fba4806ff60 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4ceab640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4806f530 con 0x7fba480cd820 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fba480cd820 0x7fba4815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fba480cd820 0x7fba4815a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53344/0 (socket says 172.21.15.162:53344) 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba45d9b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fba480cf190 0x7fba480cd110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55694/0 (socket says 172.21.15.162:55694) 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 -- 172.21.15.162:0/2513707127 learned_addr learned my addr 172.21.15.162:0/2513707127 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4659c640 1 --2- 172.21.15.162:0/2513707127 >> v2:172.21.15.99:3300/0 conn(0x7fba4806ff60 0x7fba480b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 -- 172.21.15.162:0/2513707127 >> v2:172.21.15.99:3300/0 conn(0x7fba4806ff60 msgr2=0x7fba480b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 --2- 172.21.15.162:0/2513707127 >> v2:172.21.15.99:3300/0 conn(0x7fba4806ff60 0x7fba480b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 -- 172.21.15.162:0/2513707127 >> v1:172.21.15.162:6789/0 conn(0x7fba480cf190 legacy=0x7fba480cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4559a640 1 -- 172.21.15.162:0/2513707127 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171399042 0 0) 0x7fba48070340 con 0x7fba480cf190 2024-03-04T10:03:15.939 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba46d9d640 1 -- 172.21.15.162:0/2513707127 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480cb440 con 0x7fba480cd820 2024-03-04T10:03:15.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.936+0000 7fba4659c640 1 --2- 172.21.15.162:0/2513707127 >> v2:172.21.15.99:3300/0 conn(0x7fba4806ff60 0x7fba480b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:15.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.937+0000 7fba46d9d640 1 --2- 172.21.15.162:0/2513707127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cd820 0x7fba4815a350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fba4815f610 tx=0x7fba4002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa881e8992fd035d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:15.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.937+0000 7fba4559a640 1 -- 172.21.15.162:0/2513707127 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4002cb20 con 0x7fba480cd820 2024-03-04T10:03:15.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.937+0000 7fba4559a640 1 -- 172.21.15.162:0/2513707127 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4002ccc0 con 0x7fba480cd820 2024-03-04T10:03:15.940 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.937+0000 7fba4559a640 1 -- 172.21.15.162:0/2513707127 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40034610 con 0x7fba480cd820 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 -- 172.21.15.162:0/2513707127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cd820 msgr2=0x7fba4815a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 --2- 172.21.15.162:0/2513707127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cd820 0x7fba4815a350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fba4815f610 tx=0x7fba4002bf80 comp rx=0 tx=0).stop 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 -- 172.21.15.162:0/2513707127 shutdown_connections 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 --2- 172.21.15.162:0/2513707127 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cd820 0x7fba4815a350 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 --2- 172.21.15.162:0/2513707127 >> v2:172.21.15.99:3300/0 conn(0x7fba4806ff60 0x7fba480b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 -- 172.21.15.162:0/2513707127 >> 172.21.15.162:0/2513707127 conn(0x7fba480b2250 msgr2=0x7fba480b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 -- 172.21.15.162:0/2513707127 shutdown_connections 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.938+0000 7fba4ceab640 1 -- 172.21.15.162:0/2513707127 wait complete. 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.939+0000 7fba4ceab640 1 Processor -- start 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.939+0000 7fba4ceab640 1 -- start start 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4ceab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba4806ff60 0x7fba480c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4ceab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba480cd820 0x7fba481f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.943 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4ceab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 0x7fba480b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4ceab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cbce0 con 0x7fba480cd820 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4ceab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cba00 con 0x7fba480cf190 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4ceab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba480cc4e0 con 0x7fba4806ff60 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 0x7fba480b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 0x7fba480b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53354/0 (socket says 172.21.15.162:53354) 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 -- 172.21.15.162:0/669046496 learned_addr learned my addr 172.21.15.162:0/669046496 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba4659c640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba4806ff60 0x7fba480c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba45d9b640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba480cd820 0x7fba481f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 -- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba4806ff60 msgr2=0x7fba480c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba4806ff60 0x7fba480c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 -- 172.21.15.162:0/669046496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba480cd820 msgr2=0x7fba481f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba480cd820 0x7fba481f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba46d9d640 1 -- 172.21.15.162:0/669046496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba481f4b10 con 0x7fba480cf190 2024-03-04T10:03:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.940+0000 7fba45d9b640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba480cd820 0x7fba481f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba46d9d640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 0x7fba480b3810 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fba40009f40 tx=0x7fba40004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40043020 con 0x7fba480cf190 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4003e450 con 0x7fba480cf190 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40009510 con 0x7fba480cf190 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba480b4ba0 con 0x7fba480cf190 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba480b5160 con 0x7fba480cf190 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fba40046020 con 0x7fba480cf190 2024-03-04T10:03:15.945 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.941+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fba40048450 con 0x7fba480cf190 2024-03-04T10:03:15.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.942+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba480b5930 con 0x7fba480cf190 2024-03-04T10:03:15.946 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:15.944+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fba4004d030 con 0x7fba480cf190 2024-03-04T10:03:16.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.016+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3873f63d-4fe8-4fd9-8e74-98a9d964f0fc", "id": 0} v 0) v1 -- 0x7fba480b35f0 con 0x7fba480cf190 2024-03-04T10:03:16.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.019+0000 7fba36ffd640 1 -- 172.21.15.162:0/669046496 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3873f63d-4fe8-4fd9-8e74-98a9d964f0fc", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fba481f4b10 con 0x7fba480cf190 2024-03-04T10:03:16.021 INFO:teuthology.orchestra.run.smithi162.stdout:0 2024-03-04T10:03:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 msgr2=0x7fba480b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 0x7fba480b3810 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fba40009f40 tx=0x7fba40004a40 comp rx=0 tx=0).stop 2024-03-04T10:03:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 shutdown_connections 2024-03-04T10:03:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba480cf190 0x7fba480b3810 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba480cd820 0x7fba481f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.022 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 --2- 172.21.15.162:0/669046496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba4806ff60 0x7fba480c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 >> 172.21.15.162:0/669046496 conn(0x7fba480b2250 msgr2=0x7fba4806acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 shutdown_connections 2024-03-04T10:03:16.023 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.020+0000 7fba4ceab640 1 -- 172.21.15.162:0/669046496 wait complete. 2024-03-04T10:03:16.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 76020627-44f6-4141-93e8-3c45e26c6ee6 1 2024-03-04T10:03:16.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e68cd640 1 Processor -- start 2024-03-04T10:03:16.133 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e68cd640 1 -- start start 2024-03-04T10:03:16.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e68cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e006ff60 0x7f29e00b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e68cd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e0070340 con 0x7f29e00cf190 2024-03-04T10:03:16.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e68cd640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e00cb440 con 0x7f29e00cd820 2024-03-04T10:03:16.134 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e68cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00cc050 con 0x7f29e006ff60 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29dffff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e006ff60 0x7f29e00b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e4e6a640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f29e00cd820 0x7f29e015a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55710/0 (socket says 172.21.15.162:55710) 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.132+0000 7f29e4e6a640 1 -- 172.21.15.162:0/3682591075 learned_addr learned my addr 172.21.15.162:0/3682591075 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.133+0000 7f29deffd640 1 -- 172.21.15.162:0/3682591075 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941700495 0 0) 0x7f29e00cb440 con 0x7f29e00cd820 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.133+0000 7f29dffff640 1 -- 172.21.15.162:0/3682591075 >> v1:172.21.15.162:6789/0 conn(0x7f29e00cd820 legacy=0x7f29e015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.133+0000 7f29dffff640 1 -- 172.21.15.162:0/3682591075 >> v1:172.21.15.99:6789/0 conn(0x7f29e00cf190 legacy=0x7f29e00cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.135 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.133+0000 7f29dffff640 1 -- 172.21.15.162:0/3682591075 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e00cb720 con 0x7f29e006ff60 2024-03-04T10:03:16.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.133+0000 7f29dffff640 1 --2- 172.21.15.162:0/3682591075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e006ff60 0x7f29e00b2d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29d000ba70 tx=0x7f29d002e110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e19a1dae13c057d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29deffd640 1 -- 172.21.15.162:0/3682591075 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d003b070 con 0x7f29e006ff60 2024-03-04T10:03:16.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29deffd640 1 -- 172.21.15.162:0/3682591075 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d0036440 con 0x7f29e006ff60 2024-03-04T10:03:16.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29deffd640 1 -- 172.21.15.162:0/3682591075 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d0036810 con 0x7f29e006ff60 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 -- 172.21.15.162:0/3682591075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e006ff60 msgr2=0x7f29e00b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 --2- 172.21.15.162:0/3682591075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e006ff60 0x7f29e00b2d70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29d000ba70 tx=0x7f29d002e110 comp rx=0 tx=0).stop 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 -- 172.21.15.162:0/3682591075 shutdown_connections 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 --2- 172.21.15.162:0/3682591075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e006ff60 0x7f29e00b2d70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 -- 172.21.15.162:0/3682591075 >> 172.21.15.162:0/3682591075 conn(0x7f29e00b2250 msgr2=0x7f29e00b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 -- 172.21.15.162:0/3682591075 shutdown_connections 2024-03-04T10:03:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.134+0000 7f29e68cd640 1 -- 172.21.15.162:0/3682591075 wait complete. 2024-03-04T10:03:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.135+0000 7f29e68cd640 1 Processor -- start 2024-03-04T10:03:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 -- start start 2024-03-04T10:03:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29e006ff60 0x7f29e00b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e00cd820 0x7f29e00c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 0x7f29e0159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00cbce0 con 0x7f29e006ff60 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e006f530 con 0x7f29e00cd820 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e68cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e00cc4e0 con 0x7f29e00cf190 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e4e6a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 0x7f29e0159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29df7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e00cd820 0x7f29e00c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e4e6a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 0x7f29e0159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40294/0 (socket says 172.21.15.162:40294) 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.136+0000 7f29e4e6a640 1 -- 172.21.15.162:0/3635874926 learned_addr learned my addr 172.21.15.162:0/3635874926 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dffff640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29e006ff60 0x7f29e00b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29df7fe640 1 -- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 msgr2=0x7f29e0159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29df7fe640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 0x7f29e0159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29df7fe640 1 -- 172.21.15.162:0/3635874926 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29e006ff60 msgr2=0x7f29e00b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29df7fe640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29e006ff60 0x7f29e00b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29df7fe640 1 -- 172.21.15.162:0/3635874926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e01f2d20 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29e4e6a640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 0x7f29e0159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dffff640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29e006ff60 0x7f29e00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29df7fe640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e00cd820 0x7f29e00c6b70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f29d400c3c0 tx=0x7f29d400c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4017070 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d40052d0 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4003810 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e01f3000 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e01f35c0 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f29d401a070 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.137+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f29d4011cb0 con 0x7f29e00cd820 2024-03-04T10:03:16.141 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.138+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e0074f60 con 0x7f29e00cd820 2024-03-04T10:03:16.142 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.140+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f29d4021070 con 0x7f29e00cd820 2024-03-04T10:03:16.215 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.214+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd new", "uuid": "76020627-44f6-4141-93e8-3c45e26c6ee6", "id": 1} v 0) v1 -- 0x7f29e0072760 con 0x7f29e00cd820 2024-03-04T10:03:16.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.217+0000 7f29dcff9640 1 -- 172.21.15.162:0/3635874926 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76020627-44f6-4141-93e8-3c45e26c6ee6", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f29d4011590 con 0x7f29e00cd820 2024-03-04T10:03:16.219 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e00cd820 msgr2=0x7f29e00c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e00cd820 0x7f29e00c6b70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f29d400c3c0 tx=0x7f29d400c7c0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 shutdown_connections 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29e00cf190 0x7f29e0159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29e00cd820 0x7f29e00c6b70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 --2- 172.21.15.162:0/3635874926 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29e006ff60 0x7f29e00b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 >> 172.21.15.162:0/3635874926 conn(0x7f29e00b2250 msgr2=0x7f29e015ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.218+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 shutdown_connections 2024-03-04T10:03:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.219+0000 7f29e68cd640 1 -- 172.21.15.162:0/3635874926 wait complete. 2024-03-04T10:03:16.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 9c243730-6234-466a-8471-62263dda0cc2 2 2024-03-04T10:03:16.282 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.280+0000 7f780b63a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T10:03:16.300 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.298+0000 7f713ce4a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T10:03:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.332+0000 7fae7efd4640 1 Processor -- start 2024-03-04T10:03:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.332+0000 7fae7efd4640 1 -- start start 2024-03-04T10:03:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7efd4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fae7806ff60 0x7fae780b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7efd4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fae780cf190 0x7fae780cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7efd4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae78070340 con 0x7fae780cd650 2024-03-04T10:03:16.335 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7efd4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae780cc050 con 0x7fae780cf190 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7efd4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7806f530 con 0x7fae7806ff60 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7cd70640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fae7806ff60 0x7fae780b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7cd70640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fae7806ff60 0x7fae780b2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53364/0 (socket says 172.21.15.162:53364) 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7cd70640 1 -- 172.21.15.162:0/999961629 learned_addr learned my addr 172.21.15.162:0/999961629 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae7d571640 1 --1- 172.21.15.162:0/999961629 >> v1:172.21.15.162:6789/0 conn(0x7fae780cd650 0x7fae7815a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55724/0 (socket says 172.21.15.162:55724) 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.333+0000 7fae77fff640 1 --2- 172.21.15.162:0/999961629 >> v2:172.21.15.99:3300/0 conn(0x7fae780cf190 0x7fae780cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae7cd70640 1 -- 172.21.15.162:0/999961629 >> v2:172.21.15.99:3300/0 conn(0x7fae780cf190 msgr2=0x7fae780cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae7cd70640 1 --2- 172.21.15.162:0/999961629 >> v2:172.21.15.99:3300/0 conn(0x7fae780cf190 0x7fae780cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae7cd70640 1 -- 172.21.15.162:0/999961629 >> v1:172.21.15.162:6789/0 conn(0x7fae780cd650 legacy=0x7fae7815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae7cd70640 1 -- 172.21.15.162:0/999961629 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780cb440 con 0x7fae7806ff60 2024-03-04T10:03:16.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae777fe640 1 -- 172.21.15.162:0/999961629 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6149645 0 0) 0x7fae78070340 con 0x7fae780cd650 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae7cd70640 1 --2- 172.21.15.162:0/999961629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae7806ff60 0x7fae780b2d70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae780ce280 tx=0x7fae6c02e170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b01fb165a0f2f9b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae777fe640 1 -- 172.21.15.162:0/999961629 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c03b070 con 0x7fae7806ff60 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.334+0000 7fae777fe640 1 -- 172.21.15.162:0/999961629 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c036440 con 0x7fae7806ff60 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae777fe640 1 -- 172.21.15.162:0/999961629 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c036810 con 0x7fae7806ff60 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 -- 172.21.15.162:0/999961629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae7806ff60 msgr2=0x7fae780b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 --2- 172.21.15.162:0/999961629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae7806ff60 0x7fae780b2d70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae780ce280 tx=0x7fae6c02e170 comp rx=0 tx=0).stop 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 -- 172.21.15.162:0/999961629 shutdown_connections 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 --2- 172.21.15.162:0/999961629 >> v2:172.21.15.99:3300/0 conn(0x7fae780cf190 0x7fae780cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 --2- 172.21.15.162:0/999961629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae7806ff60 0x7fae780b2d70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 -- 172.21.15.162:0/999961629 >> 172.21.15.162:0/999961629 conn(0x7fae780b2250 msgr2=0x7fae780b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 -- 172.21.15.162:0/999961629 shutdown_connections 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.335+0000 7fae7efd4640 1 -- 172.21.15.162:0/999961629 wait complete. 2024-03-04T10:03:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.336+0000 7fae7efd4640 1 Processor -- start 2024-03-04T10:03:16.339 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.336+0000 7fae7efd4640 1 -- start start 2024-03-04T10:03:16.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7efd4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fae7806ff60 0x7fae780c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7efd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 0x7fae780d0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7efd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 0x7fae781538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7efd4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae780cbce0 con 0x7fae7806ff60 2024-03-04T10:03:16.340 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7efd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae780cba00 con 0x7fae780cf190 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7efd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae780cc4e0 con 0x7fae780cd650 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7d571640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 0x7fae781538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae77fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 0x7fae780d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7d571640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 0x7fae781538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53370/0 (socket says 172.21.15.162:53370) 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae77fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 0x7fae780d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40306/0 (socket says 172.21.15.162:40306) 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7d571640 1 -- 172.21.15.162:0/585898202 learned_addr learned my addr 172.21.15.162:0/585898202 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.337+0000 7fae7cd70640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fae7806ff60 0x7fae780c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7d571640 1 -- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 msgr2=0x7fae780d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7d571640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 0x7fae780d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7d571640 1 -- 172.21.15.162:0/585898202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fae7806ff60 msgr2=0x7fae780c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7d571640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fae7806ff60 0x7fae780c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7d571640 1 -- 172.21.15.162:0/585898202 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780c7dc0 con 0x7fae780cf190 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7cd70640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fae7806ff60 0x7fae780c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.341 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae77fff640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 0x7fae780d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.342 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.338+0000 7f780b63a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7d571640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 0x7fae781538f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae7000c700 tx=0x7fae7000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70018070 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7000ce10 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae780c8350 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7000a860 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae78154ca0 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fae7001b070 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.338+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fae700077b0 con 0x7fae780cf190 2024-03-04T10:03:16.343 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.339+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae78074f60 con 0x7fae780cf190 2024-03-04T10:03:16.344 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.343+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fae700070d0 con 0x7fae780cf190 2024-03-04T10:03:16.354 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.352+0000 7f713ce4a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T10:03:16.387 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.386+0000 7f780b63a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T10:03:16.399 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.397+0000 7f713ce4a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T10:03:16.416 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.415+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9c243730-6234-466a-8471-62263dda0cc2", "id": 2} v 0) v1 -- 0x7fae78072760 con 0x7fae780cf190 2024-03-04T10:03:16.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.421+0000 7fae757fa640 1 -- 172.21.15.162:0/585898202 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c243730-6234-466a-8471-62263dda0cc2", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fae7000aaa0 con 0x7fae780cf190 2024-03-04T10:03:16.422 INFO:teuthology.orchestra.run.smithi162.stdout:2 2024-03-04T10:03:16.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 msgr2=0x7fae781538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 0x7fae781538f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fae7000c700 tx=0x7fae7000cbc0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 shutdown_connections 2024-03-04T10:03:16.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fae780cf190 0x7fae781538f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.424 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fae780cd650 0x7fae780d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 --2- 172.21.15.162:0/585898202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fae7806ff60 0x7fae780c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 >> 172.21.15.162:0/585898202 conn(0x7fae780b2250 msgr2=0x7fae7815d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 shutdown_connections 2024-03-04T10:03:16.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.422+0000 7fae7efd4640 1 -- 172.21.15.162:0/585898202 wait complete. 2024-03-04T10:03:16.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new c9d1258d-a78b-484c-a559-f05e662abdda 3 2024-03-04T10:03:16.444 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.443+0000 7f780b63a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T10:03:16.453 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.452+0000 7f713ce4a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T10:03:16.490 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.489+0000 7f780b63a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T10:03:16.495 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.494+0000 7f713ce4a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T10:03:16.535 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.534+0000 7f926f4f1640 1 Processor -- start 2024-03-04T10:03:16.536 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.534+0000 7f926f4f1640 1 -- start start 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926f4f1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f926806ff60 0x7f92680b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926f4f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f92680cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926f4f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f92680cd650 0x7f926815a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926f4f1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92680cc050 con 0x7f926806ff60 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926f4f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9268070340 con 0x7f92680cd650 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926f4f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92680cb440 con 0x7f92680cf190 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926ca8c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f92680cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926da8e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f92680cd650 0x7f926815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.538 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926da8e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f92680cd650 0x7f926815a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53384/0 (socket says 172.21.15.162:53384) 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926da8e640 1 -- 172.21.15.162:0/4058684440 learned_addr learned my addr 172.21.15.162:0/4058684440 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926ca8c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f92680cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40310/0 (socket says 172.21.15.162:40310) 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.535+0000 7f926d28d640 1 --2- 172.21.15.162:0/4058684440 >> v2:172.21.15.99:3300/0 conn(0x7f926806ff60 0x7f92680b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.536+0000 7f926da8e640 1 -- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 msgr2=0x7f92680cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.536+0000 7f926da8e640 1 --2- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f92680cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.536+0000 7f926da8e640 1 -- 172.21.15.162:0/4058684440 >> v2:172.21.15.99:3300/0 conn(0x7f926806ff60 msgr2=0x7f92680b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.536+0000 7f926da8e640 1 --2- 172.21.15.162:0/4058684440 >> v2:172.21.15.99:3300/0 conn(0x7f926806ff60 0x7f92680b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.536+0000 7f926da8e640 1 -- 172.21.15.162:0/4058684440 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680cb720 con 0x7f92680cd650 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.536+0000 7f926da8e640 1 --2- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926815a350 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f926806cfe0 tx=0x7f926002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3a4c2ae9d4246fd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f925ffff640 1 -- 172.21.15.162:0/4058684440 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926002ca80 con 0x7f92680cd650 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f925ffff640 1 -- 172.21.15.162:0/4058684440 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926002cc20 con 0x7f92680cd650 2024-03-04T10:03:16.539 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f925ffff640 1 -- 172.21.15.162:0/4058684440 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92600343f0 con 0x7f92680cd650 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 -- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 msgr2=0x7f926815a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926815a350 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f926806cfe0 tx=0x7f926002bee0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 -- 172.21.15.162:0/4058684440 shutdown_connections 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926815a350 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4058684440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f92680cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4058684440 >> v2:172.21.15.99:3300/0 conn(0x7f926806ff60 0x7f92680b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 -- 172.21.15.162:0/4058684440 >> 172.21.15.162:0/4058684440 conn(0x7f92680b2250 msgr2=0x7f92680b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 -- 172.21.15.162:0/4058684440 shutdown_connections 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.537+0000 7f926f4f1640 1 -- 172.21.15.162:0/4058684440 wait complete. 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.539+0000 7f926f4f1640 1 Processor -- start 2024-03-04T10:03:16.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.539+0000 7f926f4f1640 1 -- start start 2024-03-04T10:03:16.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926f4f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f926806ff60 0x7f926813b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926f4f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926813bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926f4f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f9268150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.545 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926f4f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680cc4c0 con 0x7f926806ff60 2024-03-04T10:03:16.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926f4f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680cbce0 con 0x7f92680cd650 2024-03-04T10:03:16.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926f4f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92680cc640 con 0x7f92680cf190 2024-03-04T10:03:16.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926ca8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926813bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926da8e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f9268150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926da8e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f9268150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40322/0 (socket says 172.21.15.162:40322) 2024-03-04T10:03:16.546 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926da8e640 1 -- 172.21.15.162:0/4095045457 learned_addr learned my addr 172.21.15.162:0/4095045457 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:16.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926ca8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926813bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53396/0 (socket says 172.21.15.162:53396) 2024-03-04T10:03:16.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.540+0000 7f926d28d640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f926806ff60 0x7f926813b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926da8e640 1 -- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 msgr2=0x7f926813bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.547 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926da8e640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926813bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926da8e640 1 -- 172.21.15.162:0/4095045457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f926806ff60 msgr2=0x7f926813b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926da8e640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f926806ff60 0x7f926813b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926da8e640 1 -- 172.21.15.162:0/4095045457 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92681515c0 con 0x7f92680cf190 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926ca8c640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926813bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926d28d640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f926806ff60 0x7f926813b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926da8e640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f9268150730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9260036040 tx=0x7f9260004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260043020 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926003e450 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9260009510 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92681518a0 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.541+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9268151e60 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.542+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9260046020 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.542+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f9260048890 con 0x7f92680cf190 2024-03-04T10:03:16.549 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.542+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9268074f60 con 0x7f92680cf190 2024-03-04T10:03:16.550 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.544+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f926004d030 con 0x7f92680cf190 2024-03-04T10:03:16.610 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.608+0000 7f713ce4a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T10:03:16.614 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.612+0000 7f780b63a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T10:03:16.618 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.617+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c9d1258d-a78b-484c-a559-f05e662abdda", "id": 3} v 0) v1 -- 0x7f92680c6250 con 0x7f92680cf190 2024-03-04T10:03:16.623 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.622+0000 7f925dffb640 1 -- 172.21.15.162:0/4095045457 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c9d1258d-a78b-484c-a559-f05e662abdda", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f926002ca90 con 0x7f92680cf190 2024-03-04T10:03:16.624 INFO:teuthology.orchestra.run.smithi162.stdout:3 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 msgr2=0x7f9268150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f9268150730 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9260036040 tx=0x7f9260004a40 comp rx=0 tx=0).stop 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 shutdown_connections 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92680cf190 0x7f9268150730 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f92680cd650 0x7f926813bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 --2- 172.21.15.162:0/4095045457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f926806ff60 0x7f926813b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 >> 172.21.15.162:0/4095045457 conn(0x7f92680b2250 msgr2=0x7f92681580f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 shutdown_connections 2024-03-04T10:03:16.625 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.623+0000 7f926f4f1640 1 -- 172.21.15.162:0/4095045457 wait complete. 2024-03-04T10:03:16.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 017c5017-a6a3-4d06-b77d-b1de332c24cc 4 2024-03-04T10:03:16.678 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.677+0000 7f713ce4a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T10:03:16.688 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.687+0000 7f780b63a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T10:03:16.725 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.723+0000 7f713ce4a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T10:03:16.737 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.736+0000 7f780b63a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T10:03:16.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.776+0000 7fab768d1640 1 Processor -- start 2024-03-04T10:03:16.778 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.776+0000 7fab768d1640 1 -- start start 2024-03-04T10:03:16.778 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.776+0000 7f713ce4a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab768d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fab7006ff60 0x7fab700b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab768d1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab70070340 con 0x7fab700cd820 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab768d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700cb440 con 0x7fab700cf190 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab768d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700cc050 con 0x7fab7006ff60 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab6f7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fab700cf190 0x7fab700cd110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55726/0 (socket says 172.21.15.162:55726) 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab6f7fe640 1 -- 172.21.15.162:0/3509549066 learned_addr learned my addr 172.21.15.162:0/3509549066 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab6ffff640 1 --2- 172.21.15.162:0/3509549066 >> v2:172.21.15.99:3300/0 conn(0x7fab7006ff60 0x7fab700b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab6effd640 1 -- 172.21.15.162:0/3509549066 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895124638 0 0) 0x7fab700cb440 con 0x7fab700cf190 2024-03-04T10:03:16.780 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.777+0000 7fab6effd640 1 -- 172.21.15.162:0/3509549066 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab58003520 con 0x7fab700cf190 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.778+0000 7fab6ffff640 1 -- 172.21.15.162:0/3509549066 >> v1:172.21.15.162:6789/0 conn(0x7fab700cf190 legacy=0x7fab700cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.778+0000 7fab6ffff640 1 -- 172.21.15.162:0/3509549066 >> v1:172.21.15.99:6789/0 conn(0x7fab700cd820 legacy=0x7fab7015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.778+0000 7fab6ffff640 1 -- 172.21.15.162:0/3509549066 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700cb720 con 0x7fab7006ff60 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.778+0000 7fab6ffff640 1 --2- 172.21.15.162:0/3509549066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab7006ff60 0x7fab700b2d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fab640097a0 tx=0x7fab6402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62064acc0e19afc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab6effd640 1 -- 172.21.15.162:0/3509549066 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64037070 con 0x7fab7006ff60 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab6effd640 1 -- 172.21.15.162:0/3509549066 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab64030710 con 0x7fab7006ff60 2024-03-04T10:03:16.781 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab6effd640 1 -- 172.21.15.162:0/3509549066 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64030ae0 con 0x7fab7006ff60 2024-03-04T10:03:16.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 -- 172.21.15.162:0/3509549066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab7006ff60 msgr2=0x7fab700b2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 --2- 172.21.15.162:0/3509549066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab7006ff60 0x7fab700b2d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fab640097a0 tx=0x7fab6402b790 comp rx=0 tx=0).stop 2024-03-04T10:03:16.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 -- 172.21.15.162:0/3509549066 shutdown_connections 2024-03-04T10:03:16.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 --2- 172.21.15.162:0/3509549066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab7006ff60 0x7fab700b2d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.784 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 -- 172.21.15.162:0/3509549066 >> 172.21.15.162:0/3509549066 conn(0x7fab700b2250 msgr2=0x7fab700b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 -- 172.21.15.162:0/3509549066 shutdown_connections 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.779+0000 7fab768d1640 1 -- 172.21.15.162:0/3509549066 wait complete. 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.780+0000 7fab768d1640 1 Processor -- start 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 -- start start 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab7006ff60 0x7fab701e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 0x7fab7015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab700cf190 0x7fab700c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700cbce0 con 0x7fab700cd820 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7006f530 con 0x7fab7006ff60 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab768d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700cc4e0 con 0x7fab700cf190 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab6ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab7006ff60 0x7fab701e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab6f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 0x7fab7015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab74e6e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab700cf190 0x7fab700c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:16.785 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab6f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 0x7fab7015acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.162:35542/0 (socket says 172.21.15.162:35542) 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.781+0000 7fab6f7fe640 1 -- 172.21.15.162:0/3143422543 learned_addr learned my addr 172.21.15.162:0/3143422543 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6f7fe640 1 -- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab700cf190 msgr2=0x7fab700c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6f7fe640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab700cf190 0x7fab700c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6f7fe640 1 -- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab7006ff60 msgr2=0x7fab701e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6f7fe640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab7006ff60 0x7fab701e43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6f7fe640 1 -- 172.21.15.162:0/3143422543 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700c7500 con 0x7fab700cd820 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6ffff640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab7006ff60 0x7fab701e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6f7fe640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 0x7fab7015acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fab6000c500 tx=0x7fab6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60017070 con 0x7fab700cd820 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab60005ce0 con 0x7fab700cd820 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab700aee60 con 0x7fab700cd820 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60007330 con 0x7fab700cd820 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab701e4e10 con 0x7fab700cd820 2024-03-04T10:03:16.786 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fab6001a070 con 0x7fab700cd820 2024-03-04T10:03:16.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.782+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fab60012910 con 0x7fab700cd820 2024-03-04T10:03:16.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.783+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700732f0 con 0x7fab700cd820 2024-03-04T10:03:16.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.785+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fab700732f0 con 0x7fab700cd820 2024-03-04T10:03:16.793 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.792+0000 7f780b63a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T10:03:16.826 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:16.824+0000 7f713ce4a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T10:03:16.842 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:16.841+0000 7f780b63a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T10:03:16.859 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.858+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd new", "uuid": "017c5017-a6a3-4d06-b77d-b1de332c24cc", "id": 4} v 0) v1 -- 0x7fab70074f60 con 0x7fab700cd820 2024-03-04T10:03:16.863 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.862+0000 7fab6cff9640 1 -- 172.21.15.162:0/3143422543 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "017c5017-a6a3-4d06-b77d-b1de332c24cc", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fab60021070 con 0x7fab700cd820 2024-03-04T10:03:16.864 INFO:teuthology.orchestra.run.smithi162.stdout:4 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 msgr2=0x7fab7015acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 0x7fab7015acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fab6000c500 tx=0x7fab6000c9c0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 shutdown_connections 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab700cf190 0x7fab700c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab700cd820 0x7fab7015acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 --2- 172.21.15.162:0/3143422543 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab7006ff60 0x7fab701e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 >> 172.21.15.162:0/3143422543 conn(0x7fab700b2250 msgr2=0x7fab700b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:16.865 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 shutdown_connections 2024-03-04T10:03:16.866 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:16.863+0000 7fab768d1640 1 -- 172.21.15.162:0/3143422543 wait complete. 2024-03-04T10:03:16.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new 92708c04-f7da-4999-b907-96500f70e153 5 2024-03-04T10:03:17.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.015+0000 7fe08a7b2640 1 Processor -- start 2024-03-04T10:03:17.017 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe08a7b2640 1 -- start start 2024-03-04T10:03:17.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe08a7b2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe08406acd0 0x7fe0840ad530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.018 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe08a7b2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0840cc7a0 con 0x7fe0840ada70 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe08a7b2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0840cca80 con 0x7fe08415bc50 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe08a7b2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0840adee0 con 0x7fe08406acd0 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe0837fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe08406acd0 0x7fe0840ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe083fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe08415bc50 0x7fe0840cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55740/0 (socket says 172.21.15.162:55740) 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe083fff640 1 -- 172.21.15.162:0/1405636421 learned_addr learned my addr 172.21.15.162:0/1405636421 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.016+0000 7fe0837fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe08406acd0 0x7fe0840ad530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53416/0 (socket says 172.21.15.162:53416) 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.017+0000 7fe0837fe640 1 -- 172.21.15.162:0/1405636421 >> v1:172.21.15.162:6789/0 conn(0x7fe08415bc50 legacy=0x7fe0840cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.017+0000 7fe0837fe640 1 -- 172.21.15.162:0/1405636421 >> v1:172.21.15.99:6789/0 conn(0x7fe0840ada70 legacy=0x7fe0840cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.017+0000 7fe082ffd640 1 -- 172.21.15.162:0/1405636421 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360451216 0 0) 0x7fe0840cca80 con 0x7fe08415bc50 2024-03-04T10:03:17.019 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.017+0000 7fe0837fe640 1 -- 172.21.15.162:0/1405636421 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840ccd60 con 0x7fe08406acd0 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.018+0000 7fe0837fe640 1 --2- 172.21.15.162:0/1405636421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe08406acd0 0x7fe0840ad530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe0740097a0 tx=0x7fe07402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133f76c60a1ffca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.018+0000 7fe082ffd640 1 -- 172.21.15.162:0/1405636421 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074034070 con 0x7fe08406acd0 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.018+0000 7fe082ffd640 1 -- 172.21.15.162:0/1405636421 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe074030610 con 0x7fe08406acd0 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.018+0000 7fe082ffd640 1 -- 172.21.15.162:0/1405636421 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0740309e0 con 0x7fe08406acd0 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 -- 172.21.15.162:0/1405636421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe08406acd0 msgr2=0x7fe0840ad530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 --2- 172.21.15.162:0/1405636421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe08406acd0 0x7fe0840ad530 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe0740097a0 tx=0x7fe07402b960 comp rx=0 tx=0).stop 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 -- 172.21.15.162:0/1405636421 shutdown_connections 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 --2- 172.21.15.162:0/1405636421 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe08406acd0 0x7fe0840ad530 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 -- 172.21.15.162:0/1405636421 >> 172.21.15.162:0/1405636421 conn(0x7fe0840b2250 msgr2=0x7fe0840b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 -- 172.21.15.162:0/1405636421 shutdown_connections 2024-03-04T10:03:17.021 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.019+0000 7fe08a7b2640 1 -- 172.21.15.162:0/1405636421 wait complete. 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.020+0000 7fe08a7b2640 1 Processor -- start 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.020+0000 7fe08a7b2640 1 -- start start 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe08a7b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe08406acd0 0x7fe0840c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe08a7b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 0x7fe0841557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe08a7b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe08415bc50 0x7fe0841563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe08a7b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840cdf50 con 0x7fe08406acd0 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe08a7b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840cd320 con 0x7fe0840ada70 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe08a7b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840cc330 con 0x7fe08415bc50 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 0x7fe0841557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe08415bc50 0x7fe0841563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 0x7fe0841557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53426/0 (socket says 172.21.15.162:53426) 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe0837fe640 1 -- 172.21.15.162:0/2628084338 learned_addr learned my addr 172.21.15.162:0/2628084338 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe083fff640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe08406acd0 0x7fe0840c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 -- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 msgr2=0x7fe0841557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 0x7fe0841557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 -- 172.21.15.162:0/2628084338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe08406acd0 msgr2=0x7fe0840c7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe08406acd0 0x7fe0840c7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 -- 172.21.15.162:0/2628084338 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084155ce0 con 0x7fe08415bc50 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe0837fe640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 0x7fe0841557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe088d4f640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe08415bc50 0x7fe0841563a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c690 tx=0x7fe07c00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.021+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c003b70 con 0x7fe08415bc50 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.022+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c003d10 con 0x7fe08415bc50 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.022+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0840c8390 con 0x7fe08415bc50 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.022+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c005310 con 0x7fe08415bc50 2024-03-04T10:03:17.025 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.022+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0840c8950 con 0x7fe08415bc50 2024-03-04T10:03:17.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.022+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe07c0054b0 con 0x7fe08415bc50 2024-03-04T10:03:17.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.022+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fe07c0129a0 con 0x7fe08415bc50 2024-03-04T10:03:17.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.023+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08406cd70 con 0x7fe08415bc50 2024-03-04T10:03:17.026 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.025+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe07c012c80 con 0x7fe08415bc50 2024-03-04T10:03:17.099 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.097+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd new", "uuid": "92708c04-f7da-4999-b907-96500f70e153", "id": 5} v 0) v1 -- 0x7fe084074f60 con 0x7fe08415bc50 2024-03-04T10:03:17.104 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.103+0000 7fe080ff9640 1 -- 172.21.15.162:0/2628084338 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "92708c04-f7da-4999-b907-96500f70e153", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe07c012420 con 0x7fe08415bc50 2024-03-04T10:03:17.104 INFO:teuthology.orchestra.run.smithi162.stdout:5 2024-03-04T10:03:17.105 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe08415bc50 msgr2=0x7fe0841563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe08415bc50 0x7fe0841563a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c690 tx=0x7fe07c00cb50 comp rx=0 tx=0).stop 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 shutdown_connections 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe08415bc50 0x7fe0841563a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0840ada70 0x7fe0841557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 --2- 172.21.15.162:0/2628084338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe08406acd0 0x7fe0840c7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 >> 172.21.15.162:0/2628084338 conn(0x7fe0840b2250 msgr2=0x7fe0840d0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 shutdown_connections 2024-03-04T10:03:17.106 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.104+0000 7fe08a7b2640 1 -- 172.21.15.162:0/2628084338 wait complete. 2024-03-04T10:03:17.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new c506d2a4-2d0c-41d6-8042-91242075b90d 6 2024-03-04T10:03:17.204 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.202+0000 7f713ce4a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T10:03:17.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.216+0000 7f0d6e7e6640 1 Processor -- start 2024-03-04T10:03:17.217 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.216+0000 7f0d6e7e6640 1 -- start start 2024-03-04T10:03:17.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6e7e6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 0x7f0d680b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6e7e6640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 0x7f0d680b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.219 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6e7e6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6815ca50 con 0x7f0d6815c2e0 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6e7e6640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6815d2f0 con 0x7f0d680ceec0 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6e7e6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6815d6d0 con 0x7f0d680ccbd0 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d67fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 0x7f0d680b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6cd83640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 0x7f0d680b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6cd83640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 0x7f0d680b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53432/0 (socket says 172.21.15.162:53432) 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d67fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 0x7f0d680b7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40340/0 (socket says 172.21.15.162:40340) 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6cd83640 1 -- 172.21.15.162:0/534610639 learned_addr learned my addr 172.21.15.162:0/534610639 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d67fff640 1 -- 172.21.15.162:0/534610639 >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 msgr2=0x7f0d680b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d67fff640 1 --2- 172.21.15.162:0/534610639 >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 0x7f0d680b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d67fff640 1 -- 172.21.15.162:0/534610639 >> v1:172.21.15.99:6789/0 conn(0x7f0d6815c2e0 legacy=0x7f0d680ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d67fff640 1 -- 172.21.15.162:0/534610639 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6815cd30 con 0x7f0d680ccbd0 2024-03-04T10:03:17.220 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.217+0000 7f0d6cd83640 1 --2- 172.21.15.162:0/534610639 >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 0x7f0d680b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:17.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.218+0000 7f0d67fff640 1 --2- 172.21.15.162:0/534610639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 0x7f0d680b7320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097a0 tx=0x7f0d5c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aba2f4e600724d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.218+0000 7f0d66ffd640 1 -- 172.21.15.162:0/534610639 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c035020 con 0x7f0d680ccbd0 2024-03-04T10:03:17.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.218+0000 7f0d66ffd640 1 -- 172.21.15.162:0/534610639 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c031810 con 0x7f0d680ccbd0 2024-03-04T10:03:17.221 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.218+0000 7f0d66ffd640 1 -- 172.21.15.162:0/534610639 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c031be0 con 0x7f0d680ccbd0 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/534610639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 msgr2=0x7f0d680b7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/534610639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 0x7f0d680b7320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097a0 tx=0x7f0d5c02b8b0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/534610639 shutdown_connections 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/534610639 >> v2:172.21.15.162:3300/0 conn(0x7f0d680ceec0 0x7f0d680b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/534610639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d680ccbd0 0x7f0d680b7320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/534610639 >> 172.21.15.162:0/534610639 conn(0x7f0d68066dc0 msgr2=0x7f0d680671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.222 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/534610639 shutdown_connections 2024-03-04T10:03:17.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.219+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/534610639 wait complete. 2024-03-04T10:03:17.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.220+0000 7f0d6e7e6640 1 Processor -- start 2024-03-04T10:03:17.223 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.220+0000 7f0d6e7e6640 1 -- start start 2024-03-04T10:03:17.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6e7e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 0x7f0d68150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6e7e6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d680ceec0 0x7f0d6806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6e7e6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d6815c2e0 0x7f0d680b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6e7e6640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680ccfb0 con 0x7f0d680ceec0 2024-03-04T10:03:17.225 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6e7e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6815c760 con 0x7f0d680ccbd0 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6e7e6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680b0aa0 con 0x7f0d6815c2e0 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 0x7f0d68150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6cd83640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d6815c2e0 0x7f0d680b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d67fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 0x7f0d68150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53438/0 (socket says 172.21.15.162:53438) 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d67fff640 1 -- 172.21.15.162:0/1281737786 learned_addr learned my addr 172.21.15.162:0/1281737786 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d677fe640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d680ceec0 0x7f0d6806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6cd83640 1 -- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 msgr2=0x7f0d68150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6cd83640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 0x7f0d68150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.221+0000 7f0d6cd83640 1 -- 172.21.15.162:0/1281737786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d680ceec0 msgr2=0x7f0d6806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d6cd83640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d680ceec0 0x7f0d6806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.226 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d6cd83640 1 -- 172.21.15.162:0/1281737786 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6806c130 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d67fff640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 0x7f0d68150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d6cd83640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d6815c2e0 0x7f0d680b5510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0d60009e50 tx=0x7f0d6000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6000cb80 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6000cd40 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6806c410 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60003bd0 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6806c9d0 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0d6001b030 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.222+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f0d6001dc80 con 0x7f0d6815c2e0 2024-03-04T10:03:17.227 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.223+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d68074f60 con 0x7f0d6815c2e0 2024-03-04T10:03:17.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.225+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0d60022020 con 0x7f0d6815c2e0 2024-03-04T10:03:17.246 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.245+0000 7f780b63a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T10:03:17.300 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.298+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c506d2a4-2d0c-41d6-8042-91242075b90d", "id": 6} v 0) v1 -- 0x7f0d680cd1c0 con 0x7f0d6815c2e0 2024-03-04T10:03:17.304 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.303+0000 7f0d64ff9640 1 -- 172.21.15.162:0/1281737786 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c506d2a4-2d0c-41d6-8042-91242075b90d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0d60010040 con 0x7f0d6815c2e0 2024-03-04T10:03:17.304 INFO:teuthology.orchestra.run.smithi162.stdout:6 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d6815c2e0 msgr2=0x7f0d680b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d6815c2e0 0x7f0d680b5510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0d60009e50 tx=0x7f0d6000c490 comp rx=0 tx=0).stop 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 shutdown_connections 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d6815c2e0 0x7f0d680b5510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d680ceec0 0x7f0d6806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 --2- 172.21.15.162:0/1281737786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d680ccbd0 0x7f0d68150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 >> 172.21.15.162:0/1281737786 conn(0x7f0d68066dc0 msgr2=0x7f0d680b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 shutdown_connections 2024-03-04T10:03:17.306 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.304+0000 7f0d6e7e6640 1 -- 172.21.15.162:0/1281737786 wait complete. 2024-03-04T10:03:17.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd new d68320e0-246d-4455-9926-d79d30c19803 7 2024-03-04T10:03:17.340 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.338+0000 7f713ce4a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T10:03:17.388 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.387+0000 7f780b63a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T10:03:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.416+0000 7fb43b64a640 1 Processor -- start 2024-03-04T10:03:17.417 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.416+0000 7fb43b64a640 1 -- start start 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb43b64a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb43415bc50 0x7fb4340cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb43b64a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 0x7fb4340ad530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb43b64a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4340cc7a0 con 0x7fb43406acd0 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb43b64a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4340cca80 con 0x7fb43415bc50 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb43b64a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340ccd60 con 0x7fb4340cdd80 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 0x7fb4340ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 0x7fb4340ad530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.162:40366/0 (socket says 172.21.15.162:40366) 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 -- 172.21.15.162:0/2873816963 learned_addr learned my addr 172.21.15.162:0/2873816963 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb4393e6640 1 --2- 172.21.15.162:0/2873816963 >> v2:172.21.15.99:3300/0 conn(0x7fb43415bc50 0x7fb4340cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 -- 172.21.15.162:0/2873816963 >> v2:172.21.15.99:3300/0 conn(0x7fb43415bc50 msgr2=0x7fb4340cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:17.419 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 --2- 172.21.15.162:0/2873816963 >> v2:172.21.15.99:3300/0 conn(0x7fb43415bc50 0x7fb4340cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 -- 172.21.15.162:0/2873816963 >> v1:172.21.15.162:6789/0 conn(0x7fb43406acd0 legacy=0x7fb4340cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.417+0000 7fb438be5640 1 -- 172.21.15.162:0/2873816963 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340cd040 con 0x7fb4340cdd80 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.418+0000 7fb438be5640 1 --2- 172.21.15.162:0/2873816963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 0x7fb4340ad530 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb43002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed96f12e6b1048c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.418+0000 7fb42bfff640 1 -- 172.21.15.162:0/2873816963 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430034070 con 0x7fb4340cdd80 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.418+0000 7fb42bfff640 1 -- 172.21.15.162:0/2873816963 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb430030840 con 0x7fb4340cdd80 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.418+0000 7fb42bfff640 1 -- 172.21.15.162:0/2873816963 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430030c10 con 0x7fb4340cdd80 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 -- 172.21.15.162:0/2873816963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 msgr2=0x7fb4340ad530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 --2- 172.21.15.162:0/2873816963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 0x7fb4340ad530 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb430009870 tx=0x7fb43002b980 comp rx=0 tx=0).stop 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 -- 172.21.15.162:0/2873816963 shutdown_connections 2024-03-04T10:03:17.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 --2- 172.21.15.162:0/2873816963 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4340cdd80 0x7fb4340ad530 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 --2- 172.21.15.162:0/2873816963 >> v2:172.21.15.99:3300/0 conn(0x7fb43415bc50 0x7fb4340cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 -- 172.21.15.162:0/2873816963 >> 172.21.15.162:0/2873816963 conn(0x7fb4340b2250 msgr2=0x7fb4340b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 -- 172.21.15.162:0/2873816963 shutdown_connections 2024-03-04T10:03:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.419+0000 7fb43b64a640 1 -- 172.21.15.162:0/2873816963 wait complete. 2024-03-04T10:03:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.421+0000 7fb43b64a640 1 Processor -- start 2024-03-04T10:03:17.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.421+0000 7fb43b64a640 1 -- start start 2024-03-04T10:03:17.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb43b64a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb43406acd0 0x7fb43414dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb43b64a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 0x7fb4340b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb43b64a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb43415bc50 0x7fb43415d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb43b64a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340cc4b0 con 0x7fb43415bc50 2024-03-04T10:03:17.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb43b64a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340cc330 con 0x7fb4340cdd80 2024-03-04T10:03:17.425 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb43b64a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4340ada70 con 0x7fb43406acd0 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb438be5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 0x7fb4340b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb4393e6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb43406acd0 0x7fb43414dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb438be5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 0x7fb4340b5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:53444/0 (socket says 172.21.15.162:53444) 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb438be5640 1 -- 172.21.15.162:0/4274246048 learned_addr learned my addr 172.21.15.162:0/4274246048 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.422+0000 7fb439be7640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb43415bc50 0x7fb43415d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb438be5640 1 -- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb43406acd0 msgr2=0x7fb43414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb438be5640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb43406acd0 0x7fb43414dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb438be5640 1 -- 172.21.15.162:0/4274246048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb43415bc50 msgr2=0x7fb43415d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb438be5640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb43415bc50 0x7fb43415d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb438be5640 1 -- 172.21.15.162:0/4274246048 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43415c4c0 con 0x7fb4340cdd80 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb4393e6640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb43406acd0 0x7fb43414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb438be5640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 0x7fb4340b5450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb430000c00 tx=0x7fb430004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.426 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430048070 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb430030640 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430034040 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43415c7a0 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb43415cd60 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb430037070 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.423+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fb430043c30 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.424+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43406cd70 con 0x7fb4340cdd80 2024-03-04T10:03:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.426+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb430051020 con 0x7fb4340cdd80 2024-03-04T10:03:17.470 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.468+0000 7f713ce4a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T10:03:17.500 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.499+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d68320e0-246d-4455-9926-d79d30c19803", "id": 7} v 0) v1 -- 0x7fb434074f60 con 0x7fb4340cdd80 2024-03-04T10:03:17.506 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.505+0000 7fb429ffb640 1 -- 172.21.15.162:0/4274246048 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d68320e0-246d-4455-9926-d79d30c19803", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb4300307f0 con 0x7fb4340cdd80 2024-03-04T10:03:17.506 INFO:teuthology.orchestra.run.smithi162.stdout:7 2024-03-04T10:03:17.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 msgr2=0x7fb4340b5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 0x7fb4340b5450 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb430000c00 tx=0x7fb430004730 comp rx=0 tx=0).stop 2024-03-04T10:03:17.507 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 shutdown_connections 2024-03-04T10:03:17.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb43415bc50 0x7fb43415d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4340cdd80 0x7fb4340b5450 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 --2- 172.21.15.162:0/4274246048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb43406acd0 0x7fb43414dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 >> 172.21.15.162:0/4274246048 conn(0x7fb4340b2250 msgr2=0x7fb4340af400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 shutdown_connections 2024-03-04T10:03:17.508 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-04T10:03:17.506+0000 7fb43b64a640 1 -- 172.21.15.162:0/4274246048 wait complete. 2024-03-04T10:03:17.516 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.514+0000 7f713ce4a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T10:03:17.518 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-04T10:03:17.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-04T10:03:17.519 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.517+0000 7f780b63a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T10:03:17.521 INFO:tasks.ceph.osd.0:Started 2024-03-04T10:03:17.521 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-04T10:03:17.521 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-04T10:03:17.526 INFO:tasks.ceph.osd.1:Started 2024-03-04T10:03:17.526 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-04T10:03:17.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-04T10:03:17.529 INFO:tasks.ceph.osd.2:Started 2024-03-04T10:03:17.529 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-04T10:03:17.529 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-04T10:03:17.531 INFO:tasks.ceph.osd.3:Started 2024-03-04T10:03:17.531 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-04T10:03:17.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-04T10:03:17.533 INFO:tasks.ceph.osd.4:Started 2024-03-04T10:03:17.533 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-04T10:03:17.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-04T10:03:17.535 INFO:tasks.ceph.osd.5:Started 2024-03-04T10:03:17.536 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-04T10:03:17.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-04T10:03:17.538 INFO:tasks.ceph.osd.6:Started 2024-03-04T10:03:17.538 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-04T10:03:17.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-04T10:03:17.540 INFO:tasks.ceph.osd.7:Started 2024-03-04T10:03:17.540 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:17.575 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.573+0000 7f780b63a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T10:03:17.685 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.683+0000 7f713ce4a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T10:03:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.683+0000 7f0bf9e61640 1 Processor -- start 2024-03-04T10:03:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.684+0000 7f0bf9e61640 1 -- start start 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.684+0000 7f0bf9e61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40d1290 0x7f0bf40cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.684+0000 7f0bf9e61640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf406ae50 con 0x7f0bf40cfae0 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.684+0000 7f0bf9e61640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf406b130 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.684+0000 7f0bf9e61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf406b410 con 0x7f0bf40d1290 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf37fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0bf40cfae0 0x7f0bf40cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55202/0 (socket says 172.21.15.99:55202) 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf37fe640 1 -- 172.21.15.99:0/849024267 learned_addr learned my addr 172.21.15.99:0/849024267 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf3fff640 1 --2- 172.21.15.99:0/849024267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40d1290 0x7f0bf40cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441850300 0 0) 0x7f0bf406ae50 con 0x7f0bf40cfae0 2024-03-04T10:03:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be0003520 con 0x7f0bf40cfae0 2024-03-04T10:03:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248685778 0 0) 0x7f0bf406b130 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf406ae50 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf3fff640 1 -- 172.21.15.99:0/849024267 >> v1:172.21.15.162:6789/0 conn(0x7f0bf40cc8e0 legacy=0x7f0bf40cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf3fff640 1 -- 172.21.15.99:0/849024267 >> v1:172.21.15.99:6789/0 conn(0x7f0bf40cfae0 legacy=0x7f0bf40cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.685+0000 7f0bf3fff640 1 -- 172.21.15.99:0/849024267 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf406b6f0 con 0x7f0bf40d1290 2024-03-04T10:03:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.686+0000 7f0bf3fff640 1 --2- 172.21.15.99:0/849024267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40d1290 0x7f0bf40cd440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009dc0 tx=0x7f0bec02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dc99e33a83e725e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.686+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec039070 con 0x7f0bf40d1290 2024-03-04T10:03:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.686+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec02ccd0 con 0x7f0bf40d1290 2024-03-04T10:03:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.686+0000 7f0bf2ffd640 1 -- 172.21.15.99:0/849024267 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec034640 con 0x7f0bf40d1290 2024-03-04T10:03:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 -- 172.21.15.99:0/849024267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40d1290 msgr2=0x7f0bf40cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 --2- 172.21.15.99:0/849024267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40d1290 0x7f0bf40cd440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0bec009dc0 tx=0x7f0bec02bf20 comp rx=0 tx=0).stop 2024-03-04T10:03:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 -- 172.21.15.99:0/849024267 shutdown_connections 2024-03-04T10:03:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 --2- 172.21.15.99:0/849024267 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40d1290 0x7f0bf40cd440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 -- 172.21.15.99:0/849024267 >> 172.21.15.99:0/849024267 conn(0x7f0bf40d0770 msgr2=0x7f0bf40d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 -- 172.21.15.99:0/849024267 shutdown_connections 2024-03-04T10:03:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.687+0000 7f0bf9e61640 1 -- 172.21.15.99:0/849024267 wait complete. 2024-03-04T10:03:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.688+0000 7f0bf9e61640 1 Processor -- start 2024-03-04T10:03:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.688+0000 7f0bf9e61640 1 -- start start 2024-03-04T10:03:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 0x7f0bf40b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40cfae0 0x7f0bf40cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bf40d1290 0x7f0bf414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40cd980 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf406b9d0 con 0x7f0bf40d1290 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40cdb00 con 0x7f0bf40cfae0 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 0x7f0bf40b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 0x7f0bf40b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37248/0 (socket says 172.21.15.99:37248) 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 -- 172.21.15.99:0/2758023965 learned_addr learned my addr 172.21.15.99:0/2758023965 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bebfff640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40cfae0 0x7f0bf40cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf3fff640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bf40d1290 0x7f0bf414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 -- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40cfae0 msgr2=0x7f0bf40cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40cfae0 0x7f0bf40cb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 -- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bf40d1290 msgr2=0x7f0bf414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bf40d1290 0x7f0bf414d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 -- 172.21.15.99:0/2758023965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40b53d0 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bebfff640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40cfae0 0x7f0bf40cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf37fe640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 0x7f0bf40b4ad0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0be400c500 tx=0x7f0be400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4017070 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be4005d00 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4007330 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf40b5710 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.689+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf40b5f50 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.690+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0be401a070 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.690+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0be4012ad0 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.690+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf406cd70 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.692+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0be4021020 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.696 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.695+0000 7f780b63a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T10:03:17.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.753+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bf40b6150 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.754+0000 7f0bf0ff9640 1 -- 172.21.15.99:0/2758023965 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f0bf40b6150 con 0x7f0bf40cc8e0 2024-03-04T10:03:17.756 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:17.756 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":10,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:17.500854+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 msgr2=0x7f0bf40b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 0x7f0bf40b4ad0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0be400c500 tx=0x7f0be400c9c0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 shutdown_connections 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0bf40d1290 0x7f0bf414d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0bf40cfae0 0x7f0bf40cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 --2- 172.21.15.99:0/2758023965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0bf40cc8e0 0x7f0bf40b4ad0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 >> 172.21.15.99:0/2758023965 conn(0x7f0bf40d0770 msgr2=0x7f0bf40cada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 shutdown_connections 2024-03-04T10:03:17.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:17.755+0000 7f0bf9e61640 1 -- 172.21.15.99:0/2758023965 wait complete. 2024-03-04T10:03:17.768 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-04T10:03:17.768 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-04T10:03:17.786 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.784+0000 7f713ce4a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T10:03:17.794 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.792+0000 7f780b63a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T10:03:17.942 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.940+0000 7f713ce4a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T10:03:17.949 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.948+0000 7f780b63a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T10:03:17.983 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:17.982+0000 7f713ce4a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T10:03:17.991 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:17.990+0000 7f780b63a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T10:03:18.123 INFO:tasks.ceph.osd.0.smithi099.stderr:2024-03-04T10:03:18.121+0000 7fb506e11640 -1 Falling back to public interface 2024-03-04T10:03:18.132 INFO:tasks.ceph.osd.3.smithi099.stderr:2024-03-04T10:03:18.130+0000 7fc28bde1640 -1 Falling back to public interface 2024-03-04T10:03:18.132 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-03-04T10:03:18.131+0000 7f084a316640 -1 Falling back to public interface 2024-03-04T10:03:18.135 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-03-04T10:03:18.134+0000 7f1ca7e11640 -1 Falling back to public interface 2024-03-04T10:03:18.138 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:18.136+0000 7f713ce4a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T10:03:18.142 INFO:tasks.ceph.osd.2.smithi099.stderr:2024-03-04T10:03:18.140+0000 7f31be60d640 -1 Falling back to public interface 2024-03-04T10:03:18.145 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:18.144+0000 7f780b63a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T10:03:18.148 INFO:tasks.ceph.osd.1.smithi099.stderr:2024-03-04T10:03:18.146+0000 7f1f00416640 -1 Falling back to public interface 2024-03-04T10:03:18.151 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-03-04T10:03:18.150+0000 7fbcfc201640 -1 Falling back to public interface 2024-03-04T10:03:18.155 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-03-04T10:03:18.154+0000 7f0f5ecc8640 -1 Falling back to public interface 2024-03-04T10:03:18.574 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:18.663 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.661+0000 7f54572d1640 1 Processor -- start 2024-03-04T10:03:18.664 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.662+0000 7f54572d1640 1 -- start start 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.662+0000 7f54572d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.662+0000 7f54572d1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f54500d0f90 0x7f545006c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.662+0000 7f54572d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500cfef0 con 0x7f545009a0b0 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.662+0000 7f54572d1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54500d0b00 con 0x7f54500d0f90 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.662+0000 7f54572d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54500d01d0 con 0x7f5450099cd0 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545506d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545586e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f54500d0f90 0x7f545006c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545486c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f545009a0b0 0x7f54500cbe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:51528/0 (socket says 172.21.15.99:51528) 2024-03-04T10:03:18.665 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545486c640 1 -- 172.21.15.99:0/2326569110 learned_addr learned my addr 172.21.15.99:0/2326569110 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:18.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545506d640 1 -- 172.21.15.99:0/2326569110 >> v2:172.21.15.162:3300/0 conn(0x7f54500d0f90 msgr2=0x7f545006c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:18.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545506d640 1 --2- 172.21.15.99:0/2326569110 >> v2:172.21.15.162:3300/0 conn(0x7f54500d0f90 0x7f545006c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545506d640 1 -- 172.21.15.99:0/2326569110 >> v1:172.21.15.162:6789/0 conn(0x7f545009a0b0 legacy=0x7f54500cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:18.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545506d640 1 -- 172.21.15.99:0/2326569110 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500d04b0 con 0x7f5450099cd0 2024-03-04T10:03:18.666 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.663+0000 7f545586e640 1 --2- 172.21.15.99:0/2326569110 >> v2:172.21.15.162:3300/0 conn(0x7f54500d0f90 0x7f545006c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:18.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.664+0000 7f545506d640 1 --2- 172.21.15.99:0/2326569110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b2880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5448009d90 tx=0x7f544802bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc4a36350758133 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.664+0000 7f544ffff640 1 -- 172.21.15.99:0/2326569110 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448039070 con 0x7f5450099cd0 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.664+0000 7f544ffff640 1 -- 172.21.15.99:0/2326569110 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544802ca10 con 0x7f5450099cd0 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.664+0000 7f544ffff640 1 -- 172.21.15.99:0/2326569110 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544802cde0 con 0x7f5450099cd0 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 -- 172.21.15.99:0/2326569110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 msgr2=0x7f54500b2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 --2- 172.21.15.99:0/2326569110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b2880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5448009d90 tx=0x7f544802bce0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 -- 172.21.15.99:0/2326569110 shutdown_connections 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 --2- 172.21.15.99:0/2326569110 >> v2:172.21.15.162:3300/0 conn(0x7f54500d0f90 0x7f545006c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 --2- 172.21.15.99:0/2326569110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b2880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 -- 172.21.15.99:0/2326569110 >> 172.21.15.99:0/2326569110 conn(0x7f54500b32b0 msgr2=0x7f54500b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 -- 172.21.15.99:0/2326569110 shutdown_connections 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.665+0000 7f54572d1640 1 -- 172.21.15.99:0/2326569110 wait complete. 2024-03-04T10:03:18.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.666+0000 7f54572d1640 1 Processor -- start 2024-03-04T10:03:18.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.666+0000 7f54572d1640 1 -- start start 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 0x7f545015f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54500d0f90 0x7f54500c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500cc520 con 0x7f545009a0b0 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54500cfba0 con 0x7f54500d0f90 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54500cc6a0 con 0x7f5450099cd0 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 0x7f545015f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 0x7f545015f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37250/0 (socket says 172.21.15.99:37250) 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 -- 172.21.15.99:0/986448886 learned_addr learned my addr 172.21.15.99:0/986448886 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545506d640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:18.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545586e640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54500d0f90 0x7f54500c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 -- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 msgr2=0x7f54500b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 -- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54500d0f90 msgr2=0x7f54500c6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54500d0f90 0x7f54500c6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 -- 172.21.15.99:0/986448886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f545015fcd0 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f545486c640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 0x7f545015f790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f544400c780 tx=0x7f544400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444018070 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54440072d0 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545015ffa0 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.667+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54440047e0 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.668+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54500c8d50 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.668+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f544401b070 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.668+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f544400acf0 con 0x7f545009a0b0 2024-03-04T10:03:18.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.669+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545004f090 con 0x7f545009a0b0 2024-03-04T10:03:18.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.671+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f544400a640 con 0x7f545009a0b0 2024-03-04T10:03:18.734 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.732+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f545006c650 con 0x7f545009a0b0 2024-03-04T10:03:18.735 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.733+0000 7f544dffb640 1 -- 172.21.15.99:0/986448886 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f544401d440 con 0x7f545009a0b0 2024-03-04T10:03:18.735 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-04T10:03:18.735 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:17.500854+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:18.736 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 msgr2=0x7f545015f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 0x7f545015f790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f544400c780 tx=0x7f544400cb80 comp rx=0 tx=0).stop 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 shutdown_connections 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54500d0f90 0x7f54500c6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545009a0b0 0x7f545015f790 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 --2- 172.21.15.99:0/986448886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5450099cd0 0x7f54500b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 >> 172.21.15.99:0/986448886 conn(0x7f54500b32b0 msgr2=0x7f545015cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 shutdown_connections 2024-03-04T10:03:18.737 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:18.734+0000 7f54572d1640 1 -- 172.21.15.99:0/986448886 wait complete. 2024-03-04T10:03:18.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-04T10:03:18.962 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:18.961+0000 7f780b63a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T10:03:18.967 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:18.966+0000 7f713ce4a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T10:03:19.243 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:19.242+0000 7f780b63a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T10:03:19.246 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:19.244+0000 7f713ce4a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T10:03:19.296 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:19.295+0000 7f780b63a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T10:03:19.298 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:19.296+0000 7f713ce4a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T10:03:19.411 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:19.409+0000 7f780b63a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T10:03:19.412 INFO:tasks.ceph.mgr.y.smithi099.stderr:2024-03-04T10:03:19.410+0000 7f713ce4a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T10:03:20.441 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:20.439+0000 7f77ac391640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-04T10:03:20.441 INFO:tasks.ceph.mgr.x.smithi162.stderr:2024-03-04T10:03:20.439+0000 7f77ac391640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-04T10:03:21.587 INFO:tasks.ceph.osd.0.smithi099.stderr:2024-03-04T10:03:21.585+0000 7fb506e11640 -1 osd.0 0 log_to_monitors true 2024-03-04T10:03:21.619 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-03-04T10:03:21.617+0000 7f084a316640 -1 osd.4 0 log_to_monitors true 2024-03-04T10:03:21.620 INFO:tasks.ceph.osd.3.smithi099.stderr:2024-03-04T10:03:21.619+0000 7fc28bde1640 -1 osd.3 0 log_to_monitors true 2024-03-04T10:03:21.649 INFO:tasks.ceph.osd.2.smithi099.stderr:2024-03-04T10:03:21.647+0000 7f31be60d640 -1 osd.2 0 log_to_monitors true 2024-03-04T10:03:21.653 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-03-04T10:03:21.652+0000 7fbcfc201640 -1 osd.7 0 log_to_monitors true 2024-03-04T10:03:21.659 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-03-04T10:03:21.658+0000 7f1ca7e11640 -1 osd.5 0 log_to_monitors true 2024-03-04T10:03:21.681 INFO:tasks.ceph.osd.1.smithi099.stderr:2024-03-04T10:03:21.679+0000 7f1f00416640 -1 osd.1 0 log_to_monitors true 2024-03-04T10:03:21.688 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-03-04T10:03:21.687+0000 7f0f5ecc8640 -1 osd.6 0 log_to_monitors true 2024-03-04T10:03:25.457 INFO:tasks.ceph.osd.0.smithi099.stderr:2024-03-04T10:03:25.456+0000 7fb50270f640 -1 osd.0 0 waiting for initial osdmap 2024-03-04T10:03:25.469 INFO:tasks.ceph.osd.0.smithi099.stderr:2024-03-04T10:03:25.468+0000 7fb4fa52f640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.553 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:25.566 INFO:tasks.ceph.osd.2.smithi099.stderr:2024-03-04T10:03:25.565+0000 7f31b9f7d640 -1 osd.2 0 waiting for initial osdmap 2024-03-04T10:03:25.573 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-03-04T10:03:25.572+0000 7fbcf8197640 -1 osd.7 0 waiting for initial osdmap 2024-03-04T10:03:25.578 INFO:tasks.ceph.osd.2.smithi099.stderr:2024-03-04T10:03:25.577+0000 7f31b1d9d640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.581 INFO:tasks.ceph.osd.3.smithi099.stderr:2024-03-04T10:03:25.580+0000 7fc287d77640 -1 osd.3 0 waiting for initial osdmap 2024-03-04T10:03:25.595 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-03-04T10:03:25.594+0000 7f1ca370f640 -1 osd.5 0 waiting for initial osdmap 2024-03-04T10:03:25.596 INFO:tasks.ceph.osd.7.smithi162.stderr:2024-03-04T10:03:25.595+0000 7fbceffb7640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.598 INFO:tasks.ceph.osd.1.smithi099.stderr:2024-03-04T10:03:25.597+0000 7f1efbdec640 -1 osd.1 0 waiting for initial osdmap 2024-03-04T10:03:25.600 INFO:tasks.ceph.osd.3.smithi099.stderr:2024-03-04T10:03:25.600+0000 7fc27fb97640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.604 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-03-04T10:03:25.603+0000 7f0f5b472640 -1 osd.6 0 waiting for initial osdmap 2024-03-04T10:03:25.609 INFO:tasks.ceph.osd.5.smithi162.stderr:2024-03-04T10:03:25.608+0000 7f1c9b52f640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.610 INFO:tasks.ceph.osd.1.smithi099.stderr:2024-03-04T10:03:25.610+0000 7f1ef3c0c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.612 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-03-04T10:03:25.611+0000 7f08462ac640 -1 osd.4 0 waiting for initial osdmap 2024-03-04T10:03:25.623 INFO:tasks.ceph.osd.6.smithi162.stderr:2024-03-04T10:03:25.622+0000 7f0f52a7e640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.624 INFO:tasks.ceph.osd.4.smithi162.stderr:2024-03-04T10:03:25.623+0000 7f083e0cc640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T10:03:25.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.654+0000 7f7cbfe4b640 1 Processor -- start 2024-03-04T10:03:25.655 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.654+0000 7f7cbfe4b640 1 -- start start 2024-03-04T10:03:25.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbfe4b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 0x7f7cb80d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbfe4b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb80d0010 con 0x7f7cb8099a60 2024-03-04T10:03:25.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbfe4b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb80d02f0 con 0x7f7cb80b2c20 2024-03-04T10:03:25.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbfe4b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb806d6f0 con 0x7f7cb80d0dd0 2024-03-04T10:03:25.667 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbdbe7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7cb8099a60 0x7f7cb8099e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35480/0 (socket says 172.21.15.99:35480) 2024-03-04T10:03:25.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbdbe7640 1 -- 172.21.15.99:0/3893693844 learned_addr learned my addr 172.21.15.99:0/3893693844 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:25.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbe3e8640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 0x7f7cb80d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316612647 0 0) 0x7f7cb80d0010 con 0x7f7cb8099a60 2024-03-04T10:03:25.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ca4003520 con 0x7f7cb8099a60 2024-03-04T10:03:25.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047540170 0 0) 0x7f7cb80d02f0 con 0x7f7cb80b2c20 2024-03-04T10:03:25.668 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb80d0010 con 0x7f7cb80b2c20 2024-03-04T10:03:25.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3396349966 0 0) 0x7f7ca4003520 con 0x7f7cb8099a60 2024-03-04T10:03:25.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 msgr2=0x7f7cb80d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:25.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 0x7f7cb80d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 >> v1:172.21.15.162:6789/0 conn(0x7f7cb80b2c20 legacy=0x7f7cb806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80d05d0 con 0x7f7cb8099a60 2024-03-04T10:03:25.669 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7cb40015f0 con 0x7f7cb8099a60 2024-03-04T10:03:25.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 >> v1:172.21.15.99:6789/0 conn(0x7f7cb8099a60 legacy=0x7f7cb8099e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 0x7f7ca4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4004120 0x7f7ca4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7ca40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb806d6f0 con 0x7f7ca4004d40 2024-03-04T10:03:25.670 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca400bb00 con 0x7f7ca4004120 2024-03-04T10:03:25.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.655+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca400bd00 con 0x7f7cb80d0dd0 2024-03-04T10:03:25.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7ca40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbdbe7640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 0x7f7ca4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbd3e6640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4004120 0x7f7ca4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 -- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 msgr2=0x7f7ca4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:25.671 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb80d0dd0 0x7f7ca4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 -- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4004120 msgr2=0x7f7ca4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:25.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ca4004120 0x7f7ca4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 -- 172.21.15.99:0/3893693844 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca4003520 con 0x7f7ca4004d40 2024-03-04T10:03:25.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.656+0000 7f7cbe3e8640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7ca40051c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fc0 tx=0x7f7ca4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a22f5a2f1cecd2c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:25.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0039070 con 0x7f7ca4004d40 2024-03-04T10:03:25.672 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb002cc20 con 0x7f7ca4004d40 2024-03-04T10:03:25.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbcbe5640 1 -- 172.21.15.99:0/3893693844 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0009c10 con 0x7f7ca4004d40 2024-03-04T10:03:25.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/3893693844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 msgr2=0x7f7ca40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/3893693844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7ca40051c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fc0 tx=0x7f7ca4005830 comp rx=0 tx=0).stop 2024-03-04T10:03:25.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbe3e8640 1 -- 172.21.15.99:0/3893693844 reap_dead start 2024-03-04T10:03:25.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/3893693844 shutdown_connections 2024-03-04T10:03:25.673 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/3893693844 >> 172.21.15.99:0/3893693844 conn(0x7f7cb80b3290 msgr2=0x7f7cb80b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:25.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/3893693844 shutdown_connections 2024-03-04T10:03:25.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.657+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/3893693844 wait complete. 2024-03-04T10:03:25.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.658+0000 7f7cbfe4b640 1 Processor -- start 2024-03-04T10:03:25.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.658+0000 7f7cbfe4b640 1 -- start start 2024-03-04T10:03:25.674 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbfe4b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7cb81f3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbfe4b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb81597c0 0x7f7cb8159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbfe4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb80b4f80 0x7f7cb80b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbfe4b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb80cfcc0 con 0x7f7ca4004d40 2024-03-04T10:03:25.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbfe4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb806d3f0 con 0x7f7cb80b4f80 2024-03-04T10:03:25.675 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbfe4b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb81617d0 con 0x7f7cb81597c0 2024-03-04T10:03:25.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7cb81f3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7cb81f3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55764/0 (socket says 172.21.15.99:55764) 2024-03-04T10:03:25.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 -- 172.21.15.99:0/873066701 learned_addr learned my addr 172.21.15.99:0/873066701 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:25.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbe3e8640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb80b4f80 0x7f7cb80b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbd3e6640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb81597c0 0x7f7cb8159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.676 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 -- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb81597c0 msgr2=0x7f7cb8159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb81597c0 0x7f7cb8159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 -- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb80b4f80 msgr2=0x7f7cb80b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.659+0000 7f7cbdbe7640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb80b4f80 0x7f7cb80b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cbdbe7640 1 -- 172.21.15.99:0/873066701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb80b7210 con 0x7f7ca4004d40 2024-03-04T10:03:25.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cbdbe7640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7cb81f3ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40072f0 tx=0x7f7cb40077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:25.677 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400f9c0 con 0x7f7ca4004d40 2024-03-04T10:03:25.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4018070 con 0x7f7ca4004d40 2024-03-04T10:03:25.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb815a180 con 0x7f7ca4004d40 2024-03-04T10:03:25.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400fda0 con 0x7f7ca4004d40 2024-03-04T10:03:25.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.660+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb815a3f0 con 0x7f7ca4004d40 2024-03-04T10:03:25.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.661+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb804f090 con 0x7f7ca4004d40 2024-03-04T10:03:25.678 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.661+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 4) v1 ==== 89722+0+0 (secure 0 0 0) 0x7f7cb401d460 con 0x7f7ca4004d40 2024-03-04T10:03:25.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.662+0000 7f7cae7fc640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7c9c06b470 0x7f7c9c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:25.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.662+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f7cb40893e0 con 0x7f7ca4004d40 2024-03-04T10:03:25.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.662+0000 7f7cbd3e6640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7c9c06b470 0x7f7c9c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:25.679 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.662+0000 7f7cbd3e6640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7c9c06b470 0x7f7c9c06d930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80b6ed0 tx=0x7f7ca8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:25.680 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.666+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7cb40590d0 con 0x7f7ca4004d40 2024-03-04T10:03:25.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.837+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cb80b4140 con 0x7f7ca4004d40 2024-03-04T10:03:25.838 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.837+0000 7f7cae7fc640 1 -- 172.21.15.99:0/873066701 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f7cb40589d0 con 0x7f7ca4004d40 2024-03-04T10:03:25.838 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-04T10:03:25.838 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":12,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:23.438966+0000","last_up_change":"0.000000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7c9c06b470 msgr2=0x7f7c9c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7c9c06b470 0x7f7c9c06d930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80b6ed0 tx=0x7f7ca8009040 comp rx=0 tx=0).stop 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 msgr2=0x7f7cb81f3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7cb81f3ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40072f0 tx=0x7f7cb40077b0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 shutdown_connections 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7c9c06b470 0x7f7c9c06d930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cb80b4f80 0x7f7cb80b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7cb81597c0 0x7f7cb8159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 --2- 172.21.15.99:0/873066701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7ca4004d40 0x7f7cb81f3ee0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 >> 172.21.15.99:0/873066701 conn(0x7f7cb80b3290 msgr2=0x7f7cb8070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 shutdown_connections 2024-03-04T10:03:25.841 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:25.839+0000 7f7cbfe4b640 1 -- 172.21.15.99:0/873066701 wait complete. 2024-03-04T10:03:25.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-04T10:03:32.657 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:32.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.756+0000 7fa3a3e52640 1 Processor -- start 2024-03-04T10:03:32.757 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.756+0000 7fa3a3e52640 1 -- start start 2024-03-04T10:03:32.758 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a3e52640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0ce7e0 0x7fa39c0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a3e52640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa39c0b6bb0 0x7fa39c0b7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a3e52640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39c0d2210 con 0x7fa39c15dd20 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a3e52640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c0cf160 con 0x7fa39c0b6bb0 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a3e52640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0d24e0 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa39c0b6bb0 0x7fa39c0b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa39c0b6bb0 0x7fa39c0b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55780/0 (socket says 172.21.15.99:55780) 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 -- 172.21.15.99:0/1321761973 learned_addr learned my addr 172.21.15.99:0/1321761973 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a13ed640 1 --2- 172.21.15.99:0/1321761973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0ce7e0 0x7fa39c0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 -- 172.21.15.99:0/1321761973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0ce7e0 msgr2=0x7fa39c0cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 --2- 172.21.15.99:0/1321761973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0ce7e0 0x7fa39c0cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 -- 172.21.15.99:0/1321761973 >> v1:172.21.15.162:6789/0 conn(0x7fa39c15dd20 legacy=0x7fa39c0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.757+0000 7fa3a23ef640 1 -- 172.21.15.99:0/1321761973 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0d27c0 con 0x7fa39c0b6bb0 2024-03-04T10:03:32.759 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.758+0000 7fa3a23ef640 1 --2- 172.21.15.99:0/1321761973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0b6bb0 0x7fa39c0b7040 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa394009e50 tx=0x7fa39402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8823479ae7d95afe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.758+0000 7fa3a0bec640 1 -- 172.21.15.99:0/1321761973 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39402ca70 con 0x7fa39c0b6bb0 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.758+0000 7fa3a0bec640 1 -- 172.21.15.99:0/1321761973 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39402cc10 con 0x7fa39c0b6bb0 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.758+0000 7fa3a0bec640 1 -- 172.21.15.99:0/1321761973 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3940343f0 con 0x7fa39c0b6bb0 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1321761973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0b6bb0 msgr2=0x7fa39c0b7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1321761973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0b6bb0 0x7fa39c0b7040 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa394009e50 tx=0x7fa39402bed0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1321761973 shutdown_connections 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1321761973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0b6bb0 0x7fa39c0b7040 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1321761973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0ce7e0 0x7fa39c0cec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1321761973 >> 172.21.15.99:0/1321761973 conn(0x7fa39c0d0490 msgr2=0x7fa39c0d08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1321761973 shutdown_connections 2024-03-04T10:03:32.761 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.759+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1321761973 wait complete. 2024-03-04T10:03:32.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.760+0000 7fa3a3e52640 1 Processor -- start 2024-03-04T10:03:32.762 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.760+0000 7fa3a3e52640 1 -- start start 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a3e52640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0b6bb0 0x7fa39c1e76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a3e52640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 0x7fa39c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a3e52640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa39c15dd20 0x7fa39c1f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a3e52640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0d3060 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a3e52640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0d2d80 con 0x7fa39c15dd20 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a3e52640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15e100 con 0x7fa39c0b6bb0 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 0x7fa39c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 0x7fa39c0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55788/0 (socket says 172.21.15.99:55788) 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 -- 172.21.15.99:0/1970862983 learned_addr learned my addr 172.21.15.99:0/1970862983 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a23ef640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa39c15dd20 0x7fa39c1f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:32.764 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a1bee640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0b6bb0 0x7fa39c1e76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 -- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0b6bb0 msgr2=0x7fa39c1e76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0b6bb0 0x7fa39c1e76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 -- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa39c15dd20 msgr2=0x7fa39c1f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa39c15dd20 0x7fa39c1f75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 -- 172.21.15.99:0/1970862983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c1e82f0 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a1bee640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0b6bb0 0x7fa39c1e76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.761+0000 7fa3a13ed640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 0x7fa39c0cc710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00c820 tx=0x7fa38c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.762+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c015070 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.762+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c007a30 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.762+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c1f4080 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.762+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c007de0 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.762+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c1f4640 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.763+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c1f4840 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.765 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.763+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa38c018070 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.763+0000 7fa3927fc640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa38006b4d0 0x7fa38006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:32.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.764+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa38c089d50 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.764+0000 7fa3a1bee640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa38006b4d0 0x7fa38006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:32.766 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.764+0000 7fa3a1bee640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa38006b4d0 0x7fa38006d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa398003eb0 tx=0x7fa398009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:32.769 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.768+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa39c1f4840 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.941+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa39c04f090 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.942 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.941+0000 7fa3927fc640 1 -- 172.21.15.99:0/1970862983 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12848 (secure 0 0 0) 0x7fa38c08d030 con 0x7fa39c0ce7e0 2024-03-04T10:03:32.943 INFO:teuthology.misc.health.smithi099.stdout: 2024-03-04T10:03:32.943 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":16,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:29.481552+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa38006b4d0 msgr2=0x7fa38006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa38006b4d0 0x7fa38006d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa398003eb0 tx=0x7fa398009040 comp rx=0 tx=0).stop 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 msgr2=0x7fa39c0cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 0x7fa39c0cc710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00c820 tx=0x7fa38c00cce0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 shutdown_connections 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa38006b4d0 0x7fa38006d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa39c15dd20 0x7fa39c1f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa39c0ce7e0 0x7fa39c0cc710 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 --2- 172.21.15.99:0/1970862983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa39c0b6bb0 0x7fa39c1e76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 >> 172.21.15.99:0/1970862983 conn(0x7fa39c0d0490 msgr2=0x7fa39c0b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 shutdown_connections 2024-03-04T10:03:32.945 INFO:teuthology.misc.health.smithi099.stderr:2024-03-04T10:03:32.943+0000 7fa3a3e52640 1 -- 172.21.15.99:0/1970862983 wait complete. 2024-03-04T10:03:32.954 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-04T10:03:32.954 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-04T10:03:32.954 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:03:32.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:03:32.957 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:03:32.957 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:03:32.957 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:03:32.959 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:03:32.959 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T10:03:32.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T10:03:32.962 INFO:tasks.ceph.mds.b:Started 2024-03-04T10:03:32.962 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T10:03:32.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T10:03:32.964 INFO:tasks.ceph.mds.d:Started 2024-03-04T10:03:32.965 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-04T10:03:32.965 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:33.033 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:03:33.033 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:03:33.048 INFO:tasks.ceph.mds.d.smithi162.stdout:starting mds.d at 2024-03-04T10:03:33.049 INFO:tasks.ceph.mds.b.smithi162.stdout:starting mds.b at 2024-03-04T10:03:33.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.069+0000 7fe619709640 1 Processor -- start 2024-03-04T10:03:33.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.069+0000 7fe619709640 1 -- start start 2024-03-04T10:03:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe619709640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140d1290 0x7fe6140cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe619709640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61406ae50 con 0x7fe6140cfae0 2024-03-04T10:03:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe619709640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61406b130 con 0x7fe6140cc8e0 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe619709640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61406b410 con 0x7fe6140d1290 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe612ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe6140cfae0 0x7fe6140cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35504/0 (socket says 172.21.15.99:35504) 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe612ffd640 1 -- 172.21.15.99:0/3338870394 learned_addr learned my addr 172.21.15.99:0/3338870394 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe6137fe640 1 --2- 172.21.15.99:0/3338870394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140d1290 0x7fe6140cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe611ffb640 1 -- 172.21.15.99:0/3338870394 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749385406 0 0) 0x7fe61406ae50 con 0x7fe6140cfae0 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe611ffb640 1 -- 172.21.15.99:0/3338870394 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5fc003520 con 0x7fe6140cfae0 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe6137fe640 1 -- 172.21.15.99:0/3338870394 >> v1:172.21.15.162:6789/0 conn(0x7fe6140cc8e0 legacy=0x7fe6140cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe6137fe640 1 -- 172.21.15.99:0/3338870394 >> v1:172.21.15.99:6789/0 conn(0x7fe6140cfae0 legacy=0x7fe6140cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe611ffb640 1 -- 172.21.15.99:0/3338870394 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792659915 0 0) 0x7fe61406b130 con 0x7fe6140cc8e0 2024-03-04T10:03:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.070+0000 7fe6137fe640 1 -- 172.21.15.99:0/3338870394 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61406b6f0 con 0x7fe6140d1290 2024-03-04T10:03:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.071+0000 7fe6137fe640 1 --2- 172.21.15.99:0/3338870394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140d1290 0x7fe6140cd440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009dc0 tx=0x7fe60c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3894fe7099702b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.071+0000 7fe611ffb640 1 -- 172.21.15.99:0/3338870394 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c039070 con 0x7fe6140d1290 2024-03-04T10:03:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.071+0000 7fe611ffb640 1 -- 172.21.15.99:0/3338870394 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c02ccd0 con 0x7fe6140d1290 2024-03-04T10:03:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.071+0000 7fe611ffb640 1 -- 172.21.15.99:0/3338870394 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c034640 con 0x7fe6140d1290 2024-03-04T10:03:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 -- 172.21.15.99:0/3338870394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140d1290 msgr2=0x7fe6140cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 --2- 172.21.15.99:0/3338870394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140d1290 0x7fe6140cd440 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009dc0 tx=0x7fe60c02bf20 comp rx=0 tx=0).stop 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 -- 172.21.15.99:0/3338870394 shutdown_connections 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 --2- 172.21.15.99:0/3338870394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140d1290 0x7fe6140cd440 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 -- 172.21.15.99:0/3338870394 >> 172.21.15.99:0/3338870394 conn(0x7fe6140d0770 msgr2=0x7fe6140d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 -- 172.21.15.99:0/3338870394 shutdown_connections 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.072+0000 7fe619709640 1 -- 172.21.15.99:0/3338870394 wait complete. 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.073+0000 7fe619709640 1 Processor -- start 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.073+0000 7fe619709640 1 -- start start 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe619709640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140cc8e0 0x7fe6140b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe619709640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 0x7fe6140c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe619709640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6140d1290 0x7fe6140c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe619709640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6140cd980 con 0x7fe6140cfae0 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe619709640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61406b9d0 con 0x7fe6140d1290 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe619709640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6140cdb00 con 0x7fe6140cc8e0 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 0x7fe6140c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 0x7fe6140c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55820/0 (socket says 172.21.15.99:55820) 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 -- 172.21.15.99:0/1056020161 learned_addr learned my addr 172.21.15.99:0/1056020161 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe612ffd640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140cc8e0 0x7fe6140b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6137fe640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6140d1290 0x7fe6140c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 -- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140cc8e0 msgr2=0x7fe6140b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140cc8e0 0x7fe6140b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 -- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6140d1290 msgr2=0x7fe6140c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6140d1290 0x7fe6140c9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 -- 172.21.15.99:0/1056020161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6140b3db0 con 0x7fe6140cfae0 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe612ffd640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140cc8e0 0x7fe6140b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe6127fc640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 0x7fe6140c9a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c4b0 tx=0x7fe60800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.074+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608017070 con 0x7fe6140cfae0 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.075+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe608005480 con 0x7fe6140cfae0 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.075+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6140b4090 con 0x7fe6140cfae0 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.075+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608003830 con 0x7fe6140cfae0 2024-03-04T10:03:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.075+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6141f3b50 con 0x7fe6140cfae0 2024-03-04T10:03:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.076+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61406cd70 con 0x7fe6140cfae0 2024-03-04T10:03:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.076+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe60801a070 con 0x7fe6140cfae0 2024-03-04T10:03:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.076+0000 7fe5fb7fe640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe60006b4d0 0x7fe60006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.076+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe608089120 con 0x7fe6140cfae0 2024-03-04T10:03:33.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.077+0000 7fe612ffd640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe60006b4d0 0x7fe60006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.077+0000 7fe612ffd640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe60006b4d0 0x7fe60006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe604007fd0 tx=0x7fe604009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.081+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe608057d30 con 0x7fe6140cfae0 2024-03-04T10:03:33.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.251+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe614074f60 con 0x7fe6140cfae0 2024-03-04T10:03:33.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.252+0000 7fe5fb7fe640 1 -- 172.21.15.99:0/1056020161 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12848 (secure 0 0 0) 0x7fe60805bdd0 con 0x7fe6140cfae0 2024-03-04T10:03:33.253 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:33.253 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":16,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:29.481552+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:33.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe60006b4d0 msgr2=0x7fe60006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe60006b4d0 0x7fe60006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe604007fd0 tx=0x7fe604009040 comp rx=0 tx=0).stop 2024-03-04T10:03:33.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 msgr2=0x7fe6140c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 0x7fe6140c9a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c4b0 tx=0x7fe60800c970 comp rx=0 tx=0).stop 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 shutdown_connections 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe60006b4d0 0x7fe60006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6140d1290 0x7fe6140c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6140cfae0 0x7fe6140c9a10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 --2- 172.21.15.99:0/1056020161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe6140cc8e0 0x7fe6140b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 >> 172.21.15.99:0/1056020161 conn(0x7fe6140d0770 msgr2=0x7fe6140cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 shutdown_connections 2024-03-04T10:03:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.254+0000 7fe619709640 1 -- 172.21.15.99:0/1056020161 wait complete. 2024-03-04T10:03:33.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:03:33.266 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:03:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 Processor -- start 2024-03-04T10:03:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 -- start start 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f25840b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25840cf020 0x7f258415d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25840ca970 con 0x7f25840d3300 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2584161d90 con 0x7f25840cf020 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f2588cfc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2584162070 con 0x7f25840cec40 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f257bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25840cf020 0x7f258415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f257bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25840cf020 0x7f258415d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55826/0 (socket says 172.21.15.99:55826) 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.368+0000 7f257bfff640 1 -- 172.21.15.99:0/1795784921 learned_addr learned my addr 172.21.15.99:0/1795784921 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.369+0000 7f257bfff640 1 -- 172.21.15.99:0/1795784921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 msgr2=0x7f25840b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.369+0000 7f258259c640 1 --2- 172.21.15.99:0/1795784921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f25840b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.369+0000 7f257bfff640 1 --2- 172.21.15.99:0/1795784921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f25840b6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.369+0000 7f257bfff640 1 -- 172.21.15.99:0/1795784921 >> v1:172.21.15.162:6789/0 conn(0x7f25840d3300 legacy=0x7f25840d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.369+0000 7f257bfff640 1 -- 172.21.15.99:0/1795784921 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2584162350 con 0x7f25840cf020 2024-03-04T10:03:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.369+0000 7f257bfff640 1 --2- 172.21.15.99:0/1795784921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840cf020 0x7f258415d730 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f256c009870 tx=0x7f256c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d446bc718efb4d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2581d9b640 1 -- 172.21.15.99:0/1795784921 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c035020 con 0x7f25840cf020 2024-03-04T10:03:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2581d9b640 1 -- 172.21.15.99:0/1795784921 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c031640 con 0x7f25840cf020 2024-03-04T10:03:33.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2581d9b640 1 -- 172.21.15.99:0/1795784921 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c02d040 con 0x7f25840cf020 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 -- 172.21.15.99:0/1795784921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840cf020 msgr2=0x7f258415d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1795784921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840cf020 0x7f258415d730 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f256c009870 tx=0x7f256c02b8f0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 -- 172.21.15.99:0/1795784921 shutdown_connections 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1795784921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840cf020 0x7f258415d730 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1795784921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f25840b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 -- 172.21.15.99:0/1795784921 >> 172.21.15.99:0/1795784921 conn(0x7f25840d0580 msgr2=0x7f25840d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 -- 172.21.15.99:0/1795784921 shutdown_connections 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.370+0000 7f2588cfc640 1 -- 172.21.15.99:0/1795784921 wait complete. 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.371+0000 7f2588cfc640 1 Processor -- start 2024-03-04T10:03:33.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 -- start start 2024-03-04T10:03:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f2584152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25840cf020 0x7f25841e7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 0x7f25841e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25841617d0 con 0x7f25840d3300 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25840d1bf0 con 0x7f25840cf020 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2588cfc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2584161950 con 0x7f25840cec40 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2582d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 0x7f25841e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2582d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 0x7f25841e7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55834/0 (socket says 172.21.15.99:55834) 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.372+0000 7f2582d9d640 1 -- 172.21.15.99:0/1054645249 learned_addr learned my addr 172.21.15.99:0/1054645249 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f258259c640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f2584152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f257bfff640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25840cf020 0x7f25841e7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2582d9d640 1 -- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 msgr2=0x7f2584152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2582d9d640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f2584152f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2582d9d640 1 -- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25840cf020 msgr2=0x7f25841e7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2582d9d640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25840cf020 0x7f25841e7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2582d9d640 1 -- 172.21.15.99:0/1054645249 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25840cc7d0 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2582d9d640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 0x7f25841e7f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f257c00b9a0 tx=0x7f257c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c019070 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f257c00cba0 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f257c014440 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25840ccab0 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.373+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25840cd070 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.374+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2584074f60 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.374+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f257c01c070 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.375+0000 7f257affd640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f256406b4d0 0x7f256406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.375+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f257c089ad0 con 0x7f25840d3300 2024-03-04T10:03:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.375+0000 7f258259c640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f256406b4d0 0x7f256406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.375+0000 7f258259c640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f256406b4d0 0x7f256406d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2574006fd0 tx=0x7f2574008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.379+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f257c058d30 con 0x7f25840d3300 2024-03-04T10:03:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.553+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25841e7830 con 0x7f25840d3300 2024-03-04T10:03:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.553+0000 7f257affd640 1 -- 172.21.15.99:0/1054645249 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f257c058630 con 0x7f25840d3300 2024-03-04T10:03:33.554 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:03:33.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f256406b4d0 msgr2=0x7f256406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f256406b4d0 0x7f256406d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2574006fd0 tx=0x7f2574008040 comp rx=0 tx=0).stop 2024-03-04T10:03:33.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 msgr2=0x7f25841e7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 0x7f25841e7f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f257c00b9a0 tx=0x7f257c00be60 comp rx=0 tx=0).stop 2024-03-04T10:03:33.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 shutdown_connections 2024-03-04T10:03:33.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f256406b4d0 0x7f256406d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25840d3300 0x7f25841e7f90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25840cf020 0x7f25841e7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 --2- 172.21.15.99:0/1054645249 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25840cec40 0x7f2584152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 >> 172.21.15.99:0/1054645249 conn(0x7f25840d0580 msgr2=0x7f2584155fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 shutdown_connections 2024-03-04T10:03:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.555+0000 7f2588cfc640 1 -- 172.21.15.99:0/1054645249 wait complete. 2024-03-04T10:03:33.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:03:33.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.665+0000 7fba68836640 1 Processor -- start 2024-03-04T10:03:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.665+0000 7fba68836640 1 -- start start 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.666+0000 7fba68836640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6406d3b0 0x7fba6415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.666+0000 7fba68836640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba6415b410 0x7fba6415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.666+0000 7fba68836640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6415be00 con 0x7fba6415b030 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.666+0000 7fba68836640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640b2720 con 0x7fba6415b410 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.666+0000 7fba68836640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406d8b0 con 0x7fba6406d3b0 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba6415b410 0x7fba6415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba6415b410 0x7fba6415b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56506/0 (socket says 172.21.15.99:56506) 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 -- 172.21.15.99:0/2699186976 learned_addr learned my addr 172.21.15.99:0/2699186976 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba61d9b640 1 --2- 172.21.15.99:0/2699186976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6406d3b0 0x7fba6415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 -- 172.21.15.99:0/2699186976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6406d3b0 msgr2=0x7fba6415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 --2- 172.21.15.99:0/2699186976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6406d3b0 0x7fba6415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 -- 172.21.15.99:0/2699186976 >> v1:172.21.15.162:6789/0 conn(0x7fba6415b030 legacy=0x7fba6415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.667+0000 7fba62d9d640 1 -- 172.21.15.99:0/2699186976 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6406db90 con 0x7fba6415b410 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba62d9d640 1 --2- 172.21.15.99:0/2699186976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b410 0x7fba6415b8c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009870 tx=0x7fba5c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467e157b65717575 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba6159a640 1 -- 172.21.15.99:0/2699186976 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c0317c0 con 0x7fba6415b410 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba6159a640 1 -- 172.21.15.99:0/2699186976 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c037070 con 0x7fba6415b410 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba6159a640 1 -- 172.21.15.99:0/2699186976 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c031b80 con 0x7fba6415b410 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 -- 172.21.15.99:0/2699186976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b410 msgr2=0x7fba6415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 --2- 172.21.15.99:0/2699186976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b410 0x7fba6415b8c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009870 tx=0x7fba5c02b860 comp rx=0 tx=0).stop 2024-03-04T10:03:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 -- 172.21.15.99:0/2699186976 shutdown_connections 2024-03-04T10:03:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 --2- 172.21.15.99:0/2699186976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b410 0x7fba6415b8c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 --2- 172.21.15.99:0/2699186976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6406d3b0 0x7fba6415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 -- 172.21.15.99:0/2699186976 >> 172.21.15.99:0/2699186976 conn(0x7fba6406acd0 msgr2=0x7fba6406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 -- 172.21.15.99:0/2699186976 shutdown_connections 2024-03-04T10:03:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.668+0000 7fba68836640 1 -- 172.21.15.99:0/2699186976 wait complete. 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 Processor -- start 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 -- start start 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba6406d3b0 0x7fba64157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 0x7fba64159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6415b410 0x7fba640ce480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406e690 con 0x7fba6415b030 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406e150 con 0x7fba6406d3b0 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba68836640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406e810 con 0x7fba6415b410 2024-03-04T10:03:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.670+0000 7fba61d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 0x7fba64159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 0x7fba64159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56522/0 (socket says 172.21.15.99:56522) 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 -- 172.21.15.99:0/2906530802 learned_addr learned my addr 172.21.15.99:0/2906530802 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba6259c640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba6406d3b0 0x7fba64157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba62d9d640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6415b410 0x7fba640ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 -- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6415b410 msgr2=0x7fba640ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6415b410 0x7fba640ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 -- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba6406d3b0 msgr2=0x7fba64157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba6406d3b0 0x7fba64157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 -- 172.21.15.99:0/2906530802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64157da0 con 0x7fba6415b030 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba61d9b640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 0x7fba64159bc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba5800d5f0 tx=0x7fba5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba580125f0 con 0x7fba6415b030 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58019070 con 0x7fba6415b030 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba64158040 con 0x7fba6415b030 2024-03-04T10:03:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba580129d0 con 0x7fba6415b030 2024-03-04T10:03:33.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.671+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba640cdc80 con 0x7fba6415b030 2024-03-04T10:03:33.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.672+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64074f60 con 0x7fba6415b030 2024-03-04T10:03:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.673+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fba58012b70 con 0x7fba6415b030 2024-03-04T10:03:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.673+0000 7fba52ffd640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.673+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba5808a130 con 0x7fba6415b030 2024-03-04T10:03:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.673+0000 7fba6259c640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.674+0000 7fba6259c640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba4406b4d0 0x7fba4406d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba54005660 tx=0x7fba54003d90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:33.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.677+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fba58058d40 con 0x7fba6415b030 2024-03-04T10:03:33.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.886+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba640cfe50 con 0x7fba6415b030 2024-03-04T10:03:33.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.886+0000 7fba52ffd640 1 -- 172.21.15.99:0/2906530802 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fba5805cde0 con 0x7fba6415b030 2024-03-04T10:03:33.888 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:33.888 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-04T10:03:33.888 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 2 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba4406b4d0 msgr2=0x7fba4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba4406b4d0 0x7fba4406d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba54005660 tx=0x7fba54003d90 comp rx=0 tx=0).stop 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 msgr2=0x7fba64159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 0x7fba64159bc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba5800d5f0 tx=0x7fba5800dab0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 shutdown_connections 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba4406b4d0 0x7fba4406d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba6415b410 0x7fba640ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba6415b030 0x7fba64159bc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 --2- 172.21.15.99:0/2906530802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba6406d3b0 0x7fba64157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:33.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 >> 172.21.15.99:0/2906530802 conn(0x7fba6406acd0 msgr2=0x7fba6406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:33.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 shutdown_connections 2024-03-04T10:03:33.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:33.888+0000 7fba68836640 1 -- 172.21.15.99:0/2906530802 wait complete. 2024-03-04T10:03:35.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:03:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.011+0000 7f88f3b53640 1 Processor -- start 2024-03-04T10:03:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.011+0000 7f88f3b53640 1 -- start start 2024-03-04T10:03:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f3b53640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec0cf650 0x7f88ec0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f3b53640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f88ec070de0 0x7f88ec071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f3b53640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec15f020 con 0x7f88ec154a80 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f3b53640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0cfff0 con 0x7f88ec070de0 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f3b53640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0cd010 con 0x7f88ec0cf650 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f20f0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f88ec070de0 0x7f88ec071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f20f0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f88ec070de0 0x7f88ec071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56530/0 (socket says 172.21.15.99:56530) 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f20f0640 1 -- 172.21.15.99:0/592763283 learned_addr learned my addr 172.21.15.99:0/592763283 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.012+0000 7f88f10ee640 1 --2- 172.21.15.99:0/592763283 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec0cf650 0x7f88ec0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f20f0640 1 -- 172.21.15.99:0/592763283 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec0cf650 msgr2=0x7f88ec0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f20f0640 1 --2- 172.21.15.99:0/592763283 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec0cf650 0x7f88ec0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f20f0640 1 -- 172.21.15.99:0/592763283 >> v1:172.21.15.162:6789/0 conn(0x7f88ec154a80 legacy=0x7f88ec15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f20f0640 1 -- 172.21.15.99:0/592763283 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec0cdaa0 con 0x7f88ec070de0 2024-03-04T10:03:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f20f0640 1 --2- 172.21.15.99:0/592763283 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec070de0 0x7f88ec071270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009e50 tx=0x7f88e402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa792ec4e9fa2701 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f08ed640 1 -- 172.21.15.99:0/592763283 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e402cb00 con 0x7f88ec070de0 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f08ed640 1 -- 172.21.15.99:0/592763283 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e402cca0 con 0x7f88ec070de0 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.013+0000 7f88f08ed640 1 -- 172.21.15.99:0/592763283 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4034610 con 0x7f88ec070de0 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 -- 172.21.15.99:0/592763283 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec070de0 msgr2=0x7f88ec071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 --2- 172.21.15.99:0/592763283 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec070de0 0x7f88ec071270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009e50 tx=0x7f88e402bf60 comp rx=0 tx=0).stop 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 -- 172.21.15.99:0/592763283 shutdown_connections 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 --2- 172.21.15.99:0/592763283 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec070de0 0x7f88ec071270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 --2- 172.21.15.99:0/592763283 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec0cf650 0x7f88ec0cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 -- 172.21.15.99:0/592763283 >> 172.21.15.99:0/592763283 conn(0x7f88ec06f5f0 msgr2=0x7f88ec06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 -- 172.21.15.99:0/592763283 shutdown_connections 2024-03-04T10:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.014+0000 7f88f3b53640 1 -- 172.21.15.99:0/592763283 wait complete. 2024-03-04T10:03:36.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.015+0000 7f88f3b53640 1 Processor -- start 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.015+0000 7f88f3b53640 1 -- start start 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f3b53640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88ec070de0 0x7f88ec156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f3b53640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 0x7f88ec160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f3b53640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec154a80 0x7f88ec153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f3b53640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0ce340 con 0x7f88ec0cf650 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f3b53640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0ce060 con 0x7f88ec070de0 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f3b53640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0ce620 con 0x7f88ec154a80 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 0x7f88ec160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 0x7f88ec160980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56544/0 (socket says 172.21.15.99:56544) 2024-03-04T10:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 -- 172.21.15.99:0/364609862 learned_addr learned my addr 172.21.15.99:0/364609862 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f18ef640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88ec070de0 0x7f88ec156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f20f0640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec154a80 0x7f88ec153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 -- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec154a80 msgr2=0x7f88ec153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec154a80 0x7f88ec153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 -- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88ec070de0 msgr2=0x7f88ec156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88ec070de0 0x7f88ec156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f10ee640 1 -- 172.21.15.99:0/364609862 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec1574f0 con 0x7f88ec0cf650 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.016+0000 7f88f20f0640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec154a80 0x7f88ec153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.017+0000 7f88f10ee640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 0x7f88ec160980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f88e000e9b0 tx=0x7f88e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.017+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0017070 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.017+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e000c9a0 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.017+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec1577d0 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.017+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e000cd90 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.017+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec159ea0 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.018+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec074f60 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.018+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f88e001a070 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.018+0000 7f88de7fc640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88d006b4d0 0x7f88d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.019+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f88e008a0a0 con 0x7f88ec0cf650 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.019+0000 7f88f18ef640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88d006b4d0 0x7f88d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.019+0000 7f88f18ef640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88d006b4d0 0x7f88d006d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f88e80016e0 tx=0x7f88e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.023+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f88e0058cb0 con 0x7f88ec0cf650 2024-03-04T10:03:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.229+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88ec0d2620 con 0x7f88ec0cf650 2024-03-04T10:03:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.229+0000 7f88de7fc640 1 -- 172.21.15.99:0/364609862 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f88e005cd50 con 0x7f88ec0cf650 2024-03-04T10:03:36.230 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:36.230 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-04T10:03:36.231 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 3 2024-03-04T10:03:36.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88d006b4d0 msgr2=0x7f88d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88d006b4d0 0x7f88d006d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f88e80016e0 tx=0x7f88e8009040 comp rx=0 tx=0).stop 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 msgr2=0x7f88ec160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 0x7f88ec160980 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f88e000e9b0 tx=0x7f88e000ee70 comp rx=0 tx=0).stop 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 shutdown_connections 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88d006b4d0 0x7f88d006d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88ec154a80 0x7f88ec153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88ec0cf650 0x7f88ec160980 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 --2- 172.21.15.99:0/364609862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88ec070de0 0x7f88ec156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 >> 172.21.15.99:0/364609862 conn(0x7f88ec06f5f0 msgr2=0x7f88ec06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 shutdown_connections 2024-03-04T10:03:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.231+0000 7f88f3b53640 1 -- 172.21.15.99:0/364609862 wait complete. 2024-03-04T10:03:36.243 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.341+0000 7f73802dc640 1 Processor -- start 2024-03-04T10:03:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.342+0000 7f73802dc640 1 -- start start 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.342+0000 7f73802dc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7378154a80 0x7f737815e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.342+0000 7f73802dc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f73780cfab0 0x7f7378161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.342+0000 7f73802dc640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73780cdaa0 con 0x7f73780cf650 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.342+0000 7f73802dc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7378071df0 con 0x7f73780cfab0 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.342+0000 7f73802dc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73780cd010 con 0x7f7378154a80 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e879640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f73780cfab0 0x7f7378161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737d877640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f73780cf650 0x7f73781616c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51000/0 (socket says 172.21.15.99:51000) 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737d877640 1 -- 172.21.15.99:0/1848329662 learned_addr learned my addr 172.21.15.99:0/1848329662 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e879640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f73780cfab0 0x7f7378161dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56552/0 (socket says 172.21.15.99:56552) 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e078640 1 --2- 172.21.15.99:0/1848329662 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7378154a80 0x7f737815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e879640 1 -- 172.21.15.99:0/1848329662 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7378154a80 msgr2=0x7f737815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e879640 1 --2- 172.21.15.99:0/1848329662 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7378154a80 0x7f737815e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e879640 1 -- 172.21.15.99:0/1848329662 >> v1:172.21.15.99:6789/0 conn(0x7f73780cf650 legacy=0x7f73781616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737e879640 1 -- 172.21.15.99:0/1848329662 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73780cdd80 con 0x7f73780cfab0 2024-03-04T10:03:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.343+0000 7f737d076640 1 -- 172.21.15.99:0/1848329662 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970044888 0 0) 0x7f73780cdaa0 con 0x7f73780cf650 2024-03-04T10:03:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f737e879640 1 --2- 172.21.15.99:0/1848329662 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f7378161dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f737402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f215de252b02f026 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f737d076640 1 -- 172.21.15.99:0/1848329662 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374039070 con 0x7f73780cfab0 2024-03-04T10:03:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f737d076640 1 -- 172.21.15.99:0/1848329662 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737402cc60 con 0x7f73780cfab0 2024-03-04T10:03:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f737d076640 1 -- 172.21.15.99:0/1848329662 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374034660 con 0x7f73780cfab0 2024-03-04T10:03:36.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 -- 172.21.15.99:0/1848329662 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 msgr2=0x7f7378161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 --2- 172.21.15.99:0/1848329662 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f7378161dd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f737402beb0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 -- 172.21.15.99:0/1848329662 shutdown_connections 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 --2- 172.21.15.99:0/1848329662 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f7378161dd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 --2- 172.21.15.99:0/1848329662 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7378154a80 0x7f737815e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 -- 172.21.15.99:0/1848329662 >> 172.21.15.99:0/1848329662 conn(0x7f737806f5f0 msgr2=0x7f737806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 -- 172.21.15.99:0/1848329662 shutdown_connections 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.344+0000 7f73802dc640 1 -- 172.21.15.99:0/1848329662 wait complete. 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.345+0000 7f73802dc640 1 Processor -- start 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.345+0000 7f73802dc640 1 -- start start 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f73802dc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780cf650 0x7f7378153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f73802dc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f737806d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f73802dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7378154a80 0x7f737806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f73802dc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780ce620 con 0x7f73780cfab0 2024-03-04T10:03:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f73802dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73780ce340 con 0x7f7378154a80 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f73802dc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73780cff40 con 0x7f73780cf650 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f737d877640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f737806d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f737d877640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f737806d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56558/0 (socket says 172.21.15.99:56558) 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f737d877640 1 -- 172.21.15.99:0/953626687 learned_addr learned my addr 172.21.15.99:0/953626687 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.346+0000 7f737e078640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780cf650 0x7f7378153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737e879640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7378154a80 0x7f737806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737d877640 1 -- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780cf650 msgr2=0x7f7378153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737d877640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780cf650 0x7f7378153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737d877640 1 -- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7378154a80 msgr2=0x7f737806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737d877640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7378154a80 0x7f737806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737d877640 1 -- 172.21.15.99:0/953626687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73781f7c30 con 0x7f73780cfab0 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737e078640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780cf650 0x7f7378153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737e879640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7378154a80 0x7f737806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f737d877640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f737806d310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f736800c4d0 tx=0x7f736800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368018070 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7368007500 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73781f3b40 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368004a20 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.347+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73781f4100 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.348+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7378074f60 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.348+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f736801b070 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.348+0000 7f736effd640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f735c06b4d0 0x7f735c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.349+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7368089120 con 0x7f73780cfab0 2024-03-04T10:03:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.349+0000 7f737e078640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f735c06b4d0 0x7f735c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.349+0000 7f737e078640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f735c06b4d0 0x7f735c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7370006fd0 tx=0x7f7370008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.353+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7368058380 con 0x7f73780cfab0 2024-03-04T10:03:36.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.525+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73780b20a0 con 0x7f73780cfab0 2024-03-04T10:03:36.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.525+0000 7f736effd640 1 -- 172.21.15.99:0/953626687 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12848 (secure 0 0 0) 0x7f7368057c80 con 0x7f73780cfab0 2024-03-04T10:03:36.526 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:36.527 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":16,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:29.481552+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f735c06b4d0 msgr2=0x7f735c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f735c06b4d0 0x7f735c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7370006fd0 tx=0x7f7370008040 comp rx=0 tx=0).stop 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 msgr2=0x7f737806d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f737806d310 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f736800c4d0 tx=0x7f736800c990 comp rx=0 tx=0).stop 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 shutdown_connections 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f735c06b4d0 0x7f735c06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7378154a80 0x7f737806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73780cfab0 0x7f737806d310 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 --2- 172.21.15.99:0/953626687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73780cf650 0x7f7378153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 >> 172.21.15.99:0/953626687 conn(0x7f737806f5f0 msgr2=0x7f7378161830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 shutdown_connections 2024-03-04T10:03:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.527+0000 7f73802dc640 1 -- 172.21.15.99:0/953626687 wait complete. 2024-03-04T10:03:36.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:03:36.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:03:36.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.641+0000 7f2d5a873640 1 Processor -- start 2024-03-04T10:03:36.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.641+0000 7f2d5a873640 1 -- start start 2024-03-04T10:03:36.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d5a873640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2d540cec40 0x7f2d540b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d5a873640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540cf020 0x7f2d5415d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d5a873640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d540ca970 con 0x7f2d540d3300 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d5a873640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d54161d90 con 0x7f2d540cec40 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d5a873640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54162070 con 0x7f2d540cf020 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d53fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2d540cec40 0x7f2d540b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540cf020 0x7f2d5415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d58e10640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2d540d3300 0x7f2d540d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:59388/0 (socket says 172.21.15.99:59388) 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d58e10640 1 -- 172.21.15.99:0/345746540 learned_addr learned my addr 172.21.15.99:0/345746540 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d53fff640 1 -- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540cf020 msgr2=0x7f2d5415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d53fff640 1 --2- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540cf020 0x7f2d5415d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d53fff640 1 -- 172.21.15.99:0/345746540 >> v1:172.21.15.162:6789/0 conn(0x7f2d540d3300 legacy=0x7f2d540d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d53fff640 1 -- 172.21.15.99:0/345746540 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d54162350 con 0x7f2d540cec40 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d537fe640 1 --2- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540cf020 0x7f2d5415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:36.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.642+0000 7f2d52ffd640 1 -- 172.21.15.99:0/345746540 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230626570 0 0) 0x7f2d540ca970 con 0x7f2d540d3300 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.643+0000 7f2d53fff640 1 --2- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540b6ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d440097a0 tx=0x7f2d4402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58c84e0829d91bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.643+0000 7f2d52ffd640 1 -- 172.21.15.99:0/345746540 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44035020 con 0x7f2d540cec40 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.643+0000 7f2d52ffd640 1 -- 172.21.15.99:0/345746540 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d44031830 con 0x7f2d540cec40 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.643+0000 7f2d52ffd640 1 -- 172.21.15.99:0/345746540 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44031c00 con 0x7f2d540cec40 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 -- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 msgr2=0x7f2d540b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 --2- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540b6ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d440097a0 tx=0x7f2d4402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 -- 172.21.15.99:0/345746540 shutdown_connections 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 --2- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540cf020 0x7f2d5415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 --2- 172.21.15.99:0/345746540 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540b6ea0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 -- 172.21.15.99:0/345746540 >> 172.21.15.99:0/345746540 conn(0x7f2d540d0580 msgr2=0x7f2d540d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 -- 172.21.15.99:0/345746540 shutdown_connections 2024-03-04T10:03:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.644+0000 7f2d5a873640 1 -- 172.21.15.99:0/345746540 wait complete. 2024-03-04T10:03:36.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.645+0000 7f2d5a873640 1 Processor -- start 2024-03-04T10:03:36.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.645+0000 7f2d5a873640 1 -- start start 2024-03-04T10:03:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d5a873640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d5a873640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 0x7f2d5415fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d5a873640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540d3300 0x7f2d54160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d5a873640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541617d0 con 0x7f2d540cf020 2024-03-04T10:03:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d5a873640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d540d1bf0 con 0x7f2d540cec40 2024-03-04T10:03:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d5a873640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d54161950 con 0x7f2d540d3300 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 0x7f2d5415fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 0x7f2d5415fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56560/0 (socket says 172.21.15.99:56560) 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 -- 172.21.15.99:0/2669982100 learned_addr learned my addr 172.21.15.99:0/2669982100 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d53fff640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d58e10640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540d3300 0x7f2d54160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 -- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540d3300 msgr2=0x7f2d54160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540d3300 0x7f2d54160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 -- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 msgr2=0x7f2d540ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 -- 172.21.15.99:0/2669982100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d541f3830 con 0x7f2d540cf020 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d53fff640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d58e10640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540d3300 0x7f2d54160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d537fe640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 0x7f2d5415fb10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800ebe0 tx=0x7f2d4800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.646+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48017070 con 0x7f2d540cf020 2024-03-04T10:03:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.647+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4800cc30 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.647+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d541f3b10 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.647+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48013470 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.647+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d541f40d0 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.648+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d54074f60 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.648+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2d4801a070 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.648+0000 7f2d50ff9640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d4006b4d0 0x7f2d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.648+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2d4808a1c0 con 0x7f2d540cf020 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.648+0000 7f2d53fff640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d4006b4d0 0x7f2d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.649+0000 7f2d53fff640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d4006b4d0 0x7f2d4006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d44036000 tx=0x7f2d44008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.652+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2d48059390 con 0x7f2d540cf020 2024-03-04T10:03:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.823+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d541607b0 con 0x7f2d540cf020 2024-03-04T10:03:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.824+0000 7f2d50ff9640 1 -- 172.21.15.99:0/2669982100 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d48058c90 con 0x7f2d540cf020 2024-03-04T10:03:36.825 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d4006b4d0 msgr2=0x7f2d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d4006b4d0 0x7f2d4006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2d44036000 tx=0x7f2d44008030 comp rx=0 tx=0).stop 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 msgr2=0x7f2d5415fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 0x7f2d5415fb10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800ebe0 tx=0x7f2d4800c520 comp rx=0 tx=0).stop 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 shutdown_connections 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d4006b4d0 0x7f2d4006d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d540d3300 0x7f2d54160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d540cf020 0x7f2d5415fb10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 --2- 172.21.15.99:0/2669982100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d540cec40 0x7f2d540ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 >> 172.21.15.99:0/2669982100 conn(0x7f2d540d0580 msgr2=0x7f2d54159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 shutdown_connections 2024-03-04T10:03:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.826+0000 7f2d5a873640 1 -- 172.21.15.99:0/2669982100 wait complete. 2024-03-04T10:03:36.837 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T10:03:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.936+0000 7f0e69b7e640 1 Processor -- start 2024-03-04T10:03:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.936+0000 7f0e69b7e640 1 -- start start 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e69b7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e6415b030 0x7f0e6415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e69b7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0e6415b410 0x7f0e6415b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e69b7e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6406d8b0 con 0x7f0e6406d3b0 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e69b7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e640b2720 con 0x7f0e6415b030 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e69b7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6406db90 con 0x7f0e6415b410 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e637fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e6415b030 0x7f0e6415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e62ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0e6406d3b0 0x7f0e6415dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51012/0 (socket says 172.21.15.99:51012) 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e62ffd640 1 -- 172.21.15.99:0/2525871252 learned_addr learned my addr 172.21.15.99:0/2525871252 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e63fff640 1 --2- 172.21.15.99:0/2525871252 >> v2:172.21.15.162:3300/0 conn(0x7f0e6415b410 0x7f0e6415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e637fe640 1 -- 172.21.15.99:0/2525871252 >> v2:172.21.15.162:3300/0 conn(0x7f0e6415b410 msgr2=0x7f0e6415b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e637fe640 1 --2- 172.21.15.99:0/2525871252 >> v2:172.21.15.162:3300/0 conn(0x7f0e6415b410 0x7f0e6415b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e637fe640 1 -- 172.21.15.99:0/2525871252 >> v1:172.21.15.99:6789/0 conn(0x7f0e6406d3b0 legacy=0x7f0e6415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e637fe640 1 -- 172.21.15.99:0/2525871252 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6406de70 con 0x7f0e6415b030 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.937+0000 7f0e627fc640 1 -- 172.21.15.99:0/2525871252 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537048972 0 0) 0x7f0e6406d8b0 con 0x7f0e6406d3b0 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.938+0000 7f0e637fe640 1 --2- 172.21.15.99:0/2525871252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6415b030 0x7f0e6415c120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e640b28a0 tx=0x7f0e5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec7046f1ccf79ad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.938+0000 7f0e627fc640 1 -- 172.21.15.99:0/2525871252 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e540315a0 con 0x7f0e6415b030 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.938+0000 7f0e627fc640 1 -- 172.21.15.99:0/2525871252 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54037070 con 0x7f0e6415b030 2024-03-04T10:03:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.938+0000 7f0e627fc640 1 -- 172.21.15.99:0/2525871252 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54031960 con 0x7f0e6415b030 2024-03-04T10:03:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 -- 172.21.15.99:0/2525871252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6415b030 msgr2=0x7f0e6415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/2525871252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6415b030 0x7f0e6415c120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e640b28a0 tx=0x7f0e5402b840 comp rx=0 tx=0).stop 2024-03-04T10:03:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 -- 172.21.15.99:0/2525871252 shutdown_connections 2024-03-04T10:03:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/2525871252 >> v2:172.21.15.162:3300/0 conn(0x7f0e6415b410 0x7f0e6415b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/2525871252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6415b030 0x7f0e6415c120 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 -- 172.21.15.99:0/2525871252 >> 172.21.15.99:0/2525871252 conn(0x7f0e6406acd0 msgr2=0x7f0e6406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 -- 172.21.15.99:0/2525871252 shutdown_connections 2024-03-04T10:03:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.939+0000 7f0e69b7e640 1 -- 172.21.15.99:0/2525871252 wait complete. 2024-03-04T10:03:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.940+0000 7f0e69b7e640 1 Processor -- start 2024-03-04T10:03:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.940+0000 7f0e69b7e640 1 -- start start 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e69b7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 0x7f0e64144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e69b7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6415b030 0x7f0e640cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e69b7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6415b410 0x7f0e64159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e69b7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6406e690 con 0x7f0e6406d3b0 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e69b7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6415bdc0 con 0x7f0e6415b410 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e69b7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6406e810 con 0x7f0e6415b030 2024-03-04T10:03:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 0x7f0e64144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 0x7f0e64144180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56578/0 (socket says 172.21.15.99:56578) 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 -- 172.21.15.99:0/3510231130 learned_addr learned my addr 172.21.15.99:0/3510231130 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e63fff640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6415b410 0x7f0e64159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e62ffd640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6415b030 0x7f0e640cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 -- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6415b030 msgr2=0x7f0e640cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6415b030 0x7f0e640cf0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 -- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6415b410 msgr2=0x7f0e64159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6415b410 0x7f0e64159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 -- 172.21.15.99:0/3510231130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e641509f0 con 0x7f0e6406d3b0 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e63fff640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6415b410 0x7f0e64159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.941+0000 7f0e637fe640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 0x7f0e64144180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e54000c00 tx=0x7f0e540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.942+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54032640 con 0x7f0e6406d3b0 2024-03-04T10:03:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.942+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54032800 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.942+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e64150cd0 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.942+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54032bf0 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.942+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e64151290 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.943+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e64074f60 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.943+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0e5404d3f0 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.943+0000 7f0e4bfff640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e5006b4d0 0x7f0e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.943+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0e5402f030 con 0x7f0e6406d3b0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.943+0000 7f0e62ffd640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e5006b4d0 0x7f0e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.944+0000 7f0e62ffd640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e5006b4d0 0x7f0e5006d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e58007fd0 tx=0x7f0e58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:36.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:36.948+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0e54040070 con 0x7f0e6406d3b0 2024-03-04T10:03:37.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.123+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0e6415c780 con 0x7f0e6406d3b0 2024-03-04T10:03:37.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.123+0000 7f0e4bfff640 1 -- 172.21.15.99:0/3510231130 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0e5407ac60 con 0x7f0e6406d3b0 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:03:37.125 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e5006b4d0 msgr2=0x7f0e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e5006b4d0 0x7f0e5006d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0e58007fd0 tx=0x7f0e58009040 comp rx=0 tx=0).stop 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 msgr2=0x7f0e64144180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 0x7f0e64144180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0e54000c00 tx=0x7f0e540046c0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 shutdown_connections 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e5006b4d0 0x7f0e5006d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e6415b410 0x7f0e64159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e6415b030 0x7f0e640cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 --2- 172.21.15.99:0/3510231130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e6406d3b0 0x7f0e64144180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.125+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 >> 172.21.15.99:0/3510231130 conn(0x7f0e6406acd0 msgr2=0x7f0e640b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.126+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 shutdown_connections 2024-03-04T10:03:37.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.126+0000 7f0e69b7e640 1 -- 172.21.15.99:0/3510231130 wait complete. 2024-03-04T10:03:37.137 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T10:03:37.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T10:03:37.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.240+0000 7f5b83a85640 1 Processor -- start 2024-03-04T10:03:37.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.240+0000 7f5b83a85640 1 -- start start 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.240+0000 7f5b83a85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c0d0150 0x7f5b7c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b83a85640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5b7c15ca00 0x7f5b7c0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b83a85640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c15bf80 con 0x7f5b7c06b200 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b83a85640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c0cc860 con 0x7f5b7c15ca00 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b83a85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c15c570 con 0x7f5b7c0d0150 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b82022640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5b7c15ca00 0x7f5b7c0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b81821640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5b7c06b200 0x7f5b7c0cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51022/0 (socket says 172.21.15.99:51022) 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b81821640 1 -- 172.21.15.99:0/3204510553 learned_addr learned my addr 172.21.15.99:0/3204510553 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b82022640 1 -- 172.21.15.99:0/3204510553 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c0d0150 msgr2=0x7f5b7c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b82022640 1 --2- 172.21.15.99:0/3204510553 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c0d0150 0x7f5b7c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b82022640 1 -- 172.21.15.99:0/3204510553 >> v1:172.21.15.99:6789/0 conn(0x7f5b7c06b200 legacy=0x7f5b7c0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b82022640 1 -- 172.21.15.99:0/3204510553 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c15ceb0 con 0x7f5b7c15ca00 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.241+0000 7f5b8081f640 1 -- 172.21.15.99:0/3204510553 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167419196 0 0) 0x7f5b78001110 con 0x7f5b7c06b200 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b82022640 1 --2- 172.21.15.99:0/3204510553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c15ca00 0x7f5b7c0cc320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009dc0 tx=0x7f5b7402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c8e0dde722a435a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b8081f640 1 -- 172.21.15.99:0/3204510553 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74039070 con 0x7f5b7c15ca00 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b8081f640 1 -- 172.21.15.99:0/3204510553 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7402cc40 con 0x7f5b7c15ca00 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b8081f640 1 -- 172.21.15.99:0/3204510553 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b74034440 con 0x7f5b7c15ca00 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 -- 172.21.15.99:0/3204510553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c15ca00 msgr2=0x7f5b7c0cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 --2- 172.21.15.99:0/3204510553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c15ca00 0x7f5b7c0cc320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009dc0 tx=0x7f5b7402be90 comp rx=0 tx=0).stop 2024-03-04T10:03:37.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 -- 172.21.15.99:0/3204510553 shutdown_connections 2024-03-04T10:03:37.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 --2- 172.21.15.99:0/3204510553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c15ca00 0x7f5b7c0cc320 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 --2- 172.21.15.99:0/3204510553 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c0d0150 0x7f5b7c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 -- 172.21.15.99:0/3204510553 >> 172.21.15.99:0/3204510553 conn(0x7f5b7c099a00 msgr2=0x7f5b7c0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:37.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 -- 172.21.15.99:0/3204510553 shutdown_connections 2024-03-04T10:03:37.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.242+0000 7f5b83a85640 1 -- 172.21.15.99:0/3204510553 wait complete. 2024-03-04T10:03:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 Processor -- start 2024-03-04T10:03:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 -- start start 2024-03-04T10:03:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b7c06b200 0x7f5b7c0b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 0x7f5b7c0c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c15ca00 0x7f5b7c0c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c15c200 con 0x7f5b7c0d0150 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0ccd70 con 0x7f5b7c06b200 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b83a85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c06b5e0 con 0x7f5b7c15ca00 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b81020640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 0x7f5b7c0c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b81020640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 0x7f5b7c0c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56602/0 (socket says 172.21.15.99:56602) 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.244+0000 7f5b81020640 1 -- 172.21.15.99:0/2788452958 learned_addr learned my addr 172.21.15.99:0/2788452958 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81020640 1 -- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c15ca00 msgr2=0x7f5b7c0c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81020640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c15ca00 0x7f5b7c0c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81020640 1 -- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b7c06b200 msgr2=0x7f5b7c0b4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81821640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b7c06b200 0x7f5b7c0b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81020640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b7c06b200 0x7f5b7c0b4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81020640 1 -- 172.21.15.99:0/2788452958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c158580 con 0x7f5b7c0d0150 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b81020640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 0x7f5b7c0c6510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00b4d0 tx=0x7f5b6c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c014070 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c010870 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c010c60 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c0c6f90 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.245+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c158b00 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.246+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c074f60 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.246+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5b6c017070 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.247+0000 7f5b727fc640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5b6006b4d0 0x7f5b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.247+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b6c089c40 con 0x7f5b7c0d0150 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.247+0000 7f5b81821640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5b6006b4d0 0x7f5b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.247+0000 7f5b81821640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5b6006b4d0 0x7f5b6006d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000f60 tx=0x7f5b78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:37.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.251+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5b6c058ea0 con 0x7f5b7c0d0150 2024-03-04T10:03:37.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.421+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5b7c0c71d0 con 0x7f5b7c0d0150 2024-03-04T10:03:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.511+0000 7f5b727fc640 1 -- 172.21.15.99:0/2788452958 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5b6c0587a0 con 0x7f5b7c0d0150 2024-03-04T10:03:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2024-03-04T10:03:37.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5b6006b4d0 msgr2=0x7f5b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5b6006b4d0 0x7f5b6006d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5b78000f60 tx=0x7f5b78009040 comp rx=0 tx=0).stop 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 msgr2=0x7f5b7c0c6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 0x7f5b7c0c6510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00b4d0 tx=0x7f5b6c00b990 comp rx=0 tx=0).stop 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 shutdown_connections 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5b6006b4d0 0x7f5b6006d990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b7c15ca00 0x7f5b7c0c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b7c0d0150 0x7f5b7c0c6510 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00b4d0 tx=0x7f5b6c00b990 comp rx=0 tx=0).stop 2024-03-04T10:03:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 --2- 172.21.15.99:0/2788452958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5b7c06b200 0x7f5b7c0b4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.514+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 >> 172.21.15.99:0/2788452958 conn(0x7f5b7c099a00 msgr2=0x7f5b7c15a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:37.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.515+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 shutdown_connections 2024-03-04T10:03:37.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.515+0000 7f5b83a85640 1 -- 172.21.15.99:0/2788452958 wait complete. 2024-03-04T10:03:37.530 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-04T10:03:37.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.636+0000 7fc8bfa20640 1 Processor -- start 2024-03-04T10:03:37.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.636+0000 7fc8bfa20640 1 -- start start 2024-03-04T10:03:37.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bfa20640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc8b815da20 0x7fc8b80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bfa20640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b8161df0 0x7fc8b80cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bfa20640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b8162450 con 0x7fc8b80cf660 2024-03-04T10:03:37.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bfa20640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80d0000 con 0x7fc8b815da20 2024-03-04T10:03:37.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bfa20640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cc920 con 0x7fc8b8161df0 2024-03-04T10:03:37.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bd7bc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc8b815da20 0x7fc8b80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bdfbd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b8161df0 0x7fc8b80cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bcfbb640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc8b80cf660 0x7fc8b81616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:59404/0 (socket says 172.21.15.99:59404) 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bcfbb640 1 -- 172.21.15.99:0/3160632410 learned_addr learned my addr 172.21.15.99:0/3160632410 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bd7bc640 1 -- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b8161df0 msgr2=0x7fc8b80cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bd7bc640 1 --2- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b8161df0 0x7fc8b80cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bd7bc640 1 -- 172.21.15.99:0/3160632410 >> v1:172.21.15.162:6789/0 conn(0x7fc8b80cf660 legacy=0x7fc8b81616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8affff640 1 -- 172.21.15.99:0/3160632410 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880681888 0 0) 0x7fc8b8162450 con 0x7fc8b80cf660 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bd7bc640 1 -- 172.21.15.99:0/3160632410 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80cc2d0 con 0x7fc8b815da20 2024-03-04T10:03:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.637+0000 7fc8bdfbd640 1 --2- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b8161df0 0x7fc8b80cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:37.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.638+0000 7fc8bd7bc640 1 --2- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b815da20 0x7fc8b80ccdb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40097a0 tx=0x7fc8b402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7c339f50519e981 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:37.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.638+0000 7fc8affff640 1 -- 172.21.15.99:0/3160632410 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4035020 con 0x7fc8b815da20 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.638+0000 7fc8affff640 1 -- 172.21.15.99:0/3160632410 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b4031830 con 0x7fc8b815da20 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.638+0000 7fc8affff640 1 -- 172.21.15.99:0/3160632410 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b4031c00 con 0x7fc8b815da20 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 -- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b815da20 msgr2=0x7fc8b80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b815da20 0x7fc8b80ccdb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40097a0 tx=0x7fc8b402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 -- 172.21.15.99:0/3160632410 shutdown_connections 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b8161df0 0x7fc8b80cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/3160632410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b815da20 0x7fc8b80ccdb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 -- 172.21.15.99:0/3160632410 >> 172.21.15.99:0/3160632410 conn(0x7fc8b80b33a0 msgr2=0x7fc8b80b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 -- 172.21.15.99:0/3160632410 shutdown_connections 2024-03-04T10:03:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.639+0000 7fc8bfa20640 1 -- 172.21.15.99:0/3160632410 wait complete. 2024-03-04T10:03:37.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.640+0000 7fc8bfa20640 1 Processor -- start 2024-03-04T10:03:37.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.640+0000 7fc8bfa20640 1 -- start start 2024-03-04T10:03:37.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bfa20640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b80cf660 0x7fc8b80b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bfa20640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 0x7fc8b81f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bfa20640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b8161df0 0x7fc8b81f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bfa20640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cbf80 con 0x7fc8b815da20 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bfa20640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b80cd2f0 con 0x7fc8b8161df0 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bfa20640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b815de00 con 0x7fc8b80cf660 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 0x7fc8b81f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 0x7fc8b81f67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56616/0 (socket says 172.21.15.99:56616) 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 -- 172.21.15.99:0/448560173 learned_addr learned my addr 172.21.15.99:0/448560173 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bd7bc640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b80cf660 0x7fc8b80b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bdfbd640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b8161df0 0x7fc8b81f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 -- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b80cf660 msgr2=0x7fc8b80b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b80cf660 0x7fc8b80b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 -- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b8161df0 msgr2=0x7fc8b81f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b8161df0 0x7fc8b81f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 -- 172.21.15.99:0/448560173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b81f7230 con 0x7fc8b815da20 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bd7bc640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b80cf660 0x7fc8b80b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.641+0000 7fc8bcfbb640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 0x7fc8b81f67b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c500 tx=0x7fc8a800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.642+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a80074e0 con 0x7fc8b815da20 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.642+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a80047e0 con 0x7fc8b815da20 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.642+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b81f74a0 con 0x7fc8b815da20 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.642+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a8004bd0 con 0x7fc8b815da20 2024-03-04T10:03:37.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.642+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8155040 con 0x7fc8b815da20 2024-03-04T10:03:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.643+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b8074f60 con 0x7fc8b815da20 2024-03-04T10:03:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.643+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc8a8011070 con 0x7fc8b815da20 2024-03-04T10:03:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.643+0000 7fc8adffb640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc89c06b4d0 0x7fc89c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.643+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc8a8089ba0 con 0x7fc8b815da20 2024-03-04T10:03:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.644+0000 7fc8bd7bc640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc89c06b4d0 0x7fc89c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.644+0000 7fc8bd7bc640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc89c06b4d0 0x7fc89c06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4036000 tx=0x7fc8b4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:37.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.648+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc8a800e280 con 0x7fc8b815da20 2024-03-04T10:03:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:37.822+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fc8b806e620 con 0x7fc8b815da20 2024-03-04T10:03:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.513+0000 7fc8adffb640 1 -- 172.21.15.99:0/448560173 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7fc8a8058c00 con 0x7fc8b815da20 2024-03-04T10:03:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc89c06b4d0 msgr2=0x7fc89c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc89c06b4d0 0x7fc89c06d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8b4036000 tx=0x7fc8b4008030 comp rx=0 tx=0).stop 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 msgr2=0x7fc8b81f67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 0x7fc8b81f67b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8a800c500 tx=0x7fc8a800c900 comp rx=0 tx=0).stop 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 shutdown_connections 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc89c06b4d0 0x7fc89c06d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8b8161df0 0x7fc8b81f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8b815da20 0x7fc8b81f67b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 --2- 172.21.15.99:0/448560173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8b80cf660 0x7fc8b80b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 >> 172.21.15.99:0/448560173 conn(0x7fc8b80b33a0 msgr2=0x7fc8b80cd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:38.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 shutdown_connections 2024-03-04T10:03:38.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.516+0000 7fc8bfa20640 1 -- 172.21.15.99:0/448560173 wait complete. 2024-03-04T10:03:38.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T10:03:38.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.639+0000 7f79604a4640 1 Processor -- start 2024-03-04T10:03:38.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.639+0000 7f79604a4640 1 -- start start 2024-03-04T10:03:38.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f79604a4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f795815da20 0x7f79580ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f79604a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7958161df0 0x7f79580cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f79604a4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7958162450 con 0x7f79580cf660 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f79604a4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79580d0000 con 0x7f795815da20 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f79604a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79580cc920 con 0x7f7958161df0 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f795da3f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f79580cf660 0x7f79581616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51030/0 (socket says 172.21.15.99:51030) 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f795e240640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f795815da20 0x7f79580ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.640+0000 7f795da3f640 1 -- 172.21.15.99:0/567117871 learned_addr learned my addr 172.21.15.99:0/567117871 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795ea41640 1 --2- 172.21.15.99:0/567117871 >> v2:172.21.15.162:3300/0 conn(0x7f7958161df0 0x7f79580cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795e240640 1 -- 172.21.15.99:0/567117871 >> v2:172.21.15.162:3300/0 conn(0x7f7958161df0 msgr2=0x7f79580cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795e240640 1 --2- 172.21.15.99:0/567117871 >> v2:172.21.15.162:3300/0 conn(0x7f7958161df0 0x7f79580cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795e240640 1 -- 172.21.15.99:0/567117871 >> v1:172.21.15.99:6789/0 conn(0x7f79580cf660 legacy=0x7f79581616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795d23e640 1 -- 172.21.15.99:0/567117871 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455843665 0 0) 0x7f7958162450 con 0x7f79580cf660 2024-03-04T10:03:38.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795e240640 1 -- 172.21.15.99:0/567117871 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79580cc2d0 con 0x7f795815da20 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795e240640 1 --2- 172.21.15.99:0/567117871 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f795815da20 0x7f79580ccdb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79580d2890 tx=0x7f795402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2416ae9689e62352 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795d23e640 1 -- 172.21.15.99:0/567117871 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954039070 con 0x7f795815da20 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795d23e640 1 -- 172.21.15.99:0/567117871 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795402cc80 con 0x7f795815da20 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.641+0000 7f795d23e640 1 -- 172.21.15.99:0/567117871 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954034660 con 0x7f795815da20 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 -- 172.21.15.99:0/567117871 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f795815da20 msgr2=0x7f79580ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 --2- 172.21.15.99:0/567117871 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f795815da20 0x7f79580ccdb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79580d2890 tx=0x7f795402bed0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 -- 172.21.15.99:0/567117871 shutdown_connections 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 --2- 172.21.15.99:0/567117871 >> v2:172.21.15.162:3300/0 conn(0x7f7958161df0 0x7f79580cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 --2- 172.21.15.99:0/567117871 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f795815da20 0x7f79580ccdb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 -- 172.21.15.99:0/567117871 >> 172.21.15.99:0/567117871 conn(0x7f79580b33a0 msgr2=0x7f79580b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:38.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 -- 172.21.15.99:0/567117871 shutdown_connections 2024-03-04T10:03:38.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.642+0000 7f79604a4640 1 -- 172.21.15.99:0/567117871 wait complete. 2024-03-04T10:03:38.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.643+0000 7f79604a4640 1 Processor -- start 2024-03-04T10:03:38.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.643+0000 7f79604a4640 1 -- start start 2024-03-04T10:03:38.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f79604a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f79580cf660 0x7f79580c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f79604a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f795815da20 0x7f79581602f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f79604a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 0x7f79581598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f79604a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580cbf80 con 0x7f7958161df0 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f79604a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580cd2f0 con 0x7f795815da20 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f79604a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795815de00 con 0x7f79580cf660 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 0x7f79581598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 0x7f79581598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56636/0 (socket says 172.21.15.99:56636) 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 -- 172.21.15.99:0/2278569881 learned_addr learned my addr 172.21.15.99:0/2278569881 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795da3f640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f795815da20 0x7f79581602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795e240640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f79580cf660 0x7f79580c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 -- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f79580cf660 msgr2=0x7f79580c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f79580cf660 0x7f79580c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 -- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f795815da20 msgr2=0x7f79581602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f795815da20 0x7f79581602f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 -- 172.21.15.99:0/2278569881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79580b43a0 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f795ea41640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 0x7f79581598c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f795000c580 tx=0x7f795000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.644+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950015070 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.645+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7950004220 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.645+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79580b4640 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.645+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7950003610 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.645+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79580b4c00 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.645+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958074f60 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.646+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f79500037b0 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.646+0000 7f794effd640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f793c06b4d0 0x7f793c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.646+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7950089f20 con 0x7f7958161df0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.646+0000 7f795e240640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f793c06b4d0 0x7f793c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:38.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.647+0000 7f795e240640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f793c06b4d0 0x7f793c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7954036000 tx=0x7f7954030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:38.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.651+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7950058d80 con 0x7f7958161df0 2024-03-04T10:03:38.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:38.858+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f79580cfde0 con 0x7f7958161df0 2024-03-04T10:03:38.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:38.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:03:39.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.539+0000 7f794effd640 1 -- 172.21.15.99:0/2278569881 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7950058680 con 0x7f7958161df0 2024-03-04T10:03:39.541 INFO:teuthology.orchestra.run.smithi099.stderr:new fs with metadata pool 2 and data pool 3 2024-03-04T10:03:39.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.545+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f793c06b4d0 msgr2=0x7f793c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:39.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.545+0000 7f79604a4640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f793c06b4d0 0x7f793c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7954036000 tx=0x7f7954030000 comp rx=0 tx=0).stop 2024-03-04T10:03:39.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.545+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 msgr2=0x7f79581598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:39.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.545+0000 7f79604a4640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 0x7f79581598c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f795000c580 tx=0x7f795000ca40 comp rx=0 tx=0).stop 2024-03-04T10:03:39.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.548+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 shutdown_connections 2024-03-04T10:03:39.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.548+0000 7f79604a4640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f793c06b4d0 0x7f793c06d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.548+0000 7f79604a4640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7958161df0 0x7f79581598c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.548+0000 7f79604a4640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f795815da20 0x7f79581602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.548+0000 7f79604a4640 1 --2- 172.21.15.99:0/2278569881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f79580cf660 0x7f79580c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.548+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 >> 172.21.15.99:0/2278569881 conn(0x7f79580b33a0 msgr2=0x7f79580b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:39.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.551+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 shutdown_connections 2024-03-04T10:03:39.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.551+0000 7f79604a4640 1 -- 172.21.15.99:0/2278569881 wait complete. 2024-03-04T10:03:39.570 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-04T10:03:39.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-04T10:03:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.674+0000 7f64a92d7640 1 Processor -- start 2024-03-04T10:03:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.674+0000 7f64a92d7640 1 -- start start 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a92d7640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f64a415dcd0 0x7f64a40b6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a92d7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64a40b5be0 0x7f64a40d2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a92d7640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a415e0b0 con 0x7f64a40b93e0 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a92d7640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a40d1e20 con 0x7f64a40b5be0 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a92d7640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a40b9870 con 0x7f64a415dcd0 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64a40b5be0 0x7f64a40d2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a27fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f64a40b93e0 0x7f64a40b54d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51038/0 (socket says 172.21.15.99:51038) 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64a40b5be0 0x7f64a40d2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56640/0 (socket says 172.21.15.99:56640) 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a27fc640 1 -- 172.21.15.99:0/4228439808 learned_addr learned my addr 172.21.15.99:0/4228439808 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a37fe640 1 -- 172.21.15.99:0/4228439808 >> v2:172.21.15.162:3300/0 conn(0x7f64a415dcd0 msgr2=0x7f64a40b6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a37fe640 1 --2- 172.21.15.99:0/4228439808 >> v2:172.21.15.162:3300/0 conn(0x7f64a415dcd0 0x7f64a40b6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a37fe640 1 -- 172.21.15.99:0/4228439808 >> v1:172.21.15.99:6789/0 conn(0x7f64a40b93e0 legacy=0x7f64a40b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a1ffb640 1 -- 172.21.15.99:0/4228439808 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497946929 0 0) 0x7f64a415e0b0 con 0x7f64a40b93e0 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.675+0000 7f64a37fe640 1 -- 172.21.15.99:0/4228439808 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40d2070 con 0x7f64a40b5be0 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.676+0000 7f64a37fe640 1 --2- 172.21.15.99:0/4228439808 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a40d2b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f649c0097a0 tx=0x7f649c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c699e5982a6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.676+0000 7f64a1ffb640 1 -- 172.21.15.99:0/4228439808 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c035020 con 0x7f64a40b5be0 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.676+0000 7f64a1ffb640 1 -- 172.21.15.99:0/4228439808 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c031830 con 0x7f64a40b5be0 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.676+0000 7f64a1ffb640 1 -- 172.21.15.99:0/4228439808 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c031c00 con 0x7f64a40b5be0 2024-03-04T10:03:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 -- 172.21.15.99:0/4228439808 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 msgr2=0x7f64a40d2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 --2- 172.21.15.99:0/4228439808 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a40d2b50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f649c0097a0 tx=0x7f649c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 -- 172.21.15.99:0/4228439808 shutdown_connections 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 --2- 172.21.15.99:0/4228439808 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a40d2b50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 --2- 172.21.15.99:0/4228439808 >> v2:172.21.15.162:3300/0 conn(0x7f64a415dcd0 0x7f64a40b6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 -- 172.21.15.99:0/4228439808 >> 172.21.15.99:0/4228439808 conn(0x7f64a40d04d0 msgr2=0x7f64a40d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 -- 172.21.15.99:0/4228439808 shutdown_connections 2024-03-04T10:03:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.677+0000 7f64a92d7640 1 -- 172.21.15.99:0/4228439808 wait complete. 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.678+0000 7f64a92d7640 1 Processor -- start 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.678+0000 7f64a92d7640 1 -- start start 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a92d7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a4148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a92d7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64a415dcd0 0x7f64a40c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a92d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64a41f4300 0x7f64a415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a92d7640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a40b7130 con 0x7f64a40b5be0 2024-03-04T10:03:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a92d7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a40d2630 con 0x7f64a41f4300 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a92d7640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a40ce0a0 con 0x7f64a415dcd0 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a4148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a4148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56654/0 (socket says 172.21.15.99:56654) 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 -- 172.21.15.99:0/2565138930 learned_addr learned my addr 172.21.15.99:0/2565138930 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a37fe640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64a41f4300 0x7f64a415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a27fc640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64a415dcd0 0x7f64a40c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 -- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64a415dcd0 msgr2=0x7f64a40c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64a415dcd0 0x7f64a40c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 -- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64a41f4300 msgr2=0x7f64a415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64a41f4300 0x7f64a415bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 -- 172.21.15.99:0/2565138930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40c8730 con 0x7f64a40b5be0 2024-03-04T10:03:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a27fc640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64a415dcd0 0x7f64a40c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f64a2ffd640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a4148eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f649400ba50 tx=0x7f649400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494019070 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.679+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649400cc00 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.680+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a40c89b0 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.680+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6494014440 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.680+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a40c8f70 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.681+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a4074f60 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.681+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f649401c070 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.681+0000 7f648b7fe640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.682+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6494089ed0 con 0x7f64a40b5be0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.682+0000 7f64a27fc640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.682+0000 7f64a27fc640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6498000f60 tx=0x7f6498009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:39.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.686+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f64a40c8730 con 0x7f64a40b5be0 2024-03-04T10:03:39.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:39.866+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f64a40b41c0 con 0x7f64a40b5be0 2024-03-04T10:03:40.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.531+0000 7f648b7fe640 1 -- 172.21.15.99:0/2565138930 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f6494058de0 con 0x7f64a40b5be0 2024-03-04T10:03:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 msgr2=0x7f649006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6498000f60 tx=0x7f6498009040 comp rx=0 tx=0).stop 2024-03-04T10:03:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 msgr2=0x7f64a4148eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a4148eb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f649400ba50 tx=0x7f649400bf10 comp rx=0 tx=0).stop 2024-03-04T10:03:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 shutdown_connections 2024-03-04T10:03:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64a41f4300 0x7f64a415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64a415dcd0 0x7f64a40c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 --2- 172.21.15.99:0/2565138930 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64a40b5be0 0x7f64a4148eb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 >> 172.21.15.99:0/2565138930 conn(0x7f64a40d04d0 msgr2=0x7f64a415cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 shutdown_connections 2024-03-04T10:03:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.534+0000 7f64a92d7640 1 -- 172.21.15.99:0/2565138930 wait complete. 2024-03-04T10:03:40.548 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T10:03:40.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.659+0000 7fca636a4640 1 Processor -- start 2024-03-04T10:03:40.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.659+0000 7fca636a4640 1 -- start start 2024-03-04T10:03:40.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca636a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fca5c0b5700 0x7fca5c0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:40.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca636a4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fca5c0b9410 0x7fca5c06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:40.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca636a4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5c0d2400 con 0x7fca5c0d34a0 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca636a4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5c0d26e0 con 0x7fca5c0b9410 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca636a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5c0d29c0 con 0x7fca5c0b5700 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca61c41640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fca5c0b9410 0x7fca5c06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca60c3f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fca5c0d34a0 0x7fca5c06c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51050/0 (socket says 172.21.15.99:51050) 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca60c3f640 1 -- 172.21.15.99:0/313075222 learned_addr learned my addr 172.21.15.99:0/313075222 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.660+0000 7fca61440640 1 --2- 172.21.15.99:0/313075222 >> v2:172.21.15.162:3300/0 conn(0x7fca5c0b5700 0x7fca5c0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca61c41640 1 -- 172.21.15.99:0/313075222 >> v2:172.21.15.162:3300/0 conn(0x7fca5c0b5700 msgr2=0x7fca5c0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca61c41640 1 --2- 172.21.15.99:0/313075222 >> v2:172.21.15.162:3300/0 conn(0x7fca5c0b5700 0x7fca5c0d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca61c41640 1 -- 172.21.15.99:0/313075222 >> v1:172.21.15.99:6789/0 conn(0x7fca5c0d34a0 legacy=0x7fca5c06c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:40.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca61c41640 1 -- 172.21.15.99:0/313075222 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c0d2ca0 con 0x7fca5c0b9410 2024-03-04T10:03:40.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca53fff640 1 -- 172.21.15.99:0/313075222 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775924653 0 0) 0x7fca5c0d2400 con 0x7fca5c0d34a0 2024-03-04T10:03:40.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca61c41640 1 --2- 172.21.15.99:0/313075222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0b9410 0x7fca5c06c7f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fca5c06dad0 tx=0x7fca5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66290cee89b353b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:40.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca53fff640 1 -- 172.21.15.99:0/313075222 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca540315a0 con 0x7fca5c0b9410 2024-03-04T10:03:40.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca53fff640 1 -- 172.21.15.99:0/313075222 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54037070 con 0x7fca5c0b9410 2024-03-04T10:03:40.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.661+0000 7fca53fff640 1 -- 172.21.15.99:0/313075222 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54031960 con 0x7fca5c0b9410 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 -- 172.21.15.99:0/313075222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0b9410 msgr2=0x7fca5c06c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 --2- 172.21.15.99:0/313075222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0b9410 0x7fca5c06c7f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fca5c06dad0 tx=0x7fca5402b840 comp rx=0 tx=0).stop 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 -- 172.21.15.99:0/313075222 shutdown_connections 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 --2- 172.21.15.99:0/313075222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0b9410 0x7fca5c06c7f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 --2- 172.21.15.99:0/313075222 >> v2:172.21.15.162:3300/0 conn(0x7fca5c0b5700 0x7fca5c0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 -- 172.21.15.99:0/313075222 >> 172.21.15.99:0/313075222 conn(0x7fca5c0d0490 msgr2=0x7fca5c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 -- 172.21.15.99:0/313075222 shutdown_connections 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.662+0000 7fca636a4640 1 -- 172.21.15.99:0/313075222 wait complete. 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.663+0000 7fca636a4640 1 Processor -- start 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.663+0000 7fca636a4640 1 -- start start 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca636a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca5c0b5700 0x7fca5c15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca636a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fca5c0b9410 0x7fca5c0ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca636a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 0x7fca5c0ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca636a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c06ceb0 con 0x7fca5c0d34a0 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca636a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c06cd30 con 0x7fca5c0b5700 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca636a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c0b5ae0 con 0x7fca5c0b9410 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 0x7fca5c0ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 0x7fca5c0ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56680/0 (socket says 172.21.15.99:56680) 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 -- 172.21.15.99:0/3582862279 learned_addr learned my addr 172.21.15.99:0/3582862279 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61440640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca5c0b5700 0x7fca5c15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca60c3f640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fca5c0b9410 0x7fca5c0ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 -- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fca5c0b9410 msgr2=0x7fca5c0ac770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fca5c0b9410 0x7fca5c0ac770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 -- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca5c0b5700 msgr2=0x7fca5c15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca5c0b5700 0x7fca5c15c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 -- 172.21.15.99:0/3582862279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c0cab80 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61440640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca5c0b5700 0x7fca5c15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca60c3f640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fca5c0b9410 0x7fca5c0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.664+0000 7fca61c41640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 0x7fca5c0ccf40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca54009890 tx=0x7fca540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.665+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54032840 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.665+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54032a00 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.665+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54032df0 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.665+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c0cae60 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.665+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c0cb420 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.666+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5c074f60 con 0x7fca5c0d34a0 2024-03-04T10:03:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.666+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fca5404c020 con 0x7fca5c0d34a0 2024-03-04T10:03:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.667+0000 7fca51ffb640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fca4006b4d0 0x7fca4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.667+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7fca5402f020 con 0x7fca5c0d34a0 2024-03-04T10:03:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.667+0000 7fca61440640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fca4006b4d0 0x7fca4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.667+0000 7fca61440640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fca4006b4d0 0x7fca4006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca58009b20 tx=0x7fca580039d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.671+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fca54040070 con 0x7fca5c0d34a0 2024-03-04T10:03:40.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:40.842+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fca5c0cb5f0 con 0x7fca5c0d34a0 2024-03-04T10:03:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.546+0000 7fca51ffb640 1 -- 172.21.15.99:0/3582862279 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7fca5407a7d0 con 0x7fca5c0d34a0 2024-03-04T10:03:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data_ec' created 2024-03-04T10:03:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fca4006b4d0 msgr2=0x7fca4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fca4006b4d0 0x7fca4006d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca58009b20 tx=0x7fca580039d0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 msgr2=0x7fca5c0ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 0x7fca5c0ccf40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca54009890 tx=0x7fca540046c0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 shutdown_connections 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fca4006b4d0 0x7fca4006d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fca5c0d34a0 0x7fca5c0ccf40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fca5c0b9410 0x7fca5c0ac770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 --2- 172.21.15.99:0/3582862279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca5c0b5700 0x7fca5c15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.549+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 >> 172.21.15.99:0/3582862279 conn(0x7fca5c0d0490 msgr2=0x7fca5c0b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.550+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 shutdown_connections 2024-03-04T10:03:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.550+0000 7fca636a4640 1 -- 172.21.15.99:0/3582862279 wait complete. 2024-03-04T10:03:42.571 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-04T10:03:42.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.679+0000 7f327e7df640 1 Processor -- start 2024-03-04T10:03:42.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.679+0000 7f327e7df640 1 -- start start 2024-03-04T10:03:42.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327e7df640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f327815da90 0x7f32780b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327e7df640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f32780d0820 0x7f32780d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327e7df640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780b7910 con 0x7f327815de70 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327e7df640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780b7bf0 con 0x7f32780d0820 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327e7df640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32780b7ed0 con 0x7f327815da90 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327cd7c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f32780d0820 0x7f32780d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f32777fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f327815de70 0x7f32780cea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51066/0 (socket says 172.21.15.99:51066) 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f32777fe640 1 -- 172.21.15.99:0/1433511440 learned_addr learned my addr 172.21.15.99:0/1433511440 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f3277fff640 1 --2- 172.21.15.99:0/1433511440 >> v2:172.21.15.162:3300/0 conn(0x7f327815da90 0x7f32780b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327cd7c640 1 -- 172.21.15.99:0/1433511440 >> v2:172.21.15.162:3300/0 conn(0x7f327815da90 msgr2=0x7f32780b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327cd7c640 1 --2- 172.21.15.99:0/1433511440 >> v2:172.21.15.162:3300/0 conn(0x7f327815da90 0x7f32780b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327cd7c640 1 -- 172.21.15.99:0/1433511440 >> v1:172.21.15.99:6789/0 conn(0x7f327815de70 legacy=0x7f32780cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f3276ffd640 1 -- 172.21.15.99:0/1433511440 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049598952 0 0) 0x7f32780b7910 con 0x7f327815de70 2024-03-04T10:03:42.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.680+0000 7f327cd7c640 1 -- 172.21.15.99:0/1433511440 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780b81b0 con 0x7f32780d0820 2024-03-04T10:03:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.681+0000 7f327cd7c640 1 --2- 172.21.15.99:0/1433511440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32780d0cb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3278073380 tx=0x7f327002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07a980c1f41cfdd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.681+0000 7f3276ffd640 1 -- 172.21.15.99:0/1433511440 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270039070 con 0x7f32780d0820 2024-03-04T10:03:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.681+0000 7f3276ffd640 1 -- 172.21.15.99:0/1433511440 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327002cc80 con 0x7f32780d0820 2024-03-04T10:03:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.681+0000 7f3276ffd640 1 -- 172.21.15.99:0/1433511440 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270034660 con 0x7f32780d0820 2024-03-04T10:03:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 -- 172.21.15.99:0/1433511440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 msgr2=0x7f32780d0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 --2- 172.21.15.99:0/1433511440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32780d0cb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3278073380 tx=0x7f327002bed0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 -- 172.21.15.99:0/1433511440 shutdown_connections 2024-03-04T10:03:42.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 --2- 172.21.15.99:0/1433511440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32780d0cb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 --2- 172.21.15.99:0/1433511440 >> v2:172.21.15.162:3300/0 conn(0x7f327815da90 0x7f32780b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 -- 172.21.15.99:0/1433511440 >> 172.21.15.99:0/1433511440 conn(0x7f32780725e0 msgr2=0x7f32780729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:42.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 -- 172.21.15.99:0/1433511440 shutdown_connections 2024-03-04T10:03:42.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.682+0000 7f327e7df640 1 -- 172.21.15.99:0/1433511440 wait complete. 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.683+0000 7f327e7df640 1 Processor -- start 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.683+0000 7f327e7df640 1 -- start start 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327e7df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32781e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327e7df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f327815da90 0x7f32780d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327e7df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f327815de70 0x7f327814b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327e7df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780ce490 con 0x7f32780d0820 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327e7df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32780cf1a0 con 0x7f327815de70 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327e7df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32780b8ef0 con 0x7f327815da90 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32781e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32781e7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56694/0 (socket says 172.21.15.99:56694) 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 -- 172.21.15.99:0/3727732513 learned_addr learned my addr 172.21.15.99:0/3727732513 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f327cd7c640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f327815de70 0x7f327814b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f32777fe640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f327815da90 0x7f32780d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 -- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f327815da90 msgr2=0x7f32780d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f327815da90 0x7f32780d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 -- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f327815de70 msgr2=0x7f327814b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f327815de70 0x7f327814b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 -- 172.21.15.99:0/3727732513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780c8aa0 con 0x7f32780d0820 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f32777fe640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f327815da90 0x7f32780d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:42.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.684+0000 7f3277fff640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32781e7680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f326800e960 tx=0x7f326800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.685+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3268017070 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.685+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326800ca00 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.685+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326800cdf0 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.685+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32780c8d80 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.685+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32780c9340 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.686+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3278074f60 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.686+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f326801a070 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.687+0000 7f3274ff9640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326406b4d0 0x7f326406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.687+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f326808a800 con 0x7f32780d0820 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.687+0000 7f32777fe640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326406b4d0 0x7f326406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:42.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.687+0000 7f32777fe640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326406b4d0 0x7f326406d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32781554f0 tx=0x7f326c007ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:42.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.691+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3268058da0 con 0x7f32780d0820 2024-03-04T10:03:42.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:42.862+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f32780cab90 con 0x7f32780d0820 2024-03-04T10:03:43.044 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:03:43.043+0000 7fbe61960640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:03:43.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.552+0000 7f3274ff9640 1 -- 172.21.15.99:0/3727732513 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f32780cab90 con 0x7f32780d0820 2024-03-04T10:03:43.553 INFO:teuthology.orchestra.run.smithi099.stderr:set pool 4 allow_ec_overwrites to true 2024-03-04T10:03:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.556+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326406b4d0 msgr2=0x7f326406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.556+0000 7f327e7df640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326406b4d0 0x7f326406d990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32781554f0 tx=0x7f326c007ca0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 msgr2=0x7f32781e7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32781e7680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f326800e960 tx=0x7f326800ee20 comp rx=0 tx=0).stop 2024-03-04T10:03:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 shutdown_connections 2024-03-04T10:03:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326406b4d0 0x7f326406d990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f327815de70 0x7f327814b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f327815da90 0x7f32780d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 --2- 172.21.15.99:0/3727732513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32780d0820 0x7f32781e7680 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f326800e960 tx=0x7f326800ee20 comp rx=0 tx=0).stop 2024-03-04T10:03:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.557+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 >> 172.21.15.99:0/3727732513 conn(0x7f32780725e0 msgr2=0x7f327815ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:43.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.559+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 shutdown_connections 2024-03-04T10:03:43.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.562+0000 7f327e7df640 1 -- 172.21.15.99:0/3727732513 wait complete. 2024-03-04T10:03:43.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-04T10:03:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.694+0000 7fb850e8e640 1 Processor -- start 2024-03-04T10:03:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.694+0000 7fb850e8e640 1 -- start start 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.695+0000 7fb850e8e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 0x7fb84c15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.695+0000 7fb850e8e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c070340 con 0x7fb84c06ff60 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.695+0000 7fb850e8e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c0cb440 con 0x7fb84c0b2d70 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.695+0000 7fb850e8e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0cc050 con 0x7fb84c0cd820 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.695+0000 7fb84a59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb84c06ff60 0x7fb84c0cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45332/0 (socket says 172.21.15.99:45332) 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.695+0000 7fb84a59c640 1 -- 172.21.15.99:0/177915100 learned_addr learned my addr 172.21.15.99:0/177915100 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84ad9d640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 0x7fb84c15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325705711 0 0) 0x7fb84c070340 con 0x7fb84c06ff60 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834003520 con 0x7fb84c06ff60 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3162172295 0 0) 0x7fb834003520 con 0x7fb84c06ff60 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 msgr2=0x7fb84c15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 0x7fb84c15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 >> v1:172.21.15.162:6789/0 conn(0x7fb84c0b2d70 legacy=0x7fb84c0cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c0cb720 con 0x7fb84c06ff60 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fb83c0019f0 con 0x7fb84c06ff60 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 >> v1:172.21.15.99:6789/0 conn(0x7fb84c06ff60 legacy=0x7fb84c0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 0x7fb8340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb834004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb834004d40 0x7fb8340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0cc050 con 0x7fb834003f20 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83400ba20 con 0x7fb834004d40 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83400bc20 con 0x7fb84c0cd820 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb849d9b640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb834004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84a59c640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 0x7fb8340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb84ad9d640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb834004d40 0x7fb8340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb849d9b640 1 -- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 msgr2=0x7fb8340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb849d9b640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0cd820 0x7fb8340039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb849d9b640 1 -- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb834004d40 msgr2=0x7fb8340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb849d9b640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb834004d40 0x7fb8340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.696+0000 7fb849d9b640 1 -- 172.21.15.99:0/177915100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834003520 con 0x7fb834003f20 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.697+0000 7fb849d9b640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb834004800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb840001860 tx=0x7fb834005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767cd5a6a16bb769 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.697+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840001430 con 0x7fb834003f20 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.697+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb840032620 con 0x7fb834003f20 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.697+0000 7fb84959a640 1 -- 172.21.15.99:0/177915100 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840032a10 con 0x7fb834003f20 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb850e8e640 1 -- 172.21.15.99:0/177915100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 msgr2=0x7fb834004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb850e8e640 1 --2- 172.21.15.99:0/177915100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb834004800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb840001860 tx=0x7fb834005830 comp rx=0 tx=0).stop 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb84ad9d640 1 -- 172.21.15.99:0/177915100 reap_dead start 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb850e8e640 1 -- 172.21.15.99:0/177915100 shutdown_connections 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb850e8e640 1 -- 172.21.15.99:0/177915100 >> 172.21.15.99:0/177915100 conn(0x7fb84c0b2250 msgr2=0x7fb84c0b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb850e8e640 1 -- 172.21.15.99:0/177915100 shutdown_connections 2024-03-04T10:03:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.698+0000 7fb850e8e640 1 -- 172.21.15.99:0/177915100 wait complete. 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.699+0000 7fb850e8e640 1 Processor -- start 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.699+0000 7fb850e8e640 1 -- start start 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb84c0c4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0b5fe0 0x7fb84c151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb84c0c8dc0 0x7fb84c0c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0cbce0 con 0x7fb834003f20 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c06f530 con 0x7fb84c0c8dc0 2024-03-04T10:03:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c0cc4e0 con 0x7fb84c0b5fe0 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb84c0c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb84c0c4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38820/0 (socket says 172.21.15.99:38820) 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 -- 172.21.15.99:0/3609565561 learned_addr learned my addr 172.21.15.99:0/3609565561 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb849d9b640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0b5fe0 0x7fb84c151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84ad9d640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb84c0c8dc0 0x7fb84c0c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 -- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0b5fe0 msgr2=0x7fb84c151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0b5fe0 0x7fb84c151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 -- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb84c0c8dc0 msgr2=0x7fb84c0c9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb84c0c8dc0 0x7fb84c0c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 -- 172.21.15.99:0/3609565561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c1ef990 con 0x7fb834003f20 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb849d9b640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0b5fe0 0x7fb84c151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb84a59c640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb84c0c4880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0077f0 tx=0x7fb83c00a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0133f0 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb83c018070 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c0b3340 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c0137b0 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.700+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c1efc30 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.701+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c074f60 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.702+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb83c013950 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.702+0000 7fb83affd640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb82c06b500 0x7fb82c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.702+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7fb83c08ade0 con 0x7fb834003f20 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.702+0000 7fb849d9b640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb82c06b500 0x7fb82c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.703+0000 7fb849d9b640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb82c06b500 0x7fb82c06d9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb840035000 tx=0x7fb840030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:43.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.706+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb83c0592f0 con 0x7fb834003f20 2024-03-04T10:03:43.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:43.915+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fb84c0ccef0 con 0x7fb834003f20 2024-03-04T10:03:44.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.553+0000 7fb83affd640 1 -- 172.21.15.99:0/3609565561 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb83c05d390 con 0x7fb834003f20 2024-03-04T10:03:44.554 INFO:teuthology.orchestra.run.smithi099.stderr:added data pool 4 to fsmap 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb82c06b500 msgr2=0x7fb82c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb82c06b500 0x7fb82c06d9c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb840035000 tx=0x7fb840030000 comp rx=0 tx=0).stop 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 msgr2=0x7fb84c0c4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb84c0c4880 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0077f0 tx=0x7fb83c00a750 comp rx=0 tx=0).stop 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 shutdown_connections 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb82c06b500 0x7fb82c06d9c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb84c0c8dc0 0x7fb84c0c9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb84c0b5fe0 0x7fb84c151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 --2- 172.21.15.99:0/3609565561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb834003f20 0x7fb84c0c4880 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 >> 172.21.15.99:0/3609565561 conn(0x7fb84c0b2250 msgr2=0x7fb84c06acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:44.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 shutdown_connections 2024-03-04T10:03:44.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.556+0000 7fb850e8e640 1 -- 172.21.15.99:0/3609565561 wait complete. 2024-03-04T10:03:44.569 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:03:44.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.670+0000 7f4960d44640 1 Processor -- start 2024-03-04T10:03:44.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.670+0000 7f4960d44640 1 -- start start 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f4960d44640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f495c15de00 0x7f495c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f4960d44640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f495c0b9380 0x7f495c0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f4960d44640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495c15e1e0 con 0x7f495c0b70d0 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f4960d44640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c0d2820 con 0x7f495c15de00 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f4960d44640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495c0d1c10 con 0x7f495c0b9380 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495a59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f495c15de00 0x7f495c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495ad9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f495c0b70d0 0x7f495c0b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45344/0 (socket says 172.21.15.99:45344) 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495ad9d640 1 -- 172.21.15.99:0/2144052702 learned_addr learned my addr 172.21.15.99:0/2144052702 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495a59c640 1 -- 172.21.15.99:0/2144052702 >> v2:172.21.15.162:3300/0 conn(0x7f495c0b9380 msgr2=0x7f495c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495a59c640 1 --2- 172.21.15.99:0/2144052702 >> v2:172.21.15.162:3300/0 conn(0x7f495c0b9380 0x7f495c0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495a59c640 1 -- 172.21.15.99:0/2144052702 >> v1:172.21.15.99:6789/0 conn(0x7f495c0b70d0 legacy=0x7f495c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f495a59c640 1 -- 172.21.15.99:0/2144052702 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c0d1ef0 con 0x7f495c15de00 2024-03-04T10:03:44.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.671+0000 7f4959d9b640 1 -- 172.21.15.99:0/2144052702 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586659583 0 0) 0x7f495c15e1e0 con 0x7f495c0b70d0 2024-03-04T10:03:44.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.672+0000 7f495a59c640 1 --2- 172.21.15.99:0/2144052702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c15de00 0x7f495c0b2e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f495c161830 tx=0x7f494c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8669161ca0efd43 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:44.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.672+0000 7f4959d9b640 1 -- 172.21.15.99:0/2144052702 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c035020 con 0x7f495c15de00 2024-03-04T10:03:44.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.672+0000 7f4959d9b640 1 -- 172.21.15.99:0/2144052702 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c031830 con 0x7f495c15de00 2024-03-04T10:03:44.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.672+0000 7f4959d9b640 1 -- 172.21.15.99:0/2144052702 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c031c00 con 0x7f495c15de00 2024-03-04T10:03:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.672+0000 7f4960d44640 1 -- 172.21.15.99:0/2144052702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c15de00 msgr2=0x7f495c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.672+0000 7f4960d44640 1 --2- 172.21.15.99:0/2144052702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c15de00 0x7f495c0b2e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f495c161830 tx=0x7f494c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.673+0000 7f4960d44640 1 -- 172.21.15.99:0/2144052702 shutdown_connections 2024-03-04T10:03:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.673+0000 7f4960d44640 1 --2- 172.21.15.99:0/2144052702 >> v2:172.21.15.162:3300/0 conn(0x7f495c0b9380 0x7f495c0b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.673+0000 7f4960d44640 1 --2- 172.21.15.99:0/2144052702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c15de00 0x7f495c0b2e60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.673+0000 7f4960d44640 1 -- 172.21.15.99:0/2144052702 >> 172.21.15.99:0/2144052702 conn(0x7f495c0d0470 msgr2=0x7f495c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.673+0000 7f4960d44640 1 -- 172.21.15.99:0/2144052702 shutdown_connections 2024-03-04T10:03:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.673+0000 7f4960d44640 1 -- 172.21.15.99:0/2144052702 wait complete. 2024-03-04T10:03:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.674+0000 7f4960d44640 1 Processor -- start 2024-03-04T10:03:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.674+0000 7f4960d44640 1 -- start start 2024-03-04T10:03:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f495c0b70d0 0x7f495c1f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 0x7f495c0ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f495c15de00 0x7f495c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0b97e0 con 0x7f495c0b9380 2024-03-04T10:03:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0d24b0 con 0x7f495c0b70d0 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495c0b5b80 con 0x7f495c15de00 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 0x7f495c0ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 0x7f495c0ce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38840/0 (socket says 172.21.15.99:38840) 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 -- 172.21.15.99:0/2500121142 learned_addr learned my addr 172.21.15.99:0/2500121142 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 -- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f495c15de00 msgr2=0x7f495c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f495c15de00 0x7f495c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 -- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f495c0b70d0 msgr2=0x7f495c1f3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f495a59c640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f495c0b70d0 0x7f495c1f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f495c0b70d0 0x7f495c1f3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 -- 172.21.15.99:0/2500121142 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c1f4590 con 0x7f495c0b9380 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4953fff640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 0x7f495c0ce500 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f494400b4a0 tx=0x7f494400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944014070 con 0x7f495c0b9380 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4944010610 con 0x7f495c0b9380 2024-03-04T10:03:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c1f4870 con 0x7f495c0b9380 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944010a00 con 0x7f495c0b9380 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.675+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c1f4e30 con 0x7f495c0b9380 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.676+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c074f60 con 0x7f495c0b9380 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.676+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4944017070 con 0x7f495c0b9380 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.677+0000 7f4952ffd640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f493c06b4d0 0x7f493c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.677+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4944089eb0 con 0x7f495c0b9380 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.677+0000 7f495a59c640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f493c06b4d0 0x7f493c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.678+0000 7f495a59c640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f493c06b4d0 0x7f493c06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f494c036000 tx=0x7f494c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:44.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.681+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4944058a80 con 0x7f495c0b9380 2024-03-04T10:03:44.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.888+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f495c0b2820 con 0x7f495c0b9380 2024-03-04T10:03:44.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.889+0000 7f4952ffd640 1 -- 172.21.15.99:0/2500121142 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f4944058380 con 0x7f495c0b9380 2024-03-04T10:03:44.890 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:44.890 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:03:43.916275+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T10:03:44.890 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 6 2024-03-04T10:03:44.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f493c06b4d0 msgr2=0x7f493c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f493c06b4d0 0x7f493c06d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f494c036000 tx=0x7f494c03c040 comp rx=0 tx=0).stop 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 msgr2=0x7f495c0ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 0x7f495c0ce500 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f494400b4a0 tx=0x7f494400b960 comp rx=0 tx=0).stop 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 shutdown_connections 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f493c06b4d0 0x7f493c06d990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f495c15de00 0x7f495c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f495c0b9380 0x7f495c0ce500 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 --2- 172.21.15.99:0/2500121142 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f495c0b70d0 0x7f495c1f3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 >> 172.21.15.99:0/2500121142 conn(0x7f495c0d0470 msgr2=0x7f495c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 shutdown_connections 2024-03-04T10:03:44.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:44.891+0000 7f4960d44640 1 -- 172.21.15.99:0/2500121142 wait complete. 2024-03-04T10:03:44.905 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:45.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.004+0000 7f3279ff1640 1 Processor -- start 2024-03-04T10:03:45.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.004+0000 7f3279ff1640 1 -- start start 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3279ff1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32740cf650 0x7f32740cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3279ff1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3274070de0 0x7f3274071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3279ff1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327415f020 con 0x7f3274154a80 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3279ff1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32740cfff0 con 0x7f3274070de0 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3279ff1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32740cd010 con 0x7f32740cf650 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3274070de0 0x7f3274071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3274070de0 0x7f3274071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38842/0 (socket says 172.21.15.99:38842) 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 -- 172.21.15.99:0/2987734327 learned_addr learned my addr 172.21.15.99:0/2987734327 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3272ffd640 1 --2- 172.21.15.99:0/2987734327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32740cf650 0x7f32740cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 -- 172.21.15.99:0/2987734327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32740cf650 msgr2=0x7f32740cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 --2- 172.21.15.99:0/2987734327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32740cf650 0x7f32740cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 -- 172.21.15.99:0/2987734327 >> v1:172.21.15.162:6789/0 conn(0x7f3274154a80 legacy=0x7f327415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.005+0000 7f3273fff640 1 -- 172.21.15.99:0/2987734327 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32740cdaa0 con 0x7f3274070de0 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.006+0000 7f3273fff640 1 --2- 172.21.15.99:0/2987734327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274070de0 0x7f3274071270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f326c009e50 tx=0x7f326c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b9d3fb093413e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.006+0000 7f32727fc640 1 -- 172.21.15.99:0/2987734327 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c02ca70 con 0x7f3274070de0 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.006+0000 7f32727fc640 1 -- 172.21.15.99:0/2987734327 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c02cc10 con 0x7f3274070de0 2024-03-04T10:03:45.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.006+0000 7f32727fc640 1 -- 172.21.15.99:0/2987734327 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0343f0 con 0x7f3274070de0 2024-03-04T10:03:45.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.006+0000 7f3279ff1640 1 -- 172.21.15.99:0/2987734327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274070de0 msgr2=0x7f3274071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 --2- 172.21.15.99:0/2987734327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274070de0 0x7f3274071270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f326c009e50 tx=0x7f326c02bed0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 -- 172.21.15.99:0/2987734327 shutdown_connections 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 --2- 172.21.15.99:0/2987734327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274070de0 0x7f3274071270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 --2- 172.21.15.99:0/2987734327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32740cf650 0x7f32740cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 -- 172.21.15.99:0/2987734327 >> 172.21.15.99:0/2987734327 conn(0x7f327406f5f0 msgr2=0x7f327406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 -- 172.21.15.99:0/2987734327 shutdown_connections 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.007+0000 7f3279ff1640 1 -- 172.21.15.99:0/2987734327 wait complete. 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.008+0000 7f3279ff1640 1 Processor -- start 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.008+0000 7f3279ff1640 1 -- start start 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3274070de0 0x7f32741538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32740cf650 0x7f3274156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 0x7f3274156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32740ce340 con 0x7f3274154a80 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32740ce060 con 0x7f32740cf650 2024-03-04T10:03:45.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32740ce620 con 0x7f3274070de0 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 0x7f3274156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 0x7f3274156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38844/0 (socket says 172.21.15.99:38844) 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 -- 172.21.15.99:0/4121010212 learned_addr learned my addr 172.21.15.99:0/4121010212 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f32737fe640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3274070de0 0x7f32741538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3272ffd640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32740cf650 0x7f3274156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 -- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3274070de0 msgr2=0x7f32741538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3274070de0 0x7f32741538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 -- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32740cf650 msgr2=0x7f3274156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32740cf650 0x7f3274156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 -- 172.21.15.99:0/4121010212 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32741f7c30 con 0x7f3274154a80 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3272ffd640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32740cf650 0x7f3274156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3273fff640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 0x7f3274156ab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f326c009f70 tx=0x7f326c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c043020 con 0x7f3274154a80 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c03e450 con 0x7f3274154a80 2024-03-04T10:03:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327415f3a0 con 0x7f3274154a80 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c009510 con 0x7f3274154a80 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.009+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3274153e20 con 0x7f3274154a80 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.010+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3274074f60 con 0x7f3274154a80 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.011+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f326c046020 con 0x7f3274154a80 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.011+0000 7f325bfff640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326006b500 0x7f326006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.011+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f326c0acce0 con 0x7f3274154a80 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.011+0000 7f32737fe640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326006b500 0x7f326006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.012+0000 7f32737fe640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326006b500 0x7f326006d9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3264004f60 tx=0x7f3264007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.016+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f326c04d030 con 0x7f3274154a80 2024-03-04T10:03:45.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.188+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f327415e170 con 0x7f3274154a80 2024-03-04T10:03:45.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.189+0000 7f325bfff640 1 -- 172.21.15.99:0/4121010212 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f327415f020 con 0x7f3274154a80 2024-03-04T10:03:45.191 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:45.191 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326006b500 msgr2=0x7f326006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326006b500 0x7f326006d9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3264004f60 tx=0x7f3264007e30 comp rx=0 tx=0).stop 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 msgr2=0x7f3274156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 0x7f3274156ab0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f326c009f70 tx=0x7f326c004a40 comp rx=0 tx=0).stop 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 shutdown_connections 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f326006b500 0x7f326006d9c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3274154a80 0x7f3274156ab0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32740cf650 0x7f3274156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 --2- 172.21.15.99:0/4121010212 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3274070de0 0x7f32741538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 >> 172.21.15.99:0/4121010212 conn(0x7f327406f5f0 msgr2=0x7f32740b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 shutdown_connections 2024-03-04T10:03:45.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.192+0000 7f3279ff1640 1 -- 172.21.15.99:0/4121010212 wait complete. 2024-03-04T10:03:45.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:45.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.304+0000 7f3509ae8640 1 Processor -- start 2024-03-04T10:03:45.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.304+0000 7f3509ae8640 1 -- start start 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.305+0000 7f3509ae8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f35040cc8e0 0x7f35040cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.305+0000 7f3509ae8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040d1290 0x7f35040cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.305+0000 7f3509ae8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35040d0400 con 0x7f35040cfae0 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.305+0000 7f3509ae8640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350406ae50 con 0x7f35040cc8e0 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.305+0000 7f3509ae8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350406b130 con 0x7f35040d1290 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3503fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040d1290 0x7f35040cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f35037fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f35040cfae0 0x7f35040cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:50962/0 (socket says 172.21.15.99:50962) 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f35037fe640 1 -- 172.21.15.99:0/2097546906 learned_addr learned my addr 172.21.15.99:0/2097546906 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:45.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3502ffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f35040cc8e0 0x7f35040cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3502ffd640 1 -- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040d1290 msgr2=0x7f35040cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3502ffd640 1 --2- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040d1290 0x7f35040cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3502ffd640 1 -- 172.21.15.99:0/2097546906 >> v1:172.21.15.162:6789/0 conn(0x7f35040cfae0 legacy=0x7f35040cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3502ffd640 1 -- 172.21.15.99:0/2097546906 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350406b410 con 0x7f35040cc8e0 2024-03-04T10:03:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f35027fc640 1 -- 172.21.15.99:0/2097546906 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288030789 0 0) 0x7f35040d0400 con 0x7f35040cfae0 2024-03-04T10:03:45.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.306+0000 7f3503fff640 1 --2- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040d1290 0x7f35040cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:45.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3502ffd640 1 --2- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cc8e0 0x7f35040cfec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f34f80097a0 tx=0x7f34f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea7bf0c3b40b65ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f35027fc640 1 -- 172.21.15.99:0/2097546906 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8035020 con 0x7f35040cc8e0 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f35027fc640 1 -- 172.21.15.99:0/2097546906 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8031830 con 0x7f35040cc8e0 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f35027fc640 1 -- 172.21.15.99:0/2097546906 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8031c00 con 0x7f35040cc8e0 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 -- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cc8e0 msgr2=0x7f35040cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 --2- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cc8e0 0x7f35040cfec0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f34f80097a0 tx=0x7f34f802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 -- 172.21.15.99:0/2097546906 shutdown_connections 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 --2- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040d1290 0x7f35040cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 --2- 172.21.15.99:0/2097546906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cc8e0 0x7f35040cfec0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 -- 172.21.15.99:0/2097546906 >> 172.21.15.99:0/2097546906 conn(0x7f35040d0770 msgr2=0x7f35040d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 -- 172.21.15.99:0/2097546906 shutdown_connections 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.307+0000 7f3509ae8640 1 -- 172.21.15.99:0/2097546906 wait complete. 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.308+0000 7f3509ae8640 1 Processor -- start 2024-03-04T10:03:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 -- start start 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040cc8e0 0x7f3504155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cfae0 0x7f35040b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 0x7f35040cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350406b9d0 con 0x7f35040d1290 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35040ce440 con 0x7f35040cfae0 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3509ae8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35040cd980 con 0x7f35040cc8e0 2024-03-04T10:03:45.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3503fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 0x7f35040cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3503fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 0x7f35040cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38848/0 (socket says 172.21.15.99:38848) 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.309+0000 7f3503fff640 1 -- 172.21.15.99:0/844487738 learned_addr learned my addr 172.21.15.99:0/844487738 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3502ffd640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cfae0 0x7f35040b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f35037fe640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040cc8e0 0x7f3504155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3503fff640 1 -- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040cc8e0 msgr2=0x7f3504155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3503fff640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040cc8e0 0x7f3504155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3503fff640 1 -- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cfae0 msgr2=0x7f35040b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3503fff640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cfae0 0x7f35040b5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3503fff640 1 -- 172.21.15.99:0/844487738 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35040b5970 con 0x7f35040d1290 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f35037fe640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040cc8e0 0x7f3504155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3503fff640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 0x7f35040cad80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00efc0 tx=0x7f34fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc018030 con 0x7f35040d1290 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc00cdf0 con 0x7f35040d1290 2024-03-04T10:03:45.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35040b5c50 con 0x7f35040d1290 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc003bb0 con 0x7f35040d1290 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.310+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3504156b60 con 0x7f35040d1290 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.311+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350406cd70 con 0x7f35040d1290 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.311+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f34fc01b070 con 0x7f35040d1290 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.312+0000 7f34ebfff640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f34f006b4d0 0x7f34f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.312+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f34fc08ae30 con 0x7f35040d1290 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.312+0000 7f35037fe640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f34f006b4d0 0x7f34f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.312+0000 7f35037fe640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f34f006b4d0 0x7f34f006d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34f4000f60 tx=0x7f34f4007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.316+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f34fc010040 con 0x7f35040d1290 2024-03-04T10:03:45.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.490+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3504074f60 con 0x7f35040d1290 2024-03-04T10:03:45.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.491+0000 7f34ebfff640 1 -- 172.21.15.99:0/844487738 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f34fc059970 con 0x7f35040d1290 2024-03-04T10:03:45.492 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:45.493 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f34f006b4d0 msgr2=0x7f34f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f34f006b4d0 0x7f34f006d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34f4000f60 tx=0x7f34f4007c10 comp rx=0 tx=0).stop 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 msgr2=0x7f35040cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 0x7f35040cad80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00efc0 tx=0x7f34fc00c490 comp rx=0 tx=0).stop 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 shutdown_connections 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f34f006b4d0 0x7f34f006d990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35040d1290 0x7f35040cad80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35040cfae0 0x7f35040b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 --2- 172.21.15.99:0/844487738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35040cc8e0 0x7f3504155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 >> 172.21.15.99:0/844487738 conn(0x7f35040d0770 msgr2=0x7f35040cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 shutdown_connections 2024-03-04T10:03:45.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.493+0000 7f3509ae8640 1 -- 172.21.15.99:0/844487738 wait complete. 2024-03-04T10:03:45.529 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:03:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.631+0000 7f2e7fc59640 1 Processor -- start 2024-03-04T10:03:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.631+0000 7f2e7fc59640 1 -- start start 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7fc59640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2e780b5b80 0x7f2e780b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7fc59640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7815e1e0 con 0x7f2e780b2e60 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7fc59640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780d1c10 con 0x7f2e7815de00 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7fc59640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e780b98c0 con 0x7f2e780b5b80 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7e1f6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2e780b5b80 0x7f2e780b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7d1f4640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2e780b2e60 0x7f2e780b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45352/0 (socket says 172.21.15.99:45352) 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7d1f4640 1 -- 172.21.15.99:0/4265800376 learned_addr learned my addr 172.21.15.99:0/4265800376 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7e1f6640 1 -- 172.21.15.99:0/4265800376 >> v1:172.21.15.162:6789/0 conn(0x7f2e7815de00 legacy=0x7f2e780b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7e1f6640 1 -- 172.21.15.99:0/4265800376 >> v1:172.21.15.99:6789/0 conn(0x7f2e780b2e60 legacy=0x7f2e780b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.632+0000 7f2e7e1f6640 1 -- 172.21.15.99:0/4265800376 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780d1ef0 con 0x7f2e780b5b80 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7e1f6640 1 --2- 172.21.15.99:0/4265800376 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b5b80 0x7f2e780b9380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009dc0 tx=0x7f2e7002bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c47209bb240b6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7c9f3640 1 -- 172.21.15.99:0/4265800376 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70039070 con 0x7f2e780b5b80 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7c9f3640 1 -- 172.21.15.99:0/4265800376 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7002ccd0 con 0x7f2e780b5b80 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7c9f3640 1 -- 172.21.15.99:0/4265800376 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e70034660 con 0x7f2e780b5b80 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7fc59640 1 -- 172.21.15.99:0/4265800376 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b5b80 msgr2=0x7f2e780b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/4265800376 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b5b80 0x7f2e780b9380 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009dc0 tx=0x7f2e7002bf20 comp rx=0 tx=0).stop 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7fc59640 1 -- 172.21.15.99:0/4265800376 shutdown_connections 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/4265800376 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b5b80 0x7f2e780b9380 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.633+0000 7f2e7fc59640 1 -- 172.21.15.99:0/4265800376 >> 172.21.15.99:0/4265800376 conn(0x7f2e780d0470 msgr2=0x7f2e780d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.634+0000 7f2e7fc59640 1 -- 172.21.15.99:0/4265800376 shutdown_connections 2024-03-04T10:03:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.634+0000 7f2e7fc59640 1 -- 172.21.15.99:0/4265800376 wait complete. 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 Processor -- start 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 -- start start 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 0x7f2e781f7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e780b5b80 0x7f2e781e7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e7815de00 0x7f2e7815b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780d2820 con 0x7f2e780b2e60 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780d24b0 con 0x7f2e7815de00 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7fc59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e780b72a0 con 0x7f2e780b5b80 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7d9f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 0x7f2e781f7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7d9f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 0x7f2e781f7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38858/0 (socket says 172.21.15.99:38858) 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.635+0000 7f2e7d9f5640 1 -- 172.21.15.99:0/510965025 learned_addr learned my addr 172.21.15.99:0/510965025 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7e1f6640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e7815de00 0x7f2e7815b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d1f4640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e780b5b80 0x7f2e781e7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d9f5640 1 -- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e780b5b80 msgr2=0x7f2e781e7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d9f5640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e780b5b80 0x7f2e781e7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d9f5640 1 -- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e7815de00 msgr2=0x7f2e7815b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d9f5640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e7815de00 0x7f2e7815b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d9f5640 1 -- 172.21.15.99:0/510965025 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780cb7d0 con 0x7f2e780b2e60 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7d9f5640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 0x7f2e781f7910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400c430 tx=0x7f2e7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e740052d0 con 0x7f2e780b2e60 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e74005490 con 0x7f2e780b2e60 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e780cbab0 con 0x7f2e780b2e60 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e74007260 con 0x7f2e780b2e60 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.636+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e780cc070 con 0x7f2e780b2e60 2024-03-04T10:03:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.637+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e78074f60 con 0x7f2e780b2e60 2024-03-04T10:03:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.637+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2e74007400 con 0x7f2e780b2e60 2024-03-04T10:03:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.638+0000 7f2e6e7fc640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2e5c06b4d0 0x7f2e5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.638+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2e740899a0 con 0x7f2e780b2e60 2024-03-04T10:03:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.638+0000 7f2e7d1f4640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2e5c06b4d0 0x7f2e5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.638+0000 7f2e7d1f4640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2e5c06b4d0 0x7f2e5c06d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e781e7e30 tx=0x7f2e68009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.642+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2e74058570 con 0x7f2e780b2e60 2024-03-04T10:03:45.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.852+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e7815d390 con 0x7f2e780b2e60 2024-03-04T10:03:45.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.852+0000 7f2e6e7fc640 1 -- 172.21.15.99:0/510965025 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f2e74057e70 con 0x7f2e780b2e60 2024-03-04T10:03:45.854 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:45.854 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:03:43.916275+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T10:03:45.854 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 6 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2e5c06b4d0 msgr2=0x7f2e5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2e5c06b4d0 0x7f2e5c06d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2e781e7e30 tx=0x7f2e68009040 comp rx=0 tx=0).stop 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 msgr2=0x7f2e781f7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 0x7f2e781f7910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e7400c430 tx=0x7f2e7400c8f0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 shutdown_connections 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2e5c06b4d0 0x7f2e5c06d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2e7815de00 0x7f2e7815b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2e780b5b80 0x7f2e781e7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 --2- 172.21.15.99:0/510965025 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2e780b2e60 0x7f2e781f7910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.854+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 >> 172.21.15.99:0/510965025 conn(0x7f2e780d0470 msgr2=0x7f2e780d2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.855+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 shutdown_connections 2024-03-04T10:03:45.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.855+0000 7f2e7fc59640 1 -- 172.21.15.99:0/510965025 wait complete. 2024-03-04T10:03:45.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:45.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.964+0000 7f5cc328e640 1 Processor -- start 2024-03-04T10:03:45.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc328e640 1 -- start start 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc328e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5cbc15ac40 0x7f5cbc0d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc328e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5cbc0cf080 0x7f5cbc0ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc328e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cbc0cc760 con 0x7f5cbc0af7b0 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc328e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cbc15b020 con 0x7f5cbc15ac40 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc328e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cbc0ce720 con 0x7f5cbc0cf080 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.965+0000 7f5cc102a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5cbc15ac40 0x7f5cbc0d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc102a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5cbc15ac40 0x7f5cbc0d1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38866/0 (socket says 172.21.15.99:38866) 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc102a640 1 -- 172.21.15.99:0/672457995 learned_addr learned my addr 172.21.15.99:0/672457995 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc182b640 1 --2- 172.21.15.99:0/672457995 >> v2:172.21.15.162:3300/0 conn(0x7f5cbc0cf080 0x7f5cbc0ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc102a640 1 -- 172.21.15.99:0/672457995 >> v2:172.21.15.162:3300/0 conn(0x7f5cbc0cf080 msgr2=0x7f5cbc0ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc102a640 1 --2- 172.21.15.99:0/672457995 >> v2:172.21.15.162:3300/0 conn(0x7f5cbc0cf080 0x7f5cbc0ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc102a640 1 -- 172.21.15.99:0/672457995 >> v1:172.21.15.162:6789/0 conn(0x7f5cbc0af7b0 legacy=0x7f5cbc0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.966+0000 7f5cc102a640 1 -- 172.21.15.99:0/672457995 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0cbe30 con 0x7f5cbc15ac40 2024-03-04T10:03:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc102a640 1 --2- 172.21.15.99:0/672457995 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc15ac40 0x7f5cbc0d1290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0cce70 tx=0x7f5cb802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1499e1fcb358cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cb3fff640 1 -- 172.21.15.99:0/672457995 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb802f850 con 0x7f5cbc15ac40 2024-03-04T10:03:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cb3fff640 1 -- 172.21.15.99:0/672457995 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb802f9f0 con 0x7f5cbc15ac40 2024-03-04T10:03:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cb3fff640 1 -- 172.21.15.99:0/672457995 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb802fdc0 con 0x7f5cbc15ac40 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 -- 172.21.15.99:0/672457995 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc15ac40 msgr2=0x7f5cbc0d1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 --2- 172.21.15.99:0/672457995 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc15ac40 0x7f5cbc0d1290 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0cce70 tx=0x7f5cb802ba30 comp rx=0 tx=0).stop 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 -- 172.21.15.99:0/672457995 shutdown_connections 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 --2- 172.21.15.99:0/672457995 >> v2:172.21.15.162:3300/0 conn(0x7f5cbc0cf080 0x7f5cbc0ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 --2- 172.21.15.99:0/672457995 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc15ac40 0x7f5cbc0d1290 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 -- 172.21.15.99:0/672457995 >> 172.21.15.99:0/672457995 conn(0x7f5cbc0d0770 msgr2=0x7f5cbc0d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 -- 172.21.15.99:0/672457995 shutdown_connections 2024-03-04T10:03:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.967+0000 7f5cc328e640 1 -- 172.21.15.99:0/672457995 wait complete. 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.968+0000 7f5cc328e640 1 Processor -- start 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.968+0000 7f5cc328e640 1 -- start start 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc328e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 0x7f5cbc1f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc328e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cbc15ac40 0x7f5cbc09a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc328e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cbc09abf0 0x7f5cbc0ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc328e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0ce340 con 0x7f5cbc0cf080 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc328e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0cc3f0 con 0x7f5cbc09abf0 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc328e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cbc0ce4c0 con 0x7f5cbc15ac40 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 0x7f5cbc1f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 0x7f5cbc1f46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38872/0 (socket says 172.21.15.99:38872) 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 -- 172.21.15.99:0/3142606965 learned_addr learned my addr 172.21.15.99:0/3142606965 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc182b640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cbc09abf0 0x7f5cbc0ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc0829640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cbc15ac40 0x7f5cbc09a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 -- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cbc15ac40 msgr2=0x7f5cbc09a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cbc15ac40 0x7f5cbc09a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 -- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cbc09abf0 msgr2=0x7f5cbc0ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cbc09abf0 0x7f5cbc0ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.969+0000 7f5cc102a640 1 -- 172.21.15.99:0/3142606965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0aef50 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cc182b640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cbc09abf0 0x7f5cbc0ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cc102a640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 0x7f5cbc1f46b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8000c00 tx=0x7f5cb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8047070 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb8037d30 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb80426c0 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cbc09c9f0 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.970+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cbc09cfb0 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.971+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cbc06cd70 con 0x7f5cbc0cf080 2024-03-04T10:03:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.971+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5cb804a070 con 0x7f5cbc0cf080 2024-03-04T10:03:45.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.972+0000 7f5cb1ffb640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5ca006b4d0 0x7f5ca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:45.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.972+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5cb80ac760 con 0x7f5cbc0cf080 2024-03-04T10:03:45.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.972+0000 7f5cc0829640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5ca006b4d0 0x7f5ca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:45.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.973+0000 7f5cc0829640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5ca006b4d0 0x7f5ca006d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5cac007fd0 tx=0x7f5cac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:45.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:45.976+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5cb8033030 con 0x7f5cbc0cf080 2024-03-04T10:03:46.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.149+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cbc09b000 con 0x7f5cbc0cf080 2024-03-04T10:03:46.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.150+0000 7f5cb1ffb640 1 -- 172.21.15.99:0/3142606965 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f5cbc09b000 con 0x7f5cbc0cf080 2024-03-04T10:03:46.153 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:46.153 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:46.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5ca006b4d0 msgr2=0x7f5ca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5ca006b4d0 0x7f5ca006d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5cac007fd0 tx=0x7f5cac009040 comp rx=0 tx=0).stop 2024-03-04T10:03:46.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 msgr2=0x7f5cbc1f46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 0x7f5cbc1f46b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5cb8000c00 tx=0x7f5cb8004730 comp rx=0 tx=0).stop 2024-03-04T10:03:46.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 shutdown_connections 2024-03-04T10:03:46.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5ca006b4d0 0x7f5ca006d990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cbc09abf0 0x7f5cbc0ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cbc15ac40 0x7f5cbc09a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.152+0000 7f5cc328e640 1 --2- 172.21.15.99:0/3142606965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5cbc0cf080 0x7f5cbc1f46b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.153+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 >> 172.21.15.99:0/3142606965 conn(0x7f5cbc0d0770 msgr2=0x7f5cbc0cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.153+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 shutdown_connections 2024-03-04T10:03:46.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.153+0000 7f5cc328e640 1 -- 172.21.15.99:0/3142606965 wait complete. 2024-03-04T10:03:46.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:03:38.859750+0000', 'modified': '2024-03-04T10:03:43.916275+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6835/593641651', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 593641651}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 593641651}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-04T10:03:46.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:03:46.567 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T10:03:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.669+0000 7f7a18cb2640 1 Processor -- start 2024-03-04T10:03:46.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.669+0000 7f7a18cb2640 1 -- start start 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a18cb2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7a14154a80 0x7f7a1415e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a18cb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a140cf650 0x7f7a140cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a18cb2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7a141616c0 0x7f7a14161b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a18cb2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140cfff0 con 0x7f7a141616c0 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a18cb2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140cdaa0 con 0x7f7a14154a80 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a18cb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140cdd80 con 0x7f7a140cf650 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a12d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7a141616c0 0x7f7a14161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a12d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7a141616c0 0x7f7a14161b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38888/0 (socket says 172.21.15.99:38888) 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.670+0000 7f7a12d9d640 1 -- 172.21.15.99:0/3804773675 learned_addr learned my addr 172.21.15.99:0/3804773675 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3804773675 >> v2:172.21.15.162:3300/0 conn(0x7f7a14154a80 0x7f7a1415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3804773675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a140cf650 0x7f7a140cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a12d9d640 1 -- 172.21.15.99:0/3804773675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a140cf650 msgr2=0x7f7a140cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3804773675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a140cf650 0x7f7a140cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a12d9d640 1 -- 172.21.15.99:0/3804773675 >> v2:172.21.15.162:3300/0 conn(0x7f7a14154a80 msgr2=0x7f7a1415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3804773675 >> v2:172.21.15.162:3300/0 conn(0x7f7a14154a80 0x7f7a1415e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a12d9d640 1 -- 172.21.15.99:0/3804773675 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140ce060 con 0x7f7a141616c0 2024-03-04T10:03:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.671+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3804773675 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a14161b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7a1406e970 tx=0x7f7a0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c738f6c1bcbfb546 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a1159a640 1 -- 172.21.15.99:0/3804773675 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c035020 con 0x7f7a141616c0 2024-03-04T10:03:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a1159a640 1 -- 172.21.15.99:0/3804773675 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c031870 con 0x7f7a141616c0 2024-03-04T10:03:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a1159a640 1 -- 172.21.15.99:0/3804773675 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c031c40 con 0x7f7a141616c0 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 -- 172.21.15.99:0/3804773675 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 msgr2=0x7f7a14161b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/3804773675 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a14161b50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7a1406e970 tx=0x7f7a0c02b910 comp rx=0 tx=0).stop 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 -- 172.21.15.99:0/3804773675 shutdown_connections 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/3804773675 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a14161b50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/3804773675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a140cf650 0x7f7a140cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/3804773675 >> v2:172.21.15.162:3300/0 conn(0x7f7a14154a80 0x7f7a1415e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 -- 172.21.15.99:0/3804773675 >> 172.21.15.99:0/3804773675 conn(0x7f7a1406f5f0 msgr2=0x7f7a1406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 -- 172.21.15.99:0/3804773675 shutdown_connections 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.672+0000 7f7a18cb2640 1 -- 172.21.15.99:0/3804773675 wait complete. 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.673+0000 7f7a18cb2640 1 Processor -- start 2024-03-04T10:03:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 -- start start 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a140cf650 0x7f7a140c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a14154a80 0x7f7a140c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a140c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14154e60 con 0x7f7a141616c0 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140ce620 con 0x7f7a140cf650 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a18cb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14154fe0 con 0x7f7a14154a80 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a12d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a140c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a12d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a140c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38896/0 (socket says 172.21.15.99:38896) 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a12d9d640 1 -- 172.21.15.99:0/2814792021 learned_addr learned my addr 172.21.15.99:0/2814792021 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a1259c640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a140cf650 0x7f7a140c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.674+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a14154a80 0x7f7a140c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a12d9d640 1 -- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a14154a80 msgr2=0x7f7a140c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a14154a80 0x7f7a140c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a12d9d640 1 -- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a140cf650 msgr2=0x7f7a140c92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a140cf650 0x7f7a140c92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a12d9d640 1 -- 172.21.15.99:0/2814792021 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140ca2a0 con 0x7f7a141616c0 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a1259c640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a140cf650 0x7f7a140c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a140c9d60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0090b0 tx=0x7f7a0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a02ffd640 1 -- 172.21.15.99:0/2814792021 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c03acb0 con 0x7f7a141616c0 2024-03-04T10:03:46.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a02ffd640 1 -- 172.21.15.99:0/2814792021 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c03ae50 con 0x7f7a141616c0 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a14157bf0 con 0x7f7a141616c0 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a02ffd640 1 -- 172.21.15.99:0/2814792021 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c044630 con 0x7f7a141616c0 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.675+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a140c8ae0 con 0x7f7a141616c0 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.676+0000 7f7a02ffd640 1 -- 172.21.15.99:0/2814792021 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7a0c04b060 con 0x7f7a141616c0 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.677+0000 7f7a02ffd640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f79f406b4d0 0x7f79f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.677+0000 7f7a02ffd640 1 -- 172.21.15.99:0/2814792021 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7a0c0ac690 con 0x7f7a141616c0 2024-03-04T10:03:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.677+0000 7f7a1259c640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f79f406b4d0 0x7f79f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.677+0000 7f7a1259c640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f79f406b4d0 0x7f79f406d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009b20 tx=0x7f7a040039d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.678+0000 7f7a1b971740 1 Processor -- start 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.678+0000 7f7a1b971740 1 -- start start 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 0x564143284e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564143285380 0x5641432b5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1b971740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641431ef7c0 con 0x5641432b1ea0 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x564143198b50 con 0x564143285380 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641431eb860 con 0x5641432b2280 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 0x564143284e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 0x564143284e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38900/0 (socket says 172.21.15.99:38900) 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 -- 172.21.15.99:0/3570839515 learned_addr learned my addr 172.21.15.99:0/3570839515 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564143285380 0x5641432b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 -- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 msgr2=0x5641432b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 -- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564143285380 msgr2=0x5641432b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564143285380 0x5641432b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 -- 172.21.15.99:0/3570839515 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641431ac9a0 con 0x5641432b1ea0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564143285380 0x5641432b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 0x564143284e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a040166b0 tx=0x7f7a04012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a521826fe1da5c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f79f3fff640 1 -- 172.21.15.99:0/3570839515 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04026070 con 0x5641432b1ea0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f79f3fff640 1 -- 172.21.15.99:0/3570839515 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a040148a0 con 0x5641432b1ea0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.679+0000 7f79f3fff640 1 -- 172.21.15.99:0/3570839515 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04014c90 con 0x5641432b1ea0 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 -- 172.21.15.99:0/3570839515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 msgr2=0x564143284e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 0x564143284e40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a040166b0 tx=0x7f7a04012000 comp rx=0 tx=0).stop 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 -- 172.21.15.99:0/3570839515 shutdown_connections 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564143285380 0x5641432b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3570839515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432b1ea0 0x564143284e40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 -- 172.21.15.99:0/3570839515 >> 172.21.15.99:0/3570839515 conn(0x5641432870b0 msgr2=0x5641432cfc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 -- 172.21.15.99:0/3570839515 shutdown_connections 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.680+0000 7f7a1b971740 1 -- 172.21.15.99:0/3570839515 wait complete. 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.681+0000 7f7a1b971740 1 -- start start 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.681+0000 7f7a1b971740 1 Processor -- start 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432aae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 0x5641432b3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x564143289750 con 0x5641432d0240 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641431abf90 con 0x5641432b1ea0 2024-03-04T10:03:46.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56414328bf90 con 0x5641432b2280 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 10 client.0 _ll_register_callbacks cb 0x5641431cdd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 0x5641432b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 0x5641432b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38902/0 (socket says 172.21.15.99:38902) 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 -- 172.21.15.99:0/4185647343 learned_addr learned my addr 172.21.15.99:0/4185647343 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 -- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 msgr2=0x5641432ab3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432ab3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 -- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 msgr2=0x5641432aae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1259c640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432aae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432aae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 -- 172.21.15.99:0/4185647343 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564142f042d0 con 0x5641432d0240 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 0x5641432b3730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c07a730 tx=0x7f7a0c07a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df74cf115ddedd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f79f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f79f17fa640 1 -- 172.21.15.99:0/4185647343 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c082eb0 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f79f17fa640 1 -- 172.21.15.99:0/4185647343 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7a0c079e80 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a040148a0 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f79f17fa640 1 -- 172.21.15.99:0/4185647343 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a0c07aa30 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 10 client.4431 fetch_fsmap learned FSMap version 6 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x564142fce6b0 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.682+0000 7f79f17fa640 1 -- 172.21.15.99:0/4185647343 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a040148a0 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f79f17fa640 1 -- 172.21.15.99:0/4185647343 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 6) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f7a0c099420 con 0x5641432d0240 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1b971740 10 client.4431 fetch_fsmap finished waiting for FSMap version 6 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1b971740 10 client.4431 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1b971740 20 client.4431 populate_metadata read hostname 'smithi099' 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1b971740 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432ad2b0 0x5641432ad690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1b971740 4 client.4431 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 --> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641431fcd00 con 0x5641432ad2b0 2024-03-04T10:03:46.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1259c640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432ad2b0 0x5641432ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f79f17fa640 10 client.4431 ms_handle_connect on v2:172.21.15.99:6833/593641651 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.683+0000 7f7a1259c640 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432ad2b0 0x5641432ad690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bcd1d53df4ed13a server_cookie=7e70ec07d56e1c41 in_seq=0 out_seq=0 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f79f17fa640 1 -- 172.21.15.99:0/4185647343 <== mds.0 v2:172.21.15.99:6833/593641651 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5641431fcd00 con 0x5641432ad2b0 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f79f17fa640 10 client.4431 handle_command_reply: tid=0 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 client.4431 shutdown 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 msgr2=0x5641432b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 0x5641432b3730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c07a730 tx=0x7f7a0c07a450 comp rx=0 tx=0).stop 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 shutdown_connections 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432ad2b0 0x5641432ad690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432d0240 0x5641432b3730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f79f17fa640 0 client.4431 ms_handle_reset on v2:172.21.15.99:6833/593641651 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 --2- 172.21.15.99:0/4185647343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432aae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 >> 172.21.15.99:0/4185647343 conn(0x564143285380 msgr2=0x5641432d3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 shutdown_connections 2024-03-04T10:03:46.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 1 -- 172.21.15.99:0/4185647343 wait complete. 2024-03-04T10:03:46.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.684+0000 7f7a1b971740 20 client.4431 trim_cache size 0 max 16384 2024-03-04T10:03:46.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.699+0000 7f7a1b971740 1 Processor -- start 2024-03-04T10:03:46.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.699+0000 7f7a1b971740 1 -- start start 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x5641432b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432d2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432d29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1b971740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a142004b0 con 0x5641432ad2b0 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a14146710 con 0x5641432b1ea0 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a141ffeb0 con 0x5641432b2280 2024-03-04T10:03:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x5641432b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x5641432b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38910/0 (socket says 172.21.15.99:38910) 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1259c640 1 -- 172.21.15.99:0/3394371385 learned_addr learned my addr 172.21.15.99:0/3394371385 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432d29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432d2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1259c640 1 -- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 msgr2=0x5641432d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.700+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432d29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1259c640 1 -- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 msgr2=0x5641432d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432d2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1259c640 1 -- 172.21.15.99:0/3394371385 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641432d1df0 con 0x5641432ad2b0 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432d29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432d2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x5641432b48e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7a04024040 tx=0x7f7a04014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4ca5833adcb15 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f79f37fe640 1 -- 172.21.15.99:0/3394371385 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04026040 con 0x5641432ad2b0 2024-03-04T10:03:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f79f37fe640 1 -- 172.21.15.99:0/3394371385 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04014980 con 0x5641432ad2b0 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f79f37fe640 1 -- 172.21.15.99:0/3394371385 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04014d30 con 0x5641432ad2b0 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 -- 172.21.15.99:0/3394371385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 msgr2=0x5641432b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x5641432b48e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7a04024040 tx=0x7f7a04014000 comp rx=0 tx=0).stop 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 -- 172.21.15.99:0/3394371385 shutdown_connections 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b2280 0x5641432d29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b1ea0 0x5641432d2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3394371385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x5641432b48e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 -- 172.21.15.99:0/3394371385 >> 172.21.15.99:0/3394371385 conn(0x564143285380 msgr2=0x5641432d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 -- 172.21.15.99:0/3394371385 shutdown_connections 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.701+0000 7f7a1b971740 1 -- 172.21.15.99:0/3394371385 wait complete. 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.702+0000 7f7a1b971740 1 -- start start 2024-03-04T10:03:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.702+0000 7f7a1b971740 1 Processor -- start 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x56414333c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b1ea0 0x5641432f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b2280 0x5641432f2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641432b8c70 con 0x5641432ad2b0 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641432b7690 con 0x5641432b2280 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5641432b4190 con 0x5641432b1ea0 2024-03-04T10:03:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1b971740 10 client.0 _ll_register_callbacks cb 0x7f7a040097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x56414333c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x56414333c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38924/0 (socket says 172.21.15.99:38924) 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 -- 172.21.15.99:0/3652585431 learned_addr learned my addr 172.21.15.99:0/3652585431 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b1ea0 0x5641432f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a12d9d640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b2280 0x5641432f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 -- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b1ea0 msgr2=0x5641432f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b1ea0 0x5641432f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 -- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b2280 msgr2=0x5641432f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b2280 0x5641432f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 -- 172.21.15.99:0/3652585431 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641432d0970 con 0x5641432ad2b0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f7a1259c640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x56414333c220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a04004010 tx=0x7f7a04004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfab3c840779dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f79f17fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.703+0000 7f79f17fa640 1 -- 172.21.15.99:0/3652585431 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04026030 con 0x5641432ad2b0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 1 -- 172.21.15.99:0/3652585431 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7a0403adf0 con 0x5641432ad2b0 2024-03-04T10:03:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a04014980 con 0x5641432ad2b0 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 1 -- 172.21.15.99:0/3652585431 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a04035070 con 0x5641432ad2b0 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 10 client.4437 fetch_fsmap learned FSMap version 6 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x5641432b3810 con 0x5641432ad2b0 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 1 -- 172.21.15.99:0/3652585431 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04014980 con 0x5641432ad2b0 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 1 -- 172.21.15.99:0/3652585431 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 6) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f7a04014b30 con 0x5641432ad2b0 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 10 client.4437 fetch_fsmap finished waiting for FSMap version 6 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 10 client.4437 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 20 client.4437 populate_metadata read hostname 'smithi099' 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432d0240 0x56414333edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 4 client.4437 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 --> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5641431f7050 con 0x5641432d0240 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432d0240 0x56414333edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 10 client.4437 ms_handle_connect on v2:172.21.15.99:6833/593641651 2024-03-04T10:03:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a11d9b640 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432d0240 0x56414333edc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=109984d15b57ebbd server_cookie=d75ac417b2986454 in_seq=0 out_seq=0 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 1 -- 172.21.15.99:0/3652585431 <== mds.0 v2:172.21.15.99:6833/593641651 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x5641431f7050 con 0x5641432d0240 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f79f17fa640 10 client.4437 handle_command_reply: tid=0 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.704+0000 7f7a1b971740 1 client.4437 shutdown 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 msgr2=0x56414333c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x56414333c220 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a04004010 tx=0x7f7a04004ab0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 shutdown_connections 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x5641432d0240 0x56414333edc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5641432b2280 0x5641432f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5641432b1ea0 0x5641432f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 --2- 172.21.15.99:0/3652585431 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5641432ad2b0 0x56414333c220 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f79f17fa640 0 client.4437 ms_handle_reset on v2:172.21.15.99:6833/593641651 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 >> 172.21.15.99:0/3652585431 conn(0x564143285380 msgr2=0x5641432d38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 shutdown_connections 2024-03-04T10:03:46.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 1 -- 172.21.15.99:0/3652585431 wait complete. 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a1b971740 20 client.4437 trim_cache size 0 max 16384 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 4275, 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 7.1659221559999997, 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 6, 2024-03-04T10:03:46.713 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 24, 2024-03-04T10:03:46.714 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 23, 2024-03-04T10:03:46.714 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 13.670894415999999 2024-03-04T10:03:46.714 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:03:46.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f79f406b4d0 msgr2=0x7f79f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f79f406b4d0 0x7f79f406d990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7a04009b20 tx=0x7f7a040039d0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 msgr2=0x7f7a140c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a140c9d60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0090b0 tx=0x7f7a0c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 shutdown_connections 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f79f406b4d0 0x7f79f406d990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a141616c0 0x7f7a140c9d60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a14154a80 0x7f7a140c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 --2- 172.21.15.99:0/2814792021 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a140cf650 0x7f7a140c92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 >> 172.21.15.99:0/2814792021 conn(0x7f7a1406f5f0 msgr2=0x7f7a140b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 shutdown_connections 2024-03-04T10:03:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.705+0000 7f7a18cb2640 1 -- 172.21.15.99:0/2814792021 wait complete. 2024-03-04T10:03:46.717 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T10:03:46.717 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:46.717 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:03:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.816+0000 7f65d09b9640 1 Processor -- start 2024-03-04T10:03:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.816+0000 7f65d09b9640 1 -- start start 2024-03-04T10:03:46.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65d09b9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f65cc15dea0 0x7f65cc0cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65d09b9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f65cc06b3e0 0x7f65cc0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65d09b9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc06fd20 con 0x7f65cc0cc8d0 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65d09b9640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc06fad0 con 0x7f65cc15dea0 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65d09b9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc070000 con 0x7f65cc06b3e0 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65ca59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f65cc15dea0 0x7f65cc0cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65ca59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f65cc15dea0 0x7f65cc0cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38932/0 (socket says 172.21.15.99:38932) 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65ca59c640 1 -- 172.21.15.99:0/945493674 learned_addr learned my addr 172.21.15.99:0/945493674 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.817+0000 7f65cad9d640 1 --2- 172.21.15.99:0/945493674 >> v2:172.21.15.162:3300/0 conn(0x7f65cc06b3e0 0x7f65cc0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65ca59c640 1 -- 172.21.15.99:0/945493674 >> v2:172.21.15.162:3300/0 conn(0x7f65cc06b3e0 msgr2=0x7f65cc0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65ca59c640 1 --2- 172.21.15.99:0/945493674 >> v2:172.21.15.162:3300/0 conn(0x7f65cc06b3e0 0x7f65cc0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65ca59c640 1 -- 172.21.15.99:0/945493674 >> v1:172.21.15.162:6789/0 conn(0x7f65cc0cc8d0 legacy=0x7f65cc06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65ca59c640 1 -- 172.21.15.99:0/945493674 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc0702e0 con 0x7f65cc15dea0 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65cad9d640 1 --2- 172.21.15.99:0/945493674 >> v2:172.21.15.162:3300/0 conn(0x7f65cc06b3e0 0x7f65cc0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65ca59c640 1 --2- 172.21.15.99:0/945493674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc0cad60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0cceb0 tx=0x7f65bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f162bd7ecd868d73 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65c959a640 1 -- 172.21.15.99:0/945493674 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc035020 con 0x7f65cc15dea0 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65c959a640 1 -- 172.21.15.99:0/945493674 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc031870 con 0x7f65cc15dea0 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.818+0000 7f65c959a640 1 -- 172.21.15.99:0/945493674 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc031c40 con 0x7f65cc15dea0 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 -- 172.21.15.99:0/945493674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 msgr2=0x7f65cc0cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 --2- 172.21.15.99:0/945493674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc0cad60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f65cc0cceb0 tx=0x7f65bc02b910 comp rx=0 tx=0).stop 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 -- 172.21.15.99:0/945493674 shutdown_connections 2024-03-04T10:03:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 --2- 172.21.15.99:0/945493674 >> v2:172.21.15.162:3300/0 conn(0x7f65cc06b3e0 0x7f65cc0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 --2- 172.21.15.99:0/945493674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc0cad60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 -- 172.21.15.99:0/945493674 >> 172.21.15.99:0/945493674 conn(0x7f65cc0d0710 msgr2=0x7f65cc0d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 -- 172.21.15.99:0/945493674 shutdown_connections 2024-03-04T10:03:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.819+0000 7f65d09b9640 1 -- 172.21.15.99:0/945493674 wait complete. 2024-03-04T10:03:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.820+0000 7f65d09b9640 1 Processor -- start 2024-03-04T10:03:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.820+0000 7f65d09b9640 1 -- start start 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65d09b9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc06b3e0 0x7f65cc156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65d09b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0cc8d0 0x7f65cc0c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65d09b9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc14d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65d09b9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0d1380 con 0x7f65cc15dea0 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65d09b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0d1200 con 0x7f65cc0cc8d0 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65d09b9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0d1500 con 0x7f65cc06b3e0 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65cad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc14d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65cad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc14d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38946/0 (socket says 172.21.15.99:38946) 2024-03-04T10:03:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65cad9d640 1 -- 172.21.15.99:0/806783435 learned_addr learned my addr 172.21.15.99:0/806783435 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65c9d9b640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0cc8d0 0x7f65cc0c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.821+0000 7f65ca59c640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc06b3e0 0x7f65cc156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65cad9d640 1 -- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc06b3e0 msgr2=0x7f65cc156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65cad9d640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc06b3e0 0x7f65cc156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65cad9d640 1 -- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0cc8d0 msgr2=0x7f65cc0c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65cad9d640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0cc8d0 0x7f65cc0c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65cad9d640 1 -- 172.21.15.99:0/806783435 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc156ca0 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65c9d9b640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0cc8d0 0x7f65cc0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65cad9d640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc14d9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f65c400cea0 tx=0x7f65c400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4003b90 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4003d50 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc14df20 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4004260 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.822+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc0c81c0 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.823+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc074f60 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.823+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f65c4004400 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.824+0000 7f65baffd640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65ac06b4d0 0x7f65ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.824+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f65c408aed0 con 0x7f65cc15dea0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.824+0000 7f65ca59c640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65ac06b4d0 0x7f65ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.824+0000 7f65ca59c640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65ac06b4d0 0x7f65ac06d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65bc036000 tx=0x7f65bc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:46.828+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f65c4059330 con 0x7f65cc15dea0 2024-03-04T10:03:47.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.040+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f65cc15cd00 con 0x7f65cc15dea0 2024-03-04T10:03:47.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.040+0000 7f65baffd640 1 -- 172.21.15.99:0/806783435 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f65c405d3d0 con 0x7f65cc15dea0 2024-03-04T10:03:47.042 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65ac06b4d0 msgr2=0x7f65ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65ac06b4d0 0x7f65ac06d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65bc036000 tx=0x7f65bc03c040 comp rx=0 tx=0).stop 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 msgr2=0x7f65cc14d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc14d9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f65c400cea0 tx=0x7f65c400cf90 comp rx=0 tx=0).stop 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 shutdown_connections 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65ac06b4d0 0x7f65ac06d990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc15dea0 0x7f65cc14d9e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0cc8d0 0x7f65cc0c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 --2- 172.21.15.99:0/806783435 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc06b3e0 0x7f65cc156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 >> 172.21.15.99:0/806783435 conn(0x7f65cc0d0710 msgr2=0x7f65cc0cae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 shutdown_connections 2024-03-04T10:03:47.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:47.042+0000 7f65d09b9640 1 -- 172.21.15.99:0/806783435 wait complete. 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:03:47.054 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-04T10:03:47.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.055 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:03:47.069 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:03:47.069 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:03:47.070 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:03:47.071 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:03:47.071 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:03:47.071 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42758sec preferred_lft 42758sec 2024-03-04T10:03:47.071 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:03:47.071 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:03:47.071 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T10:03:47.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link add name ceph-brx type bridge 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip addr flush dev ceph-brx 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set ceph-brx up 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T10:03:47.072 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:03:47.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.248 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:03:47.274 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:03:47.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.276 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:03:47.348 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:03:47.348 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:03:47.348 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:03:47.348 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:03:47.348 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:03:47.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.349 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:03:47.349 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:03:47.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:03:47.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:03:47.349 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:03:47.349 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:03:47.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.602 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:03:47.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.617 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list-id 2024-03-04T10:03:47.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.672 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:03:47.672 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:03:47.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T10:03:47.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-04T10:03:47.672 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:03:47.755 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-04T10:03:47.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.755 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:03:47.755 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:03:47.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-04T10:03:47.755 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T10:03:47.756 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-04T10:03:47.756 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-04T10:03:47.756 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-04T10:03:47.756 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:03:47.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:47.892 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:03:47.892 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:03:47.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:03:47.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T10:03:47.892 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:03:47.933 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-04T10:03:47.933 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:03:47.933 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:47.989 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-04T10:03:47.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:03:47.989 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:48.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo modprobe fuse 2024-03-04T10:03:48.112 DEBUG:teuthology.orchestra.run.smithi099:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T10:03:48.167 INFO:teuthology.orchestra.run.smithi099.stdout:/proc 2024-03-04T10:03:48.167 INFO:teuthology.orchestra.run.smithi099.stdout:/sys 2024-03-04T10:03:48.167 INFO:teuthology.orchestra.run.smithi099.stdout:/dev 2024-03-04T10:03:48.167 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/security 2024-03-04T10:03:48.167 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/shm 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/pts 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/run 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/cgroup 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/pstore 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/bpf 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/ 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/selinux 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/mqueue 2024-03-04T10:03:48.168 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/hugepages 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/debug 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/tracing 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/fuse/connections 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/config 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/run/user/1000 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-0 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-1 2024-03-04T10:03:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-2 2024-03-04T10:03:48.170 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-3 2024-03-04T10:03:48.170 INFO:teuthology.orchestra.run.smithi099.stdout:/run/netns 2024-03-04T10:03:48.170 INFO:teuthology.orchestra.run.smithi099.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T10:03:48.170 INFO:teuthology.orchestra.run.smithi099.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T10:03:48.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:48.170 DEBUG:teuthology.orchestra.run.smithi099:> ls /sys/fs/fuse/connections 2024-03-04T10:03:48.222 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-04T10:03:48.222 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-04T10:03:48.265 DEBUG:teuthology.orchestra.run.smithi099:> sudo modprobe fuse 2024-03-04T10:03:48.291 DEBUG:teuthology.orchestra.run.smithi099:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T10:03:48.346 INFO:teuthology.orchestra.run.smithi099.stdout:/proc 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/sys 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/dev 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/security 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/shm 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/pts 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/run 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/cgroup 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/pstore 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/bpf 2024-03-04T10:03:48.347 INFO:teuthology.orchestra.run.smithi099.stdout:/ 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/selinux 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/mqueue 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/dev/hugepages 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/debug 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/tracing 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/fs/fuse/connections 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/sys/kernel/config 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/run/user/1000 2024-03-04T10:03:48.348 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-0 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-1 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-2 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/var/lib/ceph/osd/ceph-3 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/run/netns 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run.smithi099.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T10:03:48.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:48.349 DEBUG:teuthology.orchestra.run.smithi099:> ls /sys/fs/fuse/connections 2024-03-04T10:03:48.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi099.stderr:2024-03-04T10:03:48.349+0000 7f38e8e42480 -1 init, newargv = 0x56045f88f7f0 newargc=15 2024-03-04T10:03:48.352 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi099.stderr:ceph-fuse[44829]: starting ceph client 2024-03-04T10:03:48.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi099.stderr:ceph-fuse[44829]: starting fuse 2024-03-04T10:03:48.366 INFO:teuthology.orchestra.run.smithi099.stdout:41 2024-03-04T10:03:48.366 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo stdin-killer -- python3 -c ' 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> import glob 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> import re 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> import os 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> import subprocess 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> def _find_admin_socket(client_name): 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> files = glob.glob(asok_path) 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> # Given a non-glob path, it better be there 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> if "*" not in asok_path: 2024-03-04T10:03:48.367 DEBUG:teuthology.orchestra.run.smithi099:> assert(len(files) == 1) 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> return files[0] 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> for f in files: 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> contents = proc_f.read() 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> if mountpoint in contents: 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> return f 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> print(_find_admin_socket("client.admin")) 2024-03-04T10:03:48.368 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:03:48.487 INFO:teuthology.orchestra.run.smithi099.stdout:/var/run/ceph/ceph-client.admin.44829.asok 2024-03-04T10:03:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T10:03:48.497 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44829.asok 2024-03-04T10:03:48.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:48.498 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44829.asok status 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "metadata": { 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "ceph_sha1": "717ce3cce8d0166e4e577faa8004e6f9cb4128c0", 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "ceph_version": "ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev)", 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "entity_id": "admin", 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "hostname": "smithi099", 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "pid": "44829", 2024-03-04T10:03:48.635 INFO:teuthology.orchestra.run.smithi099.stdout: "root": "/" 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "dentry_count": 0, 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "dentry_pinned_count": 0, 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 4452, 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "inst": { 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "name": { 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "client", 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "num": 4452 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": { 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "192.168.0.1:0", 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4213795546 2024-03-04T10:03:48.636 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": { 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "type": "v1", 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "addr": "192.168.0.1:0", 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "nonce": 4213795546 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "inst_str": "client.4452 192.168.0.1:0/4213795546", 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "addr_str": "192.168.0.1:0/4213795546", 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "inode_count": 1, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "mds_epoch": 6, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_epoch": 24, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "osd_epoch_barrier": 0, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "blocklisted": false, 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs" 2024-03-04T10:03:48.637 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:03:48.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:48.643 DEBUG:teuthology.orchestra.run.smithi099:> stat --file-system '--printf=%T 2024-03-04T10:03:48.643 DEBUG:teuthology.orchestra.run.smithi099:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:48.659 INFO:teuthology.orchestra.run.smithi099.stdout:fuseblk 2024-03-04T10:03:48.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:48.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:48.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:48.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:48.727 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-04T10:03:48.795 INFO:teuthology.orchestra.run.smithi099.stdout:# file: . 2024-03-04T10:03:48.795 INFO:teuthology.orchestra.run.smithi099.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-04T10:03:48.796 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:48.796 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:48.796 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:03:48.851 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:48.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.989+0000 7f1daa762640 1 Processor -- start 2024-03-04T10:03:48.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.989+0000 7f1daa762640 1 -- start start 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1daa762640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 0x7f1da40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1daa762640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da40ccad0 con 0x7f1da40b0700 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1daa762640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da40cfab0 con 0x7f1da415b960 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1daa762640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40cf860 con 0x7f1da40b4ed0 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1da37fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1da40b0700 0x7f1da40b47c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45364/0 (socket says 172.21.15.99:45364) 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1da37fe640 1 -- 172.21.15.99:0/1927202074 learned_addr learned my addr 172.21.15.99:0/1927202074 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1da8cff640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 0x7f1da40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210264212 0 0) 0x7f1da40ccad0 con 0x7f1da40b0700 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.990+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003520 con 0x7f1da40b0700 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915825208 0 0) 0x7f1d8c003520 con 0x7f1da40b0700 2024-03-04T10:03:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 msgr2=0x7f1da40b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 0x7f1da40b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 >> v1:172.21.15.162:6789/0 conn(0x7f1da415b960 legacy=0x7f1da40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40cfd90 con 0x7f1da40b0700 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f1d98001670 con 0x7f1da40b0700 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 >> v1:172.21.15.99:6789/0 conn(0x7f1da40b0700 legacy=0x7f1da40b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1d8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 0x7f1d8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d8c004d40 0x7f1d8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40cf860 con 0x7f1d8c0039e0 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c00bb20 con 0x7f1d8c004d40 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c00bd20 con 0x7f1da40b4ed0 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da3fff640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1d8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da37fe640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 0x7f1d8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da8cff640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d8c004d40 0x7f1d8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da3fff640 1 -- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 msgr2=0x7f1d8c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da3fff640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40b4ed0 0x7f1d8c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da3fff640 1 -- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d8c004d40 msgr2=0x7f1d8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da3fff640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d8c004d40 0x7f1d8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.991+0000 7f1da3fff640 1 -- 172.21.15.99:0/1927202074 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c003520 con 0x7f1d8c0039e0 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.992+0000 7f1da3fff640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1d8c003e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d94001750 tx=0x7f1d8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fc2d7ec42d7efe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.992+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94001370 con 0x7f1d8c0039e0 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.992+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d940052d0 con 0x7f1d8c0039e0 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.992+0000 7f1da2ffd640 1 -- 172.21.15.99:0/1927202074 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94003830 con 0x7f1d8c0039e0 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1daa762640 1 -- 172.21.15.99:0/1927202074 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 msgr2=0x7f1d8c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1daa762640 1 --2- 172.21.15.99:0/1927202074 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1d8c003e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d94001750 tx=0x7f1d8c005830 comp rx=0 tx=0).stop 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1da8cff640 1 -- 172.21.15.99:0/1927202074 reap_dead start 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1daa762640 1 -- 172.21.15.99:0/1927202074 shutdown_connections 2024-03-04T10:03:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1daa762640 1 -- 172.21.15.99:0/1927202074 >> 172.21.15.99:0/1927202074 conn(0x7f1da406f5f0 msgr2=0x7f1da406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:48.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1daa762640 1 -- 172.21.15.99:0/1927202074 shutdown_connections 2024-03-04T10:03:48.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.993+0000 7f1daa762640 1 -- 172.21.15.99:0/1927202074 wait complete. 2024-03-04T10:03:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.994+0000 7f1daa762640 1 Processor -- start 2024-03-04T10:03:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.994+0000 7f1daa762640 1 -- start start 2024-03-04T10:03:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1da415e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da40c8940 0x7f1da40c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40ae370 0x7f1da40ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40d0ca0 con 0x7f1d8c0039e0 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40d0100 con 0x7f1da40c8940 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40d0e20 con 0x7f1da40ae370 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1da415e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1da415e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38952/0 (socket says 172.21.15.99:38952) 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 -- 172.21.15.99:0/2232402259 learned_addr learned my addr 172.21.15.99:0/2232402259 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da37fe640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da40c8940 0x7f1da40c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da8cff640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40ae370 0x7f1da40ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 -- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40ae370 msgr2=0x7f1da40ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40ae370 0x7f1da40ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 -- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da40c8940 msgr2=0x7f1da40c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da40c8940 0x7f1da40c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 -- 172.21.15.99:0/2232402259 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da40aca20 con 0x7f1d8c0039e0 2024-03-04T10:03:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da3fff640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1da415e480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d94008fd0 tx=0x7f1d94001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94041070 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d94005480 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da406ddd0 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94001370 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.995+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da40aef10 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.996+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da4074f60 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.997+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1d94044070 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.997+0000 7f1da0ff9640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d9006b500 0x7f1d9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.997+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1d940ac0f0 con 0x7f1d8c0039e0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.997+0000 7f1da37fe640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d9006b500 0x7f1d9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:48.998+0000 7f1da37fe640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d9006b500 0x7f1d9006d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d98000fb0 tx=0x7f1d980078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.001+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1d9407a5e0 con 0x7f1d8c0039e0 2024-03-04T10:03:49.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.174+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1da406e620 con 0x7f1d8c0039e0 2024-03-04T10:03:49.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.176+0000 7f1da0ff9640 1 -- 172.21.15.99:0/2232402259 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f1d9407e680 con 0x7f1d8c0039e0 2024-03-04T10:03:49.178 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:49.178 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:49.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d9006b500 msgr2=0x7f1d9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d9006b500 0x7f1d9006d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d98000fb0 tx=0x7f1d980078f0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 msgr2=0x7f1da415e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1da415e480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d94008fd0 tx=0x7f1d94001120 comp rx=0 tx=0).stop 2024-03-04T10:03:49.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 shutdown_connections 2024-03-04T10:03:49.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d9006b500 0x7f1d9006d9c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1da40ae370 0x7f1da40ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1da40c8940 0x7f1da40c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 --2- 172.21.15.99:0/2232402259 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d8c0039e0 0x7f1da415e480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 >> 172.21.15.99:0/2232402259 conn(0x7f1da406f5f0 msgr2=0x7f1da40b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 shutdown_connections 2024-03-04T10:03:49.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.178+0000 7f1daa762640 1 -- 172.21.15.99:0/2232402259 wait complete. 2024-03-04T10:03:49.191 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:03:49.191 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.292+0000 7f47c3d97640 1 Processor -- start 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.293+0000 7f47c3d97640 1 -- start start 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.293+0000 7f47c3d97640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47bc0cec40 0x7f47bc0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.293+0000 7f47c3d97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc0d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.293+0000 7f47c3d97640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47bc0ca970 con 0x7f47bc0cf020 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.293+0000 7f47c3d97640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47bc161d90 con 0x7f47bc0cec40 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.293+0000 7f47c3d97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc162070 con 0x7f47bc15de40 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47bc0cec40 0x7f47bc0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47bc0cec40 0x7f47bc0b6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38964/0 (socket says 172.21.15.99:38964) 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 -- 172.21.15.99:0/3743437275 learned_addr learned my addr 172.21.15.99:0/3743437275 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c2334640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc0d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 -- 172.21.15.99:0/3743437275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 msgr2=0x7f47bc0d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc0d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 -- 172.21.15.99:0/3743437275 >> v1:172.21.15.99:6789/0 conn(0x7f47bc0cf020 legacy=0x7f47bc15d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c1b33640 1 -- 172.21.15.99:0/3743437275 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc162350 con 0x7f47bc0cec40 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c0b31640 1 -- 172.21.15.99:0/3743437275 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178377505 0 0) 0x7f47bc0ca970 con 0x7f47bc0cf020 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.294+0000 7f47c2334640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc0d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c1b33640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cec40 0x7f47bc0b6ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009870 tx=0x7f47ac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0909047e3a0fd0e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c0b31640 1 -- 172.21.15.99:0/3743437275 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac034070 con 0x7f47bc0cec40 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c0b31640 1 -- 172.21.15.99:0/3743437275 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac030840 con 0x7f47bc0cec40 2024-03-04T10:03:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c0b31640 1 -- 172.21.15.99:0/3743437275 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac030c10 con 0x7f47bc0cec40 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 -- 172.21.15.99:0/3743437275 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cec40 msgr2=0x7f47bc0b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cec40 0x7f47bc0b6ea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009870 tx=0x7f47ac02b980 comp rx=0 tx=0).stop 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 -- 172.21.15.99:0/3743437275 shutdown_connections 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc0d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 --2- 172.21.15.99:0/3743437275 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cec40 0x7f47bc0b6ea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 -- 172.21.15.99:0/3743437275 >> 172.21.15.99:0/3743437275 conn(0x7f47bc0d0580 msgr2=0x7f47bc0d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.295+0000 7f47c3d97640 1 -- 172.21.15.99:0/3743437275 shutdown_connections 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.296+0000 7f47c3d97640 1 -- 172.21.15.99:0/3743437275 wait complete. 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 Processor -- start 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 -- start start 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0cec40 0x7f47bc1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 0x7f47bc15a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc15af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0d3300 con 0x7f47bc0cf020 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0d1bf0 con 0x7f47bc0cec40 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c3d97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0d3480 con 0x7f47bc15de40 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c1332640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 0x7f47bc15a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c1332640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 0x7f47bc15a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38970/0 (socket says 172.21.15.99:38970) 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.297+0000 7f47c1332640 1 -- 172.21.15.99:0/56814345 learned_addr learned my addr 172.21.15.99:0/56814345 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c2334640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc15af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1b33640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0cec40 0x7f47bc1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1332640 1 -- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 msgr2=0x7f47bc15af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1332640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc15af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1332640 1 -- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0cec40 msgr2=0x7f47bc1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1332640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0cec40 0x7f47bc1602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1332640 1 -- 172.21.15.99:0/56814345 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc1597e0 con 0x7f47bc0cf020 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1b33640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0cec40 0x7f47bc1602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c1332640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 0x7f47bc15a9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f47b800c550 tx=0x7f47b800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b80052d0 con 0x7f47bc0cf020 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b8005470 con 0x7f47bc0cf020 2024-03-04T10:03:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc159ac0 con 0x7f47bc0cf020 2024-03-04T10:03:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8012870 con 0x7f47bc0cf020 2024-03-04T10:03:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.298+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc0c8440 con 0x7f47bc0cf020 2024-03-04T10:03:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.299+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc074f60 con 0x7f47bc0cf020 2024-03-04T10:03:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.300+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f47b8012a10 con 0x7f47bc0cf020 2024-03-04T10:03:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.300+0000 7f47b27fc640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47a006b4d0 0x7f47a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.300+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f47b808a1b0 con 0x7f47bc0cf020 2024-03-04T10:03:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.300+0000 7f47c1b33640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47a006b4d0 0x7f47a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.301+0000 7f47c1b33640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47a006b4d0 0x7f47a006d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f47ac036000 tx=0x7f47ac03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.304+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f47b8058d00 con 0x7f47bc0cf020 2024-03-04T10:03:49.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.474+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47bc0ccf20 con 0x7f47bc0cf020 2024-03-04T10:03:49.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.474+0000 7f47b27fc640 1 -- 172.21.15.99:0/56814345 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f47b8058600 con 0x7f47bc0cf020 2024-03-04T10:03:49.477 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:03:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47a006b4d0 msgr2=0x7f47a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47a006b4d0 0x7f47a006d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f47ac036000 tx=0x7f47ac03b040 comp rx=0 tx=0).stop 2024-03-04T10:03:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 msgr2=0x7f47bc15a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 0x7f47bc15a9f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f47b800c550 tx=0x7f47b800ca10 comp rx=0 tx=0).stop 2024-03-04T10:03:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 shutdown_connections 2024-03-04T10:03:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47a006b4d0 0x7f47a006d990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47bc15de40 0x7f47bc15af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47bc0cf020 0x7f47bc15a9f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 --2- 172.21.15.99:0/56814345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47bc0cec40 0x7f47bc1602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 >> 172.21.15.99:0/56814345 conn(0x7f47bc0d0580 msgr2=0x7f47bc157fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:49.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 shutdown_connections 2024-03-04T10:03:49.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.476+0000 7f47c3d97640 1 -- 172.21.15.99:0/56814345 wait complete. 2024-03-04T10:03:49.490 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:03:49.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.592+0000 7f0f2f109640 1 Processor -- start 2024-03-04T10:03:49.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2f109640 1 -- start start 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2f109640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f2806c060 0x7f0f280ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2f109640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f280cf740 con 0x7f0f28161390 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2f109640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2806c7d0 con 0x7f0f280cec20 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2f109640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280cf080 con 0x7f0f2806c060 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2cea5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0f28161390 0x7f0f280d20c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45370/0 (socket says 172.21.15.99:45370) 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.593+0000 7f0f2cea5640 1 -- 172.21.15.99:0/2585715534 learned_addr learned my addr 172.21.15.99:0/2585715534 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f27fff640 1 --2- 172.21.15.99:0/2585715534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f2806c060 0x7f0f280ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000333644 0 0) 0x7f0f280cf740 con 0x7f0f28161390 2024-03-04T10:03:49.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f18003520 con 0x7f0f28161390 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405261889 0 0) 0x7f0f2806c7d0 con 0x7f0f280cec20 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f280cf740 con 0x7f0f280cec20 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f27fff640 1 -- 172.21.15.99:0/2585715534 >> v1:172.21.15.162:6789/0 conn(0x7f0f280cec20 legacy=0x7f0f280cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f27fff640 1 -- 172.21.15.99:0/2585715534 >> v1:172.21.15.99:6789/0 conn(0x7f0f28161390 legacy=0x7f0f280d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.594+0000 7f0f27fff640 1 -- 172.21.15.99:0/2585715534 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2806cab0 con 0x7f0f2806c060 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f27fff640 1 --2- 172.21.15.99:0/2585715534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f2806c060 0x7f0f280ce6e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097d0 tx=0x7f0f1402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184166b1fa751ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14035020 con 0x7f0f2806c060 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14031890 con 0x7f0f2806c060 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f277fe640 1 -- 172.21.15.99:0/2585715534 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14031c60 con 0x7f0f2806c060 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f2f109640 1 -- 172.21.15.99:0/2585715534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f2806c060 msgr2=0x7f0f280ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f2f109640 1 --2- 172.21.15.99:0/2585715534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f2806c060 0x7f0f280ce6e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097d0 tx=0x7f0f1402b930 comp rx=0 tx=0).stop 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f2f109640 1 -- 172.21.15.99:0/2585715534 shutdown_connections 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f2f109640 1 --2- 172.21.15.99:0/2585715534 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f2806c060 0x7f0f280ce6e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f2f109640 1 -- 172.21.15.99:0/2585715534 >> 172.21.15.99:0/2585715534 conn(0x7f0f28066dc0 msgr2=0x7f0f280671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:49.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.595+0000 7f0f2f109640 1 -- 172.21.15.99:0/2585715534 shutdown_connections 2024-03-04T10:03:49.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.596+0000 7f0f2f109640 1 -- 172.21.15.99:0/2585715534 wait complete. 2024-03-04T10:03:49.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.597+0000 7f0f2f109640 1 Processor -- start 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.597+0000 7f0f2f109640 1 -- start start 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.597+0000 7f0f2f109640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2806c060 0x7f0f281603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.597+0000 7f0f2f109640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 0x7f0f281f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.597+0000 7f0f2f109640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f28161390 0x7f0f280ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2f109640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2806d070 con 0x7f0f280cec20 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2f109640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2806d3e0 con 0x7f0f2806c060 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2f109640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2806c4c0 con 0x7f0f28161390 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 0x7f0f281f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 0x7f0f281f3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38974/0 (socket says 172.21.15.99:38974) 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 -- 172.21.15.99:0/1897377269 learned_addr learned my addr 172.21.15.99:0/1897377269 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2cea5640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2806c060 0x7f0f281603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2d6a6640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f28161390 0x7f0f280ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 -- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f28161390 msgr2=0x7f0f280ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f28161390 0x7f0f280ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 -- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2806c060 msgr2=0x7f0f281603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2806c060 0x7f0f281603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 -- 172.21.15.99:0/1897377269 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2815e430 con 0x7f0f280cec20 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f27fff640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 0x7f0f281f3c00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f140098f0 tx=0x7f0f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14035040 con 0x7f0f280cec20 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14030d20 con 0x7f0f280cec20 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2815e710 con 0x7f0f280cec20 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14006e10 con 0x7f0f280cec20 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.598+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2815ecd0 con 0x7f0f280cec20 2024-03-04T10:03:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.599+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f280ccf60 con 0x7f0f280cec20 2024-03-04T10:03:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.600+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0f14038020 con 0x7f0f280cec20 2024-03-04T10:03:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.600+0000 7f0f257fa640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.600+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0f140acb30 con 0x7f0f280cec20 2024-03-04T10:03:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.600+0000 7f0f2cea5640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.601+0000 7f0f2cea5640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c000f60 tx=0x7f0f1c007d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.604+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0f280ccf60 con 0x7f0f280cec20 2024-03-04T10:03:49.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.773+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f280ac9a0 con 0x7f0f280cec20 2024-03-04T10:03:49.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.773+0000 7f0f257fa640 1 -- 172.21.15.99:0/1897377269 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0f1402d040 con 0x7f0f280cec20 2024-03-04T10:03:49.776 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.775+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0c06b4d0 msgr2=0x7f0f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.775+0000 7f0f2f109640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c000f60 tx=0x7f0f1c007d70 comp rx=0 tx=0).stop 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.775+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 msgr2=0x7f0f281f3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.775+0000 7f0f2f109640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 0x7f0f281f3c00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0f140098f0 tx=0x7f0f140046c0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 shutdown_connections 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0c06b4d0 0x7f0f0c06d990 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f28161390 0x7f0f280ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f280cec20 0x7f0f281f3c00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 --2- 172.21.15.99:0/1897377269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f2806c060 0x7f0f281603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 >> 172.21.15.99:0/1897377269 conn(0x7f0f28066dc0 msgr2=0x7f0f280b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:49.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 shutdown_connections 2024-03-04T10:03:49.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.776+0000 7f0f2f109640 1 -- 172.21.15.99:0/1897377269 wait complete. 2024-03-04T10:03:49.788 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:03:49.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.889+0000 7f0928d22640 1 Processor -- start 2024-03-04T10:03:49.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.889+0000 7f0928d22640 1 -- start start 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0928d22640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f09241450e0 0x7f09240cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0928d22640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09240cebd0 con 0x7f09240cc510 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0928d22640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09240ceeb0 con 0x7f092415c9b0 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0928d22640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09240cf7e0 con 0x7f09241450e0 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f092259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f09241450e0 0x7f09240cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0922d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f09240cc510 0x7f09240b3250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45380/0 (socket says 172.21.15.99:45380) 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0922d9d640 1 -- 172.21.15.99:0/2964378526 learned_addr learned my addr 172.21.15.99:0/2964378526 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:49.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f092259c640 1 -- 172.21.15.99:0/2964378526 >> v1:172.21.15.162:6789/0 conn(0x7f092415c9b0 legacy=0x7f09240cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f092259c640 1 -- 172.21.15.99:0/2964378526 >> v1:172.21.15.99:6789/0 conn(0x7f09240cc510 legacy=0x7f09240b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f092259c640 1 -- 172.21.15.99:0/2964378526 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09240cf190 con 0x7f09241450e0 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.890+0000 7f0921d9b640 1 -- 172.21.15.99:0/2964378526 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050358492 0 0) 0x7f09240cebd0 con 0x7f09240cc510 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f092259c640 1 --2- 172.21.15.99:0/2964378526 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09241450e0 0x7f09240cfc70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f09140097a0 tx=0x7f091402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a526214c403415 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0921d9b640 1 -- 172.21.15.99:0/2964378526 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914035020 con 0x7f09241450e0 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0921d9b640 1 -- 172.21.15.99:0/2964378526 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0914031830 con 0x7f09241450e0 2024-03-04T10:03:49.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0921d9b640 1 -- 172.21.15.99:0/2964378526 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0914031c00 con 0x7f09241450e0 2024-03-04T10:03:49.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0928d22640 1 -- 172.21.15.99:0/2964378526 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09241450e0 msgr2=0x7f09240cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0928d22640 1 --2- 172.21.15.99:0/2964378526 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09241450e0 0x7f09240cfc70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f09140097a0 tx=0x7f091402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0928d22640 1 -- 172.21.15.99:0/2964378526 shutdown_connections 2024-03-04T10:03:49.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0928d22640 1 --2- 172.21.15.99:0/2964378526 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09241450e0 0x7f09240cfc70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.891+0000 7f0928d22640 1 -- 172.21.15.99:0/2964378526 >> 172.21.15.99:0/2964378526 conn(0x7f092406acd0 msgr2=0x7f092406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.892+0000 7f0928d22640 1 -- 172.21.15.99:0/2964378526 shutdown_connections 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.892+0000 7f0928d22640 1 -- 172.21.15.99:0/2964378526 wait complete. 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 Processor -- start 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 -- start start 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09240cc510 0x7f0924140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09241450e0 0x7f092413f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 0x7f092415a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09241454c0 con 0x7f092415c9b0 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092415ce60 con 0x7f09241450e0 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.893+0000 7f0928d22640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09240ccd10 con 0x7f09240cc510 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 0x7f092415a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 0x7f092415a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39002/0 (socket says 172.21.15.99:39002) 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 -- 172.21.15.99:0/2007057843 learned_addr learned my addr 172.21.15.99:0/2007057843 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f092259c640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09240cc510 0x7f0924140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f091bfff640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09241450e0 0x7f092413f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 -- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09240cc510 msgr2=0x7f0924140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09240cc510 0x7f0924140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 -- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09241450e0 msgr2=0x7f092413f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09241450e0 0x7f092413f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 -- 172.21.15.99:0/2007057843 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092413fb60 con 0x7f092415c9b0 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f091bfff640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09241450e0 0x7f092413f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:49.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f092259c640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09240cc510 0x7f0924140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0922d9d640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 0x7f092415a8c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f091c00c6d0 tx=0x7f091c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c018070 con 0x7f092415c9b0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091c004d60 con 0x7f092415c9b0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092413fe40 con 0x7f092415c9b0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c00a880 con 0x7f092415c9b0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.894+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092415b600 con 0x7f092415c9b0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.895+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0924074f60 con 0x7f092415c9b0 2024-03-04T10:03:49.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.896+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f091c01b070 con 0x7f092415c9b0 2024-03-04T10:03:49.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.896+0000 7f091affd640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f090406b500 0x7f090406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:49.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.896+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f091c089c00 con 0x7f092415c9b0 2024-03-04T10:03:49.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.896+0000 7f092259c640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f090406b500 0x7f090406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:49.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.897+0000 7f092259c640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f090406b500 0x7f090406d9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0914036000 tx=0x7f0914008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:49.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:49.901+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f091c090080 con 0x7f092415c9b0 2024-03-04T10:03:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.071+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0924140010 con 0x7f092415c9b0 2024-03-04T10:03:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.071+0000 7f091affd640 1 -- 172.21.15.99:0/2007057843 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f091c0587d0 con 0x7f092415c9b0 2024-03-04T10:03:50.074 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f090406b500 msgr2=0x7f090406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f090406b500 0x7f090406d9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0914036000 tx=0x7f0914008030 comp rx=0 tx=0).stop 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 msgr2=0x7f092415a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 0x7f092415a8c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f091c00c6d0 tx=0x7f091c00cb90 comp rx=0 tx=0).stop 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 shutdown_connections 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f090406b500 0x7f090406d9c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f092415c9b0 0x7f092415a8c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09241450e0 0x7f092413f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 --2- 172.21.15.99:0/2007057843 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09240cc510 0x7f0924140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 >> 172.21.15.99:0/2007057843 conn(0x7f092406acd0 msgr2=0x7f09240b2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 shutdown_connections 2024-03-04T10:03:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.073+0000 7f0928d22640 1 -- 172.21.15.99:0/2007057843 wait complete. 2024-03-04T10:03:50.086 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:03:50.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 Processor -- start 2024-03-04T10:03:50.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 -- start start 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f95300cf660 0x7f95300cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f95301616e0 0x7f9530161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9530162450 con 0x7f953015da20 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95300d0000 con 0x7f95300cf660 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.187+0000 7f95367b3640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95300cc920 con 0x7f95301616e0 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f953015da20 0x7f95300ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45382/0 (socket says 172.21.15.99:45382) 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952ffff640 1 -- 172.21.15.99:0/455287677 learned_addr learned my addr 172.21.15.99:0/455287677 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952f7fe640 1 --2- 172.21.15.99:0/455287677 >> v2:172.21.15.99:3300/0 conn(0x7f95300cf660 0x7f95300cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952effd640 1 -- 172.21.15.99:0/455287677 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981853748 0 0) 0x7f9530162450 con 0x7f953015da20 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f9534d50640 1 --2- 172.21.15.99:0/455287677 >> v2:172.21.15.162:3300/0 conn(0x7f95301616e0 0x7f9530161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952effd640 1 -- 172.21.15.99:0/455287677 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9518003520 con 0x7f953015da20 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952f7fe640 1 -- 172.21.15.99:0/455287677 >> v2:172.21.15.162:3300/0 conn(0x7f95301616e0 msgr2=0x7f9530161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952f7fe640 1 --2- 172.21.15.99:0/455287677 >> v2:172.21.15.162:3300/0 conn(0x7f95301616e0 0x7f9530161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952f7fe640 1 -- 172.21.15.99:0/455287677 >> v1:172.21.15.99:6789/0 conn(0x7f953015da20 legacy=0x7f95300ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f952f7fe640 1 -- 172.21.15.99:0/455287677 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95300cc2d0 con 0x7f95300cf660 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.188+0000 7f9534d50640 1 --2- 172.21.15.99:0/455287677 >> v2:172.21.15.162:3300/0 conn(0x7f95301616e0 0x7f9530161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f952f7fe640 1 --2- 172.21.15.99:0/455287677 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95300cf660 0x7f95300cfac0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95300d2890 tx=0x7f952402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47df98313184617c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f952effd640 1 -- 172.21.15.99:0/455287677 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95240315a0 con 0x7f95300cf660 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f952effd640 1 -- 172.21.15.99:0/455287677 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9524037070 con 0x7f95300cf660 2024-03-04T10:03:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f952effd640 1 -- 172.21.15.99:0/455287677 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524031960 con 0x7f95300cf660 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 -- 172.21.15.99:0/455287677 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95300cf660 msgr2=0x7f95300cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 --2- 172.21.15.99:0/455287677 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95300cf660 0x7f95300cfac0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f95300d2890 tx=0x7f952402b840 comp rx=0 tx=0).stop 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 -- 172.21.15.99:0/455287677 shutdown_connections 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 --2- 172.21.15.99:0/455287677 >> v2:172.21.15.162:3300/0 conn(0x7f95301616e0 0x7f9530161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 --2- 172.21.15.99:0/455287677 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95300cf660 0x7f95300cfac0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 -- 172.21.15.99:0/455287677 >> 172.21.15.99:0/455287677 conn(0x7f95300b33a0 msgr2=0x7f95300b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 -- 172.21.15.99:0/455287677 shutdown_connections 2024-03-04T10:03:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.189+0000 7f95367b3640 1 -- 172.21.15.99:0/455287677 wait complete. 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.190+0000 7f95367b3640 1 Processor -- start 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 -- start start 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95300cf660 0x7f95301606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 0x7f9530154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95301616e0 0x7f9530154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953015de00 con 0x7f953015da20 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95300cbf80 con 0x7f95300cf660 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f95367b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95301620b0 con 0x7f95301616e0 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f952f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 0x7f9530154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f952f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 0x7f9530154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39022/0 (socket says 172.21.15.99:39022) 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.191+0000 7f952f7fe640 1 -- 172.21.15.99:0/2395320221 learned_addr learned my addr 172.21.15.99:0/2395320221 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f9534d50640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95301616e0 0x7f9530154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952ffff640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95300cf660 0x7f95301606b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952f7fe640 1 -- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95301616e0 msgr2=0x7f9530154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952f7fe640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95301616e0 0x7f9530154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952f7fe640 1 -- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95300cf660 msgr2=0x7f95301606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952f7fe640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95300cf660 0x7f95301606b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952f7fe640 1 -- 172.21.15.99:0/2395320221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95300c8a40 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952f7fe640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 0x7f9530154a50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9524000c00 tx=0x7f95240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524032630 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95240327f0 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95300c8d00 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9524032be0 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.192+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95300c92c0 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.193+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9530074f60 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.193+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f952404d3f0 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.194+0000 7f952cff9640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f951c06b4d0 0x7f951c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.194+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f952402f030 con 0x7f953015da20 2024-03-04T10:03:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.194+0000 7f952ffff640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f951c06b4d0 0x7f951c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.194+0000 7f952ffff640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f951c06b4d0 0x7f951c06d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9520005880 tx=0x7f9520009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.198+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f95240b2080 con 0x7f953015da20 2024-03-04T10:03:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.367+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f953006e620 con 0x7f953015da20 2024-03-04T10:03:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.367+0000 7f952cff9640 1 -- 172.21.15.99:0/2395320221 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9524040070 con 0x7f953015da20 2024-03-04T10:03:50.370 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f951c06b4d0 msgr2=0x7f951c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f951c06b4d0 0x7f951c06d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9520005880 tx=0x7f9520009040 comp rx=0 tx=0).stop 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 msgr2=0x7f9530154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 0x7f9530154a50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9524000c00 tx=0x7f95240046c0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 shutdown_connections 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f951c06b4d0 0x7f951c06d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95301616e0 0x7f9530154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f953015da20 0x7f9530154a50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 --2- 172.21.15.99:0/2395320221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95300cf660 0x7f95301606b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 >> 172.21.15.99:0/2395320221 conn(0x7f95300b33a0 msgr2=0x7f953006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 shutdown_connections 2024-03-04T10:03:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.369+0000 7f95367b3640 1 -- 172.21.15.99:0/2395320221 wait complete. 2024-03-04T10:03:50.382 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:03:50.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.480+0000 7f3eb2944640 1 Processor -- start 2024-03-04T10:03:50.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.480+0000 7f3eb2944640 1 -- start start 2024-03-04T10:03:50.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.481+0000 7f3eb2944640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 0x7f3eac0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.481+0000 7f3eb2944640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac15cf30 con 0x7f3eac0cf570 2024-03-04T10:03:50.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.481+0000 7f3eb2944640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac15c170 con 0x7f3eac0cfa30 2024-03-04T10:03:50.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.481+0000 7f3eb2944640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac15caa0 con 0x7f3eac15bc40 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.481+0000 7f3eab7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3eac0cf570 0x7f3eac0cc260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45398/0 (socket says 172.21.15.99:45398) 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.481+0000 7f3eab7fe640 1 -- 172.21.15.99:0/3452924468 learned_addr learned my addr 172.21.15.99:0/3452924468 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 0x7f3eac0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302168034 0 0) 0x7f3eac15cf30 con 0x7f3eac0cf570 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e94003520 con 0x7f3eac0cf570 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674693974 0 0) 0x7f3e94003520 con 0x7f3eac0cf570 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 msgr2=0x7f3eac0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 0x7f3eac0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 >> v1:172.21.15.162:6789/0 conn(0x7f3eac0cfa30 legacy=0x7f3eac0cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac15c450 con 0x7f3eac0cf570 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3ea0001670 con 0x7f3eac0cf570 2024-03-04T10:03:50.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 >> v1:172.21.15.99:6789/0 conn(0x7f3eac0cf570 legacy=0x7f3eac0cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 0x7f3e940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e94003f60 0x7f3e94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3e94004d40 0x7f3e940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac15caa0 con 0x7f3e94004d40 2024-03-04T10:03:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9400baa0 con 0x7f3e94003f60 2024-03-04T10:03:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9400bca0 con 0x7f3eac15bc40 2024-03-04T10:03:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3e94004d40 0x7f3e940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eab7fe640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e94003f60 0x7f3e94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eabfff640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 0x7f3e940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 -- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 msgr2=0x7f3e940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac15bc40 0x7f3e940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 -- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e94003f60 msgr2=0x7f3e94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:50.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3e94003f60 0x7f3e94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.482+0000 7f3eb0ee1640 1 -- 172.21.15.99:0/3452924468 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e94003520 con 0x7f3e94004d40 2024-03-04T10:03:50.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.483+0000 7f3eb0ee1640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3e94004d40 0x7f3e940051c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009f90 tx=0x7f3e94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae00998f1ac35002 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.483+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4039070 con 0x7f3e94004d40 2024-03-04T10:03:50.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.483+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea402cbd0 con 0x7f3e94004d40 2024-03-04T10:03:50.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.483+0000 7f3eaaffd640 1 -- 172.21.15.99:0/3452924468 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4009c10 con 0x7f3e94004d40 2024-03-04T10:03:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb2944640 1 -- 172.21.15.99:0/3452924468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3e94004d40 msgr2=0x7f3e940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb2944640 1 --2- 172.21.15.99:0/3452924468 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3e94004d40 0x7f3e940051c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009f90 tx=0x7f3e94005830 comp rx=0 tx=0).stop 2024-03-04T10:03:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb0ee1640 1 -- 172.21.15.99:0/3452924468 reap_dead start 2024-03-04T10:03:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb2944640 1 -- 172.21.15.99:0/3452924468 shutdown_connections 2024-03-04T10:03:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb2944640 1 -- 172.21.15.99:0/3452924468 >> 172.21.15.99:0/3452924468 conn(0x7f3eac0d0780 msgr2=0x7f3eac0d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb2944640 1 -- 172.21.15.99:0/3452924468 shutdown_connections 2024-03-04T10:03:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.484+0000 7f3eb2944640 1 -- 172.21.15.99:0/3452924468 wait complete. 2024-03-04T10:03:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.485+0000 7f3eb2944640 1 Processor -- start 2024-03-04T10:03:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.485+0000 7f3eb2944640 1 -- start start 2024-03-04T10:03:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb2944640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac0b5c30 0x7f3eac0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb2944640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 0x7f3eac151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb2944640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eac152090 0x7f3eac0b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb2944640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac06acd0 con 0x7f3eac06ef30 2024-03-04T10:03:50.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb2944640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac0c75a0 con 0x7f3eac152090 2024-03-04T10:03:50.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb2944640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac0c3f30 con 0x7f3eac0b5c30 2024-03-04T10:03:50.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 0x7f3eac151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 0x7f3eac151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39036/0 (socket says 172.21.15.99:39036) 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 -- 172.21.15.99:0/2600478987 learned_addr learned my addr 172.21.15.99:0/2600478987 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eabfff640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac0b5c30 0x7f3eac0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eb0ee1640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eac152090 0x7f3eac0b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 -- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac0b5c30 msgr2=0x7f3eac0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac0b5c30 0x7f3eac0b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 -- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eac152090 msgr2=0x7f3eac0b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eac152090 0x7f3eac0b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.486+0000 7f3eab7fe640 1 -- 172.21.15.99:0/2600478987 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac06f780 con 0x7f3eac06ef30 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.487+0000 7f3eab7fe640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 0x7f3eac151b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00072c0 tx=0x7f3ea0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.487+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000f9a0 con 0x7f3eac06ef30 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.487+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0018070 con 0x7f3eac06ef30 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.487+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eac06fa60 con 0x7f3eac06ef30 2024-03-04T10:03:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.487+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000fd80 con 0x7f3eac06ef30 2024-03-04T10:03:50.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.487+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eac06f3b0 con 0x7f3eac06ef30 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.488+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eac06cd70 con 0x7f3eac06ef30 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.488+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3ea001d460 con 0x7f3eac06ef30 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.489+0000 7f3ea8ff9640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e9806b500 0x7f3e9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.489+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3ea008a440 con 0x7f3eac06ef30 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.489+0000 7f3eabfff640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e9806b500 0x7f3e9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.489+0000 7f3eabfff640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e9806b500 0x7f3e9806d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3eac15f030 tx=0x7f3e9c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.493+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ea00589c0 con 0x7f3eac06ef30 2024-03-04T10:03:50.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.677+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3eac066e60 con 0x7f3eac06ef30 2024-03-04T10:03:50.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.677+0000 7f3ea8ff9640 1 -- 172.21.15.99:0/2600478987 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3eac066e60 con 0x7f3eac06ef30 2024-03-04T10:03:50.680 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:03:50.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e9806b500 msgr2=0x7f3e9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e9806b500 0x7f3e9806d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3eac15f030 tx=0x7f3e9c009040 comp rx=0 tx=0).stop 2024-03-04T10:03:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 msgr2=0x7f3eac151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 0x7f3eac151b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00072c0 tx=0x7f3ea0007780 comp rx=0 tx=0).stop 2024-03-04T10:03:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 shutdown_connections 2024-03-04T10:03:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e9806b500 0x7f3e9806d9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3eac152090 0x7f3eac0b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3eac06ef30 0x7f3eac151b50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 --2- 172.21.15.99:0/2600478987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3eac0b5c30 0x7f3eac0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 >> 172.21.15.99:0/2600478987 conn(0x7f3eac0d0780 msgr2=0x7f3eac15ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:50.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 shutdown_connections 2024-03-04T10:03:50.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.679+0000 7f3eb2944640 1 -- 172.21.15.99:0/2600478987 wait complete. 2024-03-04T10:03:50.692 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:03:50.692 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:03:50.709 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:50.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.846+0000 7fa2b8aea640 1 Processor -- start 2024-03-04T10:03:50.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.846+0000 7fa2b8aea640 1 -- start start 2024-03-04T10:03:50.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b8aea640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2b415de00 0x7fa2b40b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b8aea640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b415e1e0 con 0x7fa2b40b9380 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b8aea640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40d1c10 con 0x7fa2b40b5470 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b8aea640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b40d2820 con 0x7fa2b415de00 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2b415de00 0x7fa2b40b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2b415de00 0x7fa2b40b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39046/0 (socket says 172.21.15.99:39046) 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.847+0000 7fa2b259c640 1 -- 172.21.15.99:0/1522789352 learned_addr learned my addr 172.21.15.99:0/1522789352 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b259c640 1 -- 172.21.15.99:0/1522789352 >> v1:172.21.15.162:6789/0 conn(0x7fa2b40b5470 legacy=0x7fa2b40d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b259c640 1 -- 172.21.15.99:0/1522789352 >> v1:172.21.15.99:6789/0 conn(0x7fa2b40b9380 legacy=0x7fa2b40b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b259c640 1 -- 172.21.15.99:0/1522789352 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40d1ef0 con 0x7fa2b415de00 2024-03-04T10:03:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b159a640 1 -- 172.21.15.99:0/1522789352 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533242060 0 0) 0x7fa2b415e1e0 con 0x7fa2b40b9380 2024-03-04T10:03:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b259c640 1 --2- 172.21.15.99:0/1522789352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415de00 0x7fa2b40b2e60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009e50 tx=0x7fa2a402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61bd624e2a39daf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b159a640 1 -- 172.21.15.99:0/1522789352 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a402cb20 con 0x7fa2b415de00 2024-03-04T10:03:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b159a640 1 -- 172.21.15.99:0/1522789352 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a402ccc0 con 0x7fa2b415de00 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.848+0000 7fa2b159a640 1 -- 172.21.15.99:0/1522789352 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4034610 con 0x7fa2b415de00 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1522789352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415de00 msgr2=0x7fa2b40b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1522789352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415de00 0x7fa2b40b2e60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4009e50 tx=0x7fa2a402bf80 comp rx=0 tx=0).stop 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1522789352 shutdown_connections 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1522789352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415de00 0x7fa2b40b2e60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1522789352 >> 172.21.15.99:0/1522789352 conn(0x7fa2b40d0470 msgr2=0x7fa2b40d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1522789352 shutdown_connections 2024-03-04T10:03:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.849+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1522789352 wait complete. 2024-03-04T10:03:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.850+0000 7fa2b8aea640 1 Processor -- start 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.850+0000 7fa2b8aea640 1 -- start start 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b8aea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40b5470 0x7fa2b41f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b8aea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 0x7fa2b40ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b8aea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b415de00 0x7fa2b40cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b8aea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40b97e0 con 0x7fa2b40b9380 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b8aea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40d24b0 con 0x7fa2b415de00 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b8aea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40b72a0 con 0x7fa2b40b5470 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b1d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 0x7fa2b40ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b1d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 0x7fa2b40ccad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39058/0 (socket says 172.21.15.99:39058) 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b1d9b640 1 -- 172.21.15.99:0/1928515381 learned_addr learned my addr 172.21.15.99:0/1928515381 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b259c640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40b5470 0x7fa2b41f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.851+0000 7fa2b2d9d640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b415de00 0x7fa2b40cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b1d9b640 1 -- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40b5470 msgr2=0x7fa2b41f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b1d9b640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40b5470 0x7fa2b41f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b1d9b640 1 -- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b415de00 msgr2=0x7fa2b40cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b1d9b640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b415de00 0x7fa2b40cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b1d9b640 1 -- 172.21.15.99:0/1928515381 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b415ff20 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b1d9b640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 0x7fa2b40ccad0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c460 tx=0x7fa2a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8005480 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a80035f0 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8003a00 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b4160200 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.852+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b41607c0 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.853+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b4074f60 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.853+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa2a8003ba0 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.854+0000 7fa2a2ffd640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa29406b670 0x7fa29406db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.854+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa2a8089ff0 con 0x7fa2b40b9380 2024-03-04T10:03:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.854+0000 7fa2b259c640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa29406b670 0x7fa29406db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.855+0000 7fa2b259c640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa29406b670 0x7fa29406db30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4035000 tx=0x7fa2a4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:50.858+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa2a80584e0 con 0x7fa2b40b9380 2024-03-04T10:03:51.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.029+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2b40d09d0 con 0x7fa2b40b9380 2024-03-04T10:03:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.030+0000 7fa2a2ffd640 1 -- 172.21.15.99:0/1928515381 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7fa2a805c580 con 0x7fa2b40b9380 2024-03-04T10:03:51.033 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:51.033 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa29406b670 msgr2=0x7fa29406db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa29406b670 0x7fa29406db30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4035000 tx=0x7fa2a4030000 comp rx=0 tx=0).stop 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 msgr2=0x7fa2b40ccad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 0x7fa2b40ccad0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c460 tx=0x7fa2a800c920 comp rx=0 tx=0).stop 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 shutdown_connections 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa29406b670 0x7fa29406db30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b415de00 0x7fa2b40cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b40b9380 0x7fa2b40ccad0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 --2- 172.21.15.99:0/1928515381 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40b5470 0x7fa2b41f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 >> 172.21.15.99:0/1928515381 conn(0x7fa2b40d0470 msgr2=0x7fa2b40d2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 shutdown_connections 2024-03-04T10:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.032+0000 7fa2b8aea640 1 -- 172.21.15.99:0/1928515381 wait complete. 2024-03-04T10:03:51.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:03:51.049 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:03:51.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.150+0000 7fc4212f1640 1 Processor -- start 2024-03-04T10:03:51.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.150+0000 7fc4212f1640 1 -- start start 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc4212f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc41c15da20 0x7fc41c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc4212f1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc41c0cf660 0x7fc41c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc4212f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c1616e0 0x7fc41c161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc4212f1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c0d0000 con 0x7fc41c0cf660 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc4212f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c0cc920 con 0x7fc41c15da20 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc4212f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c162450 con 0x7fc41c1616e0 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc41c0cf660 0x7fc41c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc41c0cf660 0x7fc41c0cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39062/0 (socket says 172.21.15.99:39062) 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 -- 172.21.15.99:0/2516770364 learned_addr learned my addr 172.21.15.99:0/2516770364 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc41b7fe640 1 --2- 172.21.15.99:0/2516770364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c1616e0 0x7fc41c161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc41affd640 1 --2- 172.21.15.99:0/2516770364 >> v2:172.21.15.162:3300/0 conn(0x7fc41c15da20 0x7fc41c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 -- 172.21.15.99:0/2516770364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c1616e0 msgr2=0x7fc41c161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 --2- 172.21.15.99:0/2516770364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c1616e0 0x7fc41c161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 -- 172.21.15.99:0/2516770364 >> v2:172.21.15.162:3300/0 conn(0x7fc41c15da20 msgr2=0x7fc41c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 --2- 172.21.15.99:0/2516770364 >> v2:172.21.15.162:3300/0 conn(0x7fc41c15da20 0x7fc41c0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.151+0000 7fc413fff640 1 -- 172.21.15.99:0/2516770364 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0cc2d0 con 0x7fc41c0cf660 2024-03-04T10:03:51.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.152+0000 7fc413fff640 1 --2- 172.21.15.99:0/2516770364 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c0cf660 0x7fc41c0cfac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0cf310 tx=0x7fc40402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0e826b94ef6403 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.152+0000 7fc41a7fc640 1 -- 172.21.15.99:0/2516770364 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404035020 con 0x7fc41c0cf660 2024-03-04T10:03:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.152+0000 7fc41a7fc640 1 -- 172.21.15.99:0/2516770364 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc404031870 con 0x7fc41c0cf660 2024-03-04T10:03:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.152+0000 7fc41a7fc640 1 -- 172.21.15.99:0/2516770364 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404031c40 con 0x7fc41c0cf660 2024-03-04T10:03:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 -- 172.21.15.99:0/2516770364 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c0cf660 msgr2=0x7fc41c0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 --2- 172.21.15.99:0/2516770364 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c0cf660 0x7fc41c0cfac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc41c0cf310 tx=0x7fc40402b910 comp rx=0 tx=0).stop 2024-03-04T10:03:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 -- 172.21.15.99:0/2516770364 shutdown_connections 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 --2- 172.21.15.99:0/2516770364 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c1616e0 0x7fc41c161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 --2- 172.21.15.99:0/2516770364 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c0cf660 0x7fc41c0cfac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 --2- 172.21.15.99:0/2516770364 >> v2:172.21.15.162:3300/0 conn(0x7fc41c15da20 0x7fc41c0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 -- 172.21.15.99:0/2516770364 >> 172.21.15.99:0/2516770364 conn(0x7fc41c0b33a0 msgr2=0x7fc41c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 -- 172.21.15.99:0/2516770364 shutdown_connections 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.153+0000 7fc4212f1640 1 -- 172.21.15.99:0/2516770364 wait complete. 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.154+0000 7fc4212f1640 1 Processor -- start 2024-03-04T10:03:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.154+0000 7fc4212f1640 1 -- start start 2024-03-04T10:03:51.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc41c0cf660 0x7fc41c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c15da20 0x7fc41c0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 0x7fc41c0cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0cbf80 con 0x7fc41c1616e0 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0cd2f0 con 0x7fc41c0cf660 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c15de00 con 0x7fc41c15da20 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 0x7fc41c0cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 0x7fc41c0cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39068/0 (socket says 172.21.15.99:39068) 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 -- 172.21.15.99:0/3251966903 learned_addr learned my addr 172.21.15.99:0/3251966903 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41affd640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc41c0cf660 0x7fc41c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc413fff640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c15da20 0x7fc41c0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 -- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c15da20 msgr2=0x7fc41c0cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c15da20 0x7fc41c0cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 -- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc41c0cf660 msgr2=0x7fc41c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc41c0cf660 0x7fc41c15af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 -- 172.21.15.99:0/3251966903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c0adca0 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41affd640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc41c0cf660 0x7fc41c15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc413fff640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c15da20 0x7fc41c0cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc41b7fe640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 0x7fc41c0cd9b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc414007e60 tx=0x7fc41400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414014420 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414019070 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414014820 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.155+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c0adf80 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.156+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c0ae540 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.156+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c074f60 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.157+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc4140149c0 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.157+0000 7fc4137fe640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc3fc06b4d0 0x7fc3fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.157+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc41408ad20 con 0x7fc41c1616e0 2024-03-04T10:03:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.158+0000 7fc41affd640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc3fc06b4d0 0x7fc3fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.158+0000 7fc41affd640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc3fc06b4d0 0x7fc3fc06d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc40c006fd0 tx=0x7fc40c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.162+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc414059860 con 0x7fc41c1616e0 2024-03-04T10:03:51.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.329+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc41c15c340 con 0x7fc41c1616e0 2024-03-04T10:03:51.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.330+0000 7fc4137fe640 1 -- 172.21.15.99:0/3251966903 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc414059160 con 0x7fc41c1616e0 2024-03-04T10:03:51.332 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.331+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc3fc06b4d0 msgr2=0x7fc3fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.331+0000 7fc4212f1640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc3fc06b4d0 0x7fc3fc06d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc40c006fd0 tx=0x7fc40c008040 comp rx=0 tx=0).stop 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.331+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 msgr2=0x7fc41c0cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.331+0000 7fc4212f1640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 0x7fc41c0cd9b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc414007e60 tx=0x7fc41400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 shutdown_connections 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc3fc06b4d0 0x7fc3fc06d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc41c1616e0 0x7fc41c0cd9b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc41c15da20 0x7fc41c0cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 --2- 172.21.15.99:0/3251966903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc41c0cf660 0x7fc41c15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 >> 172.21.15.99:0/3251966903 conn(0x7fc41c0b33a0 msgr2=0x7fc41c0b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 shutdown_connections 2024-03-04T10:03:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.332+0000 7fc4212f1640 1 -- 172.21.15.99:0/3251966903 wait complete. 2024-03-04T10:03:51.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:03:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.445+0000 7faaecbcd640 1 Processor -- start 2024-03-04T10:03:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.445+0000 7faaecbcd640 1 -- start start 2024-03-04T10:03:51.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faaecbcd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7faae80d34a0 0x7faae8161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faaecbcd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7faae80b9410 0x7faae8162120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faaecbcd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae80d2400 con 0x7faae80b5700 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faaecbcd640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae80d26e0 con 0x7faae80b9410 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faaecbcd640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae80d29c0 con 0x7faae80d34a0 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faae659c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7faae80b5700 0x7faae80d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45424/0 (socket says 172.21.15.99:45424) 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faae659c640 1 -- 172.21.15.99:0/472496361 learned_addr learned my addr 172.21.15.99:0/472496361 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faae6d9d640 1 --2- 172.21.15.99:0/472496361 >> v2:172.21.15.99:3300/0 conn(0x7faae80b9410 0x7faae8162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.446+0000 7faae5d9b640 1 --2- 172.21.15.99:0/472496361 >> v2:172.21.15.162:3300/0 conn(0x7faae80d34a0 0x7faae8161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae6d9d640 1 -- 172.21.15.99:0/472496361 >> v2:172.21.15.162:3300/0 conn(0x7faae80d34a0 msgr2=0x7faae8161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae6d9d640 1 --2- 172.21.15.99:0/472496361 >> v2:172.21.15.162:3300/0 conn(0x7faae80d34a0 0x7faae8161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae6d9d640 1 -- 172.21.15.99:0/472496361 >> v1:172.21.15.99:6789/0 conn(0x7faae80b5700 legacy=0x7faae80d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae559a640 1 -- 172.21.15.99:0/472496361 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234695175 0 0) 0x7faae80d2400 con 0x7faae80b5700 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae6d9d640 1 -- 172.21.15.99:0/472496361 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae80d2ca0 con 0x7faae80b9410 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae5d9b640 1 --2- 172.21.15.99:0/472496361 >> v2:172.21.15.162:3300/0 conn(0x7faae80d34a0 0x7faae8161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae6d9d640 1 --2- 172.21.15.99:0/472496361 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80b9410 0x7faae8162120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faae80c7470 tx=0x7faae002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2cd8f5a1fe61c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae559a640 1 -- 172.21.15.99:0/472496361 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0039070 con 0x7faae80b9410 2024-03-04T10:03:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae559a640 1 -- 172.21.15.99:0/472496361 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae002cbf0 con 0x7faae80b9410 2024-03-04T10:03:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.447+0000 7faae559a640 1 -- 172.21.15.99:0/472496361 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0034440 con 0x7faae80b9410 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 -- 172.21.15.99:0/472496361 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80b9410 msgr2=0x7faae8162120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 --2- 172.21.15.99:0/472496361 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80b9410 0x7faae8162120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faae80c7470 tx=0x7faae002be40 comp rx=0 tx=0).stop 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 -- 172.21.15.99:0/472496361 shutdown_connections 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 --2- 172.21.15.99:0/472496361 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80b9410 0x7faae8162120 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 --2- 172.21.15.99:0/472496361 >> v2:172.21.15.162:3300/0 conn(0x7faae80d34a0 0x7faae8161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 -- 172.21.15.99:0/472496361 >> 172.21.15.99:0/472496361 conn(0x7faae80d0490 msgr2=0x7faae80d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 -- 172.21.15.99:0/472496361 shutdown_connections 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.448+0000 7faaecbcd640 1 -- 172.21.15.99:0/472496361 wait complete. 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.449+0000 7faaecbcd640 1 Processor -- start 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.449+0000 7faaecbcd640 1 -- start start 2024-03-04T10:03:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faaecbcd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faae80b5700 0x7faae815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faaecbcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faae80b9410 0x7faae80cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faaecbcd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 0x7faae815f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faaecbcd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8162660 con 0x7faae80d34a0 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faaecbcd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae80b5cf0 con 0x7faae80b9410 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faaecbcd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae81627e0 con 0x7faae80b5700 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faae6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 0x7faae815f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faae6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 0x7faae815f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39094/0 (socket says 172.21.15.99:39094) 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faae6d9d640 1 -- 172.21.15.99:0/668638631 learned_addr learned my addr 172.21.15.99:0/668638631 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faae659c640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faae80b5700 0x7faae815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.450+0000 7faae5d9b640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faae80b9410 0x7faae80cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae6d9d640 1 -- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faae80b5700 msgr2=0x7faae815e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae6d9d640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faae80b5700 0x7faae815e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae6d9d640 1 -- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faae80b9410 msgr2=0x7faae80cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae6d9d640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faae80b9410 0x7faae80cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae6d9d640 1 -- 172.21.15.99:0/668638631 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae80cae30 con 0x7faae80d34a0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae5d9b640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faae80b9410 0x7faae80cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faae6d9d640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 0x7faae815f7f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faae0037040 tx=0x7faae0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae0039050 con 0x7faae80d34a0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae0045070 con 0x7faae80d34a0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae003e830 con 0x7faae80d34a0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae80cb110 con 0x7faae80d34a0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.451+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae80cb6d0 con 0x7faae80d34a0 2024-03-04T10:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.452+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae8074f60 con 0x7faae80d34a0 2024-03-04T10:03:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.452+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7faae003e9d0 con 0x7faae80d34a0 2024-03-04T10:03:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.453+0000 7faad6ffd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faac806b4d0 0x7faac806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.453+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7faae00ad2e0 con 0x7faae80d34a0 2024-03-04T10:03:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.453+0000 7faae659c640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faac806b4d0 0x7faac806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.453+0000 7faae659c640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faac806b4d0 0x7faac806d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faad8000f60 tx=0x7faad8007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.457+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faae007bf40 con 0x7faae80d34a0 2024-03-04T10:03:51.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.633+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faae806b1b0 con 0x7faae80d34a0 2024-03-04T10:03:51.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.634+0000 7faad6ffd640 1 -- 172.21.15.99:0/668638631 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7faae007b840 con 0x7faae80d34a0 2024-03-04T10:03:51.636 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:03:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faac806b4d0 msgr2=0x7faac806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faac806b4d0 0x7faac806d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faad8000f60 tx=0x7faad8007c00 comp rx=0 tx=0).stop 2024-03-04T10:03:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 msgr2=0x7faae815f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 0x7faae815f7f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faae0037040 tx=0x7faae0004a40 comp rx=0 tx=0).stop 2024-03-04T10:03:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 shutdown_connections 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faac806b4d0 0x7faac806d990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faae80d34a0 0x7faae815f7f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faae80b9410 0x7faae80cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 --2- 172.21.15.99:0/668638631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faae80b5700 0x7faae815e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 >> 172.21.15.99:0/668638631 conn(0x7faae80d0490 msgr2=0x7faae80b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 shutdown_connections 2024-03-04T10:03:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.636+0000 7faaecbcd640 1 -- 172.21.15.99:0/668638631 wait complete. 2024-03-04T10:03:51.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:03:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.749+0000 7f7d4f7e1640 1 Processor -- start 2024-03-04T10:03:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.750+0000 7f7d4f7e1640 1 -- start start 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.750+0000 7f7d4f7e1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7d4815bc50 0x7f7d480cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.750+0000 7f7d4f7e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 0x7f7d480cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.750+0000 7f7d4f7e1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d480cc7a0 con 0x7f7d480cdd80 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.750+0000 7f7d4f7e1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480cca80 con 0x7f7d4815bc50 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.750+0000 7f7d4f7e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480ccd60 con 0x7f7d480adc40 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4cd7c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7d480cdd80 0x7f7d480ad530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45426/0 (socket says 172.21.15.99:45426) 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4cd7c640 1 -- 172.21.15.99:0/1610767240 learned_addr learned my addr 172.21.15.99:0/1610767240 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4dd7e640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 0x7f7d480cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728483277 0 0) 0x7f7d480cc7a0 con 0x7f7d480cdd80 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/1610767240 >> v2:172.21.15.162:3300/0 conn(0x7f7d4815bc50 0x7f7d480cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d34003520 con 0x7f7d480cdd80 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 214277404 0 0) 0x7f7d34003520 con 0x7f7d480cdd80 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 msgr2=0x7f7d480cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 0x7f7d480cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 >> v2:172.21.15.162:3300/0 conn(0x7f7d4815bc50 msgr2=0x7f7d480cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 --2- 172.21.15.99:0/1610767240 >> v2:172.21.15.162:3300/0 conn(0x7f7d4815bc50 0x7f7d480cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480cd040 con 0x7f7d480cdd80 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7d38001670 con 0x7f7d480cdd80 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/1610767240 >> v2:172.21.15.162:3300/0 conn(0x7f7d4815bc50 0x7f7d480cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 >> v1:172.21.15.99:6789/0 conn(0x7f7d480cdd80 legacy=0x7f7d480ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d340039b0 0x7f7d34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 0x7f7d34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d34004d40 0x7f7d340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480cca80 con 0x7f7d340039b0 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480ccd60 con 0x7f7d34004d40 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3400b9f0 con 0x7f7d480adc40 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d340039b0 0x7f7d34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4cd7c640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 0x7f7d34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.751+0000 7f7d4dd7e640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d34004d40 0x7f7d340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d4d57d640 1 -- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 msgr2=0x7f7d34004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d480adc40 0x7f7d34004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d4d57d640 1 -- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d34004d40 msgr2=0x7f7d340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d34004d40 0x7f7d340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d4d57d640 1 -- 172.21.15.99:0/1610767240 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34003520 con 0x7f7d340039b0 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d340039b0 0x7f7d34003dd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d440098a0 tx=0x7f7d34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd26f42f9c3afbaf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4400f040 con 0x7f7d340039b0 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d44004230 con 0x7f7d340039b0 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.752+0000 7f7d3ffff640 1 -- 172.21.15.99:0/1610767240 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44003610 con 0x7f7d340039b0 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/1610767240 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d340039b0 msgr2=0x7f7d34003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/1610767240 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d340039b0 0x7f7d34003dd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d440098a0 tx=0x7f7d34005830 comp rx=0 tx=0).stop 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4dd7e640 1 -- 172.21.15.99:0/1610767240 reap_dead start 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/1610767240 shutdown_connections 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/1610767240 >> 172.21.15.99:0/1610767240 conn(0x7f7d480b2250 msgr2=0x7f7d480b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/1610767240 shutdown_connections 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.753+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/1610767240 wait complete. 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.754+0000 7f7d4f7e1640 1 Processor -- start 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.754+0000 7f7d4f7e1640 1 -- start start 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4f7e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d340039b0 0x7f7d480ca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4f7e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d4815ed80 0x7f7d4814eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4f7e1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 0x7f7d480c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4f7e1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480cc4b0 con 0x7f7d481f1880 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4f7e1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480cc330 con 0x7f7d4815ed80 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4f7e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4806acd0 con 0x7f7d340039b0 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 0x7f7d480c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 0x7f7d480c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39120/0 (socket says 172.21.15.99:39120) 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 -- 172.21.15.99:0/2206162750 learned_addr learned my addr 172.21.15.99:0/2206162750 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d340039b0 0x7f7d480ca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4cd7c640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d4815ed80 0x7f7d4814eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 -- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d340039b0 msgr2=0x7f7d480ca3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d340039b0 0x7f7d480ca3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 -- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d4815ed80 msgr2=0x7f7d4814eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d4815ed80 0x7f7d4814eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 -- 172.21.15.99:0/2206162750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4814f050 con 0x7f7d481f1880 2024-03-04T10:03:51.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4cd7c640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d4815ed80 0x7f7d4814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:03:51.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.755+0000 7f7d4dd7e640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 0x7f7d480c97e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d4806cc70 tx=0x7f7d4000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.756+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4000fde0 con 0x7f7d481f1880 2024-03-04T10:03:51.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.756+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4001d070 con 0x7f7d481f1880 2024-03-04T10:03:51.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.756+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d480b4cf0 con 0x7f7d481f1880 2024-03-04T10:03:51.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.756+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d40015670 con 0x7f7d481f1880 2024-03-04T10:03:51.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.756+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d480b52b0 con 0x7f7d481f1880 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.757+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4806cd70 con 0x7f7d481f1880 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.757+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7d40015810 con 0x7f7d481f1880 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.758+0000 7f7d3dffb640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d2c06b4d0 0x7f7d2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.758+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7d4008aee0 con 0x7f7d481f1880 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.758+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d2c06b4d0 0x7f7d2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.758+0000 7f7d4d57d640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d2c06b4d0 0x7f7d2c06d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d440098a0 tx=0x7f7d44039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.762+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7d40010040 con 0x7f7d481f1880 2024-03-04T10:03:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.934+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d48074f60 con 0x7f7d481f1880 2024-03-04T10:03:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.934+0000 7f7d3dffb640 1 -- 172.21.15.99:0/2206162750 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d40059a20 con 0x7f7d481f1880 2024-03-04T10:03:51.937 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:03:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d2c06b4d0 msgr2=0x7f7d2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d2c06b4d0 0x7f7d2c06d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d440098a0 tx=0x7f7d44039040 comp rx=0 tx=0).stop 2024-03-04T10:03:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 msgr2=0x7f7d480c97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 0x7f7d480c97e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7d4806cc70 tx=0x7f7d4000f580 comp rx=0 tx=0).stop 2024-03-04T10:03:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 shutdown_connections 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d2c06b4d0 0x7f7d2c06d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d481f1880 0x7f7d480c97e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d4815ed80 0x7f7d4814eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 --2- 172.21.15.99:0/2206162750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d340039b0 0x7f7d480ca3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 >> 172.21.15.99:0/2206162750 conn(0x7f7d480b2250 msgr2=0x7f7d480b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 shutdown_connections 2024-03-04T10:03:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:51.936+0000 7f7d4f7e1640 1 -- 172.21.15.99:0/2206162750 wait complete. 2024-03-04T10:03:51.948 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:03:52.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.049+0000 7f3b2497e640 1 Processor -- start 2024-03-04T10:03:52.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.049+0000 7f3b2497e640 1 -- start start 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b2497e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b200b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b2497e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3b200cf020 0x7f3b200d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b2497e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b200ca970 con 0x7f3b200cec40 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b2497e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b20161d90 con 0x7f3b200cf020 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b2497e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b20162070 con 0x7f3b2015de40 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3b200cf020 0x7f3b200d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3b200cf020 0x7f3b200d2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39134/0 (socket says 172.21.15.99:39134) 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1e59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3b200cec40 0x7f3b2015d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45430/0 (socket says 172.21.15.99:45430) 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 -- 172.21.15.99:0/3077319781 learned_addr learned my addr 172.21.15.99:0/3077319781 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1dd9b640 1 --2- 172.21.15.99:0/3077319781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b200b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 -- 172.21.15.99:0/3077319781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 msgr2=0x7f3b200b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 --2- 172.21.15.99:0/3077319781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b200b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 -- 172.21.15.99:0/3077319781 >> v1:172.21.15.99:6789/0 conn(0x7f3b200cec40 legacy=0x7f3b2015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1ed9d640 1 -- 172.21.15.99:0/3077319781 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20162350 con 0x7f3b200cf020 2024-03-04T10:03:52.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.050+0000 7f3b1d59a640 1 -- 172.21.15.99:0/3077319781 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471657708 0 0) 0x7f3b200ca970 con 0x7f3b200cec40 2024-03-04T10:03:52.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.051+0000 7f3b1ed9d640 1 --2- 172.21.15.99:0/3077319781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cf020 0x7f3b200d2110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea33b65229426c12 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:52.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.051+0000 7f3b1d59a640 1 -- 172.21.15.99:0/3077319781 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18034070 con 0x7f3b200cf020 2024-03-04T10:03:52.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.051+0000 7f3b1d59a640 1 -- 172.21.15.99:0/3077319781 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b18030840 con 0x7f3b200cf020 2024-03-04T10:03:52.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.051+0000 7f3b1d59a640 1 -- 172.21.15.99:0/3077319781 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18030c10 con 0x7f3b200cf020 2024-03-04T10:03:52.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 -- 172.21.15.99:0/3077319781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cf020 msgr2=0x7f3b200d2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 --2- 172.21.15.99:0/3077319781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cf020 0x7f3b200d2110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1802b980 comp rx=0 tx=0).stop 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 -- 172.21.15.99:0/3077319781 shutdown_connections 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 --2- 172.21.15.99:0/3077319781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cf020 0x7f3b200d2110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 --2- 172.21.15.99:0/3077319781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b200b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 -- 172.21.15.99:0/3077319781 >> 172.21.15.99:0/3077319781 conn(0x7f3b200d0580 msgr2=0x7f3b200d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 -- 172.21.15.99:0/3077319781 shutdown_connections 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.052+0000 7f3b2497e640 1 -- 172.21.15.99:0/3077319781 wait complete. 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.053+0000 7f3b2497e640 1 Processor -- start 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.053+0000 7f3b2497e640 1 -- start start 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b2497e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 0x7f3b200c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b2497e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b200cf020 0x7f3b2015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b2497e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b20154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b2497e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200d3300 con 0x7f3b200cec40 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b2497e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200d1bf0 con 0x7f3b200cf020 2024-03-04T10:03:52.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b2497e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200d3480 con 0x7f3b2015de40 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 0x7f3b200c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 0x7f3b200c9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39144/0 (socket says 172.21.15.99:39144) 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 -- 172.21.15.99:0/2760749700 learned_addr learned my addr 172.21.15.99:0/2760749700 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1ed9d640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b20154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1dd9b640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b200cf020 0x7f3b2015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 -- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 msgr2=0x7f3b20154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b20154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 -- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b200cf020 msgr2=0x7f3b2015e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b200cf020 0x7f3b2015e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.054+0000 7f3b1e59c640 1 -- 172.21.15.99:0/2760749700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b200c9f80 con 0x7f3b200cec40 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.055+0000 7f3b1e59c640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 0x7f3b200c9a40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c820 tx=0x7f3b1000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.055+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10017070 con 0x7f3b200cec40 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.055+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1000a640 con 0x7f3b200cec40 2024-03-04T10:03:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.055+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2015eb20 con 0x7f3b200cec40 2024-03-04T10:03:52.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.055+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1000aa30 con 0x7f3b200cec40 2024-03-04T10:03:52.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.055+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2015f0e0 con 0x7f3b200cec40 2024-03-04T10:03:52.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.056+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b20074f60 con 0x7f3b200cec40 2024-03-04T10:03:52.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.056+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3b1001a070 con 0x7f3b200cec40 2024-03-04T10:03:52.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.057+0000 7f3b0effd640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b0006b4d0 0x7f3b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.057+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3b10089b70 con 0x7f3b200cec40 2024-03-04T10:03:52.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.057+0000 7f3b1dd9b640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b0006b4d0 0x7f3b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.057+0000 7f3b1dd9b640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b0006b4d0 0x7f3b0006d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3b2006c200 tx=0x7f3b14009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.061+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3b200c9f80 con 0x7f3b200cec40 2024-03-04T10:03:52.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.230+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3b200b6650 con 0x7f3b200cec40 2024-03-04T10:03:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.230+0000 7f3b0effd640 1 -- 172.21.15.99:0/2760749700 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3b100580f0 con 0x7f3b200cec40 2024-03-04T10:03:52.233 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:03:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b0006b4d0 msgr2=0x7f3b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b0006b4d0 0x7f3b0006d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3b2006c200 tx=0x7f3b14009190 comp rx=0 tx=0).stop 2024-03-04T10:03:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 msgr2=0x7f3b200c9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 0x7f3b200c9a40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3b1000c820 tx=0x7f3b1000cce0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 shutdown_connections 2024-03-04T10:03:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b0006b4d0 0x7f3b0006d990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b2015de40 0x7f3b20154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b200cf020 0x7f3b2015e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 --2- 172.21.15.99:0/2760749700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b200cec40 0x7f3b200c9a40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 >> 172.21.15.99:0/2760749700 conn(0x7f3b200d0580 msgr2=0x7f3b200d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:52.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 shutdown_connections 2024-03-04T10:03:52.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.232+0000 7f3b2497e640 1 -- 172.21.15.99:0/2760749700 wait complete. 2024-03-04T10:03:52.246 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:03:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 Processor -- start 2024-03-04T10:03:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 -- start start 2024-03-04T10:03:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6fac0cebd0 0x7f6fac0cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 0x7f6fac0cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac06f9f0 con 0x7f6fac15aa90 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac0cf570 con 0x7f6fac0cebd0 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.345+0000 7f6fb31df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac070600 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6fac0cebd0 0x7f6fac0cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fb177c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 0x7f6fac0cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6fac0cebd0 0x7f6fac0cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:40208/0 (socket says 172.21.15.99:40208) 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fb177c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 0x7f6fac0cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:37834/0 (socket says 172.21.15.99:37834) 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 -- 172.21.15.99:0/2359690219 learned_addr learned my addr 172.21.15.99:0/2359690219 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fb0f7b640 1 --1- 172.21.15.99:0/2359690219 >> v1:172.21.15.162:6789/0 conn(0x7f6fac15aa90 0x7f6fac06ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:51054/0 (socket says 172.21.15.99:51054) 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 -- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 msgr2=0x7f6fac0cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 --2- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 0x7f6fac0cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 -- 172.21.15.99:0/2359690219 >> v1:172.21.15.162:6789/0 conn(0x7f6fac15aa90 legacy=0x7f6fac06ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fabfff640 1 -- 172.21.15.99:0/2359690219 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac06fcd0 con 0x7f6fac0cebd0 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.346+0000 7f6fb177c640 1 --2- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 0x7f6fac0cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.347+0000 7f6fabfff640 1 --2- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac0cf030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800bb10 tx=0x7f6f9802e150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=625c9e8e53ea2c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.347+0000 7f6fab7fe640 1 -- 172.21.15.99:0/2359690219 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9803b070 con 0x7f6fac0cebd0 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.347+0000 7f6fab7fe640 1 -- 172.21.15.99:0/2359690219 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f98036440 con 0x7f6fac0cebd0 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.347+0000 7f6fab7fe640 1 -- 172.21.15.99:0/2359690219 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f98036810 con 0x7f6fac0cebd0 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 -- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 msgr2=0x7f6fac0cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac0cf030 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800bb10 tx=0x7f6f9802e150 comp rx=0 tx=0).stop 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 -- 172.21.15.99:0/2359690219 shutdown_connections 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac0cd2d0 0x7f6fac0cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2359690219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac0cf030 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 -- 172.21.15.99:0/2359690219 >> 172.21.15.99:0/2359690219 conn(0x7f6fac0d0780 msgr2=0x7f6fac0d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 -- 172.21.15.99:0/2359690219 shutdown_connections 2024-03-04T10:03:52.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.348+0000 7f6fb31df640 1 -- 172.21.15.99:0/2359690219 wait complete. 2024-03-04T10:03:52.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.349+0000 7f6fb31df640 1 Processor -- start 2024-03-04T10:03:52.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.349+0000 7f6fb31df640 1 -- start start 2024-03-04T10:03:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb31df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 0x7f6fac0ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb31df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac15aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb31df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac15aa90 0x7f6fac15b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb31df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac15ae70 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb31df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac070290 con 0x7f6fac0cebd0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb31df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0afda0 con 0x7f6fac15aa90 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 0x7f6fac0ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 0x7f6fac0ae440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39146/0 (socket says 172.21.15.99:39146) 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 -- 172.21.15.99:0/2021098976 learned_addr learned my addr 172.21.15.99:0/2021098976 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fabfff640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac15aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb177c640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac15aa90 0x7f6fac15b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 -- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac15aa90 msgr2=0x7f6fac15b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac15aa90 0x7f6fac15b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 -- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 msgr2=0x7f6fac15aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac15aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.350+0000 7f6fb0f7b640 1 -- 172.21.15.99:0/2021098976 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac15c470 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.351+0000 7f6fb0f7b640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 0x7f6fac0ae440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c430 tx=0x7f6fa000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.351+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00074e0 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.351+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0007680 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.351+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac15bbd0 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.351+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0004a00 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.351+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac15c190 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.352+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac074f60 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.352+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6fa001a020 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.352+0000 7f6fa97fa640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f9006b500 0x7f6f9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.352+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6fa0089b50 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.353+0000 7f6fabfff640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f9006b500 0x7f6f9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.353+0000 7f6fabfff640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f9006b500 0x7f6f9006d9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f98038000 tx=0x7f6f98032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.357+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6fa0058720 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.540+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6fac0cab30 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.540+0000 7f6fa97fa640 1 -- 172.21.15.99:0/2021098976 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6fa0058020 con 0x7f6fac0cd2d0 2024-03-04T10:03:52.543 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:03:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f9006b500 msgr2=0x7f6f9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f9006b500 0x7f6f9006d9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6f98038000 tx=0x7f6f98032000 comp rx=0 tx=0).stop 2024-03-04T10:03:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 msgr2=0x7f6fac0ae440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 0x7f6fac0ae440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c430 tx=0x7f6fa000c830 comp rx=0 tx=0).stop 2024-03-04T10:03:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 shutdown_connections 2024-03-04T10:03:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f9006b500 0x7f6f9006d9c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6fac15aa90 0x7f6fac15b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6fac0cebd0 0x7f6fac15aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 --2- 172.21.15.99:0/2021098976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6fac0cd2d0 0x7f6fac0ae440 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 >> 172.21.15.99:0/2021098976 conn(0x7f6fac0d0780 msgr2=0x7f6fac157d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 shutdown_connections 2024-03-04T10:03:52.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:52.542+0000 7f6fb31df640 1 -- 172.21.15.99:0/2021098976 wait complete. 2024-03-04T10:03:52.556 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi099.front.sepia.ceph.com... 2024-03-04T10:03:52.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:52.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-04T10:03:52.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi099.stderr:ceph-fuse[44829]: fuse finished with error 0 and tester_r 0 2024-03-04T10:03:52.584 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:03:58.686 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:03:58.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:03:58.687 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-04T10:03:58.714 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:58.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.863+0000 7f71b1e2b640 1 Processor -- start 2024-03-04T10:03:58.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.863+0000 7f71b1e2b640 1 -- start start 2024-03-04T10:03:58.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71b1e2b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f71ac154a80 0x7f71ac15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71b1e2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac0cfae0 0x7f71ac161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71b1e2b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71ac0cff70 con 0x7f71ac0cf650 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71b1e2b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71ac071df0 con 0x7f71ac154a80 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71b1e2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0cd010 con 0x7f71ac0cfae0 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71ab7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f71ac154a80 0x7f71ac15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71aaffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f71ac0cf650 0x7f71ac1616c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49816/0 (socket says 172.21.15.99:49816) 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71aaffd640 1 -- 172.21.15.99:0/2959789786 learned_addr learned my addr 172.21.15.99:0/2959789786 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71abfff640 1 --2- 172.21.15.99:0/2959789786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac0cfae0 0x7f71ac161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2959789786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac0cfae0 msgr2=0x7f71ac161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71ab7fe640 1 --2- 172.21.15.99:0/2959789786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac0cfae0 0x7f71ac161dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2959789786 >> v1:172.21.15.99:6789/0 conn(0x7f71ac0cf650 legacy=0x7f71ac1616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2959789786 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac0cdaa0 con 0x7f71ac154a80 2024-03-04T10:03:58.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.864+0000 7f71aa7fc640 1 -- 172.21.15.99:0/2959789786 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914802195 0 0) 0x7f71ac0cff70 con 0x7f71ac0cf650 2024-03-04T10:03:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.865+0000 7f71ab7fe640 1 --2- 172.21.15.99:0/2959789786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac154a80 0x7f71ac15e910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f719c0097a0 tx=0x7f719c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66881bebc9a345fd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.865+0000 7f71aa7fc640 1 -- 172.21.15.99:0/2959789786 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c035020 con 0x7f71ac154a80 2024-03-04T10:03:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.865+0000 7f71aa7fc640 1 -- 172.21.15.99:0/2959789786 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c031830 con 0x7f71ac154a80 2024-03-04T10:03:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.865+0000 7f71aa7fc640 1 -- 172.21.15.99:0/2959789786 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c031c00 con 0x7f71ac154a80 2024-03-04T10:03:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2959789786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac154a80 msgr2=0x7f71ac15e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2959789786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac154a80 0x7f71ac15e910 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f719c0097a0 tx=0x7f719c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2959789786 shutdown_connections 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2959789786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac0cfae0 0x7f71ac161dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2959789786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac154a80 0x7f71ac15e910 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2959789786 >> 172.21.15.99:0/2959789786 conn(0x7f71ac06f5f0 msgr2=0x7f71ac06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2959789786 shutdown_connections 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.866+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2959789786 wait complete. 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.867+0000 7f71b1e2b640 1 Processor -- start 2024-03-04T10:03:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.867+0000 7f71b1e2b640 1 -- start start 2024-03-04T10:03:58.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71b1e2b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 0x7f71ac0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:58.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71b1e2b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71ac0cfae0 0x7f71ac147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:58.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71b1e2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac154a80 0x7f71ac147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71b1e2b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0ce340 con 0x7f71ac0cf650 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71b1e2b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0ce060 con 0x7f71ac0cfae0 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71b1e2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71ac0ce620 con 0x7f71ac154a80 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 0x7f71ac0c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 0x7f71ac0c7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45850/0 (socket says 172.21.15.99:45850) 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2795216130 learned_addr learned my addr 172.21.15.99:0/2795216130 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71abfff640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac154a80 0x7f71ac147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71aaffd640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71ac0cfae0 0x7f71ac147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac154a80 msgr2=0x7f71ac147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac154a80 0x7f71ac147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71ac0cfae0 msgr2=0x7f71ac147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:58.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71ac0cfae0 0x7f71ac147640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71ab7fe640 1 -- 172.21.15.99:0/2795216130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ac1e81d0 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.868+0000 7f71aaffd640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71ac0cfae0 0x7f71ac147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.869+0000 7f71ab7fe640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 0x7f71ac0c7550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f719c0097a0 tx=0x7f719c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.869+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c035040 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.869+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c004230 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.869+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71ac1e84b0 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.869+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c03a700 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.869+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71ac0b4490 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.870+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71ac074f60 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.870+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f719c038020 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.870+0000 7f7193fff640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f719806b500 0x7f719806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.871+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f719c0ac2b0 con 0x7f71ac0cf650 2024-03-04T10:03:58.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.871+0000 7f71aaffd640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f719806b500 0x7f719806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:58.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.871+0000 7f71aaffd640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f719806b500 0x7f719806d9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71ac161d20 tx=0x7f71a0007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:58.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:58.875+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f719c07ad70 con 0x7f71ac0cf650 2024-03-04T10:03:59.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.049+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71ac15dfd0 con 0x7f71ac0cf650 2024-03-04T10:03:59.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.050+0000 7f7193fff640 1 -- 172.21.15.99:0/2795216130 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f719c07a670 con 0x7f71ac0cf650 2024-03-04T10:03:59.053 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:59.053 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.052+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f719806b500 msgr2=0x7f719806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.052+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f719806b500 0x7f719806d9c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f71ac161d20 tx=0x7f71a0007c10 comp rx=0 tx=0).stop 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.052+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 msgr2=0x7f71ac0c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.052+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 0x7f71ac0c7550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f719c0097a0 tx=0x7f719c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 shutdown_connections 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f719806b500 0x7f719806d9c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71ac154a80 0x7f71ac147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71ac0cfae0 0x7f71ac147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 --2- 172.21.15.99:0/2795216130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71ac0cf650 0x7f71ac0c7550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 >> 172.21.15.99:0/2795216130 conn(0x7f71ac06f5f0 msgr2=0x7f71ac15b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:59.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 shutdown_connections 2024-03-04T10:03:59.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.053+0000 7f71b1e2b640 1 -- 172.21.15.99:0/2795216130 wait complete. 2024-03-04T10:03:59.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:03:59.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.165+0000 7f1230f62640 1 Processor -- start 2024-03-04T10:03:59.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.165+0000 7f1230f62640 1 -- start start 2024-03-04T10:03:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.166+0000 7f1230f62640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f122c0b2e60 0x7f122c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.166+0000 7f1230f62640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c15e1e0 con 0x7f122c0b5470 2024-03-04T10:03:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.166+0000 7f1230f62640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122c0d1c10 con 0x7f122c15de00 2024-03-04T10:03:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.166+0000 7f1230f62640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f122c0b98c0 con 0x7f122c0b2e60 2024-03-04T10:03:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.166+0000 7f122ad9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f122c0b5470 0x7f122c0d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49824/0 (socket says 172.21.15.99:49824) 2024-03-04T10:03:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.166+0000 7f122ad9d640 1 -- 172.21.15.99:0/3713105005 learned_addr learned my addr 172.21.15.99:0/3713105005 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f1229d9b640 1 --2- 172.21.15.99:0/3713105005 >> v2:172.21.15.162:3300/0 conn(0x7f122c0b2e60 0x7f122c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145923436 0 0) 0x7f122c15e1e0 con 0x7f122c0b5470 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1214003520 con 0x7f122c0b5470 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3318974228 0 0) 0x7f1214003520 con 0x7f122c0b5470 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 >> v2:172.21.15.162:3300/0 conn(0x7f122c0b2e60 msgr2=0x7f122c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 --2- 172.21.15.99:0/3713105005 >> v2:172.21.15.162:3300/0 conn(0x7f122c0b2e60 0x7f122c0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 >> v1:172.21.15.162:6789/0 conn(0x7f122c15de00 legacy=0x7f122c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0d1ef0 con 0x7f122c0b5470 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f12240019f0 con 0x7f122c0b5470 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 >> v1:172.21.15.99:6789/0 conn(0x7f122c0b5470 legacy=0x7f122c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1214004800 0x7f1214004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140051c0 0x7f121400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0b98c0 con 0x7f12140039b0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121400c060 con 0x7f1214004800 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f121400c260 con 0x7f12140051c0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122a59c640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f1229d9b640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1214004800 0x7f1214004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122ad9d640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140051c0 0x7f121400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122a59c640 1 -- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140051c0 msgr2=0x7f121400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122a59c640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140051c0 0x7f121400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122a59c640 1 -- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1214004800 msgr2=0x7f1214004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122a59c640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1214004800 0x7f1214004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122a59c640 1 -- 172.21.15.99:0/3713105005 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214003520 con 0x7f12140039b0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.167+0000 7f122ad9d640 1 -- 172.21.15.99:0/3713105005 reap_dead start 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.168+0000 7f122a59c640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f121c001750 tx=0x7f1214005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7908bbd074ce51d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.168+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c001370 con 0x7f12140039b0 2024-03-04T10:03:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.168+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121c0052d0 con 0x7f12140039b0 2024-03-04T10:03:59.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.168+0000 7f122959a640 1 -- 172.21.15.99:0/3713105005 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121c003830 con 0x7f12140039b0 2024-03-04T10:03:59.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 -- 172.21.15.99:0/3713105005 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f12140039b0 msgr2=0x7f1214003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f121c001750 tx=0x7f1214005830 comp rx=0 tx=0).stop 2024-03-04T10:03:59.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 -- 172.21.15.99:0/3713105005 shutdown_connections 2024-03-04T10:03:59.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 --2- 172.21.15.99:0/3713105005 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f12140039b0 0x7f1214003dd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 -- 172.21.15.99:0/3713105005 >> 172.21.15.99:0/3713105005 conn(0x7f122c0d0470 msgr2=0x7f122c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 -- 172.21.15.99:0/3713105005 shutdown_connections 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.169+0000 7f1230f62640 1 -- 172.21.15.99:0/3713105005 wait complete. 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.170+0000 7f1230f62640 1 Processor -- start 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.170+0000 7f1230f62640 1 -- start start 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1230f62640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140039b0 0x7f122c0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1230f62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f122c14c1c0 0x7f122c15fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1230f62640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 0x7f122c15e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1230f62640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0d2820 con 0x7f122c160160 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1230f62640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0d24b0 con 0x7f122c14c1c0 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1230f62640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122c0b9380 con 0x7f12140039b0 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 0x7f122c15e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 0x7f122c15e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45860/0 (socket says 172.21.15.99:45860) 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 -- 172.21.15.99:0/2819507799 learned_addr learned my addr 172.21.15.99:0/2819507799 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:59.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f1229d9b640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f122c14c1c0 0x7f122c15fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122a59c640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140039b0 0x7f122c0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 -- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140039b0 msgr2=0x7f122c0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140039b0 0x7f122c0ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 -- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f122c14c1c0 msgr2=0x7f122c15fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f122c14c1c0 0x7f122c15fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.171+0000 7f122ad9d640 1 -- 172.21.15.99:0/2819507799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c0cc7c0 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.172+0000 7f122ad9d640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 0x7f122c15e1e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f122c0ce210 tx=0x7f12240079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.172+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400aa60 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.172+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1224018070 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.172+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122c0cd0b0 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.172+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122400ae60 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.172+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122c0cba00 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.173+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f122c0b5900 con 0x7f122c160160 2024-03-04T10:03:59.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.173+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f122401d460 con 0x7f122c160160 2024-03-04T10:03:59.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.173+0000 7f121affd640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f120c06b500 0x7f120c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.173+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f122408a880 con 0x7f122c160160 2024-03-04T10:03:59.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.174+0000 7f122a59c640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f120c06b500 0x7f120c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.174+0000 7f122a59c640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f120c06b500 0x7f120c06d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f121c033000 tx=0x7f121c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:59.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.178+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f122c0b5900 con 0x7f122c160160 2024-03-04T10:03:59.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.351+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f122c0b5290 con 0x7f122c160160 2024-03-04T10:03:59.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.353+0000 7f121affd640 1 -- 172.21.15.99:0/2819507799 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f122c0b5290 con 0x7f122c160160 2024-03-04T10:03:59.355 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:03:59.355 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f120c06b500 msgr2=0x7f120c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f120c06b500 0x7f120c06d9c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f121c033000 tx=0x7f121c009030 comp rx=0 tx=0).stop 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 msgr2=0x7f122c15e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 0x7f122c15e1e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f122c0ce210 tx=0x7f12240079d0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 shutdown_connections 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f120c06b500 0x7f120c06d9c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f122c160160 0x7f122c15e1e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f122c14c1c0 0x7f122c15fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 --2- 172.21.15.99:0/2819507799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f12140039b0 0x7f122c0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 >> 172.21.15.99:0/2819507799 conn(0x7f122c0d0470 msgr2=0x7f122c0ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:59.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 shutdown_connections 2024-03-04T10:03:59.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.355+0000 7f1230f62640 1 -- 172.21.15.99:0/2819507799 wait complete. 2024-03-04T10:03:59.368 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T10:03:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.472+0000 7fb31699e640 1 Processor -- start 2024-03-04T10:03:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.472+0000 7fb31699e640 1 -- start start 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb31699e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100b5700 0x7fb3100d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb31699e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb3100d34a0 0x7fb31006c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb31699e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3100b9410 0x7fb31006c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb31699e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3100d2400 con 0x7fb3100b9410 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb31699e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3100d26e0 con 0x7fb3100d34a0 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb31699e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100d29c0 con 0x7fb3100b5700 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3100b9410 0x7fb31006c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3100b9410 0x7fb31006c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45864/0 (socket says 172.21.15.99:45864) 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 -- 172.21.15.99:0/2776640082 learned_addr learned my addr 172.21.15.99:0/2776640082 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb30ffff640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100b5700 0x7fb3100d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb30f7fe640 1 --2- 172.21.15.99:0/2776640082 >> v2:172.21.15.162:3300/0 conn(0x7fb3100d34a0 0x7fb31006c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 -- 172.21.15.99:0/2776640082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100b5700 msgr2=0x7fb3100d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100b5700 0x7fb3100d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 -- 172.21.15.99:0/2776640082 >> v2:172.21.15.162:3300/0 conn(0x7fb3100d34a0 msgr2=0x7fb31006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 --2- 172.21.15.99:0/2776640082 >> v2:172.21.15.162:3300/0 conn(0x7fb3100d34a0 0x7fb31006c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb314f3b640 1 -- 172.21.15.99:0/2776640082 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3100d2ca0 con 0x7fb3100b9410 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb30ffff640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100b5700 0x7fb3100d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.473+0000 7fb30f7fe640 1 --2- 172.21.15.99:0/2776640082 >> v2:172.21.15.162:3300/0 conn(0x7fb3100d34a0 0x7fb31006c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.474+0000 7fb314f3b640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b9410 0x7fb31006c620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb3100d4a70 tx=0x7fb30802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ecfb2bb3089395 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.474+0000 7fb30effd640 1 -- 172.21.15.99:0/2776640082 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30802c9e0 con 0x7fb3100b9410 2024-03-04T10:03:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.474+0000 7fb30effd640 1 -- 172.21.15.99:0/2776640082 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30802cb80 con 0x7fb3100b9410 2024-03-04T10:03:59.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.474+0000 7fb30effd640 1 -- 172.21.15.99:0/2776640082 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3080343f0 con 0x7fb3100b9410 2024-03-04T10:03:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 -- 172.21.15.99:0/2776640082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b9410 msgr2=0x7fb31006c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b9410 0x7fb31006c620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb3100d4a70 tx=0x7fb30802be40 comp rx=0 tx=0).stop 2024-03-04T10:03:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 -- 172.21.15.99:0/2776640082 shutdown_connections 2024-03-04T10:03:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b9410 0x7fb31006c620 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 --2- 172.21.15.99:0/2776640082 >> v2:172.21.15.162:3300/0 conn(0x7fb3100d34a0 0x7fb31006c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 --2- 172.21.15.99:0/2776640082 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100b5700 0x7fb3100d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 -- 172.21.15.99:0/2776640082 >> 172.21.15.99:0/2776640082 conn(0x7fb3100d0490 msgr2=0x7fb3100d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 -- 172.21.15.99:0/2776640082 shutdown_connections 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.475+0000 7fb31699e640 1 -- 172.21.15.99:0/2776640082 wait complete. 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.476+0000 7fb31699e640 1 Processor -- start 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.476+0000 7fb31699e640 1 -- start start 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb31699e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 0x7fb3100c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb31699e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3100b9410 0x7fb3100ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb31699e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100d34a0 0x7fb310156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb31699e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100b5c60 con 0x7fb3100b5700 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb31699e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100b5ae0 con 0x7fb3100b9410 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb31699e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100b5de0 con 0x7fb3100d34a0 2024-03-04T10:03:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 0x7fb3100c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 0x7fb3100c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45872/0 (socket says 172.21.15.99:45872) 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 -- 172.21.15.99:0/2674951917 learned_addr learned my addr 172.21.15.99:0/2674951917 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb314f3b640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100d34a0 0x7fb310156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30f7fe640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3100b9410 0x7fb3100ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 -- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100d34a0 msgr2=0x7fb310156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100d34a0 0x7fb310156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 -- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3100b9410 msgr2=0x7fb3100ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3100b9410 0x7fb3100ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.477+0000 7fb30ffff640 1 -- 172.21.15.99:0/2674951917 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3100cb690 con 0x7fb3100b5700 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.478+0000 7fb30ffff640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 0x7fb3100c9d20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb30400d800 tx=0x7fb30400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.478+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304018070 con 0x7fb3100b5700 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.478+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb304004030 con 0x7fb3100b5700 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.478+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3100cb970 con 0x7fb3100b5700 2024-03-04T10:03:59.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.478+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb304004400 con 0x7fb3100b5700 2024-03-04T10:03:59.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.478+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3100cbf30 con 0x7fb3100b5700 2024-03-04T10:03:59.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.479+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3100cc170 con 0x7fb3100b5700 2024-03-04T10:03:59.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.479+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb3040096a0 con 0x7fb3100b5700 2024-03-04T10:03:59.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.479+0000 7fb30cff9640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:03:59.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.480+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb30408a720 con 0x7fb3100b5700 2024-03-04T10:03:59.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.480+0000 7fb30f7fe640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:03:59.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.480+0000 7fb30f7fe640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb3100b3ef0 tx=0x7fb300008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:03:59.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.484+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb3100cc170 con 0x7fb3100b5700 2024-03-04T10:03:59.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:03:59.704+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb310074f60 con 0x7fb3100b5700 2024-03-04T10:04:00.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.607+0000 7fb30cff9640 1 -- 172.21.15.99:0/2674951917 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb304058ca0 con 0x7fb3100b5700 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2fc06b4d0 msgr2=0x7fb2fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb3100b3ef0 tx=0x7fb300008040 comp rx=0 tx=0).stop 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 msgr2=0x7fb3100c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 0x7fb3100c9d20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb30400d800 tx=0x7fb30400dcc0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 shutdown_connections 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2fc06b4d0 0x7fb2fc06d990 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3100d34a0 0x7fb310156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3100b9410 0x7fb3100ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 --2- 172.21.15.99:0/2674951917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3100b5700 0x7fb3100c9d20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 >> 172.21.15.99:0/2674951917 conn(0x7fb3100d0490 msgr2=0x7fb3100b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:00.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 shutdown_connections 2024-03-04T10:04:00.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.610+0000 7fb31699e640 1 -- 172.21.15.99:0/2674951917 wait complete. 2024-03-04T10:04:00.624 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-04T10:04:00.624 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T10:04:00.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.728+0000 7f546f885640 1 Processor -- start 2024-03-04T10:04:00.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.728+0000 7f546f885640 1 -- start start 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546f885640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f546815a810 0x7f54680b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546f885640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f54680b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546f885640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f54680cbfb0 0x7f546815abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546f885640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546806d8a0 con 0x7f546815a810 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546f885640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546806db80 con 0x7f54680cbfb0 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546f885640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546806de60 con 0x7f546806d3c0 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f546815a810 0x7f54680b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f546815a810 0x7f54680b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45886/0 (socket says 172.21.15.99:45886) 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 -- 172.21.15.99:0/3871284870 learned_addr learned my addr 172.21.15.99:0/3871284870 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546ce20640 1 --2- 172.21.15.99:0/3871284870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f54680b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546de22640 1 --2- 172.21.15.99:0/3871284870 >> v2:172.21.15.162:3300/0 conn(0x7f54680cbfb0 0x7f546815abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 -- 172.21.15.99:0/3871284870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 msgr2=0x7f54680b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 --2- 172.21.15.99:0/3871284870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f54680b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 -- 172.21.15.99:0/3871284870 >> v2:172.21.15.162:3300/0 conn(0x7f54680cbfb0 msgr2=0x7f546815abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 --2- 172.21.15.99:0/3871284870 >> v2:172.21.15.162:3300/0 conn(0x7f54680cbfb0 0x7f546815abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.729+0000 7f546d621640 1 -- 172.21.15.99:0/3871284870 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546806e140 con 0x7f546815a810 2024-03-04T10:04:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.730+0000 7f546d621640 1 --2- 172.21.15.99:0/3871284870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f54680b4190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f546806bc80 tx=0x7f546402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e422cbf8c072db7a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.730+0000 7f545ffff640 1 -- 172.21.15.99:0/3871284870 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546402ca70 con 0x7f546815a810 2024-03-04T10:04:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.730+0000 7f545ffff640 1 -- 172.21.15.99:0/3871284870 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546402cc10 con 0x7f546815a810 2024-03-04T10:04:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.730+0000 7f545ffff640 1 -- 172.21.15.99:0/3871284870 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54640343f0 con 0x7f546815a810 2024-03-04T10:04:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 -- 172.21.15.99:0/3871284870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 msgr2=0x7f54680b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 --2- 172.21.15.99:0/3871284870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f54680b4190 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f546806bc80 tx=0x7f546402bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 -- 172.21.15.99:0/3871284870 shutdown_connections 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 --2- 172.21.15.99:0/3871284870 >> v2:172.21.15.162:3300/0 conn(0x7f54680cbfb0 0x7f546815abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 --2- 172.21.15.99:0/3871284870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f54680b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 --2- 172.21.15.99:0/3871284870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f54680b4190 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 -- 172.21.15.99:0/3871284870 >> 172.21.15.99:0/3871284870 conn(0x7f546806acd0 msgr2=0x7f546806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 -- 172.21.15.99:0/3871284870 shutdown_connections 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.731+0000 7f546f885640 1 -- 172.21.15.99:0/3871284870 wait complete. 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.732+0000 7f546f885640 1 Processor -- start 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.732+0000 7f546f885640 1 -- start start 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546f885640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f5468165750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546f885640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54680cbfb0 0x7f54680adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546f885640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f546815cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546f885640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546815b130 con 0x7f546815a810 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546f885640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546815e5d0 con 0x7f54680cbfb0 2024-03-04T10:04:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546f885640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546815b2b0 con 0x7f546806d3c0 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f546815cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f546815cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45892/0 (socket says 172.21.15.99:45892) 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 -- 172.21.15.99:0/4281860512 learned_addr learned my addr 172.21.15.99:0/4281860512 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546d621640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f5468165750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546ce20640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54680cbfb0 0x7f54680adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 -- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 msgr2=0x7f5468165750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f5468165750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 -- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54680cbfb0 msgr2=0x7f54680adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54680cbfb0 0x7f54680adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 -- 172.21.15.99:0/4281860512 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5468165de0 con 0x7f546815a810 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.733+0000 7f546de22640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f546815cf20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f546000d800 tx=0x7f546000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.734+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546000b6c0 con 0x7f546815a810 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.734+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5460019070 con 0x7f546815a810 2024-03-04T10:04:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.734+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546000bac0 con 0x7f546815a810 2024-03-04T10:04:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.734+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54681660c0 con 0x7f546815a810 2024-03-04T10:04:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.734+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5468166680 con 0x7f546815a810 2024-03-04T10:04:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.735+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5468074f60 con 0x7f546815a810 2024-03-04T10:04:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.735+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f546000bc60 con 0x7f546815a810 2024-03-04T10:04:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.735+0000 7f545dffb640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544c06b4d0 0x7f544c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.735+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f546008a880 con 0x7f546815a810 2024-03-04T10:04:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.736+0000 7f546d621640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544c06b4d0 0x7f544c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.736+0000 7f546d621640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544c06b4d0 0x7f544c06d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5464035000 tx=0x7f5464030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:00.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.740+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5460058ce0 con 0x7f546815a810 2024-03-04T10:04:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:00.944+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f546814de70 con 0x7f546815a810 2024-03-04T10:04:01.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.611+0000 7f545dffb640 1 -- 172.21.15.99:0/4281860512 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f546005cd80 con 0x7f546815a810 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544c06b4d0 msgr2=0x7f544c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544c06b4d0 0x7f544c06d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5464035000 tx=0x7f5464030000 comp rx=0 tx=0).stop 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 msgr2=0x7f546815cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f546815cf20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f546000d800 tx=0x7f546000dcc0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 shutdown_connections 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544c06b4d0 0x7f544c06d990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f546815a810 0x7f546815cf20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54680cbfb0 0x7f54680adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 --2- 172.21.15.99:0/4281860512 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546806d3c0 0x7f5468165750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 >> 172.21.15.99:0/4281860512 conn(0x7f546806acd0 msgr2=0x7f546814e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 shutdown_connections 2024-03-04T10:04:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.613+0000 7f546f885640 1 -- 172.21.15.99:0/4281860512 wait complete. 2024-03-04T10:04:01.627 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T10:04:01.627 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:01.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.728+0000 7f7a3390a640 1 Processor -- start 2024-03-04T10:04:01.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.728+0000 7f7a3390a640 1 -- start start 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a3390a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7a2c15de00 0x7f7a2c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a3390a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c0b5470 0x7f7a2c0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a3390a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c15e1e0 con 0x7f7a2c0b9380 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a3390a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c0d2820 con 0x7f7a2c15de00 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a3390a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0d1c10 con 0x7f7a2c0b5470 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7a2c15de00 0x7f7a2c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7a2c15de00 0x7f7a2c0b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45898/0 (socket says 172.21.15.99:45898) 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 -- 172.21.15.99:0/1880492289 learned_addr learned my addr 172.21.15.99:0/1880492289 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:01.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 -- 172.21.15.99:0/1880492289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c0b5470 msgr2=0x7f7a2c0b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 --2- 172.21.15.99:0/1880492289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c0b5470 0x7f7a2c0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 -- 172.21.15.99:0/1880492289 >> v1:172.21.15.162:6789/0 conn(0x7f7a2c0b9380 legacy=0x7f7a2c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.729+0000 7f7a316a6640 1 -- 172.21.15.99:0/1880492289 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c0d1ef0 con 0x7f7a2c15de00 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a316a6640 1 --2- 172.21.15.99:0/1880492289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c15de00 0x7f7a2c0b2e60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009870 tx=0x7f7a2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c94f5ad9cc77dae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a23fff640 1 -- 172.21.15.99:0/1880492289 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28035020 con 0x7f7a2c15de00 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a23fff640 1 -- 172.21.15.99:0/1880492289 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a28031640 con 0x7f7a2c15de00 2024-03-04T10:04:01.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a23fff640 1 -- 172.21.15.99:0/1880492289 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2802d040 con 0x7f7a2c15de00 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a3390a640 1 -- 172.21.15.99:0/1880492289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c15de00 msgr2=0x7f7a2c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a3390a640 1 --2- 172.21.15.99:0/1880492289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c15de00 0x7f7a2c0b2e60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7a28009870 tx=0x7f7a2802b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a3390a640 1 -- 172.21.15.99:0/1880492289 shutdown_connections 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a3390a640 1 --2- 172.21.15.99:0/1880492289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c0b5470 0x7f7a2c0b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a3390a640 1 --2- 172.21.15.99:0/1880492289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c15de00 0x7f7a2c0b2e60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.730+0000 7f7a3390a640 1 -- 172.21.15.99:0/1880492289 >> 172.21.15.99:0/1880492289 conn(0x7f7a2c0d0470 msgr2=0x7f7a2c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.731+0000 7f7a3390a640 1 -- 172.21.15.99:0/1880492289 shutdown_connections 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.731+0000 7f7a3390a640 1 -- 172.21.15.99:0/1880492289 wait complete. 2024-03-04T10:04:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 Processor -- start 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 -- start start 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 0x7f7a2c153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a2c0b9380 0x7f7a2c15f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c15de00 0x7f7a2c145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0b97e0 con 0x7f7a2c0b5470 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0d24b0 con 0x7f7a2c0b9380 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a3390a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2c0b72a0 con 0x7f7a2c15de00 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a316a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 0x7f7a2c153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a316a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 0x7f7a2c153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45906/0 (socket says 172.21.15.99:45906) 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.732+0000 7f7a316a6640 1 -- 172.21.15.99:0/722941848 learned_addr learned my addr 172.21.15.99:0/722941848 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a31ea7640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c15de00 0x7f7a2c145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a30ea5640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a2c0b9380 0x7f7a2c15f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a316a6640 1 -- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c15de00 msgr2=0x7f7a2c145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a316a6640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c15de00 0x7f7a2c145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a316a6640 1 -- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a2c0b9380 msgr2=0x7f7a2c15f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a316a6640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a2c0b9380 0x7f7a2c15f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a316a6640 1 -- 172.21.15.99:0/722941848 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c15fd90 con 0x7f7a2c0b5470 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a30ea5640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a2c0b9380 0x7f7a2c15f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a31ea7640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c15de00 0x7f7a2c145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a316a6640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 0x7f7a2c153cf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7a2802b930 tx=0x7f7a280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2803ac60 con 0x7f7a2c0b5470 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2803ae20 con 0x7f7a2c0b5470 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28044630 con 0x7f7a2c0b5470 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c160070 con 0x7f7a2c0b5470 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.733+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c160630 con 0x7f7a2c0b5470 2024-03-04T10:04:01.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.734+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c074f60 con 0x7f7a2c0b5470 2024-03-04T10:04:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.735+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7a2804b060 con 0x7f7a2c0b5470 2024-03-04T10:04:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.735+0000 7f7a21ffb640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a1006b4d0 0x7f7a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.735+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7a280ac340 con 0x7f7a2c0b5470 2024-03-04T10:04:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.735+0000 7f7a30ea5640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a1006b4d0 0x7f7a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.736+0000 7f7a30ea5640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a1006b4d0 0x7f7a1006d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c007fd0 tx=0x7f7a1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.739+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7a2803f070 con 0x7f7a2c0b5470 2024-03-04T10:04:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.947+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a2c0b7550 con 0x7f7a2c0b5470 2024-03-04T10:04:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.947+0000 7f7a21ffb640 1 -- 172.21.15.99:0/722941848 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f7a2802f300 con 0x7f7a2c0b5470 2024-03-04T10:04:01.950 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:01.950 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a1006b4d0 msgr2=0x7f7a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a1006b4d0 0x7f7a1006d990 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c007fd0 tx=0x7f7a1c009040 comp rx=0 tx=0).stop 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 msgr2=0x7f7a2c153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 0x7f7a2c153cf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7a2802b930 tx=0x7f7a280046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 shutdown_connections 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a1006b4d0 0x7f7a1006d990 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a2c15de00 0x7f7a2c145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a2c0b9380 0x7f7a2c15f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 --2- 172.21.15.99:0/722941848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a2c0b5470 0x7f7a2c153cf0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 >> 172.21.15.99:0/722941848 conn(0x7f7a2c0d0470 msgr2=0x7f7a2c0b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 shutdown_connections 2024-03-04T10:04:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:01.950+0000 7f7a3390a640 1 -- 172.21.15.99:0/722941848 wait complete. 2024-03-04T10:04:01.963 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:02.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.061+0000 7f6aed068640 1 Processor -- start 2024-03-04T10:04:02.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.061+0000 7f6aed068640 1 -- start start 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6aed068640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 0x7f6ae80b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6aed068640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae815e1e0 con 0x7f6ae80b5470 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6aed068640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae80d1c10 con 0x7f6ae815de00 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6aed068640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b98c0 con 0x7f6ae80b2e60 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6ae759e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6ae80b5470 0x7f6ae80d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49838/0 (socket says 172.21.15.99:49838) 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6ae759e640 1 -- 172.21.15.99:0/1548995347 learned_addr learned my addr 172.21.15.99:0/1548995347 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.062+0000 7f6ae659c640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 0x7f6ae80b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599195725 0 0) 0x7f6ae815e1e0 con 0x7f6ae80b5470 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad0003520 con 0x7f6ae80b5470 2024-03-04T10:04:02.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1902130056 0 0) 0x7f6ad0003520 con 0x7f6ae80b5470 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 msgr2=0x7f6ae80b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 0x7f6ae80b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 >> v1:172.21.15.162:6789/0 conn(0x7f6ae815de00 legacy=0x7f6ae80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80d1ef0 con 0x7f6ae80b5470 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6ae0001670 con 0x7f6ae80b5470 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 >> v1:172.21.15.99:6789/0 conn(0x7f6ae80b5470 legacy=0x7f6ae80d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad00039e0 0x7f6ad0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 0x7f6ad0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ad0004d40 0x7f6ad00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b98c0 con 0x7f6ad0004d40 2024-03-04T10:04:02.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad000ba70 con 0x7f6ad00039e0 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad000bc70 con 0x7f6ae80b2e60 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae759e640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ad0004d40 0x7f6ad00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae6d9d640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad00039e0 0x7f6ad0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae659c640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 0x7f6ad0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae759e640 1 -- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 msgr2=0x7f6ad0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae759e640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ae80b2e60 0x7f6ad0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae759e640 1 -- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad00039e0 msgr2=0x7f6ad0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae759e640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ad00039e0 0x7f6ad0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.063+0000 7f6ae759e640 1 -- 172.21.15.99:0/1548995347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad0003520 con 0x7f6ad0004d40 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.064+0000 7f6ae759e640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ad0004d40 0x7f6ad00051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c210 tx=0x7f6ad0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260b2cf5b73d49c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.064+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0007cf0 con 0x7f6ad0004d40 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.064+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae0030070 con 0x7f6ad0004d40 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.064+0000 7f6ae5d9b640 1 -- 172.21.15.99:0/1548995347 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0005d30 con 0x7f6ad0004d40 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6aed068640 1 -- 172.21.15.99:0/1548995347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ad0004d40 msgr2=0x7f6ad00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6aed068640 1 --2- 172.21.15.99:0/1548995347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ad0004d40 0x7f6ad00051c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c210 tx=0x7f6ad0005830 comp rx=0 tx=0).stop 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6ae759e640 1 -- 172.21.15.99:0/1548995347 reap_dead start 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6aed068640 1 -- 172.21.15.99:0/1548995347 shutdown_connections 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6aed068640 1 -- 172.21.15.99:0/1548995347 >> 172.21.15.99:0/1548995347 conn(0x7f6ae80d0470 msgr2=0x7f6ae80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6aed068640 1 -- 172.21.15.99:0/1548995347 shutdown_connections 2024-03-04T10:04:02.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.065+0000 7f6aed068640 1 -- 172.21.15.99:0/1548995347 wait complete. 2024-03-04T10:04:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.066+0000 7f6aed068640 1 Processor -- start 2024-03-04T10:04:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.066+0000 7f6aed068640 1 -- start start 2024-03-04T10:04:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6aed068640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ad0004d40 0x7f6ae80c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6aed068640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae80ccd90 0x7f6ae81e6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6aed068640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 0x7f6ae81e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6aed068640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80d2820 con 0x7f6ae80cd210 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6aed068640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80d24b0 con 0x7f6ae80ccd90 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6aed068640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae80b72a0 con 0x7f6ad0004d40 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 0x7f6ae81e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 0x7f6ae81e6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45916/0 (socket says 172.21.15.99:45916) 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 -- 172.21.15.99:0/2011942550 learned_addr learned my addr 172.21.15.99:0/2011942550 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae6d9d640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ad0004d40 0x7f6ae80c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae659c640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae80ccd90 0x7f6ae81e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 -- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ad0004d40 msgr2=0x7f6ae80c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ad0004d40 0x7f6ae80c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 -- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae80ccd90 msgr2=0x7f6ae81e6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae80ccd90 0x7f6ae81e6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.067+0000 7f6ae759e640 1 -- 172.21.15.99:0/2011942550 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae80c9960 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.068+0000 7f6ae759e640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 0x7f6ae81e6a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c210 tx=0x7f6ae80b2d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.068+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0007cf0 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.068+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae0004bf0 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.068+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0001c30 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.068+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae815eff0 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.068+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae815f5b0 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.069+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae80b6b90 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.069+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6ae0007070 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.069+0000 7f6acf7fe640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ad406b500 0x7f6ad406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.069+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6ae00ab390 con 0x7f6ae80cd210 2024-03-04T10:04:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.070+0000 7f6ae6d9d640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ad406b500 0x7f6ad406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.070+0000 7f6ae6d9d640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ad406b500 0x7f6ad406d9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8001ee0 tx=0x7f6ad8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.074+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6ae80b6b90 con 0x7f6ae80cd210 2024-03-04T10:04:02.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.246+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ae8074f60 con 0x7f6ae80cd210 2024-03-04T10:04:02.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.247+0000 7f6acf7fe640 1 -- 172.21.15.99:0/2011942550 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f6ae0079ed0 con 0x7f6ae80cd210 2024-03-04T10:04:02.250 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:02.250 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ad406b500 msgr2=0x7f6ad406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ad406b500 0x7f6ad406d9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8001ee0 tx=0x7f6ad8009040 comp rx=0 tx=0).stop 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 msgr2=0x7f6ae81e6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 0x7f6ae81e6a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000c210 tx=0x7f6ae80b2d40 comp rx=0 tx=0).stop 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 shutdown_connections 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ad406b500 0x7f6ad406d9c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ae80cd210 0x7f6ae81e6a80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ae80ccd90 0x7f6ae81e6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 --2- 172.21.15.99:0/2011942550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ad0004d40 0x7f6ae80c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 >> 172.21.15.99:0/2011942550 conn(0x7f6ae80d0470 msgr2=0x7f6ae80cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:02.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 shutdown_connections 2024-03-04T10:04:02.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.249+0000 7f6aed068640 1 -- 172.21.15.99:0/2011942550 wait complete. 2024-03-04T10:04:02.263 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T10:04:02.263 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:04:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.362+0000 7fdfbac27640 1 Processor -- start 2024-03-04T10:04:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfbac27640 1 -- start start 2024-03-04T10:04:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfbac27640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb415bbb0 0x7fdfb40d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfbac27640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40cfb40 con 0x7fdfb40cc690 2024-03-04T10:04:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfbac27640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb40cfe20 con 0x7fdfb40d0e40 2024-03-04T10:04:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfbac27640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40d2880 con 0x7fdfb415bbb0 2024-03-04T10:04:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfb91c4640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdfb40cc690 0x7fdfb40d1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49848/0 (socket says 172.21.15.99:49848) 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.363+0000 7fdfb91c4640 1 -- 172.21.15.99:0/2644549605 learned_addr learned my addr 172.21.15.99:0/2644549605 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb89c3640 1 --2- 172.21.15.99:0/2644549605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb415bbb0 0x7fdfb40d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb37fe640 1 -- 172.21.15.99:0/2644549605 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060081264 0 0) 0x7fdfb40cfb40 con 0x7fdfb40cc690 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb37fe640 1 -- 172.21.15.99:0/2644549605 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa4003520 con 0x7fdfb40cc690 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb37fe640 1 -- 172.21.15.99:0/2644549605 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461545881 0 0) 0x7fdfb40cfe20 con 0x7fdfb40d0e40 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb89c3640 1 -- 172.21.15.99:0/2644549605 >> v1:172.21.15.162:6789/0 conn(0x7fdfb40d0e40 legacy=0x7fdfb40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb89c3640 1 -- 172.21.15.99:0/2644549605 >> v1:172.21.15.99:6789/0 conn(0x7fdfb40cc690 legacy=0x7fdfb40d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.364+0000 7fdfb89c3640 1 -- 172.21.15.99:0/2644549605 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb40d0100 con 0x7fdfb415bbb0 2024-03-04T10:04:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.365+0000 7fdfb89c3640 1 --2- 172.21.15.99:0/2644549605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb415bbb0 0x7fdfb40d0900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80097d0 tx=0x7fdfa802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=521deaaa8bbc6f9d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.365+0000 7fdfb37fe640 1 -- 172.21.15.99:0/2644549605 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8037070 con 0x7fdfb415bbb0 2024-03-04T10:04:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.365+0000 7fdfb37fe640 1 -- 172.21.15.99:0/2644549605 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa8030510 con 0x7fdfb415bbb0 2024-03-04T10:04:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.365+0000 7fdfb37fe640 1 -- 172.21.15.99:0/2644549605 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0019f0 con 0x7fdfb415bbb0 2024-03-04T10:04:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 -- 172.21.15.99:0/2644549605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb415bbb0 msgr2=0x7fdfb40d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 --2- 172.21.15.99:0/2644549605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb415bbb0 0x7fdfb40d0900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdfa80097d0 tx=0x7fdfa802b730 comp rx=0 tx=0).stop 2024-03-04T10:04:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 -- 172.21.15.99:0/2644549605 shutdown_connections 2024-03-04T10:04:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 --2- 172.21.15.99:0/2644549605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb415bbb0 0x7fdfb40d0900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 -- 172.21.15.99:0/2644549605 >> 172.21.15.99:0/2644549605 conn(0x7fdfb40b3480 msgr2=0x7fdfb40b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 -- 172.21.15.99:0/2644549605 shutdown_connections 2024-03-04T10:04:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.366+0000 7fdfbac27640 1 -- 172.21.15.99:0/2644549605 wait complete. 2024-03-04T10:04:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.367+0000 7fdfbac27640 1 Processor -- start 2024-03-04T10:04:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.367+0000 7fdfbac27640 1 -- start start 2024-03-04T10:04:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb40cc690 0x7fdfb4150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfb40d0e40 0x7fdfb406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 0x7fdfb40b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb415bf90 con 0x7fdfb415bbb0 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40d23b0 con 0x7fdfb40d0e40 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4151400 con 0x7fdfb40cc690 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 0x7fdfb40b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 0x7fdfb40b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45922/0 (socket says 172.21.15.99:45922) 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 -- 172.21.15.99:0/1746206832 learned_addr learned my addr 172.21.15.99:0/1746206832 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb3fff640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfb40d0e40 0x7fdfb406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb89c3640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb40cc690 0x7fdfb4150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 -- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb40cc690 msgr2=0x7fdfb4150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb40cc690 0x7fdfb4150ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 -- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfb40d0e40 msgr2=0x7fdfb406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfb40d0e40 0x7fdfb406d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 -- 172.21.15.99:0/1746206832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb406bf60 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb89c3640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb40cc690 0x7fdfb4150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb91c4640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 0x7fdfb40b4e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdfac007fd0 tx=0x7fdfac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac018070 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac00a7d0 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb406c240 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac00aba0 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.368+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb406c800 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.369+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb4074f60 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.370+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdfac01b070 con 0x7fdfb415bbb0 2024-03-04T10:04:02.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.370+0000 7fdfb17fa640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf9806b500 0x7fdf9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.371+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdfac089ea0 con 0x7fdfb415bbb0 2024-03-04T10:04:02.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.371+0000 7fdfb89c3640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf9806b500 0x7fdf9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.371+0000 7fdfb89c3640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf9806b500 0x7fdf9806d9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8034000 tx=0x7fdfa8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.375+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdfac0582f0 con 0x7fdfb415bbb0 2024-03-04T10:04:02.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.543+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdfb40d2090 con 0x7fdf9806b500 2024-03-04T10:04:02.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.552+0000 7fdfb17fa640 1 -- 172.21.15.99:0/1746206832 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332534 (secure 0 0 0) 0x7fdfb40d2090 con 0x7fdf9806b500 2024-03-04T10:04:02.558 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:02.562 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:04:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.555+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf9806b500 msgr2=0x7fdf9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf9806b500 0x7fdf9806d9c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8034000 tx=0x7fdfa8008030 comp rx=0 tx=0).stop 2024-03-04T10:04:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 msgr2=0x7fdfb40b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 0x7fdfb40b4e40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdfac007fd0 tx=0x7fdfac00a060 comp rx=0 tx=0).stop 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 shutdown_connections 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf9806b500 0x7fdf9806d9c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfb415bbb0 0x7fdfb40b4e40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfb40d0e40 0x7fdfb406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 --2- 172.21.15.99:0/1746206832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfb40cc690 0x7fdfb4150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 >> 172.21.15.99:0/1746206832 conn(0x7fdfb40b3480 msgr2=0x7fdfb40b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 shutdown_connections 2024-03-04T10:04:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.556+0000 7fdfbac27640 1 -- 172.21.15.99:0/1746206832 wait complete. 2024-03-04T10:04:02.569 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-04T10:04:01.441574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219388,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755332,"statfs":{"total":773094113280,"available":772869459968,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455527+0000","last_change":"2024-03-04T10:03:38.529898+0000","last_active":"2024-03-04T10:03:45.455527+0000","last_peered":"2024-03-04T10:03:45.455527+0000","last_clean":"2024-03-04T10:03:45.455527+0000","last_became_active":"2024-03-04T10:03:38.529499+0000","last_became_peered":"2024-03-04T10:03:38.529499+0000","last_unstale":"2024-03-04T10:03:45.455527+0000","last_undegraded":"2024-03-04T10:03:45.455527+0000","last_fullsized":"2024-03-04T10:03:45.455527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727461+0000","last_change":"2024-03-04T10:03:39.556883+0000","last_active":"2024-03-04T10:03:44.727461+0000","last_peered":"2024-03-04T10:03:44.727461+0000","last_clean":"2024-03-04T10:03:44.727461+0000","last_became_active":"2024-03-04T10:03:39.556616+0000","last_became_peered":"2024-03-04T10:03:39.556616+0000","last_unstale":"2024-03-04T10:03:44.727461+0000","last_undegraded":"2024-03-04T10:03:44.727461+0000","last_fullsized":"2024-03-04T10:03:44.727461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726874+0000","last_change":"2024-03-04T10:03:43.569338+0000","last_active":"2024-03-04T10:03:44.726874+0000","last_peered":"2024-03-04T10:03:44.726874+0000","last_clean":"2024-03-04T10:03:44.726874+0000","last_became_active":"2024-03-04T10:03:43.568332+0000","last_became_peered":"2024-03-04T10:03:43.568332+0000","last_unstale":"2024-03-04T10:03:44.726874+0000","last_undegraded":"2024-03-04T10:03:44.726874+0000","last_fullsized":"2024-03-04T10:03:44.726874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727559+0000","last_change":"2024-03-04T10:03:38.533956+0000","last_active":"2024-03-04T10:03:44.727559+0000","last_peered":"2024-03-04T10:03:44.727559+0000","last_clean":"2024-03-04T10:03:44.727559+0000","last_became_active":"2024-03-04T10:03:38.533665+0000","last_became_peered":"2024-03-04T10:03:38.533665+0000","last_unstale":"2024-03-04T10:03:44.727559+0000","last_undegraded":"2024-03-04T10:03:44.727559+0000","last_fullsized":"2024-03-04T10:03:44.727559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455218+0000","last_change":"2024-03-04T10:03:39.550341+0000","last_active":"2024-03-04T10:03:45.455218+0000","last_peered":"2024-03-04T10:03:45.455218+0000","last_clean":"2024-03-04T10:03:45.455218+0000","last_became_active":"2024-03-04T10:03:39.549960+0000","last_became_peered":"2024-03-04T10:03:39.549960+0000","last_unstale":"2024-03-04T10:03:45.455218+0000","last_undegraded":"2024-03-04T10:03:45.455218+0000","last_fullsized":"2024-03-04T10:03:45.455218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925071+0000","last_change":"2024-03-04T10:03:43.578315+0000","last_active":"2024-03-04T10:03:44.925071+0000","last_peered":"2024-03-04T10:03:44.925071+0000","last_clean":"2024-03-04T10:03:44.925071+0000","last_became_active":"2024-03-04T10:03:43.577864+0000","last_became_peered":"2024-03-04T10:03:43.577864+0000","last_unstale":"2024-03-04T10:03:44.925071+0000","last_undegraded":"2024-03-04T10:03:44.925071+0000","last_fullsized":"2024-03-04T10:03:44.925071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:27.973529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558121+0000","last_change":"2024-03-04T10:03:38.536815+0000","last_active":"2024-03-04T10:03:44.558121+0000","last_peered":"2024-03-04T10:03:44.558121+0000","last_clean":"2024-03-04T10:03:44.558121+0000","last_became_active":"2024-03-04T10:03:38.536526+0000","last_became_peered":"2024-03-04T10:03:38.536526+0000","last_unstale":"2024-03-04T10:03:44.558121+0000","last_undegraded":"2024-03-04T10:03:44.558121+0000","last_fullsized":"2024-03-04T10:03:44.558121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455699+0000","last_change":"2024-03-04T10:03:39.551796+0000","last_active":"2024-03-04T10:03:45.455699+0000","last_peered":"2024-03-04T10:03:45.455699+0000","last_clean":"2024-03-04T10:03:45.455699+0000","last_became_active":"2024-03-04T10:03:39.551527+0000","last_became_peered":"2024-03-04T10:03:39.551527+0000","last_unstale":"2024-03-04T10:03:45.455699+0000","last_undegraded":"2024-03-04T10:03:45.455699+0000","last_fullsized":"2024-03-04T10:03:45.455699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925354+0000","last_change":"2024-03-04T10:03:43.573372+0000","last_active":"2024-03-04T10:03:44.925354+0000","last_peered":"2024-03-04T10:03:44.925354+0000","last_clean":"2024-03-04T10:03:44.925354+0000","last_became_active":"2024-03-04T10:03:43.572926+0000","last_became_peered":"2024-03-04T10:03:43.572926+0000","last_unstale":"2024-03-04T10:03:44.925354+0000","last_undegraded":"2024-03-04T10:03:44.925354+0000","last_fullsized":"2024-03-04T10:03:44.925354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:37.210258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557408+0000","last_change":"2024-03-04T10:03:38.535163+0000","last_active":"2024-03-04T10:03:44.557408+0000","last_peered":"2024-03-04T10:03:44.557408+0000","last_clean":"2024-03-04T10:03:44.557408+0000","last_became_active":"2024-03-04T10:03:38.534751+0000","last_became_peered":"2024-03-04T10:03:38.534751+0000","last_unstale":"2024-03-04T10:03:44.557408+0000","last_undegraded":"2024-03-04T10:03:44.557408+0000","last_fullsized":"2024-03-04T10:03:44.557408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:40.978639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455945+0000","last_change":"2024-03-04T10:03:39.548628+0000","last_active":"2024-03-04T10:03:45.455945+0000","last_peered":"2024-03-04T10:03:45.455945+0000","last_clean":"2024-03-04T10:03:45.455945+0000","last_became_active":"2024-03-04T10:03:39.548297+0000","last_became_peered":"2024-03-04T10:03:39.548297+0000","last_unstale":"2024-03-04T10:03:45.455945+0000","last_undegraded":"2024-03-04T10:03:45.455945+0000","last_fullsized":"2024-03-04T10:03:45.455945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925131+0000","last_change":"2024-03-04T10:03:43.576063+0000","last_active":"2024-03-04T10:03:44.925131+0000","last_peered":"2024-03-04T10:03:44.925131+0000","last_clean":"2024-03-04T10:03:44.925131+0000","last_became_active":"2024-03-04T10:03:43.575658+0000","last_became_peered":"2024-03-04T10:03:43.575658+0000","last_unstale":"2024-03-04T10:03:44.925131+0000","last_undegraded":"2024-03-04T10:03:44.925131+0000","last_fullsized":"2024-03-04T10:03:44.925131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:47.766859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557898+0000","last_change":"2024-03-04T10:03:39.562772+0000","last_active":"2024-03-04T10:03:44.557898+0000","last_peered":"2024-03-04T10:03:44.557898+0000","last_clean":"2024-03-04T10:03:44.557898+0000","last_became_active":"2024-03-04T10:03:39.562479+0000","last_became_peered":"2024-03-04T10:03:39.562479+0000","last_unstale":"2024-03-04T10:03:44.557898+0000","last_undegraded":"2024-03-04T10:03:44.557898+0000","last_fullsized":"2024-03-04T10:03:44.557898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:48.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558549+0000","last_change":"2024-03-04T10:03:38.524487+0000","last_active":"2024-03-04T10:03:44.558549+0000","last_peered":"2024-03-04T10:03:44.558549+0000","last_clean":"2024-03-04T10:03:44.558549+0000","last_became_active":"2024-03-04T10:03:38.524186+0000","last_became_peered":"2024-03-04T10:03:38.524186+0000","last_unstale":"2024-03-04T10:03:44.558549+0000","last_undegraded":"2024-03-04T10:03:44.558549+0000","last_fullsized":"2024-03-04T10:03:44.558549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925560+0000","last_change":"2024-03-04T10:03:43.573621+0000","last_active":"2024-03-04T10:03:44.925560+0000","last_peered":"2024-03-04T10:03:44.925560+0000","last_clean":"2024-03-04T10:03:44.925560+0000","last_became_active":"2024-03-04T10:03:43.573125+0000","last_became_peered":"2024-03-04T10:03:43.573125+0000","last_unstale":"2024-03-04T10:03:44.925560+0000","last_undegraded":"2024-03-04T10:03:44.925560+0000","last_fullsized":"2024-03-04T10:03:44.925560+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727176+0000","last_change":"2024-03-04T10:03:39.553163+0000","last_active":"2024-03-04T10:03:44.727176+0000","last_peered":"2024-03-04T10:03:44.727176+0000","last_clean":"2024-03-04T10:03:44.727176+0000","last_became_active":"2024-03-04T10:03:39.552909+0000","last_became_peered":"2024-03-04T10:03:39.552909+0000","last_unstale":"2024-03-04T10:03:44.727176+0000","last_undegraded":"2024-03-04T10:03:44.727176+0000","last_fullsized":"2024-03-04T10:03:44.727176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455728+0000","last_change":"2024-03-04T10:03:38.534433+0000","last_active":"2024-03-04T10:03:45.455728+0000","last_peered":"2024-03-04T10:03:45.455728+0000","last_clean":"2024-03-04T10:03:45.455728+0000","last_became_active":"2024-03-04T10:03:38.534162+0000","last_became_peered":"2024-03-04T10:03:38.534162+0000","last_unstale":"2024-03-04T10:03:45.455728+0000","last_undegraded":"2024-03-04T10:03:45.455728+0000","last_fullsized":"2024-03-04T10:03:45.455728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925353+0000","last_change":"2024-03-04T10:03:43.572145+0000","last_active":"2024-03-04T10:03:44.925353+0000","last_peered":"2024-03-04T10:03:44.925353+0000","last_clean":"2024-03-04T10:03:44.925353+0000","last_became_active":"2024-03-04T10:03:43.571762+0000","last_became_peered":"2024-03-04T10:03:43.571762+0000","last_unstale":"2024-03-04T10:03:44.925353+0000","last_undegraded":"2024-03-04T10:03:44.925353+0000","last_fullsized":"2024-03-04T10:03:44.925353+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:43.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727042+0000","last_change":"2024-03-04T10:03:39.558092+0000","last_active":"2024-03-04T10:03:44.727042+0000","last_peered":"2024-03-04T10:03:44.727042+0000","last_clean":"2024-03-04T10:03:44.727042+0000","last_became_active":"2024-03-04T10:03:39.557838+0000","last_became_peered":"2024-03-04T10:03:39.557838+0000","last_unstale":"2024-03-04T10:03:44.727042+0000","last_undegraded":"2024-03-04T10:03:44.727042+0000","last_fullsized":"2024-03-04T10:03:44.727042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456127+0000","last_change":"2024-03-04T10:03:38.538667+0000","last_active":"2024-03-04T10:03:45.456127+0000","last_peered":"2024-03-04T10:03:45.456127+0000","last_clean":"2024-03-04T10:03:45.456127+0000","last_became_active":"2024-03-04T10:03:38.538344+0000","last_became_peered":"2024-03-04T10:03:38.538344+0000","last_unstale":"2024-03-04T10:03:45.456127+0000","last_undegraded":"2024-03-04T10:03:45.456127+0000","last_fullsized":"2024-03-04T10:03:45.456127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456240+0000","last_change":"2024-03-04T10:03:43.583702+0000","last_active":"2024-03-04T10:03:45.456240+0000","last_peered":"2024-03-04T10:03:45.456240+0000","last_clean":"2024-03-04T10:03:45.456240+0000","last_became_active":"2024-03-04T10:03:43.583287+0000","last_became_peered":"2024-03-04T10:03:43.583287+0000","last_unstale":"2024-03-04T10:03:45.456240+0000","last_undegraded":"2024-03-04T10:03:45.456240+0000","last_fullsized":"2024-03-04T10:03:45.456240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:26.836809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925694+0000","last_change":"2024-03-04T10:03:39.542379+0000","last_active":"2024-03-04T10:03:44.925694+0000","last_peered":"2024-03-04T10:03:44.925694+0000","last_clean":"2024-03-04T10:03:44.925694+0000","last_became_active":"2024-03-04T10:03:39.541991+0000","last_became_peered":"2024-03-04T10:03:39.541991+0000","last_unstale":"2024-03-04T10:03:44.925694+0000","last_undegraded":"2024-03-04T10:03:44.925694+0000","last_fullsized":"2024-03-04T10:03:44.925694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:48.539804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557756+0000","last_change":"2024-03-04T10:03:38.532133+0000","last_active":"2024-03-04T10:03:44.557756+0000","last_peered":"2024-03-04T10:03:44.557756+0000","last_clean":"2024-03-04T10:03:44.557756+0000","last_became_active":"2024-03-04T10:03:38.531622+0000","last_became_peered":"2024-03-04T10:03:38.531622+0000","last_unstale":"2024-03-04T10:03:44.557756+0000","last_undegraded":"2024-03-04T10:03:44.557756+0000","last_fullsized":"2024-03-04T10:03:44.557756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925652+0000","last_change":"2024-03-04T10:03:43.570872+0000","last_active":"2024-03-04T10:03:44.925652+0000","last_peered":"2024-03-04T10:03:44.925652+0000","last_clean":"2024-03-04T10:03:44.925652+0000","last_became_active":"2024-03-04T10:03:43.570436+0000","last_became_peered":"2024-03-04T10:03:43.570436+0000","last_unstale":"2024-03-04T10:03:44.925652+0000","last_undegraded":"2024-03-04T10:03:44.925652+0000","last_fullsized":"2024-03-04T10:03:44.925652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455648+0000","last_change":"2024-03-04T10:03:39.541376+0000","last_active":"2024-03-04T10:03:45.455648+0000","last_peered":"2024-03-04T10:03:45.455648+0000","last_clean":"2024-03-04T10:03:45.455648+0000","last_became_active":"2024-03-04T10:03:39.541117+0000","last_became_peered":"2024-03-04T10:03:39.541117+0000","last_unstale":"2024-03-04T10:03:45.455648+0000","last_undegraded":"2024-03-04T10:03:45.455648+0000","last_fullsized":"2024-03-04T10:03:45.455648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558575+0000","last_change":"2024-03-04T10:03:38.538684+0000","last_active":"2024-03-04T10:03:44.558575+0000","last_peered":"2024-03-04T10:03:44.558575+0000","last_clean":"2024-03-04T10:03:44.558575+0000","last_became_active":"2024-03-04T10:03:38.538374+0000","last_became_peered":"2024-03-04T10:03:38.538374+0000","last_unstale":"2024-03-04T10:03:44.558575+0000","last_undegraded":"2024-03-04T10:03:44.558575+0000","last_fullsized":"2024-03-04T10:03:44.558575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925011+0000","last_change":"2024-03-04T10:03:43.575117+0000","last_active":"2024-03-04T10:03:44.925011+0000","last_peered":"2024-03-04T10:03:44.925011+0000","last_clean":"2024-03-04T10:03:44.925011+0000","last_became_active":"2024-03-04T10:03:43.574283+0000","last_became_peered":"2024-03-04T10:03:43.574283+0000","last_unstale":"2024-03-04T10:03:44.925011+0000","last_undegraded":"2024-03-04T10:03:44.925011+0000","last_fullsized":"2024-03-04T10:03:44.925011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:09.615407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727535+0000","last_change":"2024-03-04T10:03:39.554004+0000","last_active":"2024-03-04T10:03:44.727535+0000","last_peered":"2024-03-04T10:03:44.727535+0000","last_clean":"2024-03-04T10:03:44.727535+0000","last_became_active":"2024-03-04T10:03:39.553717+0000","last_became_peered":"2024-03-04T10:03:39.553717+0000","last_unstale":"2024-03-04T10:03:44.727535+0000","last_undegraded":"2024-03-04T10:03:44.727535+0000","last_fullsized":"2024-03-04T10:03:44.727535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455963+0000","last_change":"2024-03-04T10:03:38.533053+0000","last_active":"2024-03-04T10:03:45.455963+0000","last_peered":"2024-03-04T10:03:45.455963+0000","last_clean":"2024-03-04T10:03:45.455963+0000","last_became_active":"2024-03-04T10:03:38.532793+0000","last_became_peered":"2024-03-04T10:03:38.532793+0000","last_unstale":"2024-03-04T10:03:45.455963+0000","last_undegraded":"2024-03-04T10:03:45.455963+0000","last_fullsized":"2024-03-04T10:03:45.455963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726840+0000","last_change":"2024-03-04T10:03:43.577818+0000","last_active":"2024-03-04T10:03:44.726840+0000","last_peered":"2024-03-04T10:03:44.726840+0000","last_clean":"2024-03-04T10:03:44.726840+0000","last_became_active":"2024-03-04T10:03:43.577239+0000","last_became_peered":"2024-03-04T10:03:43.577239+0000","last_unstale":"2024-03-04T10:03:44.726840+0000","last_undegraded":"2024-03-04T10:03:44.726840+0000","last_fullsized":"2024-03-04T10:03:44.726840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455944+0000","last_change":"2024-03-04T10:03:39.542820+0000","last_active":"2024-03-04T10:03:45.455944+0000","last_peered":"2024-03-04T10:03:45.455944+0000","last_clean":"2024-03-04T10:03:45.455944+0000","last_became_active":"2024-03-04T10:03:39.542521+0000","last_became_peered":"2024-03-04T10:03:39.542521+0000","last_unstale":"2024-03-04T10:03:45.455944+0000","last_undegraded":"2024-03-04T10:03:45.455944+0000","last_fullsized":"2024-03-04T10:03:45.455944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558219+0000","last_change":"2024-03-04T10:03:38.534337+0000","last_active":"2024-03-04T10:03:44.558219+0000","last_peered":"2024-03-04T10:03:44.558219+0000","last_clean":"2024-03-04T10:03:44.558219+0000","last_became_active":"2024-03-04T10:03:38.534027+0000","last_became_peered":"2024-03-04T10:03:38.534027+0000","last_unstale":"2024-03-04T10:03:44.558219+0000","last_undegraded":"2024-03-04T10:03:44.558219+0000","last_fullsized":"2024-03-04T10:03:44.558219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:18.373547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455176+0000","last_change":"2024-03-04T10:03:43.577959+0000","last_active":"2024-03-04T10:03:45.455176+0000","last_peered":"2024-03-04T10:03:45.455176+0000","last_clean":"2024-03-04T10:03:45.455176+0000","last_became_active":"2024-03-04T10:03:43.576536+0000","last_became_peered":"2024-03-04T10:03:43.576536+0000","last_unstale":"2024-03-04T10:03:45.455176+0000","last_undegraded":"2024-03-04T10:03:45.455176+0000","last_fullsized":"2024-03-04T10:03:45.455176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456091+0000","last_change":"2024-03-04T10:03:39.556129+0000","last_active":"2024-03-04T10:03:45.456091+0000","last_peered":"2024-03-04T10:03:45.456091+0000","last_clean":"2024-03-04T10:03:45.456091+0000","last_became_active":"2024-03-04T10:03:39.555824+0000","last_became_peered":"2024-03-04T10:03:39.555824+0000","last_unstale":"2024-03-04T10:03:45.456091+0000","last_undegraded":"2024-03-04T10:03:45.456091+0000","last_fullsized":"2024-03-04T10:03:45.456091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456030+0000","last_change":"2024-03-04T10:03:38.531643+0000","last_active":"2024-03-04T10:03:45.456030+0000","last_peered":"2024-03-04T10:03:45.456030+0000","last_clean":"2024-03-04T10:03:45.456030+0000","last_became_active":"2024-03-04T10:03:38.531299+0000","last_became_peered":"2024-03-04T10:03:38.531299+0000","last_unstale":"2024-03-04T10:03:45.456030+0000","last_undegraded":"2024-03-04T10:03:45.456030+0000","last_fullsized":"2024-03-04T10:03:45.456030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455431+0000","last_change":"2024-03-04T10:03:43.576642+0000","last_active":"2024-03-04T10:03:45.455431+0000","last_peered":"2024-03-04T10:03:45.455431+0000","last_clean":"2024-03-04T10:03:45.455431+0000","last_became_active":"2024-03-04T10:03:43.576270+0000","last_became_peered":"2024-03-04T10:03:43.576270+0000","last_unstale":"2024-03-04T10:03:45.455431+0000","last_undegraded":"2024-03-04T10:03:45.455431+0000","last_fullsized":"2024-03-04T10:03:45.455431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925763+0000","last_change":"2024-03-04T10:03:39.546721+0000","last_active":"2024-03-04T10:03:44.925763+0000","last_peered":"2024-03-04T10:03:44.925763+0000","last_clean":"2024-03-04T10:03:44.925763+0000","last_became_active":"2024-03-04T10:03:39.546438+0000","last_became_peered":"2024-03-04T10:03:39.546438+0000","last_unstale":"2024-03-04T10:03:44.925763+0000","last_undegraded":"2024-03-04T10:03:44.925763+0000","last_fullsized":"2024-03-04T10:03:44.925763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455730+0000","last_change":"2024-03-04T10:03:38.534806+0000","last_active":"2024-03-04T10:03:45.455730+0000","last_peered":"2024-03-04T10:03:45.455730+0000","last_clean":"2024-03-04T10:03:45.455730+0000","last_became_active":"2024-03-04T10:03:38.534555+0000","last_became_peered":"2024-03-04T10:03:38.534555+0000","last_unstale":"2024-03-04T10:03:45.455730+0000","last_undegraded":"2024-03-04T10:03:45.455730+0000","last_fullsized":"2024-03-04T10:03:45.455730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455898+0000","last_change":"2024-03-04T10:03:43.583916+0000","last_active":"2024-03-04T10:03:45.455898+0000","last_peered":"2024-03-04T10:03:45.455898+0000","last_clean":"2024-03-04T10:03:45.455898+0000","last_became_active":"2024-03-04T10:03:43.583537+0000","last_became_peered":"2024-03-04T10:03:43.583537+0000","last_unstale":"2024-03-04T10:03:45.455898+0000","last_undegraded":"2024-03-04T10:03:45.455898+0000","last_fullsized":"2024-03-04T10:03:45.455898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:46.674144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454949+0000","last_change":"2024-03-04T10:03:39.539248+0000","last_active":"2024-03-04T10:03:45.454949+0000","last_peered":"2024-03-04T10:03:45.454949+0000","last_clean":"2024-03-04T10:03:45.454949+0000","last_became_active":"2024-03-04T10:03:39.538978+0000","last_became_peered":"2024-03-04T10:03:39.538978+0000","last_unstale":"2024-03-04T10:03:45.454949+0000","last_undegraded":"2024-03-04T10:03:45.454949+0000","last_fullsized":"2024-03-04T10:03:45.454949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556029+0000","last_change":"2024-03-04T10:03:38.534970+0000","last_active":"2024-03-04T10:03:44.556029+0000","last_peered":"2024-03-04T10:03:44.556029+0000","last_clean":"2024-03-04T10:03:44.556029+0000","last_became_active":"2024-03-04T10:03:38.534694+0000","last_became_peered":"2024-03-04T10:03:38.534694+0000","last_unstale":"2024-03-04T10:03:44.556029+0000","last_undegraded":"2024-03-04T10:03:44.556029+0000","last_fullsized":"2024-03-04T10:03:44.556029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456270+0000","last_change":"2024-03-04T10:03:43.582855+0000","last_active":"2024-03-04T10:03:45.456270+0000","last_peered":"2024-03-04T10:03:45.456270+0000","last_clean":"2024-03-04T10:03:45.456270+0000","last_became_active":"2024-03-04T10:03:43.580626+0000","last_became_peered":"2024-03-04T10:03:43.580626+0000","last_unstale":"2024-03-04T10:03:45.456270+0000","last_undegraded":"2024-03-04T10:03:45.456270+0000","last_fullsized":"2024-03-04T10:03:45.456270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:39.012948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454952+0000","last_change":"2024-03-04T10:03:39.538900+0000","last_active":"2024-03-04T10:03:45.454952+0000","last_peered":"2024-03-04T10:03:45.454952+0000","last_clean":"2024-03-04T10:03:45.454952+0000","last_became_active":"2024-03-04T10:03:39.538624+0000","last_became_peered":"2024-03-04T10:03:39.538624+0000","last_unstale":"2024-03-04T10:03:45.454952+0000","last_undegraded":"2024-03-04T10:03:45.454952+0000","last_fullsized":"2024-03-04T10:03:45.454952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557795+0000","last_change":"2024-03-04T10:03:38.534850+0000","last_active":"2024-03-04T10:03:44.557795+0000","last_peered":"2024-03-04T10:03:44.557795+0000","last_clean":"2024-03-04T10:03:44.557795+0000","last_became_active":"2024-03-04T10:03:38.534604+0000","last_became_peered":"2024-03-04T10:03:38.534604+0000","last_unstale":"2024-03-04T10:03:44.557795+0000","last_undegraded":"2024-03-04T10:03:44.557795+0000","last_fullsized":"2024-03-04T10:03:44.557795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455383+0000","last_change":"2024-03-04T10:03:43.576525+0000","last_active":"2024-03-04T10:03:45.455383+0000","last_peered":"2024-03-04T10:03:45.455383+0000","last_clean":"2024-03-04T10:03:45.455383+0000","last_became_active":"2024-03-04T10:03:43.576149+0000","last_became_peered":"2024-03-04T10:03:43.576149+0000","last_unstale":"2024-03-04T10:03:45.455383+0000","last_undegraded":"2024-03-04T10:03:45.455383+0000","last_fullsized":"2024-03-04T10:03:45.455383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727079+0000","last_change":"2024-03-04T10:03:39.557088+0000","last_active":"2024-03-04T10:03:44.727079+0000","last_peered":"2024-03-04T10:03:44.727079+0000","last_clean":"2024-03-04T10:03:44.727079+0000","last_became_active":"2024-03-04T10:03:39.556715+0000","last_became_peered":"2024-03-04T10:03:39.556715+0000","last_unstale":"2024-03-04T10:03:44.727079+0000","last_undegraded":"2024-03-04T10:03:44.727079+0000","last_fullsized":"2024-03-04T10:03:44.727079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455694+0000","last_change":"2024-03-04T10:03:38.537571+0000","last_active":"2024-03-04T10:03:45.455694+0000","last_peered":"2024-03-04T10:03:45.455694+0000","last_clean":"2024-03-04T10:03:45.455694+0000","last_became_active":"2024-03-04T10:03:38.536646+0000","last_became_peered":"2024-03-04T10:03:38.536646+0000","last_unstale":"2024-03-04T10:03:45.455694+0000","last_undegraded":"2024-03-04T10:03:45.455694+0000","last_fullsized":"2024-03-04T10:03:45.455694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557928+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:03:44.557928+0000","last_peered":"2024-03-04T10:03:44.557928+0000","last_clean":"2024-03-04T10:03:44.557928+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:03:44.557928+0000","last_undegraded":"2024-03-04T10:03:44.557928+0000","last_fullsized":"2024-03-04T10:03:44.557928+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558259+0000","last_change":"2024-03-04T10:03:43.579804+0000","last_active":"2024-03-04T10:03:44.558259+0000","last_peered":"2024-03-04T10:03:44.558259+0000","last_clean":"2024-03-04T10:03:44.558259+0000","last_became_active":"2024-03-04T10:03:43.579420+0000","last_became_peered":"2024-03-04T10:03:43.579420+0000","last_unstale":"2024-03-04T10:03:44.558259+0000","last_undegraded":"2024-03-04T10:03:44.558259+0000","last_fullsized":"2024-03-04T10:03:44.558259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557993+0000","last_change":"2024-03-04T10:03:39.550717+0000","last_active":"2024-03-04T10:03:44.557993+0000","last_peered":"2024-03-04T10:03:44.557993+0000","last_clean":"2024-03-04T10:03:44.557993+0000","last_became_active":"2024-03-04T10:03:39.550331+0000","last_became_peered":"2024-03-04T10:03:39.550331+0000","last_unstale":"2024-03-04T10:03:44.557993+0000","last_undegraded":"2024-03-04T10:03:44.557993+0000","last_fullsized":"2024-03-04T10:03:44.557993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556242+0000","last_change":"2024-03-04T10:03:38.529200+0000","last_active":"2024-03-04T10:03:44.556242+0000","last_peered":"2024-03-04T10:03:44.556242+0000","last_clean":"2024-03-04T10:03:44.556242+0000","last_became_active":"2024-03-04T10:03:38.528943+0000","last_became_peered":"2024-03-04T10:03:38.528943+0000","last_unstale":"2024-03-04T10:03:44.556242+0000","last_undegraded":"2024-03-04T10:03:44.556242+0000","last_fullsized":"2024-03-04T10:03:44.556242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557839+0000","last_change":"2024-03-04T10:03:43.578553+0000","last_active":"2024-03-04T10:03:44.557839+0000","last_peered":"2024-03-04T10:03:44.557839+0000","last_clean":"2024-03-04T10:03:44.557839+0000","last_became_active":"2024-03-04T10:03:43.577929+0000","last_became_peered":"2024-03-04T10:03:43.577929+0000","last_unstale":"2024-03-04T10:03:44.557839+0000","last_undegraded":"2024-03-04T10:03:44.557839+0000","last_fullsized":"2024-03-04T10:03:44.557839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455715+0000","last_change":"2024-03-04T10:03:39.556135+0000","last_active":"2024-03-04T10:03:45.455715+0000","last_peered":"2024-03-04T10:03:45.455715+0000","last_clean":"2024-03-04T10:03:45.455715+0000","last_became_active":"2024-03-04T10:03:39.555846+0000","last_became_peered":"2024-03-04T10:03:39.555846+0000","last_unstale":"2024-03-04T10:03:45.455715+0000","last_undegraded":"2024-03-04T10:03:45.455715+0000","last_fullsized":"2024-03-04T10:03:45.455715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455561+0000","last_change":"2024-03-04T10:03:38.530735+0000","last_active":"2024-03-04T10:03:45.455561+0000","last_peered":"2024-03-04T10:03:45.455561+0000","last_clean":"2024-03-04T10:03:45.455561+0000","last_became_active":"2024-03-04T10:03:38.530463+0000","last_became_peered":"2024-03-04T10:03:38.530463+0000","last_unstale":"2024-03-04T10:03:45.455561+0000","last_undegraded":"2024-03-04T10:03:45.455561+0000","last_fullsized":"2024-03-04T10:03:45.455561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455391+0000","last_change":"2024-03-04T10:03:43.577622+0000","last_active":"2024-03-04T10:03:45.455391+0000","last_peered":"2024-03-04T10:03:45.455391+0000","last_clean":"2024-03-04T10:03:45.455391+0000","last_became_active":"2024-03-04T10:03:43.575914+0000","last_became_peered":"2024-03-04T10:03:43.575914+0000","last_unstale":"2024-03-04T10:03:45.455391+0000","last_undegraded":"2024-03-04T10:03:45.455391+0000","last_fullsized":"2024-03-04T10:03:45.455391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558308+0000","last_change":"2024-03-04T10:03:39.551071+0000","last_active":"2024-03-04T10:03:44.558308+0000","last_peered":"2024-03-04T10:03:44.558308+0000","last_clean":"2024-03-04T10:03:44.558308+0000","last_became_active":"2024-03-04T10:03:39.550739+0000","last_became_peered":"2024-03-04T10:03:39.550739+0000","last_unstale":"2024-03-04T10:03:44.558308+0000","last_undegraded":"2024-03-04T10:03:44.558308+0000","last_fullsized":"2024-03-04T10:03:44.558308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:14.185049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556247+0000","last_change":"2024-03-04T10:03:38.534772+0000","last_active":"2024-03-04T10:03:44.556247+0000","last_peered":"2024-03-04T10:03:44.556247+0000","last_clean":"2024-03-04T10:03:44.556247+0000","last_became_active":"2024-03-04T10:03:38.534520+0000","last_became_peered":"2024-03-04T10:03:38.534520+0000","last_unstale":"2024-03-04T10:03:44.556247+0000","last_undegraded":"2024-03-04T10:03:44.556247+0000","last_fullsized":"2024-03-04T10:03:44.556247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455292+0000","last_change":"2024-03-04T10:03:43.570591+0000","last_active":"2024-03-04T10:03:45.455292+0000","last_peered":"2024-03-04T10:03:45.455292+0000","last_clean":"2024-03-04T10:03:45.455292+0000","last_became_active":"2024-03-04T10:03:43.570154+0000","last_became_peered":"2024-03-04T10:03:43.570154+0000","last_unstale":"2024-03-04T10:03:45.455292+0000","last_undegraded":"2024-03-04T10:03:45.455292+0000","last_fullsized":"2024-03-04T10:03:45.455292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558471+0000","last_change":"2024-03-04T10:03:39.547823+0000","last_active":"2024-03-04T10:03:44.558471+0000","last_peered":"2024-03-04T10:03:44.558471+0000","last_clean":"2024-03-04T10:03:44.558471+0000","last_became_active":"2024-03-04T10:03:39.547553+0000","last_became_peered":"2024-03-04T10:03:39.547553+0000","last_unstale":"2024-03-04T10:03:44.558471+0000","last_undegraded":"2024-03-04T10:03:44.558471+0000","last_fullsized":"2024-03-04T10:03:44.558471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925780+0000","last_change":"2024-03-04T10:03:38.530675+0000","last_active":"2024-03-04T10:03:44.925780+0000","last_peered":"2024-03-04T10:03:44.925780+0000","last_clean":"2024-03-04T10:03:44.925780+0000","last_became_active":"2024-03-04T10:03:38.530406+0000","last_became_peered":"2024-03-04T10:03:38.530406+0000","last_unstale":"2024-03-04T10:03:44.925780+0000","last_undegraded":"2024-03-04T10:03:44.925780+0000","last_fullsized":"2024-03-04T10:03:44.925780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:29.050824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925002+0000","last_change":"2024-03-04T10:03:43.577882+0000","last_active":"2024-03-04T10:03:44.925002+0000","last_peered":"2024-03-04T10:03:44.925002+0000","last_clean":"2024-03-04T10:03:44.925002+0000","last_became_active":"2024-03-04T10:03:43.577500+0000","last_became_peered":"2024-03-04T10:03:43.577500+0000","last_unstale":"2024-03-04T10:03:44.925002+0000","last_undegraded":"2024-03-04T10:03:44.925002+0000","last_fullsized":"2024-03-04T10:03:44.925002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:38.890692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557934+0000","last_change":"2024-03-04T10:03:39.546483+0000","last_active":"2024-03-04T10:03:44.557934+0000","last_peered":"2024-03-04T10:03:44.557934+0000","last_clean":"2024-03-04T10:03:44.557934+0000","last_became_active":"2024-03-04T10:03:39.546211+0000","last_became_peered":"2024-03-04T10:03:39.546211+0000","last_unstale":"2024-03-04T10:03:44.557934+0000","last_undegraded":"2024-03-04T10:03:44.557934+0000","last_fullsized":"2024-03-04T10:03:44.557934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555858+0000","last_change":"2024-03-04T10:03:38.532190+0000","last_active":"2024-03-04T10:03:44.555858+0000","last_peered":"2024-03-04T10:03:44.555858+0000","last_clean":"2024-03-04T10:03:44.555858+0000","last_became_active":"2024-03-04T10:03:38.531859+0000","last_became_peered":"2024-03-04T10:03:38.531859+0000","last_unstale":"2024-03-04T10:03:44.555858+0000","last_undegraded":"2024-03-04T10:03:44.555858+0000","last_fullsized":"2024-03-04T10:03:44.555858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:30:57.662758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455223+0000","last_change":"2024-03-04T10:03:43.575386+0000","last_active":"2024-03-04T10:03:45.455223+0000","last_peered":"2024-03-04T10:03:45.455223+0000","last_clean":"2024-03-04T10:03:45.455223+0000","last_became_active":"2024-03-04T10:03:43.575009+0000","last_became_peered":"2024-03-04T10:03:43.575009+0000","last_unstale":"2024-03-04T10:03:45.455223+0000","last_undegraded":"2024-03-04T10:03:45.455223+0000","last_fullsized":"2024-03-04T10:03:45.455223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:05.911238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556867+0000","last_change":"2024-03-04T10:03:39.546825+0000","last_active":"2024-03-04T10:03:44.556867+0000","last_peered":"2024-03-04T10:03:44.556867+0000","last_clean":"2024-03-04T10:03:44.556867+0000","last_became_active":"2024-03-04T10:03:39.546567+0000","last_became_peered":"2024-03-04T10:03:39.546567+0000","last_unstale":"2024-03-04T10:03:44.556867+0000","last_undegraded":"2024-03-04T10:03:44.556867+0000","last_fullsized":"2024-03-04T10:03:44.556867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557579+0000","last_change":"2024-03-04T10:03:38.529048+0000","last_active":"2024-03-04T10:03:44.557579+0000","last_peered":"2024-03-04T10:03:44.557579+0000","last_clean":"2024-03-04T10:03:44.557579+0000","last_became_active":"2024-03-04T10:03:38.528276+0000","last_became_peered":"2024-03-04T10:03:38.528276+0000","last_unstale":"2024-03-04T10:03:44.557579+0000","last_undegraded":"2024-03-04T10:03:44.557579+0000","last_fullsized":"2024-03-04T10:03:44.557579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727557+0000","last_change":"2024-03-04T10:03:43.578301+0000","last_active":"2024-03-04T10:03:44.727557+0000","last_peered":"2024-03-04T10:03:44.727557+0000","last_clean":"2024-03-04T10:03:44.727557+0000","last_became_active":"2024-03-04T10:03:43.577909+0000","last_became_peered":"2024-03-04T10:03:43.577909+0000","last_unstale":"2024-03-04T10:03:44.727557+0000","last_undegraded":"2024-03-04T10:03:44.727557+0000","last_fullsized":"2024-03-04T10:03:44.727557+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:37.260262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558519+0000","last_change":"2024-03-04T10:03:39.547091+0000","last_active":"2024-03-04T10:03:44.558519+0000","last_peered":"2024-03-04T10:03:44.558519+0000","last_clean":"2024-03-04T10:03:44.558519+0000","last_became_active":"2024-03-04T10:03:39.546836+0000","last_became_peered":"2024-03-04T10:03:39.546836+0000","last_unstale":"2024-03-04T10:03:44.558519+0000","last_undegraded":"2024-03-04T10:03:44.558519+0000","last_fullsized":"2024-03-04T10:03:44.558519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455647+0000","last_change":"2024-03-04T10:03:38.535462+0000","last_active":"2024-03-04T10:03:45.455647+0000","last_peered":"2024-03-04T10:03:45.455647+0000","last_clean":"2024-03-04T10:03:45.455647+0000","last_became_active":"2024-03-04T10:03:38.535165+0000","last_became_peered":"2024-03-04T10:03:38.535165+0000","last_unstale":"2024-03-04T10:03:45.455647+0000","last_undegraded":"2024-03-04T10:03:45.455647+0000","last_fullsized":"2024-03-04T10:03:45.455647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727307+0000","last_change":"2024-03-04T10:03:43.576692+0000","last_active":"2024-03-04T10:03:44.727307+0000","last_peered":"2024-03-04T10:03:44.727307+0000","last_clean":"2024-03-04T10:03:44.727307+0000","last_became_active":"2024-03-04T10:03:43.576321+0000","last_became_peered":"2024-03-04T10:03:43.576321+0000","last_unstale":"2024-03-04T10:03:44.727307+0000","last_undegraded":"2024-03-04T10:03:44.727307+0000","last_fullsized":"2024-03-04T10:03:44.727307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:14:32.702691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455942+0000","last_change":"2024-03-04T10:03:39.547184+0000","last_active":"2024-03-04T10:03:45.455942+0000","last_peered":"2024-03-04T10:03:45.455942+0000","last_clean":"2024-03-04T10:03:45.455942+0000","last_became_active":"2024-03-04T10:03:39.546899+0000","last_became_peered":"2024-03-04T10:03:39.546899+0000","last_unstale":"2024-03-04T10:03:45.455942+0000","last_undegraded":"2024-03-04T10:03:45.455942+0000","last_fullsized":"2024-03-04T10:03:45.455942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455444+0000","last_change":"2024-03-04T10:03:38.529853+0000","last_active":"2024-03-04T10:03:45.455444+0000","last_peered":"2024-03-04T10:03:45.455444+0000","last_clean":"2024-03-04T10:03:45.455444+0000","last_became_active":"2024-03-04T10:03:38.529259+0000","last_became_peered":"2024-03-04T10:03:38.529259+0000","last_unstale":"2024-03-04T10:03:45.455444+0000","last_undegraded":"2024-03-04T10:03:45.455444+0000","last_fullsized":"2024-03-04T10:03:45.455444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456026+0000","last_change":"2024-03-04T10:03:43.572841+0000","last_active":"2024-03-04T10:03:45.456026+0000","last_peered":"2024-03-04T10:03:45.456026+0000","last_clean":"2024-03-04T10:03:45.456026+0000","last_became_active":"2024-03-04T10:03:43.572414+0000","last_became_peered":"2024-03-04T10:03:43.572414+0000","last_unstale":"2024-03-04T10:03:45.456026+0000","last_undegraded":"2024-03-04T10:03:45.456026+0000","last_fullsized":"2024-03-04T10:03:45.456026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557911+0000","last_change":"2024-03-04T10:03:39.552501+0000","last_active":"2024-03-04T10:03:44.557911+0000","last_peered":"2024-03-04T10:03:44.557911+0000","last_clean":"2024-03-04T10:03:44.557911+0000","last_became_active":"2024-03-04T10:03:39.551870+0000","last_became_peered":"2024-03-04T10:03:39.551870+0000","last_unstale":"2024-03-04T10:03:44.557911+0000","last_undegraded":"2024-03-04T10:03:44.557911+0000","last_fullsized":"2024-03-04T10:03:44.557911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:48.793130+0000","last_change":"2024-03-04T10:03:38.534457+0000","last_active":"2024-03-04T10:03:48.793130+0000","last_peered":"2024-03-04T10:03:48.793130+0000","last_clean":"2024-03-04T10:03:48.793130+0000","last_became_active":"2024-03-04T10:03:38.534185+0000","last_became_peered":"2024-03-04T10:03:38.534185+0000","last_unstale":"2024-03-04T10:03:48.793130+0000","last_undegraded":"2024-03-04T10:03:48.793130+0000","last_fullsized":"2024-03-04T10:03:48.793130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455693+0000","last_change":"2024-03-04T10:03:43.572170+0000","last_active":"2024-03-04T10:03:45.455693+0000","last_peered":"2024-03-04T10:03:45.455693+0000","last_clean":"2024-03-04T10:03:45.455693+0000","last_became_active":"2024-03-04T10:03:43.571770+0000","last_became_peered":"2024-03-04T10:03:43.571770+0000","last_unstale":"2024-03-04T10:03:45.455693+0000","last_undegraded":"2024-03-04T10:03:45.455693+0000","last_fullsized":"2024-03-04T10:03:45.455693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:19.867931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925655+0000","last_change":"2024-03-04T10:03:39.548855+0000","last_active":"2024-03-04T10:03:44.925655+0000","last_peered":"2024-03-04T10:03:44.925655+0000","last_clean":"2024-03-04T10:03:44.925655+0000","last_became_active":"2024-03-04T10:03:39.548568+0000","last_became_peered":"2024-03-04T10:03:39.548568+0000","last_unstale":"2024-03-04T10:03:44.925655+0000","last_undegraded":"2024-03-04T10:03:44.925655+0000","last_fullsized":"2024-03-04T10:03:44.925655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:19.378207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455946+0000","last_change":"2024-03-04T10:03:38.537010+0000","last_active":"2024-03-04T10:03:45.455946+0000","last_peered":"2024-03-04T10:03:45.455946+0000","last_clean":"2024-03-04T10:03:45.455946+0000","last_became_active":"2024-03-04T10:03:38.536784+0000","last_became_peered":"2024-03-04T10:03:38.536784+0000","last_unstale":"2024-03-04T10:03:45.455946+0000","last_undegraded":"2024-03-04T10:03:45.455946+0000","last_fullsized":"2024-03-04T10:03:45.455946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925807+0000","last_change":"2024-03-04T10:03:43.575162+0000","last_active":"2024-03-04T10:03:44.925807+0000","last_peered":"2024-03-04T10:03:44.925807+0000","last_clean":"2024-03-04T10:03:44.925807+0000","last_became_active":"2024-03-04T10:03:43.574407+0000","last_became_peered":"2024-03-04T10:03:43.574407+0000","last_unstale":"2024-03-04T10:03:44.925807+0000","last_undegraded":"2024-03-04T10:03:44.925807+0000","last_fullsized":"2024-03-04T10:03:44.925807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455373+0000","last_change":"2024-03-04T10:03:39.550513+0000","last_active":"2024-03-04T10:03:45.455373+0000","last_peered":"2024-03-04T10:03:45.455373+0000","last_clean":"2024-03-04T10:03:45.455373+0000","last_became_active":"2024-03-04T10:03:39.549951+0000","last_became_peered":"2024-03-04T10:03:39.549951+0000","last_unstale":"2024-03-04T10:03:45.455373+0000","last_undegraded":"2024-03-04T10:03:45.455373+0000","last_fullsized":"2024-03-04T10:03:45.455373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455931+0000","last_change":"2024-03-04T10:03:38.538520+0000","last_active":"2024-03-04T10:03:45.455931+0000","last_peered":"2024-03-04T10:03:45.455931+0000","last_clean":"2024-03-04T10:03:45.455931+0000","last_became_active":"2024-03-04T10:03:38.538251+0000","last_became_peered":"2024-03-04T10:03:38.538251+0000","last_unstale":"2024-03-04T10:03:45.455931+0000","last_undegraded":"2024-03-04T10:03:45.455931+0000","last_fullsized":"2024-03-04T10:03:45.455931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727165+0000","last_change":"2024-03-04T10:03:43.573784+0000","last_active":"2024-03-04T10:03:44.727165+0000","last_peered":"2024-03-04T10:03:44.727165+0000","last_clean":"2024-03-04T10:03:44.727165+0000","last_became_active":"2024-03-04T10:03:43.573040+0000","last_became_peered":"2024-03-04T10:03:43.573040+0000","last_unstale":"2024-03-04T10:03:44.727165+0000","last_undegraded":"2024-03-04T10:03:44.727165+0000","last_fullsized":"2024-03-04T10:03:44.727165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:31.435305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455930+0000","last_change":"2024-03-04T10:03:39.559601+0000","last_active":"2024-03-04T10:03:45.455930+0000","last_peered":"2024-03-04T10:03:45.455930+0000","last_clean":"2024-03-04T10:03:45.455930+0000","last_became_active":"2024-03-04T10:03:39.559321+0000","last_became_peered":"2024-03-04T10:03:39.559321+0000","last_unstale":"2024-03-04T10:03:45.455930+0000","last_undegraded":"2024-03-04T10:03:45.455930+0000","last_fullsized":"2024-03-04T10:03:45.455930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558393+0000","last_change":"2024-03-04T10:03:38.536855+0000","last_active":"2024-03-04T10:03:44.558393+0000","last_peered":"2024-03-04T10:03:44.558393+0000","last_clean":"2024-03-04T10:03:44.558393+0000","last_became_active":"2024-03-04T10:03:38.536551+0000","last_became_peered":"2024-03-04T10:03:38.536551+0000","last_unstale":"2024-03-04T10:03:44.558393+0000","last_undegraded":"2024-03-04T10:03:44.558393+0000","last_fullsized":"2024-03-04T10:03:44.558393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727003+0000","last_change":"2024-03-04T10:03:43.569782+0000","last_active":"2024-03-04T10:03:44.727003+0000","last_peered":"2024-03-04T10:03:44.727003+0000","last_clean":"2024-03-04T10:03:44.727003+0000","last_became_active":"2024-03-04T10:03:43.568928+0000","last_became_peered":"2024-03-04T10:03:43.568928+0000","last_unstale":"2024-03-04T10:03:44.727003+0000","last_undegraded":"2024-03-04T10:03:44.727003+0000","last_fullsized":"2024-03-04T10:03:44.727003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:33.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455065+0000","last_change":"2024-03-04T10:03:43.577832+0000","last_active":"2024-03-04T10:03:45.455065+0000","last_peered":"2024-03-04T10:03:45.455065+0000","last_clean":"2024-03-04T10:03:45.455065+0000","last_became_active":"2024-03-04T10:03:43.577460+0000","last_became_peered":"2024-03-04T10:03:43.577460+0000","last_unstale":"2024-03-04T10:03:45.455065+0000","last_undegraded":"2024-03-04T10:03:45.455065+0000","last_fullsized":"2024-03-04T10:03:45.455065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455950+0000","last_change":"2024-03-04T10:03:39.547155+0000","last_active":"2024-03-04T10:03:45.455950+0000","last_peered":"2024-03-04T10:03:45.455950+0000","last_clean":"2024-03-04T10:03:45.455950+0000","last_became_active":"2024-03-04T10:03:39.546882+0000","last_became_peered":"2024-03-04T10:03:39.546882+0000","last_unstale":"2024-03-04T10:03:45.455950+0000","last_undegraded":"2024-03-04T10:03:45.455950+0000","last_fullsized":"2024-03-04T10:03:45.455950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455427+0000","last_change":"2024-03-04T10:03:38.533042+0000","last_active":"2024-03-04T10:03:45.455427+0000","last_peered":"2024-03-04T10:03:45.455427+0000","last_clean":"2024-03-04T10:03:45.455427+0000","last_became_active":"2024-03-04T10:03:38.532609+0000","last_became_peered":"2024-03-04T10:03:38.532609+0000","last_unstale":"2024-03-04T10:03:45.455427+0000","last_undegraded":"2024-03-04T10:03:45.455427+0000","last_fullsized":"2024-03-04T10:03:45.455427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558268+0000","last_change":"2024-03-04T10:03:43.577800+0000","last_active":"2024-03-04T10:03:44.558268+0000","last_peered":"2024-03-04T10:03:44.558268+0000","last_clean":"2024-03-04T10:03:44.558268+0000","last_became_active":"2024-03-04T10:03:43.577379+0000","last_became_peered":"2024-03-04T10:03:43.577379+0000","last_unstale":"2024-03-04T10:03:44.558268+0000","last_undegraded":"2024-03-04T10:03:44.558268+0000","last_fullsized":"2024-03-04T10:03:44.558268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726957+0000","last_change":"2024-03-04T10:03:39.555669+0000","last_active":"2024-03-04T10:03:44.726957+0000","last_peered":"2024-03-04T10:03:44.726957+0000","last_clean":"2024-03-04T10:03:44.726957+0000","last_became_active":"2024-03-04T10:03:39.555415+0000","last_became_peered":"2024-03-04T10:03:39.555415+0000","last_unstale":"2024-03-04T10:03:44.726957+0000","last_undegraded":"2024-03-04T10:03:44.726957+0000","last_fullsized":"2024-03-04T10:03:44.726957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727092+0000","last_change":"2024-03-04T10:03:38.537206+0000","last_active":"2024-03-04T10:03:44.727092+0000","last_peered":"2024-03-04T10:03:44.727092+0000","last_clean":"2024-03-04T10:03:44.727092+0000","last_became_active":"2024-03-04T10:03:38.536938+0000","last_became_peered":"2024-03-04T10:03:38.536938+0000","last_unstale":"2024-03-04T10:03:44.727092+0000","last_undegraded":"2024-03-04T10:03:44.727092+0000","last_fullsized":"2024-03-04T10:03:44.727092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558028+0000","last_change":"2024-03-04T10:03:38.528419+0000","last_active":"2024-03-04T10:03:44.558028+0000","last_peered":"2024-03-04T10:03:44.558028+0000","last_clean":"2024-03-04T10:03:44.558028+0000","last_became_active":"2024-03-04T10:03:38.528143+0000","last_became_peered":"2024-03-04T10:03:38.528143+0000","last_unstale":"2024-03-04T10:03:44.558028+0000","last_undegraded":"2024-03-04T10:03:44.558028+0000","last_fullsized":"2024-03-04T10:03:44.558028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455342+0000","last_change":"2024-03-04T10:03:39.537531+0000","last_active":"2024-03-04T10:03:45.455342+0000","last_peered":"2024-03-04T10:03:45.455342+0000","last_clean":"2024-03-04T10:03:45.455342+0000","last_became_active":"2024-03-04T10:03:39.537282+0000","last_became_peered":"2024-03-04T10:03:39.537282+0000","last_unstale":"2024-03-04T10:03:45.455342+0000","last_undegraded":"2024-03-04T10:03:45.455342+0000","last_fullsized":"2024-03-04T10:03:45.455342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556191+0000","last_change":"2024-03-04T10:03:43.576190+0000","last_active":"2024-03-04T10:03:44.556191+0000","last_peered":"2024-03-04T10:03:44.556191+0000","last_clean":"2024-03-04T10:03:44.556191+0000","last_became_active":"2024-03-04T10:03:43.575758+0000","last_became_peered":"2024-03-04T10:03:43.575758+0000","last_unstale":"2024-03-04T10:03:44.556191+0000","last_undegraded":"2024-03-04T10:03:44.556191+0000","last_fullsized":"2024-03-04T10:03:44.556191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:27:39.111875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727248+0000","last_change":"2024-03-04T10:03:38.538563+0000","last_active":"2024-03-04T10:03:44.727248+0000","last_peered":"2024-03-04T10:03:44.727248+0000","last_clean":"2024-03-04T10:03:44.727248+0000","last_became_active":"2024-03-04T10:03:38.538284+0000","last_became_peered":"2024-03-04T10:03:38.538284+0000","last_unstale":"2024-03-04T10:03:44.727248+0000","last_undegraded":"2024-03-04T10:03:44.727248+0000","last_fullsized":"2024-03-04T10:03:44.727248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925234+0000","last_change":"2024-03-04T10:03:39.543278+0000","last_active":"2024-03-04T10:03:44.925234+0000","last_peered":"2024-03-04T10:03:44.925234+0000","last_clean":"2024-03-04T10:03:44.925234+0000","last_became_active":"2024-03-04T10:03:39.542993+0000","last_became_peered":"2024-03-04T10:03:39.542993+0000","last_unstale":"2024-03-04T10:03:44.925234+0000","last_undegraded":"2024-03-04T10:03:44.925234+0000","last_fullsized":"2024-03-04T10:03:44.925234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:44.723658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727569+0000","last_change":"2024-03-04T10:03:43.569930+0000","last_active":"2024-03-04T10:03:44.727569+0000","last_peered":"2024-03-04T10:03:44.727569+0000","last_clean":"2024-03-04T10:03:44.727569+0000","last_became_active":"2024-03-04T10:03:43.569524+0000","last_became_peered":"2024-03-04T10:03:43.569524+0000","last_unstale":"2024-03-04T10:03:44.727569+0000","last_undegraded":"2024-03-04T10:03:44.727569+0000","last_fullsized":"2024-03-04T10:03:44.727569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:28.002170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556756+0000","last_change":"2024-03-04T10:03:39.547564+0000","last_active":"2024-03-04T10:03:44.556756+0000","last_peered":"2024-03-04T10:03:44.556756+0000","last_clean":"2024-03-04T10:03:44.556756+0000","last_became_active":"2024-03-04T10:03:39.547245+0000","last_became_peered":"2024-03-04T10:03:39.547245+0000","last_unstale":"2024-03-04T10:03:44.556756+0000","last_undegraded":"2024-03-04T10:03:44.556756+0000","last_fullsized":"2024-03-04T10:03:44.556756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:15:14.213025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555940+0000","last_change":"2024-03-04T10:03:39.549137+0000","last_active":"2024-03-04T10:03:44.555940+0000","last_peered":"2024-03-04T10:03:44.555940+0000","last_clean":"2024-03-04T10:03:44.555940+0000","last_became_active":"2024-03-04T10:03:39.548867+0000","last_became_peered":"2024-03-04T10:03:39.548867+0000","last_unstale":"2024-03-04T10:03:44.555940+0000","last_undegraded":"2024-03-04T10:03:44.555940+0000","last_fullsized":"2024-03-04T10:03:44.555940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455603+0000","last_change":"2024-03-04T10:03:39.555564+0000","last_active":"2024-03-04T10:03:45.455603+0000","last_peered":"2024-03-04T10:03:45.455603+0000","last_clean":"2024-03-04T10:03:45.455603+0000","last_became_active":"2024-03-04T10:03:39.555310+0000","last_became_peered":"2024-03-04T10:03:39.555310+0000","last_unstale":"2024-03-04T10:03:45.455603+0000","last_undegraded":"2024-03-04T10:03:45.455603+0000","last_fullsized":"2024-03-04T10:03:45.455603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455247+0000","last_change":"2024-03-04T10:03:39.550377+0000","last_active":"2024-03-04T10:03:45.455247+0000","last_peered":"2024-03-04T10:03:45.455247+0000","last_clean":"2024-03-04T10:03:45.455247+0000","last_became_active":"2024-03-04T10:03:39.550028+0000","last_became_peered":"2024-03-04T10:03:39.550028+0000","last_unstale":"2024-03-04T10:03:45.455247+0000","last_undegraded":"2024-03-04T10:03:45.455247+0000","last_fullsized":"2024-03-04T10:03:45.455247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727357+0000","last_change":"2024-03-04T10:03:39.553769+0000","last_active":"2024-03-04T10:03:44.727357+0000","last_peered":"2024-03-04T10:03:44.727357+0000","last_clean":"2024-03-04T10:03:44.727357+0000","last_became_active":"2024-03-04T10:03:39.553517+0000","last_became_peered":"2024-03-04T10:03:39.553517+0000","last_unstale":"2024-03-04T10:03:44.727357+0000","last_undegraded":"2024-03-04T10:03:44.727357+0000","last_fullsized":"2024-03-04T10:03:44.727357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455401+0000","last_change":"2024-03-04T10:03:39.551104+0000","last_active":"2024-03-04T10:03:45.455401+0000","last_peered":"2024-03-04T10:03:45.455401+0000","last_clean":"2024-03-04T10:03:45.455401+0000","last_became_active":"2024-03-04T10:03:39.550823+0000","last_became_peered":"2024-03-04T10:03:39.550823+0000","last_unstale":"2024-03-04T10:03:45.455401+0000","last_undegraded":"2024-03-04T10:03:45.455401+0000","last_fullsized":"2024-03-04T10:03:45.455401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558684+0000","last_change":"2024-03-04T10:03:39.563672+0000","last_active":"2024-03-04T10:03:44.558684+0000","last_peered":"2024-03-04T10:03:44.558684+0000","last_clean":"2024-03-04T10:03:44.558684+0000","last_became_active":"2024-03-04T10:03:39.563394+0000","last_became_peered":"2024-03-04T10:03:39.563394+0000","last_unstale":"2024-03-04T10:03:44.558684+0000","last_undegraded":"2024-03-04T10:03:44.558684+0000","last_fullsized":"2024-03-04T10:03:44.558684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727442+0000","last_change":"2024-03-04T10:03:39.549746+0000","last_active":"2024-03-04T10:03:44.727442+0000","last_peered":"2024-03-04T10:03:44.727442+0000","last_clean":"2024-03-04T10:03:44.727442+0000","last_became_active":"2024-03-04T10:03:39.549349+0000","last_became_peered":"2024-03-04T10:03:39.549349+0000","last_unstale":"2024-03-04T10:03:44.727442+0000","last_undegraded":"2024-03-04T10:03:44.727442+0000","last_fullsized":"2024-03-04T10:03:44.727442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455405+0000","last_change":"2024-03-04T10:03:39.556217+0000","last_active":"2024-03-04T10:03:45.455405+0000","last_peered":"2024-03-04T10:03:45.455405+0000","last_clean":"2024-03-04T10:03:45.455405+0000","last_became_active":"2024-03-04T10:03:39.555896+0000","last_became_peered":"2024-03-04T10:03:39.555896+0000","last_unstale":"2024-03-04T10:03:45.455405+0000","last_undegraded":"2024-03-04T10:03:45.455405+0000","last_fullsized":"2024-03-04T10:03:45.455405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726990+0000","last_change":"2024-03-04T10:03:39.546403+0000","last_active":"2024-03-04T10:03:44.726990+0000","last_peered":"2024-03-04T10:03:44.726990+0000","last_clean":"2024-03-04T10:03:44.726990+0000","last_became_active":"2024-03-04T10:03:39.546136+0000","last_became_peered":"2024-03-04T10:03:39.546136+0000","last_unstale":"2024-03-04T10:03:44.726990+0000","last_undegraded":"2024-03-04T10:03:44.726990+0000","last_fullsized":"2024-03-04T10:03:44.726990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455552+0000","last_change":"2024-03-04T10:03:39.553616+0000","last_active":"2024-03-04T10:03:45.455552+0000","last_peered":"2024-03-04T10:03:45.455552+0000","last_clean":"2024-03-04T10:03:45.455552+0000","last_became_active":"2024-03-04T10:03:39.553356+0000","last_became_peered":"2024-03-04T10:03:39.553356+0000","last_unstale":"2024-03-04T10:03:45.455552+0000","last_undegraded":"2024-03-04T10:03:45.455552+0000","last_fullsized":"2024-03-04T10:03:45.455552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455120+0000","last_change":"2024-03-04T10:03:39.539406+0000","last_active":"2024-03-04T10:03:45.455120+0000","last_peered":"2024-03-04T10:03:45.455120+0000","last_clean":"2024-03-04T10:03:45.455120+0000","last_became_active":"2024-03-04T10:03:39.539139+0000","last_became_peered":"2024-03-04T10:03:39.539139+0000","last_unstale":"2024-03-04T10:03:45.455120+0000","last_undegraded":"2024-03-04T10:03:45.455120+0000","last_fullsized":"2024-03-04T10:03:45.455120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558323+0000","last_change":"2024-03-04T10:03:39.554026+0000","last_active":"2024-03-04T10:03:44.558323+0000","last_peered":"2024-03-04T10:03:44.558323+0000","last_clean":"2024-03-04T10:03:44.558323+0000","last_became_active":"2024-03-04T10:03:39.553711+0000","last_became_peered":"2024-03-04T10:03:39.553711+0000","last_unstale":"2024-03-04T10:03:44.558323+0000","last_undegraded":"2024-03-04T10:03:44.558323+0000","last_fullsized":"2024-03-04T10:03:44.558323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558201+0000","last_change":"2024-03-04T10:03:39.545468+0000","last_active":"2024-03-04T10:03:44.558201+0000","last_peered":"2024-03-04T10:03:44.558201+0000","last_clean":"2024-03-04T10:03:44.558201+0000","last_became_active":"2024-03-04T10:03:39.545202+0000","last_became_peered":"2024-03-04T10:03:39.545202+0000","last_unstale":"2024-03-04T10:03:44.558201+0000","last_undegraded":"2024-03-04T10:03:44.558201+0000","last_fullsized":"2024-03-04T10:03:44.558201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:42.641276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556175+0000","last_change":"2024-03-04T10:03:39.549637+0000","last_active":"2024-03-04T10:03:44.556175+0000","last_peered":"2024-03-04T10:03:44.556175+0000","last_clean":"2024-03-04T10:03:44.556175+0000","last_became_active":"2024-03-04T10:03:39.549384+0000","last_became_peered":"2024-03-04T10:03:39.549384+0000","last_unstale":"2024-03-04T10:03:44.556175+0000","last_undegraded":"2024-03-04T10:03:44.556175+0000","last_fullsized":"2024-03-04T10:03:44.556175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456190+0000","last_change":"2024-03-04T10:03:39.559786+0000","last_active":"2024-03-04T10:03:45.456190+0000","last_peered":"2024-03-04T10:03:45.456190+0000","last_clean":"2024-03-04T10:03:45.456190+0000","last_became_active":"2024-03-04T10:03:39.559507+0000","last_became_peered":"2024-03-04T10:03:39.559507+0000","last_unstale":"2024-03-04T10:03:45.456190+0000","last_undegraded":"2024-03-04T10:03:45.456190+0000","last_fullsized":"2024-03-04T10:03:45.456190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455259+0000","last_change":"2024-03-04T10:03:39.547601+0000","last_active":"2024-03-04T10:03:45.455259+0000","last_peered":"2024-03-04T10:03:45.455259+0000","last_clean":"2024-03-04T10:03:45.455259+0000","last_became_active":"2024-03-04T10:03:39.547309+0000","last_became_peered":"2024-03-04T10:03:39.547309+0000","last_unstale":"2024-03-04T10:03:45.455259+0000","last_undegraded":"2024-03-04T10:03:45.455259+0000","last_fullsized":"2024-03-04T10:03:45.455259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456059+0000","last_change":"2024-03-04T10:03:39.543308+0000","last_active":"2024-03-04T10:03:45.456059+0000","last_peered":"2024-03-04T10:03:45.456059+0000","last_clean":"2024-03-04T10:03:45.456059+0000","last_became_active":"2024-03-04T10:03:39.543064+0000","last_became_peered":"2024-03-04T10:03:39.543064+0000","last_unstale":"2024-03-04T10:03:45.456059+0000","last_undegraded":"2024-03-04T10:03:45.456059+0000","last_fullsized":"2024-03-04T10:03:45.456059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455862+0000","last_change":"2024-03-04T10:03:39.540533+0000","last_active":"2024-03-04T10:03:45.455862+0000","last_peered":"2024-03-04T10:03:45.455862+0000","last_clean":"2024-03-04T10:03:45.455862+0000","last_became_active":"2024-03-04T10:03:39.540274+0000","last_became_peered":"2024-03-04T10:03:39.540274+0000","last_unstale":"2024-03-04T10:03:45.455862+0000","last_undegraded":"2024-03-04T10:03:45.455862+0000","last_fullsized":"2024-03-04T10:03:45.455862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727155+0000","last_change":"2024-03-04T10:03:39.556252+0000","last_active":"2024-03-04T10:03:44.727155+0000","last_peered":"2024-03-04T10:03:44.727155+0000","last_clean":"2024-03-04T10:03:44.727155+0000","last_became_active":"2024-03-04T10:03:39.556009+0000","last_became_peered":"2024-03-04T10:03:39.556009+0000","last_unstale":"2024-03-04T10:03:44.727155+0000","last_undegraded":"2024-03-04T10:03:44.727155+0000","last_fullsized":"2024-03-04T10:03:44.727155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925508+0000","last_change":"2024-03-04T10:03:39.552666+0000","last_active":"2024-03-04T10:03:44.925508+0000","last_peered":"2024-03-04T10:03:44.925508+0000","last_clean":"2024-03-04T10:03:44.925508+0000","last_became_active":"2024-03-04T10:03:39.552178+0000","last_became_peered":"2024-03-04T10:03:39.552178+0000","last_unstale":"2024-03-04T10:03:44.925508+0000","last_undegraded":"2024-03-04T10:03:44.925508+0000","last_fullsized":"2024-03-04T10:03:44.925508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558388+0000","last_change":"2024-03-04T10:03:39.557130+0000","last_active":"2024-03-04T10:03:44.558388+0000","last_peered":"2024-03-04T10:03:44.558388+0000","last_clean":"2024-03-04T10:03:44.558388+0000","last_became_active":"2024-03-04T10:03:39.556736+0000","last_became_peered":"2024-03-04T10:03:39.556736+0000","last_unstale":"2024-03-04T10:03:44.558388+0000","last_undegraded":"2024-03-04T10:03:44.558388+0000","last_fullsized":"2024-03-04T10:03:44.558388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558631+0000","last_change":"2024-03-04T10:03:39.556929+0000","last_active":"2024-03-04T10:03:44.558631+0000","last_peered":"2024-03-04T10:03:44.558631+0000","last_clean":"2024-03-04T10:03:44.558631+0000","last_became_active":"2024-03-04T10:03:39.556672+0000","last_became_peered":"2024-03-04T10:03:39.556672+0000","last_unstale":"2024-03-04T10:03:44.558631+0000","last_undegraded":"2024-03-04T10:03:44.558631+0000","last_fullsized":"2024-03-04T10:03:44.558631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558062+0000","last_change":"2024-03-04T10:03:39.553116+0000","last_active":"2024-03-04T10:03:44.558062+0000","last_peered":"2024-03-04T10:03:44.558062+0000","last_clean":"2024-03-04T10:03:44.558062+0000","last_became_active":"2024-03-04T10:03:39.551779+0000","last_became_peered":"2024-03-04T10:03:39.551779+0000","last_unstale":"2024-03-04T10:03:44.558062+0000","last_undegraded":"2024-03-04T10:03:44.558062+0000","last_fullsized":"2024-03-04T10:03:44.558062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:34.980080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726862+0000","last_change":"2024-03-04T10:03:39.557751+0000","last_active":"2024-03-04T10:03:44.726862+0000","last_peered":"2024-03-04T10:03:44.726862+0000","last_clean":"2024-03-04T10:03:44.726862+0000","last_became_active":"2024-03-04T10:03:39.557443+0000","last_became_peered":"2024-03-04T10:03:39.557443+0000","last_unstale":"2024-03-04T10:03:44.726862+0000","last_undegraded":"2024-03-04T10:03:44.726862+0000","last_fullsized":"2024-03-04T10:03:44.726862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557825+0000","last_change":"2024-03-04T10:03:39.547100+0000","last_active":"2024-03-04T10:03:44.557825+0000","last_peered":"2024-03-04T10:03:44.557825+0000","last_clean":"2024-03-04T10:03:44.557825+0000","last_became_active":"2024-03-04T10:03:39.546794+0000","last_became_peered":"2024-03-04T10:03:39.546794+0000","last_unstale":"2024-03-04T10:03:44.557825+0000","last_undegraded":"2024-03-04T10:03:44.557825+0000","last_fullsized":"2024-03-04T10:03:44.557825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455496+0000","last_change":"2024-03-04T10:03:39.548756+0000","last_active":"2024-03-04T10:03:45.455496+0000","last_peered":"2024-03-04T10:03:45.455496+0000","last_clean":"2024-03-04T10:03:45.455496+0000","last_became_active":"2024-03-04T10:03:39.548474+0000","last_became_peered":"2024-03-04T10:03:39.548474+0000","last_unstale":"2024-03-04T10:03:45.455496+0000","last_undegraded":"2024-03-04T10:03:45.455496+0000","last_fullsized":"2024-03-04T10:03:45.455496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557997+0000","last_change":"2024-03-04T10:03:39.544100+0000","last_active":"2024-03-04T10:03:44.557997+0000","last_peered":"2024-03-04T10:03:44.557997+0000","last_clean":"2024-03-04T10:03:44.557997+0000","last_became_active":"2024-03-04T10:03:39.543581+0000","last_became_peered":"2024-03-04T10:03:39.543581+0000","last_unstale":"2024-03-04T10:03:44.557997+0000","last_undegraded":"2024-03-04T10:03:44.557997+0000","last_fullsized":"2024-03-04T10:03:44.557997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455591+0000","last_change":"2024-03-04T10:03:39.556280+0000","last_active":"2024-03-04T10:03:45.455591+0000","last_peered":"2024-03-04T10:03:45.455591+0000","last_clean":"2024-03-04T10:03:45.455591+0000","last_became_active":"2024-03-04T10:03:39.555867+0000","last_became_peered":"2024-03-04T10:03:39.555867+0000","last_unstale":"2024-03-04T10:03:45.455591+0000","last_undegraded":"2024-03-04T10:03:45.455591+0000","last_fullsized":"2024-03-04T10:03:45.455591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455074+0000","last_change":"2024-03-04T10:03:39.538030+0000","last_active":"2024-03-04T10:03:45.455074+0000","last_peered":"2024-03-04T10:03:45.455074+0000","last_clean":"2024-03-04T10:03:45.455074+0000","last_became_active":"2024-03-04T10:03:39.537783+0000","last_became_peered":"2024-03-04T10:03:39.537783+0000","last_unstale":"2024-03-04T10:03:45.455074+0000","last_undegraded":"2024-03-04T10:03:45.455074+0000","last_fullsized":"2024-03-04T10:03:45.455074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455845+0000","last_change":"2024-03-04T10:03:39.545505+0000","last_active":"2024-03-04T10:03:45.455845+0000","last_peered":"2024-03-04T10:03:45.455845+0000","last_clean":"2024-03-04T10:03:45.455845+0000","last_became_active":"2024-03-04T10:03:39.545176+0000","last_became_peered":"2024-03-04T10:03:39.545176+0000","last_unstale":"2024-03-04T10:03:45.455845+0000","last_undegraded":"2024-03-04T10:03:45.455845+0000","last_fullsized":"2024-03-04T10:03:45.455845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557636+0000","last_change":"2024-03-04T10:03:39.551482+0000","last_active":"2024-03-04T10:03:44.557636+0000","last_peered":"2024-03-04T10:03:44.557636+0000","last_clean":"2024-03-04T10:03:44.557636+0000","last_became_active":"2024-03-04T10:03:39.551234+0000","last_became_peered":"2024-03-04T10:03:39.551234+0000","last_unstale":"2024-03-04T10:03:44.557636+0000","last_undegraded":"2024-03-04T10:03:44.557636+0000","last_fullsized":"2024-03-04T10:03:44.557636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:22.803941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:04:02.575 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:04:02.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.707+0000 7f67f6f7e640 1 Processor -- start 2024-03-04T10:04:02.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.707+0000 7f67f6f7e640 1 -- start start 2024-03-04T10:04:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67f6f7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f67f0154a80 0x7f67f015e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67f6f7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f67f00cf650 0x7f67f00cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67f6f7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f0161b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67f6f7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00cdaa0 con 0x7f67f00cf650 2024-03-04T10:04:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67f6f7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00cdd80 con 0x7f67f0154a80 2024-03-04T10:04:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67f6f7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00ce060 con 0x7f67f01616c0 2024-03-04T10:04:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f67f00cf650 0x7f67f00cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f67f00cf650 0x7f67f00cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45924/0 (socket says 172.21.15.99:45924) 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 -- 172.21.15.99:0/2833798549 learned_addr learned my addr 172.21.15.99:0/2833798549 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 -- 172.21.15.99:0/2833798549 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 msgr2=0x7f67f0161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 --2- 172.21.15.99:0/2833798549 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f0161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 -- 172.21.15.99:0/2833798549 >> v2:172.21.15.162:3300/0 conn(0x7f67f0154a80 msgr2=0x7f67f015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 --2- 172.21.15.99:0/2833798549 >> v2:172.21.15.162:3300/0 conn(0x7f67f0154a80 0x7f67f015e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.708+0000 7f67effff640 1 -- 172.21.15.99:0/2833798549 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f00ce340 con 0x7f67f00cf650 2024-03-04T10:04:02.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.709+0000 7f67effff640 1 --2- 172.21.15.99:0/2833798549 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f00cf650 0x7f67f00cfae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67f006e970 tx=0x7f67e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d82da26bd75c6ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.709+0000 7f67ef7fe640 1 -- 172.21.15.99:0/2833798549 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4035020 con 0x7f67f00cf650 2024-03-04T10:04:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.709+0000 7f67ef7fe640 1 -- 172.21.15.99:0/2833798549 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e4031870 con 0x7f67f00cf650 2024-03-04T10:04:02.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.709+0000 7f67ef7fe640 1 -- 172.21.15.99:0/2833798549 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4031c40 con 0x7f67f00cf650 2024-03-04T10:04:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2833798549 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f00cf650 msgr2=0x7f67f00cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2833798549 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f00cf650 0x7f67f00cfae0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67f006e970 tx=0x7f67e402b910 comp rx=0 tx=0).stop 2024-03-04T10:04:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2833798549 shutdown_connections 2024-03-04T10:04:02.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2833798549 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f0161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2833798549 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f00cf650 0x7f67f00cfae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2833798549 >> v2:172.21.15.162:3300/0 conn(0x7f67f0154a80 0x7f67f015e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2833798549 >> 172.21.15.99:0/2833798549 conn(0x7f67f006f5f0 msgr2=0x7f67f006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2833798549 shutdown_connections 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.710+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2833798549 wait complete. 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.711+0000 7f67f6f7e640 1 Processor -- start 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.711+0000 7f67f6f7e640 1 -- start start 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f6f7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67f00cf650 0x7f67f01f7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f6f7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 0x7f67f00c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f6f7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f00b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f6f7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0154fe0 con 0x7f67f0154a80 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f6f7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0154e60 con 0x7f67f00cf650 2024-03-04T10:04:02.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f6f7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00ceb40 con 0x7f67f01616c0 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 0x7f67f00c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 0x7f67f00c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45930/0 (socket says 172.21.15.99:45930) 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 -- 172.21.15.99:0/2307445719 learned_addr learned my addr 172.21.15.99:0/2307445719 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f551b640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f00b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f4d1a640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67f00cf650 0x7f67f01f7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 -- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 msgr2=0x7f67f00b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f00b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 -- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67f00cf650 msgr2=0x7f67f01f7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67f00cf650 0x7f67f01f7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 -- 172.21.15.99:0/2307445719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f00c68d0 con 0x7f67f0154a80 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f551b640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f00b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67f4d1a640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67f00cf650 0x7f67f01f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.712+0000 7f67effff640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 0x7f67f00c5e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67e4000c00 tx=0x7f67e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.713+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e403ad60 con 0x7f67f0154a80 2024-03-04T10:04:02.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.713+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e40443f0 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.713+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e40447e0 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.713+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f01f3c00 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.713+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f01f41c0 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.714+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f0074f60 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.714+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f67e404b060 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.714+0000 7f67ed7fa640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f67dc06b4d0 0x7f67dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.715+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f67e40ac740 con 0x7f67f0154a80 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.715+0000 7f67f4d1a640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f67dc06b4d0 0x7f67dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:02.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.715+0000 7f67f4d1a640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f67dc06b4d0 0x7f67dc06d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f67e0006fd0 tx=0x7f67e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:02.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.719+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f67e403f070 con 0x7f67f0154a80 2024-03-04T10:04:02.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.884+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67f00ccb70 con 0x7f67dc06b4d0 2024-03-04T10:04:02.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.893+0000 7f67ed7fa640 1 -- 172.21.15.99:0/2307445719 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332534 (secure 0 0 0) 0x7f67f00ccb70 con 0x7f67dc06b4d0 2024-03-04T10:04:02.899 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:02.901 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f67dc06b4d0 msgr2=0x7f67dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f67dc06b4d0 0x7f67dc06d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f67e0006fd0 tx=0x7f67e0008040 comp rx=0 tx=0).stop 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 msgr2=0x7f67f00c5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 0x7f67f00c5e10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67e4000c00 tx=0x7f67e40046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 shutdown_connections 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f67dc06b4d0 0x7f67dc06d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67f01616c0 0x7f67f00b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67f0154a80 0x7f67f00c5e10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 --2- 172.21.15.99:0/2307445719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67f00cf650 0x7f67f01f7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:02.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 >> 172.21.15.99:0/2307445719 conn(0x7f67f006f5f0 msgr2=0x7f67f00cbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:02.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 shutdown_connections 2024-03-04T10:04:02.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:02.896+0000 7f67f6f7e640 1 -- 172.21.15.99:0/2307445719 wait complete. 2024-03-04T10:04:02.908 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-04T10:04:01.441574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219388,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755332,"statfs":{"total":773094113280,"available":772869459968,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455527+0000","last_change":"2024-03-04T10:03:38.529898+0000","last_active":"2024-03-04T10:03:45.455527+0000","last_peered":"2024-03-04T10:03:45.455527+0000","last_clean":"2024-03-04T10:03:45.455527+0000","last_became_active":"2024-03-04T10:03:38.529499+0000","last_became_peered":"2024-03-04T10:03:38.529499+0000","last_unstale":"2024-03-04T10:03:45.455527+0000","last_undegraded":"2024-03-04T10:03:45.455527+0000","last_fullsized":"2024-03-04T10:03:45.455527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727461+0000","last_change":"2024-03-04T10:03:39.556883+0000","last_active":"2024-03-04T10:03:44.727461+0000","last_peered":"2024-03-04T10:03:44.727461+0000","last_clean":"2024-03-04T10:03:44.727461+0000","last_became_active":"2024-03-04T10:03:39.556616+0000","last_became_peered":"2024-03-04T10:03:39.556616+0000","last_unstale":"2024-03-04T10:03:44.727461+0000","last_undegraded":"2024-03-04T10:03:44.727461+0000","last_fullsized":"2024-03-04T10:03:44.727461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726874+0000","last_change":"2024-03-04T10:03:43.569338+0000","last_active":"2024-03-04T10:03:44.726874+0000","last_peered":"2024-03-04T10:03:44.726874+0000","last_clean":"2024-03-04T10:03:44.726874+0000","last_became_active":"2024-03-04T10:03:43.568332+0000","last_became_peered":"2024-03-04T10:03:43.568332+0000","last_unstale":"2024-03-04T10:03:44.726874+0000","last_undegraded":"2024-03-04T10:03:44.726874+0000","last_fullsized":"2024-03-04T10:03:44.726874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727559+0000","last_change":"2024-03-04T10:03:38.533956+0000","last_active":"2024-03-04T10:03:44.727559+0000","last_peered":"2024-03-04T10:03:44.727559+0000","last_clean":"2024-03-04T10:03:44.727559+0000","last_became_active":"2024-03-04T10:03:38.533665+0000","last_became_peered":"2024-03-04T10:03:38.533665+0000","last_unstale":"2024-03-04T10:03:44.727559+0000","last_undegraded":"2024-03-04T10:03:44.727559+0000","last_fullsized":"2024-03-04T10:03:44.727559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455218+0000","last_change":"2024-03-04T10:03:39.550341+0000","last_active":"2024-03-04T10:03:45.455218+0000","last_peered":"2024-03-04T10:03:45.455218+0000","last_clean":"2024-03-04T10:03:45.455218+0000","last_became_active":"2024-03-04T10:03:39.549960+0000","last_became_peered":"2024-03-04T10:03:39.549960+0000","last_unstale":"2024-03-04T10:03:45.455218+0000","last_undegraded":"2024-03-04T10:03:45.455218+0000","last_fullsized":"2024-03-04T10:03:45.455218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925071+0000","last_change":"2024-03-04T10:03:43.578315+0000","last_active":"2024-03-04T10:03:44.925071+0000","last_peered":"2024-03-04T10:03:44.925071+0000","last_clean":"2024-03-04T10:03:44.925071+0000","last_became_active":"2024-03-04T10:03:43.577864+0000","last_became_peered":"2024-03-04T10:03:43.577864+0000","last_unstale":"2024-03-04T10:03:44.925071+0000","last_undegraded":"2024-03-04T10:03:44.925071+0000","last_fullsized":"2024-03-04T10:03:44.925071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:27.973529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558121+0000","last_change":"2024-03-04T10:03:38.536815+0000","last_active":"2024-03-04T10:03:44.558121+0000","last_peered":"2024-03-04T10:03:44.558121+0000","last_clean":"2024-03-04T10:03:44.558121+0000","last_became_active":"2024-03-04T10:03:38.536526+0000","last_became_peered":"2024-03-04T10:03:38.536526+0000","last_unstale":"2024-03-04T10:03:44.558121+0000","last_undegraded":"2024-03-04T10:03:44.558121+0000","last_fullsized":"2024-03-04T10:03:44.558121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455699+0000","last_change":"2024-03-04T10:03:39.551796+0000","last_active":"2024-03-04T10:03:45.455699+0000","last_peered":"2024-03-04T10:03:45.455699+0000","last_clean":"2024-03-04T10:03:45.455699+0000","last_became_active":"2024-03-04T10:03:39.551527+0000","last_became_peered":"2024-03-04T10:03:39.551527+0000","last_unstale":"2024-03-04T10:03:45.455699+0000","last_undegraded":"2024-03-04T10:03:45.455699+0000","last_fullsized":"2024-03-04T10:03:45.455699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925354+0000","last_change":"2024-03-04T10:03:43.573372+0000","last_active":"2024-03-04T10:03:44.925354+0000","last_peered":"2024-03-04T10:03:44.925354+0000","last_clean":"2024-03-04T10:03:44.925354+0000","last_became_active":"2024-03-04T10:03:43.572926+0000","last_became_peered":"2024-03-04T10:03:43.572926+0000","last_unstale":"2024-03-04T10:03:44.925354+0000","last_undegraded":"2024-03-04T10:03:44.925354+0000","last_fullsized":"2024-03-04T10:03:44.925354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:37.210258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557408+0000","last_change":"2024-03-04T10:03:38.535163+0000","last_active":"2024-03-04T10:03:44.557408+0000","last_peered":"2024-03-04T10:03:44.557408+0000","last_clean":"2024-03-04T10:03:44.557408+0000","last_became_active":"2024-03-04T10:03:38.534751+0000","last_became_peered":"2024-03-04T10:03:38.534751+0000","last_unstale":"2024-03-04T10:03:44.557408+0000","last_undegraded":"2024-03-04T10:03:44.557408+0000","last_fullsized":"2024-03-04T10:03:44.557408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:40.978639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455945+0000","last_change":"2024-03-04T10:03:39.548628+0000","last_active":"2024-03-04T10:03:45.455945+0000","last_peered":"2024-03-04T10:03:45.455945+0000","last_clean":"2024-03-04T10:03:45.455945+0000","last_became_active":"2024-03-04T10:03:39.548297+0000","last_became_peered":"2024-03-04T10:03:39.548297+0000","last_unstale":"2024-03-04T10:03:45.455945+0000","last_undegraded":"2024-03-04T10:03:45.455945+0000","last_fullsized":"2024-03-04T10:03:45.455945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925131+0000","last_change":"2024-03-04T10:03:43.576063+0000","last_active":"2024-03-04T10:03:44.925131+0000","last_peered":"2024-03-04T10:03:44.925131+0000","last_clean":"2024-03-04T10:03:44.925131+0000","last_became_active":"2024-03-04T10:03:43.575658+0000","last_became_peered":"2024-03-04T10:03:43.575658+0000","last_unstale":"2024-03-04T10:03:44.925131+0000","last_undegraded":"2024-03-04T10:03:44.925131+0000","last_fullsized":"2024-03-04T10:03:44.925131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:47.766859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557898+0000","last_change":"2024-03-04T10:03:39.562772+0000","last_active":"2024-03-04T10:03:44.557898+0000","last_peered":"2024-03-04T10:03:44.557898+0000","last_clean":"2024-03-04T10:03:44.557898+0000","last_became_active":"2024-03-04T10:03:39.562479+0000","last_became_peered":"2024-03-04T10:03:39.562479+0000","last_unstale":"2024-03-04T10:03:44.557898+0000","last_undegraded":"2024-03-04T10:03:44.557898+0000","last_fullsized":"2024-03-04T10:03:44.557898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:48.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558549+0000","last_change":"2024-03-04T10:03:38.524487+0000","last_active":"2024-03-04T10:03:44.558549+0000","last_peered":"2024-03-04T10:03:44.558549+0000","last_clean":"2024-03-04T10:03:44.558549+0000","last_became_active":"2024-03-04T10:03:38.524186+0000","last_became_peered":"2024-03-04T10:03:38.524186+0000","last_unstale":"2024-03-04T10:03:44.558549+0000","last_undegraded":"2024-03-04T10:03:44.558549+0000","last_fullsized":"2024-03-04T10:03:44.558549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925560+0000","last_change":"2024-03-04T10:03:43.573621+0000","last_active":"2024-03-04T10:03:44.925560+0000","last_peered":"2024-03-04T10:03:44.925560+0000","last_clean":"2024-03-04T10:03:44.925560+0000","last_became_active":"2024-03-04T10:03:43.573125+0000","last_became_peered":"2024-03-04T10:03:43.573125+0000","last_unstale":"2024-03-04T10:03:44.925560+0000","last_undegraded":"2024-03-04T10:03:44.925560+0000","last_fullsized":"2024-03-04T10:03:44.925560+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727176+0000","last_change":"2024-03-04T10:03:39.553163+0000","last_active":"2024-03-04T10:03:44.727176+0000","last_peered":"2024-03-04T10:03:44.727176+0000","last_clean":"2024-03-04T10:03:44.727176+0000","last_became_active":"2024-03-04T10:03:39.552909+0000","last_became_peered":"2024-03-04T10:03:39.552909+0000","last_unstale":"2024-03-04T10:03:44.727176+0000","last_undegraded":"2024-03-04T10:03:44.727176+0000","last_fullsized":"2024-03-04T10:03:44.727176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455728+0000","last_change":"2024-03-04T10:03:38.534433+0000","last_active":"2024-03-04T10:03:45.455728+0000","last_peered":"2024-03-04T10:03:45.455728+0000","last_clean":"2024-03-04T10:03:45.455728+0000","last_became_active":"2024-03-04T10:03:38.534162+0000","last_became_peered":"2024-03-04T10:03:38.534162+0000","last_unstale":"2024-03-04T10:03:45.455728+0000","last_undegraded":"2024-03-04T10:03:45.455728+0000","last_fullsized":"2024-03-04T10:03:45.455728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925353+0000","last_change":"2024-03-04T10:03:43.572145+0000","last_active":"2024-03-04T10:03:44.925353+0000","last_peered":"2024-03-04T10:03:44.925353+0000","last_clean":"2024-03-04T10:03:44.925353+0000","last_became_active":"2024-03-04T10:03:43.571762+0000","last_became_peered":"2024-03-04T10:03:43.571762+0000","last_unstale":"2024-03-04T10:03:44.925353+0000","last_undegraded":"2024-03-04T10:03:44.925353+0000","last_fullsized":"2024-03-04T10:03:44.925353+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:43.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727042+0000","last_change":"2024-03-04T10:03:39.558092+0000","last_active":"2024-03-04T10:03:44.727042+0000","last_peered":"2024-03-04T10:03:44.727042+0000","last_clean":"2024-03-04T10:03:44.727042+0000","last_became_active":"2024-03-04T10:03:39.557838+0000","last_became_peered":"2024-03-04T10:03:39.557838+0000","last_unstale":"2024-03-04T10:03:44.727042+0000","last_undegraded":"2024-03-04T10:03:44.727042+0000","last_fullsized":"2024-03-04T10:03:44.727042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456127+0000","last_change":"2024-03-04T10:03:38.538667+0000","last_active":"2024-03-04T10:03:45.456127+0000","last_peered":"2024-03-04T10:03:45.456127+0000","last_clean":"2024-03-04T10:03:45.456127+0000","last_became_active":"2024-03-04T10:03:38.538344+0000","last_became_peered":"2024-03-04T10:03:38.538344+0000","last_unstale":"2024-03-04T10:03:45.456127+0000","last_undegraded":"2024-03-04T10:03:45.456127+0000","last_fullsized":"2024-03-04T10:03:45.456127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456240+0000","last_change":"2024-03-04T10:03:43.583702+0000","last_active":"2024-03-04T10:03:45.456240+0000","last_peered":"2024-03-04T10:03:45.456240+0000","last_clean":"2024-03-04T10:03:45.456240+0000","last_became_active":"2024-03-04T10:03:43.583287+0000","last_became_peered":"2024-03-04T10:03:43.583287+0000","last_unstale":"2024-03-04T10:03:45.456240+0000","last_undegraded":"2024-03-04T10:03:45.456240+0000","last_fullsized":"2024-03-04T10:03:45.456240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:26.836809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925694+0000","last_change":"2024-03-04T10:03:39.542379+0000","last_active":"2024-03-04T10:03:44.925694+0000","last_peered":"2024-03-04T10:03:44.925694+0000","last_clean":"2024-03-04T10:03:44.925694+0000","last_became_active":"2024-03-04T10:03:39.541991+0000","last_became_peered":"2024-03-04T10:03:39.541991+0000","last_unstale":"2024-03-04T10:03:44.925694+0000","last_undegraded":"2024-03-04T10:03:44.925694+0000","last_fullsized":"2024-03-04T10:03:44.925694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:48.539804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557756+0000","last_change":"2024-03-04T10:03:38.532133+0000","last_active":"2024-03-04T10:03:44.557756+0000","last_peered":"2024-03-04T10:03:44.557756+0000","last_clean":"2024-03-04T10:03:44.557756+0000","last_became_active":"2024-03-04T10:03:38.531622+0000","last_became_peered":"2024-03-04T10:03:38.531622+0000","last_unstale":"2024-03-04T10:03:44.557756+0000","last_undegraded":"2024-03-04T10:03:44.557756+0000","last_fullsized":"2024-03-04T10:03:44.557756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925652+0000","last_change":"2024-03-04T10:03:43.570872+0000","last_active":"2024-03-04T10:03:44.925652+0000","last_peered":"2024-03-04T10:03:44.925652+0000","last_clean":"2024-03-04T10:03:44.925652+0000","last_became_active":"2024-03-04T10:03:43.570436+0000","last_became_peered":"2024-03-04T10:03:43.570436+0000","last_unstale":"2024-03-04T10:03:44.925652+0000","last_undegraded":"2024-03-04T10:03:44.925652+0000","last_fullsized":"2024-03-04T10:03:44.925652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455648+0000","last_change":"2024-03-04T10:03:39.541376+0000","last_active":"2024-03-04T10:03:45.455648+0000","last_peered":"2024-03-04T10:03:45.455648+0000","last_clean":"2024-03-04T10:03:45.455648+0000","last_became_active":"2024-03-04T10:03:39.541117+0000","last_became_peered":"2024-03-04T10:03:39.541117+0000","last_unstale":"2024-03-04T10:03:45.455648+0000","last_undegraded":"2024-03-04T10:03:45.455648+0000","last_fullsized":"2024-03-04T10:03:45.455648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558575+0000","last_change":"2024-03-04T10:03:38.538684+0000","last_active":"2024-03-04T10:03:44.558575+0000","last_peered":"2024-03-04T10:03:44.558575+0000","last_clean":"2024-03-04T10:03:44.558575+0000","last_became_active":"2024-03-04T10:03:38.538374+0000","last_became_peered":"2024-03-04T10:03:38.538374+0000","last_unstale":"2024-03-04T10:03:44.558575+0000","last_undegraded":"2024-03-04T10:03:44.558575+0000","last_fullsized":"2024-03-04T10:03:44.558575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925011+0000","last_change":"2024-03-04T10:03:43.575117+0000","last_active":"2024-03-04T10:03:44.925011+0000","last_peered":"2024-03-04T10:03:44.925011+0000","last_clean":"2024-03-04T10:03:44.925011+0000","last_became_active":"2024-03-04T10:03:43.574283+0000","last_became_peered":"2024-03-04T10:03:43.574283+0000","last_unstale":"2024-03-04T10:03:44.925011+0000","last_undegraded":"2024-03-04T10:03:44.925011+0000","last_fullsized":"2024-03-04T10:03:44.925011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:09.615407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727535+0000","last_change":"2024-03-04T10:03:39.554004+0000","last_active":"2024-03-04T10:03:44.727535+0000","last_peered":"2024-03-04T10:03:44.727535+0000","last_clean":"2024-03-04T10:03:44.727535+0000","last_became_active":"2024-03-04T10:03:39.553717+0000","last_became_peered":"2024-03-04T10:03:39.553717+0000","last_unstale":"2024-03-04T10:03:44.727535+0000","last_undegraded":"2024-03-04T10:03:44.727535+0000","last_fullsized":"2024-03-04T10:03:44.727535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455963+0000","last_change":"2024-03-04T10:03:38.533053+0000","last_active":"2024-03-04T10:03:45.455963+0000","last_peered":"2024-03-04T10:03:45.455963+0000","last_clean":"2024-03-04T10:03:45.455963+0000","last_became_active":"2024-03-04T10:03:38.532793+0000","last_became_peered":"2024-03-04T10:03:38.532793+0000","last_unstale":"2024-03-04T10:03:45.455963+0000","last_undegraded":"2024-03-04T10:03:45.455963+0000","last_fullsized":"2024-03-04T10:03:45.455963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726840+0000","last_change":"2024-03-04T10:03:43.577818+0000","last_active":"2024-03-04T10:03:44.726840+0000","last_peered":"2024-03-04T10:03:44.726840+0000","last_clean":"2024-03-04T10:03:44.726840+0000","last_became_active":"2024-03-04T10:03:43.577239+0000","last_became_peered":"2024-03-04T10:03:43.577239+0000","last_unstale":"2024-03-04T10:03:44.726840+0000","last_undegraded":"2024-03-04T10:03:44.726840+0000","last_fullsized":"2024-03-04T10:03:44.726840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455944+0000","last_change":"2024-03-04T10:03:39.542820+0000","last_active":"2024-03-04T10:03:45.455944+0000","last_peered":"2024-03-04T10:03:45.455944+0000","last_clean":"2024-03-04T10:03:45.455944+0000","last_became_active":"2024-03-04T10:03:39.542521+0000","last_became_peered":"2024-03-04T10:03:39.542521+0000","last_unstale":"2024-03-04T10:03:45.455944+0000","last_undegraded":"2024-03-04T10:03:45.455944+0000","last_fullsized":"2024-03-04T10:03:45.455944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558219+0000","last_change":"2024-03-04T10:03:38.534337+0000","last_active":"2024-03-04T10:03:44.558219+0000","last_peered":"2024-03-04T10:03:44.558219+0000","last_clean":"2024-03-04T10:03:44.558219+0000","last_became_active":"2024-03-04T10:03:38.534027+0000","last_became_peered":"2024-03-04T10:03:38.534027+0000","last_unstale":"2024-03-04T10:03:44.558219+0000","last_undegraded":"2024-03-04T10:03:44.558219+0000","last_fullsized":"2024-03-04T10:03:44.558219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:18.373547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455176+0000","last_change":"2024-03-04T10:03:43.577959+0000","last_active":"2024-03-04T10:03:45.455176+0000","last_peered":"2024-03-04T10:03:45.455176+0000","last_clean":"2024-03-04T10:03:45.455176+0000","last_became_active":"2024-03-04T10:03:43.576536+0000","last_became_peered":"2024-03-04T10:03:43.576536+0000","last_unstale":"2024-03-04T10:03:45.455176+0000","last_undegraded":"2024-03-04T10:03:45.455176+0000","last_fullsized":"2024-03-04T10:03:45.455176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456091+0000","last_change":"2024-03-04T10:03:39.556129+0000","last_active":"2024-03-04T10:03:45.456091+0000","last_peered":"2024-03-04T10:03:45.456091+0000","last_clean":"2024-03-04T10:03:45.456091+0000","last_became_active":"2024-03-04T10:03:39.555824+0000","last_became_peered":"2024-03-04T10:03:39.555824+0000","last_unstale":"2024-03-04T10:03:45.456091+0000","last_undegraded":"2024-03-04T10:03:45.456091+0000","last_fullsized":"2024-03-04T10:03:45.456091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456030+0000","last_change":"2024-03-04T10:03:38.531643+0000","last_active":"2024-03-04T10:03:45.456030+0000","last_peered":"2024-03-04T10:03:45.456030+0000","last_clean":"2024-03-04T10:03:45.456030+0000","last_became_active":"2024-03-04T10:03:38.531299+0000","last_became_peered":"2024-03-04T10:03:38.531299+0000","last_unstale":"2024-03-04T10:03:45.456030+0000","last_undegraded":"2024-03-04T10:03:45.456030+0000","last_fullsized":"2024-03-04T10:03:45.456030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455431+0000","last_change":"2024-03-04T10:03:43.576642+0000","last_active":"2024-03-04T10:03:45.455431+0000","last_peered":"2024-03-04T10:03:45.455431+0000","last_clean":"2024-03-04T10:03:45.455431+0000","last_became_active":"2024-03-04T10:03:43.576270+0000","last_became_peered":"2024-03-04T10:03:43.576270+0000","last_unstale":"2024-03-04T10:03:45.455431+0000","last_undegraded":"2024-03-04T10:03:45.455431+0000","last_fullsized":"2024-03-04T10:03:45.455431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925763+0000","last_change":"2024-03-04T10:03:39.546721+0000","last_active":"2024-03-04T10:03:44.925763+0000","last_peered":"2024-03-04T10:03:44.925763+0000","last_clean":"2024-03-04T10:03:44.925763+0000","last_became_active":"2024-03-04T10:03:39.546438+0000","last_became_peered":"2024-03-04T10:03:39.546438+0000","last_unstale":"2024-03-04T10:03:44.925763+0000","last_undegraded":"2024-03-04T10:03:44.925763+0000","last_fullsized":"2024-03-04T10:03:44.925763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455730+0000","last_change":"2024-03-04T10:03:38.534806+0000","last_active":"2024-03-04T10:03:45.455730+0000","last_peered":"2024-03-04T10:03:45.455730+0000","last_clean":"2024-03-04T10:03:45.455730+0000","last_became_active":"2024-03-04T10:03:38.534555+0000","last_became_peered":"2024-03-04T10:03:38.534555+0000","last_unstale":"2024-03-04T10:03:45.455730+0000","last_undegraded":"2024-03-04T10:03:45.455730+0000","last_fullsized":"2024-03-04T10:03:45.455730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455898+0000","last_change":"2024-03-04T10:03:43.583916+0000","last_active":"2024-03-04T10:03:45.455898+0000","last_peered":"2024-03-04T10:03:45.455898+0000","last_clean":"2024-03-04T10:03:45.455898+0000","last_became_active":"2024-03-04T10:03:43.583537+0000","last_became_peered":"2024-03-04T10:03:43.583537+0000","last_unstale":"2024-03-04T10:03:45.455898+0000","last_undegraded":"2024-03-04T10:03:45.455898+0000","last_fullsized":"2024-03-04T10:03:45.455898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:46.674144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454949+0000","last_change":"2024-03-04T10:03:39.539248+0000","last_active":"2024-03-04T10:03:45.454949+0000","last_peered":"2024-03-04T10:03:45.454949+0000","last_clean":"2024-03-04T10:03:45.454949+0000","last_became_active":"2024-03-04T10:03:39.538978+0000","last_became_peered":"2024-03-04T10:03:39.538978+0000","last_unstale":"2024-03-04T10:03:45.454949+0000","last_undegraded":"2024-03-04T10:03:45.454949+0000","last_fullsized":"2024-03-04T10:03:45.454949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556029+0000","last_change":"2024-03-04T10:03:38.534970+0000","last_active":"2024-03-04T10:03:44.556029+0000","last_peered":"2024-03-04T10:03:44.556029+0000","last_clean":"2024-03-04T10:03:44.556029+0000","last_became_active":"2024-03-04T10:03:38.534694+0000","last_became_peered":"2024-03-04T10:03:38.534694+0000","last_unstale":"2024-03-04T10:03:44.556029+0000","last_undegraded":"2024-03-04T10:03:44.556029+0000","last_fullsized":"2024-03-04T10:03:44.556029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456270+0000","last_change":"2024-03-04T10:03:43.582855+0000","last_active":"2024-03-04T10:03:45.456270+0000","last_peered":"2024-03-04T10:03:45.456270+0000","last_clean":"2024-03-04T10:03:45.456270+0000","last_became_active":"2024-03-04T10:03:43.580626+0000","last_became_peered":"2024-03-04T10:03:43.580626+0000","last_unstale":"2024-03-04T10:03:45.456270+0000","last_undegraded":"2024-03-04T10:03:45.456270+0000","last_fullsized":"2024-03-04T10:03:45.456270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:39.012948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454952+0000","last_change":"2024-03-04T10:03:39.538900+0000","last_active":"2024-03-04T10:03:45.454952+0000","last_peered":"2024-03-04T10:03:45.454952+0000","last_clean":"2024-03-04T10:03:45.454952+0000","last_became_active":"2024-03-04T10:03:39.538624+0000","last_became_peered":"2024-03-04T10:03:39.538624+0000","last_unstale":"2024-03-04T10:03:45.454952+0000","last_undegraded":"2024-03-04T10:03:45.454952+0000","last_fullsized":"2024-03-04T10:03:45.454952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557795+0000","last_change":"2024-03-04T10:03:38.534850+0000","last_active":"2024-03-04T10:03:44.557795+0000","last_peered":"2024-03-04T10:03:44.557795+0000","last_clean":"2024-03-04T10:03:44.557795+0000","last_became_active":"2024-03-04T10:03:38.534604+0000","last_became_peered":"2024-03-04T10:03:38.534604+0000","last_unstale":"2024-03-04T10:03:44.557795+0000","last_undegraded":"2024-03-04T10:03:44.557795+0000","last_fullsized":"2024-03-04T10:03:44.557795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455383+0000","last_change":"2024-03-04T10:03:43.576525+0000","last_active":"2024-03-04T10:03:45.455383+0000","last_peered":"2024-03-04T10:03:45.455383+0000","last_clean":"2024-03-04T10:03:45.455383+0000","last_became_active":"2024-03-04T10:03:43.576149+0000","last_became_peered":"2024-03-04T10:03:43.576149+0000","last_unstale":"2024-03-04T10:03:45.455383+0000","last_undegraded":"2024-03-04T10:03:45.455383+0000","last_fullsized":"2024-03-04T10:03:45.455383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727079+0000","last_change":"2024-03-04T10:03:39.557088+0000","last_active":"2024-03-04T10:03:44.727079+0000","last_peered":"2024-03-04T10:03:44.727079+0000","last_clean":"2024-03-04T10:03:44.727079+0000","last_became_active":"2024-03-04T10:03:39.556715+0000","last_became_peered":"2024-03-04T10:03:39.556715+0000","last_unstale":"2024-03-04T10:03:44.727079+0000","last_undegraded":"2024-03-04T10:03:44.727079+0000","last_fullsized":"2024-03-04T10:03:44.727079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455694+0000","last_change":"2024-03-04T10:03:38.537571+0000","last_active":"2024-03-04T10:03:45.455694+0000","last_peered":"2024-03-04T10:03:45.455694+0000","last_clean":"2024-03-04T10:03:45.455694+0000","last_became_active":"2024-03-04T10:03:38.536646+0000","last_became_peered":"2024-03-04T10:03:38.536646+0000","last_unstale":"2024-03-04T10:03:45.455694+0000","last_undegraded":"2024-03-04T10:03:45.455694+0000","last_fullsized":"2024-03-04T10:03:45.455694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557928+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:03:44.557928+0000","last_peered":"2024-03-04T10:03:44.557928+0000","last_clean":"2024-03-04T10:03:44.557928+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:03:44.557928+0000","last_undegraded":"2024-03-04T10:03:44.557928+0000","last_fullsized":"2024-03-04T10:03:44.557928+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558259+0000","last_change":"2024-03-04T10:03:43.579804+0000","last_active":"2024-03-04T10:03:44.558259+0000","last_peered":"2024-03-04T10:03:44.558259+0000","last_clean":"2024-03-04T10:03:44.558259+0000","last_became_active":"2024-03-04T10:03:43.579420+0000","last_became_peered":"2024-03-04T10:03:43.579420+0000","last_unstale":"2024-03-04T10:03:44.558259+0000","last_undegraded":"2024-03-04T10:03:44.558259+0000","last_fullsized":"2024-03-04T10:03:44.558259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557993+0000","last_change":"2024-03-04T10:03:39.550717+0000","last_active":"2024-03-04T10:03:44.557993+0000","last_peered":"2024-03-04T10:03:44.557993+0000","last_clean":"2024-03-04T10:03:44.557993+0000","last_became_active":"2024-03-04T10:03:39.550331+0000","last_became_peered":"2024-03-04T10:03:39.550331+0000","last_unstale":"2024-03-04T10:03:44.557993+0000","last_undegraded":"2024-03-04T10:03:44.557993+0000","last_fullsized":"2024-03-04T10:03:44.557993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556242+0000","last_change":"2024-03-04T10:03:38.529200+0000","last_active":"2024-03-04T10:03:44.556242+0000","last_peered":"2024-03-04T10:03:44.556242+0000","last_clean":"2024-03-04T10:03:44.556242+0000","last_became_active":"2024-03-04T10:03:38.528943+0000","last_became_peered":"2024-03-04T10:03:38.528943+0000","last_unstale":"2024-03-04T10:03:44.556242+0000","last_undegraded":"2024-03-04T10:03:44.556242+0000","last_fullsized":"2024-03-04T10:03:44.556242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557839+0000","last_change":"2024-03-04T10:03:43.578553+0000","last_active":"2024-03-04T10:03:44.557839+0000","last_peered":"2024-03-04T10:03:44.557839+0000","last_clean":"2024-03-04T10:03:44.557839+0000","last_became_active":"2024-03-04T10:03:43.577929+0000","last_became_peered":"2024-03-04T10:03:43.577929+0000","last_unstale":"2024-03-04T10:03:44.557839+0000","last_undegraded":"2024-03-04T10:03:44.557839+0000","last_fullsized":"2024-03-04T10:03:44.557839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455715+0000","last_change":"2024-03-04T10:03:39.556135+0000","last_active":"2024-03-04T10:03:45.455715+0000","last_peered":"2024-03-04T10:03:45.455715+0000","last_clean":"2024-03-04T10:03:45.455715+0000","last_became_active":"2024-03-04T10:03:39.555846+0000","last_became_peered":"2024-03-04T10:03:39.555846+0000","last_unstale":"2024-03-04T10:03:45.455715+0000","last_undegraded":"2024-03-04T10:03:45.455715+0000","last_fullsized":"2024-03-04T10:03:45.455715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455561+0000","last_change":"2024-03-04T10:03:38.530735+0000","last_active":"2024-03-04T10:03:45.455561+0000","last_peered":"2024-03-04T10:03:45.455561+0000","last_clean":"2024-03-04T10:03:45.455561+0000","last_became_active":"2024-03-04T10:03:38.530463+0000","last_became_peered":"2024-03-04T10:03:38.530463+0000","last_unstale":"2024-03-04T10:03:45.455561+0000","last_undegraded":"2024-03-04T10:03:45.455561+0000","last_fullsized":"2024-03-04T10:03:45.455561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455391+0000","last_change":"2024-03-04T10:03:43.577622+0000","last_active":"2024-03-04T10:03:45.455391+0000","last_peered":"2024-03-04T10:03:45.455391+0000","last_clean":"2024-03-04T10:03:45.455391+0000","last_became_active":"2024-03-04T10:03:43.575914+0000","last_became_peered":"2024-03-04T10:03:43.575914+0000","last_unstale":"2024-03-04T10:03:45.455391+0000","last_undegraded":"2024-03-04T10:03:45.455391+0000","last_fullsized":"2024-03-04T10:03:45.455391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558308+0000","last_change":"2024-03-04T10:03:39.551071+0000","last_active":"2024-03-04T10:03:44.558308+0000","last_peered":"2024-03-04T10:03:44.558308+0000","last_clean":"2024-03-04T10:03:44.558308+0000","last_became_active":"2024-03-04T10:03:39.550739+0000","last_became_peered":"2024-03-04T10:03:39.550739+0000","last_unstale":"2024-03-04T10:03:44.558308+0000","last_undegraded":"2024-03-04T10:03:44.558308+0000","last_fullsized":"2024-03-04T10:03:44.558308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:14.185049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556247+0000","last_change":"2024-03-04T10:03:38.534772+0000","last_active":"2024-03-04T10:03:44.556247+0000","last_peered":"2024-03-04T10:03:44.556247+0000","last_clean":"2024-03-04T10:03:44.556247+0000","last_became_active":"2024-03-04T10:03:38.534520+0000","last_became_peered":"2024-03-04T10:03:38.534520+0000","last_unstale":"2024-03-04T10:03:44.556247+0000","last_undegraded":"2024-03-04T10:03:44.556247+0000","last_fullsized":"2024-03-04T10:03:44.556247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455292+0000","last_change":"2024-03-04T10:03:43.570591+0000","last_active":"2024-03-04T10:03:45.455292+0000","last_peered":"2024-03-04T10:03:45.455292+0000","last_clean":"2024-03-04T10:03:45.455292+0000","last_became_active":"2024-03-04T10:03:43.570154+0000","last_became_peered":"2024-03-04T10:03:43.570154+0000","last_unstale":"2024-03-04T10:03:45.455292+0000","last_undegraded":"2024-03-04T10:03:45.455292+0000","last_fullsized":"2024-03-04T10:03:45.455292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558471+0000","last_change":"2024-03-04T10:03:39.547823+0000","last_active":"2024-03-04T10:03:44.558471+0000","last_peered":"2024-03-04T10:03:44.558471+0000","last_clean":"2024-03-04T10:03:44.558471+0000","last_became_active":"2024-03-04T10:03:39.547553+0000","last_became_peered":"2024-03-04T10:03:39.547553+0000","last_unstale":"2024-03-04T10:03:44.558471+0000","last_undegraded":"2024-03-04T10:03:44.558471+0000","last_fullsized":"2024-03-04T10:03:44.558471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925780+0000","last_change":"2024-03-04T10:03:38.530675+0000","last_active":"2024-03-04T10:03:44.925780+0000","last_peered":"2024-03-04T10:03:44.925780+0000","last_clean":"2024-03-04T10:03:44.925780+0000","last_became_active":"2024-03-04T10:03:38.530406+0000","last_became_peered":"2024-03-04T10:03:38.530406+0000","last_unstale":"2024-03-04T10:03:44.925780+0000","last_undegraded":"2024-03-04T10:03:44.925780+0000","last_fullsized":"2024-03-04T10:03:44.925780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:29.050824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925002+0000","last_change":"2024-03-04T10:03:43.577882+0000","last_active":"2024-03-04T10:03:44.925002+0000","last_peered":"2024-03-04T10:03:44.925002+0000","last_clean":"2024-03-04T10:03:44.925002+0000","last_became_active":"2024-03-04T10:03:43.577500+0000","last_became_peered":"2024-03-04T10:03:43.577500+0000","last_unstale":"2024-03-04T10:03:44.925002+0000","last_undegraded":"2024-03-04T10:03:44.925002+0000","last_fullsized":"2024-03-04T10:03:44.925002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:38.890692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557934+0000","last_change":"2024-03-04T10:03:39.546483+0000","last_active":"2024-03-04T10:03:44.557934+0000","last_peered":"2024-03-04T10:03:44.557934+0000","last_clean":"2024-03-04T10:03:44.557934+0000","last_became_active":"2024-03-04T10:03:39.546211+0000","last_became_peered":"2024-03-04T10:03:39.546211+0000","last_unstale":"2024-03-04T10:03:44.557934+0000","last_undegraded":"2024-03-04T10:03:44.557934+0000","last_fullsized":"2024-03-04T10:03:44.557934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555858+0000","last_change":"2024-03-04T10:03:38.532190+0000","last_active":"2024-03-04T10:03:44.555858+0000","last_peered":"2024-03-04T10:03:44.555858+0000","last_clean":"2024-03-04T10:03:44.555858+0000","last_became_active":"2024-03-04T10:03:38.531859+0000","last_became_peered":"2024-03-04T10:03:38.531859+0000","last_unstale":"2024-03-04T10:03:44.555858+0000","last_undegraded":"2024-03-04T10:03:44.555858+0000","last_fullsized":"2024-03-04T10:03:44.555858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:30:57.662758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455223+0000","last_change":"2024-03-04T10:03:43.575386+0000","last_active":"2024-03-04T10:03:45.455223+0000","last_peered":"2024-03-04T10:03:45.455223+0000","last_clean":"2024-03-04T10:03:45.455223+0000","last_became_active":"2024-03-04T10:03:43.575009+0000","last_became_peered":"2024-03-04T10:03:43.575009+0000","last_unstale":"2024-03-04T10:03:45.455223+0000","last_undegraded":"2024-03-04T10:03:45.455223+0000","last_fullsized":"2024-03-04T10:03:45.455223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:05.911238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556867+0000","last_change":"2024-03-04T10:03:39.546825+0000","last_active":"2024-03-04T10:03:44.556867+0000","last_peered":"2024-03-04T10:03:44.556867+0000","last_clean":"2024-03-04T10:03:44.556867+0000","last_became_active":"2024-03-04T10:03:39.546567+0000","last_became_peered":"2024-03-04T10:03:39.546567+0000","last_unstale":"2024-03-04T10:03:44.556867+0000","last_undegraded":"2024-03-04T10:03:44.556867+0000","last_fullsized":"2024-03-04T10:03:44.556867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557579+0000","last_change":"2024-03-04T10:03:38.529048+0000","last_active":"2024-03-04T10:03:44.557579+0000","last_peered":"2024-03-04T10:03:44.557579+0000","last_clean":"2024-03-04T10:03:44.557579+0000","last_became_active":"2024-03-04T10:03:38.528276+0000","last_became_peered":"2024-03-04T10:03:38.528276+0000","last_unstale":"2024-03-04T10:03:44.557579+0000","last_undegraded":"2024-03-04T10:03:44.557579+0000","last_fullsized":"2024-03-04T10:03:44.557579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727557+0000","last_change":"2024-03-04T10:03:43.578301+0000","last_active":"2024-03-04T10:03:44.727557+0000","last_peered":"2024-03-04T10:03:44.727557+0000","last_clean":"2024-03-04T10:03:44.727557+0000","last_became_active":"2024-03-04T10:03:43.577909+0000","last_became_peered":"2024-03-04T10:03:43.577909+0000","last_unstale":"2024-03-04T10:03:44.727557+0000","last_undegraded":"2024-03-04T10:03:44.727557+0000","last_fullsized":"2024-03-04T10:03:44.727557+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:37.260262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558519+0000","last_change":"2024-03-04T10:03:39.547091+0000","last_active":"2024-03-04T10:03:44.558519+0000","last_peered":"2024-03-04T10:03:44.558519+0000","last_clean":"2024-03-04T10:03:44.558519+0000","last_became_active":"2024-03-04T10:03:39.546836+0000","last_became_peered":"2024-03-04T10:03:39.546836+0000","last_unstale":"2024-03-04T10:03:44.558519+0000","last_undegraded":"2024-03-04T10:03:44.558519+0000","last_fullsized":"2024-03-04T10:03:44.558519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455647+0000","last_change":"2024-03-04T10:03:38.535462+0000","last_active":"2024-03-04T10:03:45.455647+0000","last_peered":"2024-03-04T10:03:45.455647+0000","last_clean":"2024-03-04T10:03:45.455647+0000","last_became_active":"2024-03-04T10:03:38.535165+0000","last_became_peered":"2024-03-04T10:03:38.535165+0000","last_unstale":"2024-03-04T10:03:45.455647+0000","last_undegraded":"2024-03-04T10:03:45.455647+0000","last_fullsized":"2024-03-04T10:03:45.455647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727307+0000","last_change":"2024-03-04T10:03:43.576692+0000","last_active":"2024-03-04T10:03:44.727307+0000","last_peered":"2024-03-04T10:03:44.727307+0000","last_clean":"2024-03-04T10:03:44.727307+0000","last_became_active":"2024-03-04T10:03:43.576321+0000","last_became_peered":"2024-03-04T10:03:43.576321+0000","last_unstale":"2024-03-04T10:03:44.727307+0000","last_undegraded":"2024-03-04T10:03:44.727307+0000","last_fullsized":"2024-03-04T10:03:44.727307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:14:32.702691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455942+0000","last_change":"2024-03-04T10:03:39.547184+0000","last_active":"2024-03-04T10:03:45.455942+0000","last_peered":"2024-03-04T10:03:45.455942+0000","last_clean":"2024-03-04T10:03:45.455942+0000","last_became_active":"2024-03-04T10:03:39.546899+0000","last_became_peered":"2024-03-04T10:03:39.546899+0000","last_unstale":"2024-03-04T10:03:45.455942+0000","last_undegraded":"2024-03-04T10:03:45.455942+0000","last_fullsized":"2024-03-04T10:03:45.455942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455444+0000","last_change":"2024-03-04T10:03:38.529853+0000","last_active":"2024-03-04T10:03:45.455444+0000","last_peered":"2024-03-04T10:03:45.455444+0000","last_clean":"2024-03-04T10:03:45.455444+0000","last_became_active":"2024-03-04T10:03:38.529259+0000","last_became_peered":"2024-03-04T10:03:38.529259+0000","last_unstale":"2024-03-04T10:03:45.455444+0000","last_undegraded":"2024-03-04T10:03:45.455444+0000","last_fullsized":"2024-03-04T10:03:45.455444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456026+0000","last_change":"2024-03-04T10:03:43.572841+0000","last_active":"2024-03-04T10:03:45.456026+0000","last_peered":"2024-03-04T10:03:45.456026+0000","last_clean":"2024-03-04T10:03:45.456026+0000","last_became_active":"2024-03-04T10:03:43.572414+0000","last_became_peered":"2024-03-04T10:03:43.572414+0000","last_unstale":"2024-03-04T10:03:45.456026+0000","last_undegraded":"2024-03-04T10:03:45.456026+0000","last_fullsized":"2024-03-04T10:03:45.456026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557911+0000","last_change":"2024-03-04T10:03:39.552501+0000","last_active":"2024-03-04T10:03:44.557911+0000","last_peered":"2024-03-04T10:03:44.557911+0000","last_clean":"2024-03-04T10:03:44.557911+0000","last_became_active":"2024-03-04T10:03:39.551870+0000","last_became_peered":"2024-03-04T10:03:39.551870+0000","last_unstale":"2024-03-04T10:03:44.557911+0000","last_undegraded":"2024-03-04T10:03:44.557911+0000","last_fullsized":"2024-03-04T10:03:44.557911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:48.793130+0000","last_change":"2024-03-04T10:03:38.534457+0000","last_active":"2024-03-04T10:03:48.793130+0000","last_peered":"2024-03-04T10:03:48.793130+0000","last_clean":"2024-03-04T10:03:48.793130+0000","last_became_active":"2024-03-04T10:03:38.534185+0000","last_became_peered":"2024-03-04T10:03:38.534185+0000","last_unstale":"2024-03-04T10:03:48.793130+0000","last_undegraded":"2024-03-04T10:03:48.793130+0000","last_fullsized":"2024-03-04T10:03:48.793130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455693+0000","last_change":"2024-03-04T10:03:43.572170+0000","last_active":"2024-03-04T10:03:45.455693+0000","last_peered":"2024-03-04T10:03:45.455693+0000","last_clean":"2024-03-04T10:03:45.455693+0000","last_became_active":"2024-03-04T10:03:43.571770+0000","last_became_peered":"2024-03-04T10:03:43.571770+0000","last_unstale":"2024-03-04T10:03:45.455693+0000","last_undegraded":"2024-03-04T10:03:45.455693+0000","last_fullsized":"2024-03-04T10:03:45.455693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:19.867931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925655+0000","last_change":"2024-03-04T10:03:39.548855+0000","last_active":"2024-03-04T10:03:44.925655+0000","last_peered":"2024-03-04T10:03:44.925655+0000","last_clean":"2024-03-04T10:03:44.925655+0000","last_became_active":"2024-03-04T10:03:39.548568+0000","last_became_peered":"2024-03-04T10:03:39.548568+0000","last_unstale":"2024-03-04T10:03:44.925655+0000","last_undegraded":"2024-03-04T10:03:44.925655+0000","last_fullsized":"2024-03-04T10:03:44.925655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:19.378207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455946+0000","last_change":"2024-03-04T10:03:38.537010+0000","last_active":"2024-03-04T10:03:45.455946+0000","last_peered":"2024-03-04T10:03:45.455946+0000","last_clean":"2024-03-04T10:03:45.455946+0000","last_became_active":"2024-03-04T10:03:38.536784+0000","last_became_peered":"2024-03-04T10:03:38.536784+0000","last_unstale":"2024-03-04T10:03:45.455946+0000","last_undegraded":"2024-03-04T10:03:45.455946+0000","last_fullsized":"2024-03-04T10:03:45.455946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925807+0000","last_change":"2024-03-04T10:03:43.575162+0000","last_active":"2024-03-04T10:03:44.925807+0000","last_peered":"2024-03-04T10:03:44.925807+0000","last_clean":"2024-03-04T10:03:44.925807+0000","last_became_active":"2024-03-04T10:03:43.574407+0000","last_became_peered":"2024-03-04T10:03:43.574407+0000","last_unstale":"2024-03-04T10:03:44.925807+0000","last_undegraded":"2024-03-04T10:03:44.925807+0000","last_fullsized":"2024-03-04T10:03:44.925807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455373+0000","last_change":"2024-03-04T10:03:39.550513+0000","last_active":"2024-03-04T10:03:45.455373+0000","last_peered":"2024-03-04T10:03:45.455373+0000","last_clean":"2024-03-04T10:03:45.455373+0000","last_became_active":"2024-03-04T10:03:39.549951+0000","last_became_peered":"2024-03-04T10:03:39.549951+0000","last_unstale":"2024-03-04T10:03:45.455373+0000","last_undegraded":"2024-03-04T10:03:45.455373+0000","last_fullsized":"2024-03-04T10:03:45.455373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455931+0000","last_change":"2024-03-04T10:03:38.538520+0000","last_active":"2024-03-04T10:03:45.455931+0000","last_peered":"2024-03-04T10:03:45.455931+0000","last_clean":"2024-03-04T10:03:45.455931+0000","last_became_active":"2024-03-04T10:03:38.538251+0000","last_became_peered":"2024-03-04T10:03:38.538251+0000","last_unstale":"2024-03-04T10:03:45.455931+0000","last_undegraded":"2024-03-04T10:03:45.455931+0000","last_fullsized":"2024-03-04T10:03:45.455931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727165+0000","last_change":"2024-03-04T10:03:43.573784+0000","last_active":"2024-03-04T10:03:44.727165+0000","last_peered":"2024-03-04T10:03:44.727165+0000","last_clean":"2024-03-04T10:03:44.727165+0000","last_became_active":"2024-03-04T10:03:43.573040+0000","last_became_peered":"2024-03-04T10:03:43.573040+0000","last_unstale":"2024-03-04T10:03:44.727165+0000","last_undegraded":"2024-03-04T10:03:44.727165+0000","last_fullsized":"2024-03-04T10:03:44.727165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:31.435305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455930+0000","last_change":"2024-03-04T10:03:39.559601+0000","last_active":"2024-03-04T10:03:45.455930+0000","last_peered":"2024-03-04T10:03:45.455930+0000","last_clean":"2024-03-04T10:03:45.455930+0000","last_became_active":"2024-03-04T10:03:39.559321+0000","last_became_peered":"2024-03-04T10:03:39.559321+0000","last_unstale":"2024-03-04T10:03:45.455930+0000","last_undegraded":"2024-03-04T10:03:45.455930+0000","last_fullsized":"2024-03-04T10:03:45.455930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558393+0000","last_change":"2024-03-04T10:03:38.536855+0000","last_active":"2024-03-04T10:03:44.558393+0000","last_peered":"2024-03-04T10:03:44.558393+0000","last_clean":"2024-03-04T10:03:44.558393+0000","last_became_active":"2024-03-04T10:03:38.536551+0000","last_became_peered":"2024-03-04T10:03:38.536551+0000","last_unstale":"2024-03-04T10:03:44.558393+0000","last_undegraded":"2024-03-04T10:03:44.558393+0000","last_fullsized":"2024-03-04T10:03:44.558393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727003+0000","last_change":"2024-03-04T10:03:43.569782+0000","last_active":"2024-03-04T10:03:44.727003+0000","last_peered":"2024-03-04T10:03:44.727003+0000","last_clean":"2024-03-04T10:03:44.727003+0000","last_became_active":"2024-03-04T10:03:43.568928+0000","last_became_peered":"2024-03-04T10:03:43.568928+0000","last_unstale":"2024-03-04T10:03:44.727003+0000","last_undegraded":"2024-03-04T10:03:44.727003+0000","last_fullsized":"2024-03-04T10:03:44.727003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:33.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455065+0000","last_change":"2024-03-04T10:03:43.577832+0000","last_active":"2024-03-04T10:03:45.455065+0000","last_peered":"2024-03-04T10:03:45.455065+0000","last_clean":"2024-03-04T10:03:45.455065+0000","last_became_active":"2024-03-04T10:03:43.577460+0000","last_became_peered":"2024-03-04T10:03:43.577460+0000","last_unstale":"2024-03-04T10:03:45.455065+0000","last_undegraded":"2024-03-04T10:03:45.455065+0000","last_fullsized":"2024-03-04T10:03:45.455065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455950+0000","last_change":"2024-03-04T10:03:39.547155+0000","last_active":"2024-03-04T10:03:45.455950+0000","last_peered":"2024-03-04T10:03:45.455950+0000","last_clean":"2024-03-04T10:03:45.455950+0000","last_became_active":"2024-03-04T10:03:39.546882+0000","last_became_peered":"2024-03-04T10:03:39.546882+0000","last_unstale":"2024-03-04T10:03:45.455950+0000","last_undegraded":"2024-03-04T10:03:45.455950+0000","last_fullsized":"2024-03-04T10:03:45.455950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455427+0000","last_change":"2024-03-04T10:03:38.533042+0000","last_active":"2024-03-04T10:03:45.455427+0000","last_peered":"2024-03-04T10:03:45.455427+0000","last_clean":"2024-03-04T10:03:45.455427+0000","last_became_active":"2024-03-04T10:03:38.532609+0000","last_became_peered":"2024-03-04T10:03:38.532609+0000","last_unstale":"2024-03-04T10:03:45.455427+0000","last_undegraded":"2024-03-04T10:03:45.455427+0000","last_fullsized":"2024-03-04T10:03:45.455427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558268+0000","last_change":"2024-03-04T10:03:43.577800+0000","last_active":"2024-03-04T10:03:44.558268+0000","last_peered":"2024-03-04T10:03:44.558268+0000","last_clean":"2024-03-04T10:03:44.558268+0000","last_became_active":"2024-03-04T10:03:43.577379+0000","last_became_peered":"2024-03-04T10:03:43.577379+0000","last_unstale":"2024-03-04T10:03:44.558268+0000","last_undegraded":"2024-03-04T10:03:44.558268+0000","last_fullsized":"2024-03-04T10:03:44.558268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726957+0000","last_change":"2024-03-04T10:03:39.555669+0000","last_active":"2024-03-04T10:03:44.726957+0000","last_peered":"2024-03-04T10:03:44.726957+0000","last_clean":"2024-03-04T10:03:44.726957+0000","last_became_active":"2024-03-04T10:03:39.555415+0000","last_became_peered":"2024-03-04T10:03:39.555415+0000","last_unstale":"2024-03-04T10:03:44.726957+0000","last_undegraded":"2024-03-04T10:03:44.726957+0000","last_fullsized":"2024-03-04T10:03:44.726957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727092+0000","last_change":"2024-03-04T10:03:38.537206+0000","last_active":"2024-03-04T10:03:44.727092+0000","last_peered":"2024-03-04T10:03:44.727092+0000","last_clean":"2024-03-04T10:03:44.727092+0000","last_became_active":"2024-03-04T10:03:38.536938+0000","last_became_peered":"2024-03-04T10:03:38.536938+0000","last_unstale":"2024-03-04T10:03:44.727092+0000","last_undegraded":"2024-03-04T10:03:44.727092+0000","last_fullsized":"2024-03-04T10:03:44.727092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558028+0000","last_change":"2024-03-04T10:03:38.528419+0000","last_active":"2024-03-04T10:03:44.558028+0000","last_peered":"2024-03-04T10:03:44.558028+0000","last_clean":"2024-03-04T10:03:44.558028+0000","last_became_active":"2024-03-04T10:03:38.528143+0000","last_became_peered":"2024-03-04T10:03:38.528143+0000","last_unstale":"2024-03-04T10:03:44.558028+0000","last_undegraded":"2024-03-04T10:03:44.558028+0000","last_fullsized":"2024-03-04T10:03:44.558028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455342+0000","last_change":"2024-03-04T10:03:39.537531+0000","last_active":"2024-03-04T10:03:45.455342+0000","last_peered":"2024-03-04T10:03:45.455342+0000","last_clean":"2024-03-04T10:03:45.455342+0000","last_became_active":"2024-03-04T10:03:39.537282+0000","last_became_peered":"2024-03-04T10:03:39.537282+0000","last_unstale":"2024-03-04T10:03:45.455342+0000","last_undegraded":"2024-03-04T10:03:45.455342+0000","last_fullsized":"2024-03-04T10:03:45.455342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556191+0000","last_change":"2024-03-04T10:03:43.576190+0000","last_active":"2024-03-04T10:03:44.556191+0000","last_peered":"2024-03-04T10:03:44.556191+0000","last_clean":"2024-03-04T10:03:44.556191+0000","last_became_active":"2024-03-04T10:03:43.575758+0000","last_became_peered":"2024-03-04T10:03:43.575758+0000","last_unstale":"2024-03-04T10:03:44.556191+0000","last_undegraded":"2024-03-04T10:03:44.556191+0000","last_fullsized":"2024-03-04T10:03:44.556191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:27:39.111875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727248+0000","last_change":"2024-03-04T10:03:38.538563+0000","last_active":"2024-03-04T10:03:44.727248+0000","last_peered":"2024-03-04T10:03:44.727248+0000","last_clean":"2024-03-04T10:03:44.727248+0000","last_became_active":"2024-03-04T10:03:38.538284+0000","last_became_peered":"2024-03-04T10:03:38.538284+0000","last_unstale":"2024-03-04T10:03:44.727248+0000","last_undegraded":"2024-03-04T10:03:44.727248+0000","last_fullsized":"2024-03-04T10:03:44.727248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925234+0000","last_change":"2024-03-04T10:03:39.543278+0000","last_active":"2024-03-04T10:03:44.925234+0000","last_peered":"2024-03-04T10:03:44.925234+0000","last_clean":"2024-03-04T10:03:44.925234+0000","last_became_active":"2024-03-04T10:03:39.542993+0000","last_became_peered":"2024-03-04T10:03:39.542993+0000","last_unstale":"2024-03-04T10:03:44.925234+0000","last_undegraded":"2024-03-04T10:03:44.925234+0000","last_fullsized":"2024-03-04T10:03:44.925234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:44.723658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727569+0000","last_change":"2024-03-04T10:03:43.569930+0000","last_active":"2024-03-04T10:03:44.727569+0000","last_peered":"2024-03-04T10:03:44.727569+0000","last_clean":"2024-03-04T10:03:44.727569+0000","last_became_active":"2024-03-04T10:03:43.569524+0000","last_became_peered":"2024-03-04T10:03:43.569524+0000","last_unstale":"2024-03-04T10:03:44.727569+0000","last_undegraded":"2024-03-04T10:03:44.727569+0000","last_fullsized":"2024-03-04T10:03:44.727569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:28.002170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556756+0000","last_change":"2024-03-04T10:03:39.547564+0000","last_active":"2024-03-04T10:03:44.556756+0000","last_peered":"2024-03-04T10:03:44.556756+0000","last_clean":"2024-03-04T10:03:44.556756+0000","last_became_active":"2024-03-04T10:03:39.547245+0000","last_became_peered":"2024-03-04T10:03:39.547245+0000","last_unstale":"2024-03-04T10:03:44.556756+0000","last_undegraded":"2024-03-04T10:03:44.556756+0000","last_fullsized":"2024-03-04T10:03:44.556756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:15:14.213025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555940+0000","last_change":"2024-03-04T10:03:39.549137+0000","last_active":"2024-03-04T10:03:44.555940+0000","last_peered":"2024-03-04T10:03:44.555940+0000","last_clean":"2024-03-04T10:03:44.555940+0000","last_became_active":"2024-03-04T10:03:39.548867+0000","last_became_peered":"2024-03-04T10:03:39.548867+0000","last_unstale":"2024-03-04T10:03:44.555940+0000","last_undegraded":"2024-03-04T10:03:44.555940+0000","last_fullsized":"2024-03-04T10:03:44.555940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455603+0000","last_change":"2024-03-04T10:03:39.555564+0000","last_active":"2024-03-04T10:03:45.455603+0000","last_peered":"2024-03-04T10:03:45.455603+0000","last_clean":"2024-03-04T10:03:45.455603+0000","last_became_active":"2024-03-04T10:03:39.555310+0000","last_became_peered":"2024-03-04T10:03:39.555310+0000","last_unstale":"2024-03-04T10:03:45.455603+0000","last_undegraded":"2024-03-04T10:03:45.455603+0000","last_fullsized":"2024-03-04T10:03:45.455603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455247+0000","last_change":"2024-03-04T10:03:39.550377+0000","last_active":"2024-03-04T10:03:45.455247+0000","last_peered":"2024-03-04T10:03:45.455247+0000","last_clean":"2024-03-04T10:03:45.455247+0000","last_became_active":"2024-03-04T10:03:39.550028+0000","last_became_peered":"2024-03-04T10:03:39.550028+0000","last_unstale":"2024-03-04T10:03:45.455247+0000","last_undegraded":"2024-03-04T10:03:45.455247+0000","last_fullsized":"2024-03-04T10:03:45.455247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727357+0000","last_change":"2024-03-04T10:03:39.553769+0000","last_active":"2024-03-04T10:03:44.727357+0000","last_peered":"2024-03-04T10:03:44.727357+0000","last_clean":"2024-03-04T10:03:44.727357+0000","last_became_active":"2024-03-04T10:03:39.553517+0000","last_became_peered":"2024-03-04T10:03:39.553517+0000","last_unstale":"2024-03-04T10:03:44.727357+0000","last_undegraded":"2024-03-04T10:03:44.727357+0000","last_fullsized":"2024-03-04T10:03:44.727357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455401+0000","last_change":"2024-03-04T10:03:39.551104+0000","last_active":"2024-03-04T10:03:45.455401+0000","last_peered":"2024-03-04T10:03:45.455401+0000","last_clean":"2024-03-04T10:03:45.455401+0000","last_became_active":"2024-03-04T10:03:39.550823+0000","last_became_peered":"2024-03-04T10:03:39.550823+0000","last_unstale":"2024-03-04T10:03:45.455401+0000","last_undegraded":"2024-03-04T10:03:45.455401+0000","last_fullsized":"2024-03-04T10:03:45.455401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558684+0000","last_change":"2024-03-04T10:03:39.563672+0000","last_active":"2024-03-04T10:03:44.558684+0000","last_peered":"2024-03-04T10:03:44.558684+0000","last_clean":"2024-03-04T10:03:44.558684+0000","last_became_active":"2024-03-04T10:03:39.563394+0000","last_became_peered":"2024-03-04T10:03:39.563394+0000","last_unstale":"2024-03-04T10:03:44.558684+0000","last_undegraded":"2024-03-04T10:03:44.558684+0000","last_fullsized":"2024-03-04T10:03:44.558684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727442+0000","last_change":"2024-03-04T10:03:39.549746+0000","last_active":"2024-03-04T10:03:44.727442+0000","last_peered":"2024-03-04T10:03:44.727442+0000","last_clean":"2024-03-04T10:03:44.727442+0000","last_became_active":"2024-03-04T10:03:39.549349+0000","last_became_peered":"2024-03-04T10:03:39.549349+0000","last_unstale":"2024-03-04T10:03:44.727442+0000","last_undegraded":"2024-03-04T10:03:44.727442+0000","last_fullsized":"2024-03-04T10:03:44.727442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455405+0000","last_change":"2024-03-04T10:03:39.556217+0000","last_active":"2024-03-04T10:03:45.455405+0000","last_peered":"2024-03-04T10:03:45.455405+0000","last_clean":"2024-03-04T10:03:45.455405+0000","last_became_active":"2024-03-04T10:03:39.555896+0000","last_became_peered":"2024-03-04T10:03:39.555896+0000","last_unstale":"2024-03-04T10:03:45.455405+0000","last_undegraded":"2024-03-04T10:03:45.455405+0000","last_fullsized":"2024-03-04T10:03:45.455405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726990+0000","last_change":"2024-03-04T10:03:39.546403+0000","last_active":"2024-03-04T10:03:44.726990+0000","last_peered":"2024-03-04T10:03:44.726990+0000","last_clean":"2024-03-04T10:03:44.726990+0000","last_became_active":"2024-03-04T10:03:39.546136+0000","last_became_peered":"2024-03-04T10:03:39.546136+0000","last_unstale":"2024-03-04T10:03:44.726990+0000","last_undegraded":"2024-03-04T10:03:44.726990+0000","last_fullsized":"2024-03-04T10:03:44.726990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455552+0000","last_change":"2024-03-04T10:03:39.553616+0000","last_active":"2024-03-04T10:03:45.455552+0000","last_peered":"2024-03-04T10:03:45.455552+0000","last_clean":"2024-03-04T10:03:45.455552+0000","last_became_active":"2024-03-04T10:03:39.553356+0000","last_became_peered":"2024-03-04T10:03:39.553356+0000","last_unstale":"2024-03-04T10:03:45.455552+0000","last_undegraded":"2024-03-04T10:03:45.455552+0000","last_fullsized":"2024-03-04T10:03:45.455552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455120+0000","last_change":"2024-03-04T10:03:39.539406+0000","last_active":"2024-03-04T10:03:45.455120+0000","last_peered":"2024-03-04T10:03:45.455120+0000","last_clean":"2024-03-04T10:03:45.455120+0000","last_became_active":"2024-03-04T10:03:39.539139+0000","last_became_peered":"2024-03-04T10:03:39.539139+0000","last_unstale":"2024-03-04T10:03:45.455120+0000","last_undegraded":"2024-03-04T10:03:45.455120+0000","last_fullsized":"2024-03-04T10:03:45.455120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558323+0000","last_change":"2024-03-04T10:03:39.554026+0000","last_active":"2024-03-04T10:03:44.558323+0000","last_peered":"2024-03-04T10:03:44.558323+0000","last_clean":"2024-03-04T10:03:44.558323+0000","last_became_active":"2024-03-04T10:03:39.553711+0000","last_became_peered":"2024-03-04T10:03:39.553711+0000","last_unstale":"2024-03-04T10:03:44.558323+0000","last_undegraded":"2024-03-04T10:03:44.558323+0000","last_fullsized":"2024-03-04T10:03:44.558323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558201+0000","last_change":"2024-03-04T10:03:39.545468+0000","last_active":"2024-03-04T10:03:44.558201+0000","last_peered":"2024-03-04T10:03:44.558201+0000","last_clean":"2024-03-04T10:03:44.558201+0000","last_became_active":"2024-03-04T10:03:39.545202+0000","last_became_peered":"2024-03-04T10:03:39.545202+0000","last_unstale":"2024-03-04T10:03:44.558201+0000","last_undegraded":"2024-03-04T10:03:44.558201+0000","last_fullsized":"2024-03-04T10:03:44.558201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:42.641276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556175+0000","last_change":"2024-03-04T10:03:39.549637+0000","last_active":"2024-03-04T10:03:44.556175+0000","last_peered":"2024-03-04T10:03:44.556175+0000","last_clean":"2024-03-04T10:03:44.556175+0000","last_became_active":"2024-03-04T10:03:39.549384+0000","last_became_peered":"2024-03-04T10:03:39.549384+0000","last_unstale":"2024-03-04T10:03:44.556175+0000","last_undegraded":"2024-03-04T10:03:44.556175+0000","last_fullsized":"2024-03-04T10:03:44.556175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456190+0000","last_change":"2024-03-04T10:03:39.559786+0000","last_active":"2024-03-04T10:03:45.456190+0000","last_peered":"2024-03-04T10:03:45.456190+0000","last_clean":"2024-03-04T10:03:45.456190+0000","last_became_active":"2024-03-04T10:03:39.559507+0000","last_became_peered":"2024-03-04T10:03:39.559507+0000","last_unstale":"2024-03-04T10:03:45.456190+0000","last_undegraded":"2024-03-04T10:03:45.456190+0000","last_fullsized":"2024-03-04T10:03:45.456190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455259+0000","last_change":"2024-03-04T10:03:39.547601+0000","last_active":"2024-03-04T10:03:45.455259+0000","last_peered":"2024-03-04T10:03:45.455259+0000","last_clean":"2024-03-04T10:03:45.455259+0000","last_became_active":"2024-03-04T10:03:39.547309+0000","last_became_peered":"2024-03-04T10:03:39.547309+0000","last_unstale":"2024-03-04T10:03:45.455259+0000","last_undegraded":"2024-03-04T10:03:45.455259+0000","last_fullsized":"2024-03-04T10:03:45.455259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456059+0000","last_change":"2024-03-04T10:03:39.543308+0000","last_active":"2024-03-04T10:03:45.456059+0000","last_peered":"2024-03-04T10:03:45.456059+0000","last_clean":"2024-03-04T10:03:45.456059+0000","last_became_active":"2024-03-04T10:03:39.543064+0000","last_became_peered":"2024-03-04T10:03:39.543064+0000","last_unstale":"2024-03-04T10:03:45.456059+0000","last_undegraded":"2024-03-04T10:03:45.456059+0000","last_fullsized":"2024-03-04T10:03:45.456059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455862+0000","last_change":"2024-03-04T10:03:39.540533+0000","last_active":"2024-03-04T10:03:45.455862+0000","last_peered":"2024-03-04T10:03:45.455862+0000","last_clean":"2024-03-04T10:03:45.455862+0000","last_became_active":"2024-03-04T10:03:39.540274+0000","last_became_peered":"2024-03-04T10:03:39.540274+0000","last_unstale":"2024-03-04T10:03:45.455862+0000","last_undegraded":"2024-03-04T10:03:45.455862+0000","last_fullsized":"2024-03-04T10:03:45.455862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727155+0000","last_change":"2024-03-04T10:03:39.556252+0000","last_active":"2024-03-04T10:03:44.727155+0000","last_peered":"2024-03-04T10:03:44.727155+0000","last_clean":"2024-03-04T10:03:44.727155+0000","last_became_active":"2024-03-04T10:03:39.556009+0000","last_became_peered":"2024-03-04T10:03:39.556009+0000","last_unstale":"2024-03-04T10:03:44.727155+0000","last_undegraded":"2024-03-04T10:03:44.727155+0000","last_fullsized":"2024-03-04T10:03:44.727155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925508+0000","last_change":"2024-03-04T10:03:39.552666+0000","last_active":"2024-03-04T10:03:44.925508+0000","last_peered":"2024-03-04T10:03:44.925508+0000","last_clean":"2024-03-04T10:03:44.925508+0000","last_became_active":"2024-03-04T10:03:39.552178+0000","last_became_peered":"2024-03-04T10:03:39.552178+0000","last_unstale":"2024-03-04T10:03:44.925508+0000","last_undegraded":"2024-03-04T10:03:44.925508+0000","last_fullsized":"2024-03-04T10:03:44.925508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558388+0000","last_change":"2024-03-04T10:03:39.557130+0000","last_active":"2024-03-04T10:03:44.558388+0000","last_peered":"2024-03-04T10:03:44.558388+0000","last_clean":"2024-03-04T10:03:44.558388+0000","last_became_active":"2024-03-04T10:03:39.556736+0000","last_became_peered":"2024-03-04T10:03:39.556736+0000","last_unstale":"2024-03-04T10:03:44.558388+0000","last_undegraded":"2024-03-04T10:03:44.558388+0000","last_fullsized":"2024-03-04T10:03:44.558388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558631+0000","last_change":"2024-03-04T10:03:39.556929+0000","last_active":"2024-03-04T10:03:44.558631+0000","last_peered":"2024-03-04T10:03:44.558631+0000","last_clean":"2024-03-04T10:03:44.558631+0000","last_became_active":"2024-03-04T10:03:39.556672+0000","last_became_peered":"2024-03-04T10:03:39.556672+0000","last_unstale":"2024-03-04T10:03:44.558631+0000","last_undegraded":"2024-03-04T10:03:44.558631+0000","last_fullsized":"2024-03-04T10:03:44.558631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558062+0000","last_change":"2024-03-04T10:03:39.553116+0000","last_active":"2024-03-04T10:03:44.558062+0000","last_peered":"2024-03-04T10:03:44.558062+0000","last_clean":"2024-03-04T10:03:44.558062+0000","last_became_active":"2024-03-04T10:03:39.551779+0000","last_became_peered":"2024-03-04T10:03:39.551779+0000","last_unstale":"2024-03-04T10:03:44.558062+0000","last_undegraded":"2024-03-04T10:03:44.558062+0000","last_fullsized":"2024-03-04T10:03:44.558062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:34.980080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726862+0000","last_change":"2024-03-04T10:03:39.557751+0000","last_active":"2024-03-04T10:03:44.726862+0000","last_peered":"2024-03-04T10:03:44.726862+0000","last_clean":"2024-03-04T10:03:44.726862+0000","last_became_active":"2024-03-04T10:03:39.557443+0000","last_became_peered":"2024-03-04T10:03:39.557443+0000","last_unstale":"2024-03-04T10:03:44.726862+0000","last_undegraded":"2024-03-04T10:03:44.726862+0000","last_fullsized":"2024-03-04T10:03:44.726862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557825+0000","last_change":"2024-03-04T10:03:39.547100+0000","last_active":"2024-03-04T10:03:44.557825+0000","last_peered":"2024-03-04T10:03:44.557825+0000","last_clean":"2024-03-04T10:03:44.557825+0000","last_became_active":"2024-03-04T10:03:39.546794+0000","last_became_peered":"2024-03-04T10:03:39.546794+0000","last_unstale":"2024-03-04T10:03:44.557825+0000","last_undegraded":"2024-03-04T10:03:44.557825+0000","last_fullsized":"2024-03-04T10:03:44.557825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455496+0000","last_change":"2024-03-04T10:03:39.548756+0000","last_active":"2024-03-04T10:03:45.455496+0000","last_peered":"2024-03-04T10:03:45.455496+0000","last_clean":"2024-03-04T10:03:45.455496+0000","last_became_active":"2024-03-04T10:03:39.548474+0000","last_became_peered":"2024-03-04T10:03:39.548474+0000","last_unstale":"2024-03-04T10:03:45.455496+0000","last_undegraded":"2024-03-04T10:03:45.455496+0000","last_fullsized":"2024-03-04T10:03:45.455496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557997+0000","last_change":"2024-03-04T10:03:39.544100+0000","last_active":"2024-03-04T10:03:44.557997+0000","last_peered":"2024-03-04T10:03:44.557997+0000","last_clean":"2024-03-04T10:03:44.557997+0000","last_became_active":"2024-03-04T10:03:39.543581+0000","last_became_peered":"2024-03-04T10:03:39.543581+0000","last_unstale":"2024-03-04T10:03:44.557997+0000","last_undegraded":"2024-03-04T10:03:44.557997+0000","last_fullsized":"2024-03-04T10:03:44.557997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455591+0000","last_change":"2024-03-04T10:03:39.556280+0000","last_active":"2024-03-04T10:03:45.455591+0000","last_peered":"2024-03-04T10:03:45.455591+0000","last_clean":"2024-03-04T10:03:45.455591+0000","last_became_active":"2024-03-04T10:03:39.555867+0000","last_became_peered":"2024-03-04T10:03:39.555867+0000","last_unstale":"2024-03-04T10:03:45.455591+0000","last_undegraded":"2024-03-04T10:03:45.455591+0000","last_fullsized":"2024-03-04T10:03:45.455591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455074+0000","last_change":"2024-03-04T10:03:39.538030+0000","last_active":"2024-03-04T10:03:45.455074+0000","last_peered":"2024-03-04T10:03:45.455074+0000","last_clean":"2024-03-04T10:03:45.455074+0000","last_became_active":"2024-03-04T10:03:39.537783+0000","last_became_peered":"2024-03-04T10:03:39.537783+0000","last_unstale":"2024-03-04T10:03:45.455074+0000","last_undegraded":"2024-03-04T10:03:45.455074+0000","last_fullsized":"2024-03-04T10:03:45.455074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455845+0000","last_change":"2024-03-04T10:03:39.545505+0000","last_active":"2024-03-04T10:03:45.455845+0000","last_peered":"2024-03-04T10:03:45.455845+0000","last_clean":"2024-03-04T10:03:45.455845+0000","last_became_active":"2024-03-04T10:03:39.545176+0000","last_became_peered":"2024-03-04T10:03:39.545176+0000","last_unstale":"2024-03-04T10:03:45.455845+0000","last_undegraded":"2024-03-04T10:03:45.455845+0000","last_fullsized":"2024-03-04T10:03:45.455845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557636+0000","last_change":"2024-03-04T10:03:39.551482+0000","last_active":"2024-03-04T10:03:44.557636+0000","last_peered":"2024-03-04T10:03:44.557636+0000","last_clean":"2024-03-04T10:03:44.557636+0000","last_became_active":"2024-03-04T10:03:39.551234+0000","last_became_peered":"2024-03-04T10:03:39.551234+0000","last_unstale":"2024-03-04T10:03:44.557636+0000","last_undegraded":"2024-03-04T10:03:44.557636+0000","last_fullsized":"2024-03-04T10:03:44.557636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:22.803941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:04:02.914 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T10:04:02.914 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T10:04:02.914 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-04T10:04:02.914 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-04T10:04:02.915 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-04T10:04:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 Processor -- start 2024-03-04T10:04:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 -- start start 2024-03-04T10:04:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f54541597f0 0x7f545415c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 0x7f54540cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545415cfe0 con 0x7f545406acd0 2024-03-04T10:04:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540d0240 con 0x7f54541597f0 2024-03-04T10:04:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.048+0000 7f5459e61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540cf630 con 0x7f545406b860 2024-03-04T10:04:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f5453fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 0x7f54540cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f54537fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f54541597f0 0x7f545415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f5452ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f545406acd0 0x7f545406b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:58284/0 (socket says 172.21.15.99:58284) 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f5453fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 0x7f54540cc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:54036/0 (socket says 172.21.15.99:54036) 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f54537fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f54541597f0 0x7f545415c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:40284/0 (socket says 172.21.15.99:40284) 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f5452ffd640 1 -- 172.21.15.99:0/2018583000 learned_addr learned my addr 172.21.15.99:0/2018583000 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f54537fe640 1 -- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 msgr2=0x7f54540cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f54537fe640 1 --2- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 0x7f54540cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f54537fe640 1 -- 172.21.15.99:0/2018583000 >> v1:172.21.15.162:6789/0 conn(0x7f545406acd0 legacy=0x7f545406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f54537fe640 1 -- 172.21.15.99:0/2018583000 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540cf910 con 0x7f54541597f0 2024-03-04T10:04:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.049+0000 7f5453fff640 1 --2- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 0x7f54540cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:03.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.050+0000 7f54537fe640 1 --2- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54541597f0 0x7f545415c030 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54440097a0 tx=0x7f544402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51ca279416b18746 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.050+0000 7f54527fc640 1 -- 172.21.15.99:0/2018583000 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54440315a0 con 0x7f54541597f0 2024-03-04T10:04:03.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.050+0000 7f54527fc640 1 -- 172.21.15.99:0/2018583000 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5444037070 con 0x7f54541597f0 2024-03-04T10:04:03.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.050+0000 7f54527fc640 1 -- 172.21.15.99:0/2018583000 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444031960 con 0x7f54541597f0 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 -- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54541597f0 msgr2=0x7f545415c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 --2- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54541597f0 0x7f545415c030 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54440097a0 tx=0x7f544402b840 comp rx=0 tx=0).stop 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 -- 172.21.15.99:0/2018583000 shutdown_connections 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 --2- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f545406b860 0x7f54540cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 --2- 172.21.15.99:0/2018583000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54541597f0 0x7f545415c030 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 -- 172.21.15.99:0/2018583000 >> 172.21.15.99:0/2018583000 conn(0x7f54540d0710 msgr2=0x7f54540d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 -- 172.21.15.99:0/2018583000 shutdown_connections 2024-03-04T10:04:03.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.051+0000 7f5459e61640 1 -- 172.21.15.99:0/2018583000 wait complete. 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.052+0000 7f5459e61640 1 Processor -- start 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.052+0000 7f5459e61640 1 -- start start 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5459e61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 0x7f5454070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5459e61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545406b860 0x7f54540c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5459e61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54541597f0 0x7f54540b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5459e61640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540d1350 con 0x7f545406acd0 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5459e61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540cfed0 con 0x7f545406b860 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5459e61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540cc7a0 con 0x7f54541597f0 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 0x7f5454070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 0x7f5454070080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45938/0 (socket says 172.21.15.99:45938) 2024-03-04T10:04:03.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 -- 172.21.15.99:0/3395614273 learned_addr learned my addr 172.21.15.99:0/3395614273 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5452ffd640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545406b860 0x7f54540c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f5453fff640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54541597f0 0x7f54540b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 -- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54541597f0 msgr2=0x7f54540b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54541597f0 0x7f54540b23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 -- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545406b860 msgr2=0x7f54540c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545406b860 0x7f54540c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 -- 172.21.15.99:0/3395614273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540c6cd0 con 0x7f545406acd0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.053+0000 7f54537fe640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 0x7f5454070080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5444034040 tx=0x7f54440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.054+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444032630 con 0x7f545406acd0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.054+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54440327f0 con 0x7f545406acd0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.054+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444032be0 con 0x7f545406acd0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.054+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54540c4490 con 0x7f545406acd0 2024-03-04T10:04:03.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.054+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54540b28e0 con 0x7f545406acd0 2024-03-04T10:04:03.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.055+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545406cd70 con 0x7f545406acd0 2024-03-04T10:04:03.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.055+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f544404e430 con 0x7f545406acd0 2024-03-04T10:04:03.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.056+0000 7f543bfff640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544006b500 0x7f544006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.056+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f544402f030 con 0x7f545406acd0 2024-03-04T10:04:03.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.056+0000 7f5452ffd640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544006b500 0x7f544006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.056+0000 7f5452ffd640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544006b500 0x7f544006d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54541f6b50 tx=0x7f5448009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.060+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5444040070 con 0x7f545406acd0 2024-03-04T10:04:03.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.274+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5454074f60 con 0x7f545406acd0 2024-03-04T10:04:03.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.277+0000 7f543bfff640 1 -- 172.21.15.99:0/3395614273 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171789 (secure 0 0 0) 0x7f544407a8b0 con 0x7f545406acd0 2024-03-04T10:04:03.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544006b500 msgr2=0x7f544006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544006b500 0x7f544006d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54541f6b50 tx=0x7f5448009040 comp rx=0 tx=0).stop 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 msgr2=0x7f5454070080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 0x7f5454070080 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5444034040 tx=0x7f54440046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 shutdown_connections 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f544006b500 0x7f544006d9c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54541597f0 0x7f54540b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545406b860 0x7f54540c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 --2- 172.21.15.99:0/3395614273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f545406acd0 0x7f5454070080 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 >> 172.21.15.99:0/3395614273 conn(0x7f54540d0710 msgr2=0x7f545415bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 shutdown_connections 2024-03-04T10:04:03.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.280+0000 7f5459e61640 1 -- 172.21.15.99:0/3395614273 wait complete. 2024-03-04T10:04:03.296 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":1452235113},{"type":"v1","addr":"172.21.15.162:6833","nonce":1452235113}]},"active_addr":"172.21.15.162:6833/1452235113","active_change":"2024-03-04T10:03:19.424637+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":3770505773}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":1352926362}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":2591514423}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.162:0","nonce":640241254}]}]} 2024-03-04T10:04:03.299 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-04T10:04:03.299 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-04T10:04:03.299 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:03.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.397+0000 7efdeff74640 1 Processor -- start 2024-03-04T10:04:03.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.397+0000 7efdeff74640 1 -- start start 2024-03-04T10:04:03.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdeff74640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7efde80b2e60 0x7efde80b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdeff74640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efde80b5470 0x7efde80b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdeff74640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde815e1e0 con 0x7efde815de00 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdeff74640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde80b98c0 con 0x7efde80b5470 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdeff74640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efde80b32c0 con 0x7efde80b2e60 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efde80b5470 0x7efde80b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efde80b5470 0x7efde80b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45954/0 (socket says 172.21.15.99:45954) 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 -- 172.21.15.99:0/2462546882 learned_addr learned my addr 172.21.15.99:0/2462546882 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efded50f640 1 --2- 172.21.15.99:0/2462546882 >> v2:172.21.15.162:3300/0 conn(0x7efde80b2e60 0x7efde80b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 -- 172.21.15.99:0/2462546882 >> v2:172.21.15.162:3300/0 conn(0x7efde80b2e60 msgr2=0x7efde80b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 --2- 172.21.15.99:0/2462546882 >> v2:172.21.15.162:3300/0 conn(0x7efde80b2e60 0x7efde80b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 -- 172.21.15.99:0/2462546882 >> v1:172.21.15.162:6789/0 conn(0x7efde815de00 legacy=0x7efde80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.398+0000 7efdee511640 1 -- 172.21.15.99:0/2462546882 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde80d1c10 con 0x7efde80b5470 2024-03-04T10:04:03.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.399+0000 7efdee511640 1 --2- 172.21.15.99:0/2462546882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde80b5470 0x7efde80b5900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efde8161830 tx=0x7efde002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7a7f742aa443b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.399+0000 7efdecd0e640 1 -- 172.21.15.99:0/2462546882 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0034070 con 0x7efde80b5470 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.399+0000 7efdecd0e640 1 -- 172.21.15.99:0/2462546882 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0030860 con 0x7efde80b5470 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.399+0000 7efdecd0e640 1 -- 172.21.15.99:0/2462546882 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0030c30 con 0x7efde80b5470 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 -- 172.21.15.99:0/2462546882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde80b5470 msgr2=0x7efde80b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 --2- 172.21.15.99:0/2462546882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde80b5470 0x7efde80b5900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efde8161830 tx=0x7efde002b9a0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 -- 172.21.15.99:0/2462546882 shutdown_connections 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 --2- 172.21.15.99:0/2462546882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde80b5470 0x7efde80b5900 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 --2- 172.21.15.99:0/2462546882 >> v2:172.21.15.162:3300/0 conn(0x7efde80b2e60 0x7efde80b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 -- 172.21.15.99:0/2462546882 >> 172.21.15.99:0/2462546882 conn(0x7efde80d0470 msgr2=0x7efde80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 -- 172.21.15.99:0/2462546882 shutdown_connections 2024-03-04T10:04:03.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.400+0000 7efdeff74640 1 -- 172.21.15.99:0/2462546882 wait complete. 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.401+0000 7efdeff74640 1 Processor -- start 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.401+0000 7efdeff74640 1 -- start start 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdeff74640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde80b2e60 0x7efde80c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdeff74640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde80b5470 0x7efde80b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdeff74640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 0x7efde80b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdeff74640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde80d21d0 con 0x7efde815de00 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdeff74640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde80d2820 con 0x7efde80b2e60 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdeff74640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde80d24b0 con 0x7efde80b5470 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 0x7efde80b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 0x7efde80b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45960/0 (socket says 172.21.15.99:45960) 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 -- 172.21.15.99:0/1321797580 learned_addr learned my addr 172.21.15.99:0/1321797580 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdedd10640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde80b2e60 0x7efde80c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 -- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde80b5470 msgr2=0x7efde80b7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efded50f640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde80b5470 0x7efde80b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde80b5470 0x7efde80b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 -- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde80b2e60 msgr2=0x7efde80c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde80b2e60 0x7efde80c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 -- 172.21.15.99:0/1321797580 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde80c9db0 con 0x7efde815de00 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdedd10640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde80b2e60 0x7efde80c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdee511640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 0x7efde80b7750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efde002b9e0 tx=0x7efde0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.402+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0048070 con 0x7efde815de00 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.403+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0030660 con 0x7efde815de00 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.403+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde80ca050 con 0x7efde815de00 2024-03-04T10:04:03.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.403+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0034040 con 0x7efde815de00 2024-03-04T10:04:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.403+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde815f460 con 0x7efde815de00 2024-03-04T10:04:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.404+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde80ca220 con 0x7efde815de00 2024-03-04T10:04:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.404+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7efde004b070 con 0x7efde815de00 2024-03-04T10:04:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.404+0000 7efdde7fc640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efdcc06b4d0 0x7efdcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.405+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efde00ac6f0 con 0x7efde815de00 2024-03-04T10:04:03.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.405+0000 7efdedd10640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efdcc06b4d0 0x7efdcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.405+0000 7efdedd10640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efdcc06b4d0 0x7efdcc06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efde4001cd0 tx=0x7efde4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.409+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efde007b2d0 con 0x7efde815de00 2024-03-04T10:04:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.583+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efde8074f60 con 0x7efde815de00 2024-03-04T10:04:03.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.584+0000 7efdde7fc640 1 -- 172.21.15.99:0/1321797580 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7efde0030830 con 0x7efde815de00 2024-03-04T10:04:03.587 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:03.587 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efdcc06b4d0 msgr2=0x7efdcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efdcc06b4d0 0x7efdcc06d990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efde4001cd0 tx=0x7efde4009040 comp rx=0 tx=0).stop 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 msgr2=0x7efde80b7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 0x7efde80b7750 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efde002b9e0 tx=0x7efde0004730 comp rx=0 tx=0).stop 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 shutdown_connections 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efdcc06b4d0 0x7efdcc06d990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde815de00 0x7efde80b7750 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde80b5470 0x7efde80b7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 --2- 172.21.15.99:0/1321797580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde80b2e60 0x7efde80c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 >> 172.21.15.99:0/1321797580 conn(0x7efde80d0470 msgr2=0x7efde806c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 shutdown_connections 2024-03-04T10:04:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.586+0000 7efdeff74640 1 -- 172.21.15.99:0/1321797580 wait complete. 2024-03-04T10:04:03.599 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-04T10:04:03.600 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:03.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.699+0000 7f2104cc0640 1 Processor -- start 2024-03-04T10:04:03.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.699+0000 7f2104cc0640 1 -- start start 2024-03-04T10:04:03.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f2104cc0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f210015de00 0x7f21000b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f2104cc0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21000b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f2104cc0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210015e1e0 con 0x7f21000b9380 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f2104cc0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21000d2820 con 0x7f210015de00 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f2104cc0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000d1c10 con 0x7f21000b5470 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f20fe59c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f210015de00 0x7f21000b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f20fed9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21000b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f20fdd9b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f21000b9380 0x7f21000b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43536/0 (socket says 172.21.15.99:43536) 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.700+0000 7f20fdd9b640 1 -- 172.21.15.99:0/4195388703 learned_addr learned my addr 172.21.15.99:0/4195388703 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.701+0000 7f20fed9d640 1 -- 172.21.15.99:0/4195388703 >> v2:172.21.15.162:3300/0 conn(0x7f210015de00 msgr2=0x7f21000b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.701+0000 7f20fed9d640 1 --2- 172.21.15.99:0/4195388703 >> v2:172.21.15.162:3300/0 conn(0x7f210015de00 0x7f21000b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.701+0000 7f20fed9d640 1 -- 172.21.15.99:0/4195388703 >> v1:172.21.15.162:6789/0 conn(0x7f21000b9380 legacy=0x7f21000b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.701+0000 7f20fed9d640 1 -- 172.21.15.99:0/4195388703 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21000d1ef0 con 0x7f21000b5470 2024-03-04T10:04:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.701+0000 7f20fe59c640 1 --2- 172.21.15.99:0/4195388703 >> v2:172.21.15.162:3300/0 conn(0x7f210015de00 0x7f21000b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:03.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f20fed9d640 1 --2- 172.21.15.99:0/4195388703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21000b5900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20f80097a0 tx=0x7f20f802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac281655c706f2fe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f20fd59a640 1 -- 172.21.15.99:0/4195388703 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8035020 con 0x7f21000b5470 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f20fd59a640 1 -- 172.21.15.99:0/4195388703 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f8031830 con 0x7f21000b5470 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f20fd59a640 1 -- 172.21.15.99:0/4195388703 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8031c00 con 0x7f21000b5470 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 -- 172.21.15.99:0/4195388703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 msgr2=0x7f21000b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 --2- 172.21.15.99:0/4195388703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21000b5900 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20f80097a0 tx=0x7f20f802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 -- 172.21.15.99:0/4195388703 shutdown_connections 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 --2- 172.21.15.99:0/4195388703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21000b5900 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 --2- 172.21.15.99:0/4195388703 >> v2:172.21.15.162:3300/0 conn(0x7f210015de00 0x7f21000b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 -- 172.21.15.99:0/4195388703 >> 172.21.15.99:0/4195388703 conn(0x7f21000d0470 msgr2=0x7f21000d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 -- 172.21.15.99:0/4195388703 shutdown_connections 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.702+0000 7f2104cc0640 1 -- 172.21.15.99:0/4195388703 wait complete. 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 Processor -- start 2024-03-04T10:04:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 -- start start 2024-03-04T10:04:03.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21001f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21000b9380 0x7f210015f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 0x7f210014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b97e0 con 0x7f210015de00 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000d24b0 con 0x7f21000b9380 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.704+0000 7f2104cc0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21000b72a0 con 0x7f21000b5470 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 0x7f210014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 0x7f210014de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37936/0 (socket says 172.21.15.99:37936) 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 -- 172.21.15.99:0/804190349 learned_addr learned my addr 172.21.15.99:0/804190349 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fdd9b640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21000b9380 0x7f210015f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fe59c640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21001f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 -- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 msgr2=0x7f21001f6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21001f6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 -- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21000b9380 msgr2=0x7f210015f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21000b9380 0x7f210015f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 -- 172.21.15.99:0/804190349 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210015e470 con 0x7f210015de00 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fe59c640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21001f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fdd9b640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21000b9380 0x7f210015f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20fed9d640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 0x7f210014de30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f20f80098c0 tx=0x7f20f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8035040 con 0x7f210015de00 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f803b840 con 0x7f210015de00 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f803bc30 con 0x7f210015de00 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21001f7440 con 0x7f210015de00 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.705+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21001f7a00 con 0x7f210015de00 2024-03-04T10:04:03.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.706+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2100074f60 con 0x7f210015de00 2024-03-04T10:04:03.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.707+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f20f8038020 con 0x7f210015de00 2024-03-04T10:04:03.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.707+0000 7f20eeffd640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20e006b4d0 0x7f20e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:03.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.707+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f20f80ac560 con 0x7f210015de00 2024-03-04T10:04:03.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.707+0000 7f20fe59c640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20e006b4d0 0x7f20e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:03.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.708+0000 7f20fe59c640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20e006b4d0 0x7f20e006d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f21000d36b0 tx=0x7f20f000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:03.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.711+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f20f8031800 con 0x7f210015de00 2024-03-04T10:04:03.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.884+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21000c8d40 con 0x7f210015de00 2024-03-04T10:04:03.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.885+0000 7f20eeffd640 1 -- 172.21.15.99:0/804190349 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f20f807b130 con 0x7f210015de00 2024-03-04T10:04:03.887 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:03.887 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20e006b4d0 msgr2=0x7f20e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20e006b4d0 0x7f20e006d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f21000d36b0 tx=0x7f20f000b040 comp rx=0 tx=0).stop 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 msgr2=0x7f210014de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 0x7f210014de30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f20f80098c0 tx=0x7f20f80046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 shutdown_connections 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20e006b4d0 0x7f20e006d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f210015de00 0x7f210014de30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21000b9380 0x7f210015f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 --2- 172.21.15.99:0/804190349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21000b5470 0x7f21001f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 >> 172.21.15.99:0/804190349 conn(0x7f21000d0470 msgr2=0x7f21000d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 shutdown_connections 2024-03-04T10:04:03.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:03.887+0000 7f2104cc0640 1 -- 172.21.15.99:0/804190349 wait complete. 2024-03-04T10:04:03.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-04T10:04:03.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-04T10:04:03.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-04T10:04:03.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-04T10:04:03.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-04T10:04:03.903 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-04T10:04:03.903 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-04T10:04:03.903 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-04T10:04:04.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7fbdcee44640 1 Processor -- start 2024-03-04T10:04:04.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7fbdcee44640 1 -- start start 2024-03-04T10:04:04.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7f5907481640 1 Processor -- start 2024-03-04T10:04:04.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7f5907481640 1 -- start start 2024-03-04T10:04:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.071+0000 7f6a88ae9640 1 Processor -- start 2024-03-04T10:04:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7f6a88ae9640 1 -- start start 2024-03-04T10:04:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7fcf7c744640 1 Processor -- start 2024-03-04T10:04:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7fcf7c744640 1 -- start start 2024-03-04T10:04:04.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f34098d1640 1 Processor -- start 2024-03-04T10:04:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7f2cdfa34640 1 Processor -- start 2024-03-04T10:04:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.072+0000 7f2cdfa34640 1 -- start start 2024-03-04T10:04:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f34098d1640 1 -- start start 2024-03-04T10:04:04.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdcee44640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbdc80cebd0 0x7fbdc80cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f6a88ae9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6a8415a700 0x7f6a8415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdcee44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cd2d0 0x7fbdc80cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdcee44640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc806f9f0 con 0x7fbdc815aa90 2024-03-04T10:04:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdcee44640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc80cf570 con 0x7fbdc80cebd0 2024-03-04T10:04:04.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdcee44640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8070600 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdc7fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbdc80cebd0 0x7fbdc80cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdcd3e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cd2d0 0x7fbdc80cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f2cdfa34640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2cd815c640 0x7f2cd806e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fcf7c744640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcf7415aa90 0x7fcf7406ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdccbe0640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fbdc815aa90 0x7fbdc806ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43548/0 (socket says 172.21.15.99:43548) 2024-03-04T10:04:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7fbdccbe0640 1 -- 172.21.15.99:0/1824243542 learned_addr learned my addr 172.21.15.99:0/1824243542 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2cd815c640 0x7f2cd806e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf7c744640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcf740cf2e0 0x7fcf740cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a8259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6a8415a700 0x7f6a8415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2cd815c640 0x7f2cd806e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37956/0 (socket says 172.21.15.99:37956) 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 -- 172.21.15.99:0/2141810401 learned_addr learned my addr 172.21.15.99:0/2141810401 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/2141810401 >> v2:172.21.15.162:3300/0 conn(0x7f2cd806e980 0x7f2cd8071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd8071890 0x7f2cd80cf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd815ca20 con 0x7f2cd815c640 2024-03-04T10:04:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd80d04f0 con 0x7f2cd806e980 2024-03-04T10:04:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81628c0 con 0x7f2cd8071890 2024-03-04T10:04:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fbdc77fe640 1 -- 172.21.15.99:0/1824243542 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690073038 0 0) 0x7fbdc806f9f0 con 0x7fbdc815aa90 2024-03-04T10:04:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fbdc77fe640 1 -- 172.21.15.99:0/1824243542 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb8003520 con 0x7fbdc815aa90 2024-03-04T10:04:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a8259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6a8415a700 0x7f6a8415aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37940/0 (socket says 172.21.15.99:37940) 2024-03-04T10:04:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fe34101e640 1 Processor -- start 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a8259c640 1 -- 172.21.15.99:0/2280431166 learned_addr learned my addr 172.21.15.99:0/2280431166 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a88ae9640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a8406e1c0 0x7f6a84071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf7c744640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7406f9f0 con 0x7fcf7406f470 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf7c744640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf74070600 con 0x7fcf740cf2e0 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf7c744640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7406fcd0 con 0x7fcf7415aa90 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf7a4e0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcf7415aa90 0x7fcf7406ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf79cdf640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fcf7406f470 0x7fcf740cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43570/0 (socket says 172.21.15.99:43570) 2024-03-04T10:04:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf79cdf640 1 -- 172.21.15.99:0/1820208498 learned_addr learned my addr 172.21.15.99:0/1820208498 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f3913112640 1 Processor -- start 2024-03-04T10:04:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fe34101e640 1 -- start start 2024-03-04T10:04:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f5907481640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590015cc80 0x7f590006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f5907481640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f59000727f0 0x7f590006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f5907481640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5900071a30 con 0x7f590015ba50 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f5907481640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590015be30 con 0x7f59000727f0 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.073+0000 7f5907481640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59000717e0 con 0x7f590015cc80 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f590521d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f590015ba50 0x7f590006d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43564/0 (socket says 172.21.15.99:43564) 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f590521d640 1 -- 172.21.15.99:0/3902988330 learned_addr learned my addr 172.21.15.99:0/3902988330 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f5904a1c640 1 --2- 172.21.15.99:0/3902988330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590015cc80 0x7f590006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdcfcf640 1 --2- 172.21.15.99:0/2141810401 >> v2:172.21.15.162:3300/0 conn(0x7f2cd806e980 0x7f2cd8071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f5905a1e640 1 --2- 172.21.15.99:0/3902988330 >> v2:172.21.15.162:3300/0 conn(0x7f59000727f0 0x7f590006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cddfd1640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd8071890 0x7f2cd80cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f3913112640 1 -- start start 2024-03-04T10:04:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8406f240 con 0x7f6a8415b510 2024-03-04T10:04:04.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a840721d0 con 0x7f6a8415a700 2024-03-04T10:04:04.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f58f7fff640 1 -- 172.21.15.99:0/3902988330 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102237576 0 0) 0x7f5900071a30 con 0x7f590015ba50 2024-03-04T10:04:04.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f58f7fff640 1 -- 172.21.15.99:0/3902988330 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58ec003520 con 0x7f590015ba50 2024-03-04T10:04:04.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a81d9b640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a8406e1c0 0x7f6a84071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8406e6c0 con 0x7f6a8406e1c0 2024-03-04T10:04:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 -- 172.21.15.99:0/2141810401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd8071890 msgr2=0x7f2cd80cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fbdcd3e1640 1 -- 172.21.15.99:0/1824243542 >> v2:172.21.15.162:3300/0 conn(0x7fbdc80cebd0 msgr2=0x7fbdc80cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fbdcd3e1640 1 --2- 172.21.15.99:0/1824243542 >> v2:172.21.15.162:3300/0 conn(0x7fbdc80cebd0 0x7fbdc80cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fbdcd3e1640 1 -- 172.21.15.99:0/1824243542 >> v1:172.21.15.162:6789/0 conn(0x7fbdc815aa90 legacy=0x7fbdc806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fbdcd3e1640 1 -- 172.21.15.99:0/1824243542 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc806fcd0 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f34098d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cf570 0x7f34040cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f34098d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f340415bc40 0x7f34040cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f34098d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340415cf30 con 0x7f34040cfa30 2024-03-04T10:04:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f34098d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f340415caa0 con 0x7f340415bc40 2024-03-04T10:04:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f34098d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340415c170 con 0x7f34040cf570 2024-03-04T10:04:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fbdcd3e1640 1 --2- 172.21.15.99:0/1824243542 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cd2d0 0x7fbdc80cd760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009d90 tx=0x7fbdc002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fcfba6dde748cdb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fe34101e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3340df210 0x7fe33403a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7fcf7ace1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcf740cf2e0 0x7fcf740cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fcf7ace1640 1 -- 172.21.15.99:0/1820208498 >> v2:172.21.15.162:3300/0 conn(0x7fcf7415aa90 msgr2=0x7fcf7406ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fcf7ace1640 1 --2- 172.21.15.99:0/1820208498 >> v2:172.21.15.162:3300/0 conn(0x7fcf7415aa90 0x7fcf7406ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fcf7ace1640 1 -- 172.21.15.99:0/1820208498 >> v1:172.21.15.162:6789/0 conn(0x7fcf7406f470 legacy=0x7fcf740cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fcf7ace1640 1 -- 172.21.15.99:0/1820208498 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7406ffb0 con 0x7fcf740cf2e0 2024-03-04T10:04:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7fbdc77fe640 1 -- 172.21.15.99:0/1824243542 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0039070 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fbdc77fe640 1 -- 172.21.15.99:0/1824243542 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc002cbd0 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34101e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 0x7fe334053ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3913112640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f390404b3c0 0x7f3904034020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3913112640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f390404f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3913112640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f390404fe90 0x7f39040eb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3913112640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39040062e0 con 0x7f390404fe90 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34101e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe334054630 con 0x7fe3340e2d70 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34101e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe334054910 con 0x7fe3340df210 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34101e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe334054bf0 con 0x7fe3340e31d0 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f5904a1c640 1 -- 172.21.15.99:0/3902988330 >> v2:172.21.15.162:3300/0 conn(0x7f59000727f0 msgr2=0x7f590006e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f5904a1c640 1 --2- 172.21.15.99:0/3902988330 >> v2:172.21.15.162:3300/0 conn(0x7f59000727f0 0x7f590006e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f5904a1c640 1 -- 172.21.15.99:0/3902988330 >> v1:172.21.15.162:6789/0 conn(0x7f590015ba50 legacy=0x7f590006d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f5904a1c640 1 -- 172.21.15.99:0/3902988330 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900071d10 con 0x7f590015cc80 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3913112640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3904054c10 con 0x7f390404b3c0 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3913112640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3904053a50 con 0x7f390404f4d0 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f390404b3c0 0x7f3904034020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3402ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f34040cfa30 0x7f34040cfe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43600/0 (socket says 172.21.15.99:43600) 2024-03-04T10:04:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3402ffd640 1 -- 172.21.15.99:0/2163491761 learned_addr learned my addr 172.21.15.99:0/2163491761 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345472761 0 0) 0x7f6a8406f240 con 0x7f6a8415b510 2024-03-04T10:04:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6c003520 con 0x7f6a8415b510 2024-03-04T10:04:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 <== mon.1 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 608446844 0 0) 0x7f6a6c003520 con 0x7f6a8415b510 2024-03-04T10:04:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a8406e1c0 msgr2=0x7f6a84071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a8406e1c0 0x7f6a84071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 >> v2:172.21.15.99:3300/0 conn(0x7f6a8415a700 msgr2=0x7f6a8415aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f33fbfff640 1 --2- 172.21.15.99:0/2163491761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cf570 0x7f34040cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 --2- 172.21.15.99:0/2280431166 >> v2:172.21.15.99:3300/0 conn(0x7f6a8415a700 0x7f6a8415aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8406e9a0 con 0x7f6a8415b510 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 <== mon.1 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6a7c0019f0 con 0x7f6a8415b510 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 >> v1:172.21.15.162:6789/0 conn(0x7f6a8415b510 legacy=0x7f6a8415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c004b90 0x7f6a6c0037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a6c003dc0 0x7f6a6c004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fbdc77fe640 1 -- 172.21.15.99:0/1824243542 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0034440 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.075+0000 7f6a8159a640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a6c009330 0x7f6a6c0097b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8406e6c0 con 0x7f6a6c003dc0 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c00a060 con 0x7f6a6c009330 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c00a260 con 0x7f6a6c004b90 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a81d9b640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a8406e1c0 0x7f6a84071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a6c009330 0x7f6a6c0097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a8259c640 1 --2- 172.21.15.99:0/2280431166 >> v2:172.21.15.99:3300/0 conn(0x7f6a8415a700 0x7f6a8415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd8071890 0x7f2cd80cf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 -- 172.21.15.99:0/2141810401 >> v2:172.21.15.162:3300/0 conn(0x7f2cd806e980 msgr2=0x7f2cd8071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 --2- 172.21.15.99:0/2141810401 >> v2:172.21.15.162:3300/0 conn(0x7f2cd806e980 0x7f2cd8071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.074+0000 7f2cdd7d0640 1 -- 172.21.15.99:0/2141810401 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80cff30 con 0x7f2cd815c640 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a8259c640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c004b90 0x7f6a6c0037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c004b90 msgr2=0x7f6a6c0037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f2cdd7d0640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd815c640 0x7f2cd806e440 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80d0810 tx=0x7f2cd002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4456465966f12d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f5904a1c640 1 --2- 172.21.15.99:0/3902988330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590015cc80 0x7f590006e2c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009d90 tx=0x7f58f802bc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f49399e03d9269 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c004b90 0x7f6a6c0037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2280431166 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a6c003dc0 msgr2=0x7f6a6c004240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a6c003dc0 0x7f6a6c004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2280431166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c003520 con 0x7f6a6c009330 2024-03-04T10:04:04.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f6a8259c640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c004b90 0x7f6a6c0037a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:04.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f390404b3c0 0x7f3904034020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39302/0 (socket says 172.21.15.99:39302) 2024-03-04T10:04:04.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 -- 172.21.15.99:0/2933175781 learned_addr learned my addr 172.21.15.99:0/2933175781 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f391190f640 1 --2- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f390404f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34081d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 0x7fe334053ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34081d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 0x7fe334053ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:42930/0 (socket says 172.21.15.99:42930) 2024-03-04T10:04:04.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7fe34081d640 1 -- 172.21.15.99:0/2919498932 learned_addr learned my addr 172.21.15.99:0/2919498932 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f2ccffff640 1 -- 172.21.15.99:0/2141810401 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd002ca70 con 0x7f2cd815c640 2024-03-04T10:04:04.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f34027fc640 1 -- 172.21.15.99:0/2163491761 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277261747 0 0) 0x7f340415cf30 con 0x7f34040cfa30 2024-03-04T10:04:04.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fe33bfff640 1 --2- 172.21.15.99:0/2919498932 >> v2:172.21.15.99:3300/0 conn(0x7fe3340df210 0x7fe33403a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f34027fc640 1 -- 172.21.15.99:0/2163491761 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33f0003520 con 0x7f34040cfa30 2024-03-04T10:04:04.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f2ccffff640 1 -- 172.21.15.99:0/2141810401 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd002cc10 con 0x7f2cd815c640 2024-03-04T10:04:04.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 -- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 msgr2=0x7f390404f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 --2- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f390404f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 -- 172.21.15.99:0/2933175781 >> v2:172.21.15.99:3300/0 conn(0x7f390404fe90 msgr2=0x7f39040eb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 --2- 172.21.15.99:0/2933175781 >> v2:172.21.15.99:3300/0 conn(0x7f390404fe90 0x7f39040eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f3912110640 1 -- 172.21.15.99:0/2933175781 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904053d30 con 0x7f390404b3c0 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f3912110640 1 --2- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404b3c0 0x7f3904034020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f390c050300 tx=0x7f390c069fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4de6fb6037682041 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 -- 172.21.15.99:0/1824243542 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cd2d0 msgr2=0x7fbdc80cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 --2- 172.21.15.99:0/1824243542 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cd2d0 0x7fbdc80cd760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0009d90 tx=0x7fbdc002be20 comp rx=0 tx=0).stop 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 -- 172.21.15.99:0/1824243542 shutdown_connections 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 --2- 172.21.15.99:0/1824243542 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cd2d0 0x7fbdc80cd760 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 --2- 172.21.15.99:0/1824243542 >> v2:172.21.15.162:3300/0 conn(0x7fbdc80cebd0 0x7fbdc80cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 -- 172.21.15.99:0/1824243542 >> 172.21.15.99:0/1824243542 conn(0x7fbdc80d0780 msgr2=0x7fbdc80d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 -- 172.21.15.99:0/1824243542 shutdown_connections 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fbdcee44640 1 -- 172.21.15.99:0/1824243542 wait complete. 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fe34081d640 1 -- 172.21.15.99:0/2919498932 >> v2:172.21.15.99:3300/0 conn(0x7fe3340df210 msgr2=0x7fe33403a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fe34081d640 1 --2- 172.21.15.99:0/2919498932 >> v2:172.21.15.99:3300/0 conn(0x7fe3340df210 0x7fe33403a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fe34081d640 1 -- 172.21.15.99:0/2919498932 >> v1:172.21.15.99:6789/0 conn(0x7fe3340e2d70 legacy=0x7fe3340533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f58f7fff640 1 -- 172.21.15.99:0/3902988330 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8039070 con 0x7f590015cc80 2024-03-04T10:04:04.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f58f7fff640 1 -- 172.21.15.99:0/3902988330 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f802c740 con 0x7f590015cc80 2024-03-04T10:04:04.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f58f7fff640 1 -- 172.21.15.99:0/3902988330 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc001670 con 0x7f590015cc80 2024-03-04T10:04:04.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.076+0000 7f2ccffff640 1 -- 172.21.15.99:0/2141810401 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00343f0 con 0x7f2cd815c640 2024-03-04T10:04:04.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fcf7ace1640 1 --2- 172.21.15.99:0/1820208498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf740cd2d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf7404f710 tx=0x7fcf700302f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1883ae48783fd541 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fcf794de640 1 -- 172.21.15.99:0/1820208498 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7003d070 con 0x7fcf740cf2e0 2024-03-04T10:04:04.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fcf794de640 1 -- 172.21.15.99:0/1820208498 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70038640 con 0x7fcf740cf2e0 2024-03-04T10:04:04.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fcf794de640 1 -- 172.21.15.99:0/1820208498 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7003d070 con 0x7fcf740cf2e0 2024-03-04T10:04:04.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7fe34081d640 1 -- 172.21.15.99:0/2919498932 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3340062d0 con 0x7fe3340e31d0 2024-03-04T10:04:04.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd815c640 msgr2=0x7f2cd806e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd815c640 0x7f2cd806e440 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2cd80d0810 tx=0x7f2cd002bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f391110e640 1 -- 172.21.15.99:0/2933175781 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c06ce60 con 0x7f390404b3c0 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f34037fe640 1 --2- 172.21.15.99:0/2163491761 >> v2:172.21.15.99:3300/0 conn(0x7f340415bc40 0x7f34040cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f34037fe640 1 -- 172.21.15.99:0/2163491761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cf570 msgr2=0x7f34040cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f34037fe640 1 --2- 172.21.15.99:0/2163491761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cf570 0x7f34040cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f34037fe640 1 -- 172.21.15.99:0/2163491761 >> v1:172.21.15.162:6789/0 conn(0x7f34040cfa30 legacy=0x7f34040cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.077+0000 7f34037fe640 1 -- 172.21.15.99:0/2163491761 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340415c450 con 0x7f340415bc40 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f391110e640 1 -- 172.21.15.99:0/2933175781 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c0ac640 con 0x7f390404b3c0 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f391110e640 1 -- 172.21.15.99:0/2933175781 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c0acab0 con 0x7f390404b3c0 2024-03-04T10:04:04.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fe33bfff640 1 --2- 172.21.15.99:0/2919498932 >> v2:172.21.15.99:3300/0 conn(0x7fe3340df210 0x7fe33403a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:04.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a6c009330 0x7f6a6c0097b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c4c0 tx=0x7f6a6c004820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19bc5dff548d4668 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fe34081d640 1 --2- 172.21.15.99:0/2919498932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 0x7fe334053ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe33c064820 tx=0x7fe33c068aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44d06ada0aedbe33 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 shutdown_connections 2024-03-04T10:04:04.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd8071890 0x7f2cd80cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/2141810401 >> v2:172.21.15.162:3300/0 conn(0x7f2cd806e980 0x7f2cd8071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/2141810401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd815c640 0x7f2cd806e440 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 >> 172.21.15.99:0/2141810401 conn(0x7f2cd806acd0 msgr2=0x7f2cd806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f5907481640 1 -- 172.21.15.99:0/3902988330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590015cc80 msgr2=0x7f590006e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f5907481640 1 --2- 172.21.15.99:0/3902988330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590015cc80 0x7f590006e2c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f58f8009d90 tx=0x7f58f802bc10 comp rx=0 tx=0).stop 2024-03-04T10:04:04.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fe33affd640 1 -- 172.21.15.99:0/2919498932 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b5070 con 0x7fe3340e31d0 2024-03-04T10:04:04.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 shutdown_connections 2024-03-04T10:04:04.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fcf7c744640 1 -- 172.21.15.99:0/1820208498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 msgr2=0x7fcf740cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f5907481640 1 -- 172.21.15.99:0/3902988330 shutdown_connections 2024-03-04T10:04:04.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f5907481640 1 --2- 172.21.15.99:0/3902988330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590015cc80 0x7f590006e2c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f5907481640 1 --2- 172.21.15.99:0/3902988330 >> v2:172.21.15.162:3300/0 conn(0x7f59000727f0 0x7f590006e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f5907481640 1 -- 172.21.15.99:0/3902988330 >> 172.21.15.99:0/3902988330 conn(0x7f59000cc4d0 msgr2=0x7f59000cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fcf7c744640 1 --2- 172.21.15.99:0/1820208498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf740cd2d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcf7404f710 tx=0x7fcf700302f0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fe33affd640 1 -- 172.21.15.99:0/2919498932 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c0a8a50 con 0x7fe3340e31d0 2024-03-04T10:04:04.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fe33affd640 1 -- 172.21.15.99:0/2919498932 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c06ce60 con 0x7fe3340e31d0 2024-03-04T10:04:04.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f2cdfa34640 1 -- 172.21.15.99:0/2141810401 wait complete. 2024-03-04T10:04:04.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f34037fe640 1 --2- 172.21.15.99:0/2163491761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 0x7f34040cc7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009d90 tx=0x7f33fc02bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f630450e5d4e305b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fcf7c744640 1 -- 172.21.15.99:0/1820208498 shutdown_connections 2024-03-04T10:04:04.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fcf7c744640 1 --2- 172.21.15.99:0/1820208498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf740cd2d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7f34027fc640 1 -- 172.21.15.99:0/2163491761 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc039070 con 0x7f340415bc40 2024-03-04T10:04:04.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fcf7c744640 1 --2- 172.21.15.99:0/1820208498 >> v2:172.21.15.162:3300/0 conn(0x7fcf7415aa90 0x7fcf7406ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.078+0000 7fcf7c744640 1 -- 172.21.15.99:0/1820208498 >> 172.21.15.99:0/1820208498 conn(0x7fcf740d0780 msgr2=0x7fcf740d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f5907481640 1 -- 172.21.15.99:0/3902988330 shutdown_connections 2024-03-04T10:04:04.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f5907481640 1 -- 172.21.15.99:0/3902988330 wait complete. 2024-03-04T10:04:04.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c039070 con 0x7f6a6c009330 2024-03-04T10:04:04.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c02cd20 con 0x7f6a6c009330 2024-03-04T10:04:04.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f6a8159a640 1 -- 172.21.15.99:0/2280431166 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c00c0e0 con 0x7f6a6c009330 2024-03-04T10:04:04.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f3913112640 1 -- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404b3c0 msgr2=0x7f3904034020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f3913112640 1 --2- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404b3c0 0x7f3904034020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f390c050300 tx=0x7f390c069fc0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fcf7c744640 1 -- 172.21.15.99:0/1820208498 shutdown_connections 2024-03-04T10:04:04.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fcf7c744640 1 -- 172.21.15.99:0/1820208498 wait complete. 2024-03-04T10:04:04.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f34027fc640 1 -- 172.21.15.99:0/2163491761 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc02c7c0 con 0x7f340415bc40 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fbdcee44640 1 Processor -- start 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fbdcee44640 1 -- start start 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fe34101e640 1 -- 172.21.15.99:0/2919498932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 msgr2=0x7fe334053ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fe34101e640 1 --2- 172.21.15.99:0/2919498932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 0x7fe334053ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe33c064820 tx=0x7fe33c068aa0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fe34101e640 1 -- 172.21.15.99:0/2919498932 shutdown_connections 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fe34101e640 1 --2- 172.21.15.99:0/2919498932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340e31d0 0x7fe334053ae0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7fe34101e640 1 --2- 172.21.15.99:0/2919498932 >> v2:172.21.15.99:3300/0 conn(0x7fe3340df210 0x7fe33403a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7fe34101e640 1 -- 172.21.15.99:0/2919498932 >> 172.21.15.99:0/2919498932 conn(0x7fe33400a3f0 msgr2=0x7fe33400a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.079+0000 7f34027fc640 1 -- 172.21.15.99:0/2163491761 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4001670 con 0x7f340415bc40 2024-03-04T10:04:04.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a6c009330 msgr2=0x7f6a6c0097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a88ae9640 1 --2- 172.21.15.99:0/2280431166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a6c009330 0x7f6a6c0097b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c4c0 tx=0x7f6a6c004820 comp rx=0 tx=0).stop 2024-03-04T10:04:04.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2280431166 reap_dead start 2024-03-04T10:04:04.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f3913112640 1 -- 172.21.15.99:0/2933175781 shutdown_connections 2024-03-04T10:04:04.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 shutdown_connections 2024-03-04T10:04:04.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 >> 172.21.15.99:0/2280431166 conn(0x7f6a8406acd0 msgr2=0x7f6a8406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f3913112640 1 --2- 172.21.15.99:0/2933175781 >> v2:172.21.15.99:3300/0 conn(0x7f390404fe90 0x7f39040eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f3913112640 1 --2- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f390404f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f3913112640 1 --2- 172.21.15.99:0/2933175781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404b3c0 0x7f3904034020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f3913112640 1 -- 172.21.15.99:0/2933175781 >> 172.21.15.99:0/2933175781 conn(0x7f390400a400 msgr2=0x7f390400a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 shutdown_connections 2024-03-04T10:04:04.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2280431166 wait complete. 2024-03-04T10:04:04.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f2cdfa34640 1 Processor -- start 2024-03-04T10:04:04.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7f3913112640 1 -- 172.21.15.99:0/2933175781 shutdown_connections 2024-03-04T10:04:04.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7f3913112640 1 -- 172.21.15.99:0/2933175781 wait complete. 2024-03-04T10:04:04.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7fe34101e640 1 -- 172.21.15.99:0/2919498932 shutdown_connections 2024-03-04T10:04:04.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7fe34101e640 1 -- 172.21.15.99:0/2919498932 wait complete. 2024-03-04T10:04:04.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f5907481640 1 Processor -- start 2024-03-04T10:04:04.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7f5907481640 1 -- start start 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f34098d1640 1 -- 172.21.15.99:0/2163491761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 msgr2=0x7f34040cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f34098d1640 1 --2- 172.21.15.99:0/2163491761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 0x7f34040cc7a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009d90 tx=0x7f33fc02bcb0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f34098d1640 1 -- 172.21.15.99:0/2163491761 shutdown_connections 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f34098d1640 1 --2- 172.21.15.99:0/2163491761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 0x7f34040cc7a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f34098d1640 1 --2- 172.21.15.99:0/2163491761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cf570 0x7f34040cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f34098d1640 1 -- 172.21.15.99:0/2163491761 >> 172.21.15.99:0/2163491761 conn(0x7f34040d0780 msgr2=0x7f34040d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.080+0000 7f2cdfa34640 1 -- start start 2024-03-04T10:04:04.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fcf7c744640 1 Processor -- start 2024-03-04T10:04:04.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fcf7c744640 1 -- start start 2024-03-04T10:04:04.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fbdcee44640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbdc80cd2d0 0x7fbdc81f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fbdccbe0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbdc80cd2d0 0x7fbdc81f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fbdccbe0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbdc80cd2d0 0x7fbdc81f3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38026/0 (socket says 172.21.15.99:38026) 2024-03-04T10:04:04.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fbdcee44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cebd0 0x7fbdc81f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7fbdccbe0640 1 -- 172.21.15.99:0/4254248434 learned_addr learned my addr 172.21.15.99:0/4254248434 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.081+0000 7f2cdfa34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd806e980 0x7f2cd814b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdfa34640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd8071890 0x7f2cd80c5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fbdcee44640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc815aa90 0x7fbdc80b5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7c744640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf7406f470 0x7fcf7414de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdfa34640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd815c640 0x7f2cd8161640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7c744640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf7414d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7c744640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf7415aa90 0x7fcf74156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7c744640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740afda0 con 0x7fcf740cf2e0 2024-03-04T10:04:04.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7c744640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7415ae70 con 0x7fcf7406f470 2024-03-04T10:04:04.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7c744640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf740cd810 con 0x7fcf7415aa90 2024-03-04T10:04:04.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdfa34640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd806df20 con 0x7f2cd806e980 2024-03-04T10:04:04.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f34098d1640 1 -- 172.21.15.99:0/2163491761 shutdown_connections 2024-03-04T10:04:04.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdcfcf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd8071890 0x7f2cd80c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf7414d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf7414d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38034/0 (socket says 172.21.15.99:38034) 2024-03-04T10:04:04.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdcfcf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd8071890 0x7f2cd80c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39304/0 (socket says 172.21.15.99:39304) 2024-03-04T10:04:04.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdcfcf640 1 -- 172.21.15.99:0/382479266 learned_addr learned my addr 172.21.15.99:0/382479266 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 -- 172.21.15.99:0/653243572 learned_addr learned my addr 172.21.15.99:0/653243572 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f34098d1640 1 -- 172.21.15.99:0/2163491761 wait complete. 2024-03-04T10:04:04.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cddfd1640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd815c640 0x7f2cd8161640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd806dc40 con 0x7f2cd8071890 2024-03-04T10:04:04.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80cf840 con 0x7f2cd815c640 2024-03-04T10:04:04.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fbdc7fff640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cebd0 0x7fbdc81f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7a4e0640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf7406f470 0x7fcf7414de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf7ace1640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf7415aa90 0x7fcf74156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fbdcd3e1640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc815aa90 0x7fbdc80b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 -- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf7415aa90 msgr2=0x7fcf74156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcf7415aa90 0x7fcf74156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 -- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf7406f470 msgr2=0x7fcf7414de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcf7406f470 0x7fcf7414de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fcf79cdf640 1 -- 172.21.15.99:0/653243572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74155c60 con 0x7fcf740cf2e0 2024-03-04T10:04:04.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f6a88ae9640 1 Processor -- start 2024-03-04T10:04:04.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fbdcee44640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc815ae70 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f6a88ae9640 1 -- start start 2024-03-04T10:04:04.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fe34101e640 1 Processor -- start 2024-03-04T10:04:04.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fbdcee44640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8070290 con 0x7fbdc815aa90 2024-03-04T10:04:04.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdcfcf640 1 -- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd815c640 msgr2=0x7f2cd8161640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fcf79cdf640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf7414d2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00cb10 tx=0x7fcf6c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7f2cdcfcf640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd815c640 0x7f2cd8161640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cdcfcf640 1 -- 172.21.15.99:0/382479266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd806e980 msgr2=0x7f2cd814b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cdcfcf640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd806e980 0x7f2cd814b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cdcfcf640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd814dc60 con 0x7f2cd8071890 2024-03-04T10:04:04.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c017070 con 0x7fcf740cf2e0 2024-03-04T10:04:04.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cddfd1640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cd815c640 0x7f2cd8161640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:04.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6c00ac70 con 0x7fcf740cf2e0 2024-03-04T10:04:04.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.082+0000 7fbdcee44640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80afda0 con 0x7fbdc80cebd0 2024-03-04T10:04:04.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fcf7c744640 1 -- 172.21.15.99:0/653243572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf74155f40 con 0x7fcf740cf2e0 2024-03-04T10:04:04.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fe34101e640 1 -- start start 2024-03-04T10:04:04.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cdcfcf640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd8071890 0x7f2cd80c5010 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400e890 tx=0x7f2cd400ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fcf7c744640 1 -- 172.21.15.99:0/653243572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf74156500 con 0x7fcf740cf2e0 2024-03-04T10:04:04.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c00ae60 con 0x7fcf740cf2e0 2024-03-04T10:04:04.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4017070 con 0x7f2cd8071890 2024-03-04T10:04:04.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd400c940 con 0x7f2cd8071890 2024-03-04T10:04:04.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd814df40 con 0x7f2cd8071890 2024-03-04T10:04:04.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fbdccbe0640 1 -- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cebd0 msgr2=0x7fbdc81f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fbdccbe0640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbdc80cebd0 0x7fbdc81f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fbdccbe0640 1 -- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc815aa90 msgr2=0x7fbdc80b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fbdccbe0640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdc815aa90 0x7fbdc80b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7fbdccbe0640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80b6000 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a88ae9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c009330 0x7f6a84155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a88ae9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a84157580 0x7f6a84158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f34098d1640 1 Processor -- start 2024-03-04T10:04:04.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7fbdccbe0640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbdc80cd2d0 0x7fbdc81f3a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00c580 tx=0x7fbdbc00ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a8259c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c009330 0x7f6a84155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f34098d1640 1 -- start start 2024-03-04T10:04:04.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a8259c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c009330 0x7f6a84155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:42972/0 (socket says 172.21.15.99:42972) 2024-03-04T10:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a8259c640 1 -- 172.21.15.99:0/2948435104 learned_addr learned my addr 172.21.15.99:0/2948435104 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd814e500 con 0x7f2cd8071890 2024-03-04T10:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.083+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd400ccf0 con 0x7f2cd8071890 2024-03-04T10:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7fe34101e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340df210 0x7fe334035e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a81d9b640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a84157580 0x7f6a84158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7fe34101e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3340e2d70 0x7fe334036fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a88ae9640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a84153990 0x7f6a841f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2cd8074f60 con 0x7f2cd8071890 2024-03-04T10:04:04.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8415b020 con 0x7f6a84153990 2024-03-04T10:04:04.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a84153990 0x7f6a841f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.084+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8406ef60 con 0x7f6a84157580 2024-03-04T10:04:04.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8406b7c0 con 0x7f6a6c009330 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe33b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3340e2d70 0x7fe334036fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe33b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3340e2d70 0x7fe334036fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38064/0 (socket says 172.21.15.99:38064) 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe33bfff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340df210 0x7fe334035e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe33bfff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340df210 0x7fe334035e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:42982/0 (socket says 172.21.15.99:42982) 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe33b7fe640 1 -- 172.21.15.99:0/1874062065 learned_addr learned my addr 172.21.15.99:0/1874062065 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34101e640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3340e31d0 0x7fe3340e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340df5f0 con 0x7fe3340e2d70 2024-03-04T10:04:04.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340e3660 con 0x7fe3340e31d0 2024-03-04T10:04:04.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340e3a50 con 0x7fe3340df210 2024-03-04T10:04:04.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc018070 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34081d640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3340e31d0 0x7fe3340e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7f34098d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34040cf570 0x7f3404157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7f34098d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cfa30 0x7f34040ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34081d640 1 -- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340df210 msgr2=0x7fe334035e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34081d640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3340df210 0x7fe334035e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34081d640 1 -- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3340e2d70 msgr2=0x7fe334036fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34081d640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3340e2d70 0x7fe334036fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fe34081d640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe334036540 con 0x7fe3340e31d0 2024-03-04T10:04:04.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fcf7c744640 1 -- 172.21.15.99:0/653243572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf5c001030 con 0x7fcf740cf2e0 2024-03-04T10:04:04.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcf6c01a070 con 0x7fcf740cf2e0 2024-03-04T10:04:04.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c009330 msgr2=0x7f6a84155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c009330 0x7f6a84155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a84157580 msgr2=0x7f6a84158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a84157580 0x7f6a84158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a84159240 con 0x7f6a84153990 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a82d9d640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a84153990 0x7f6a841f2c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c033000 tx=0x7f6a8406dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a8259c640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6a6c009330 0x7f6a84155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a81d9b640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a84157580 0x7f6a84158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f5907481640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59000727f0 0x7f59001475f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f3913112640 1 Processor -- start 2024-03-04T10:04:04.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f5907481640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590015ba50 0x7f59000b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f3913112640 1 -- start start 2024-03-04T10:04:04.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f5907481640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f590015cc80 0x7f59000b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f5907481640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000cd620 con 0x7f590015cc80 2024-03-04T10:04:04.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c039040 con 0x7f6a84153990 2024-03-04T10:04:04.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7c003d80 con 0x7f6a84153990 2024-03-04T10:04:04.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c005270 con 0x7f6a84153990 2024-03-04T10:04:04.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f5907481640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000722d0 con 0x7f590015ba50 2024-03-04T10:04:04.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f5907481640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59000cd7a0 con 0x7f59000727f0 2024-03-04T10:04:04.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a84156500 con 0x7f6a84153990 2024-03-04T10:04:04.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59000727f0 0x7f59001475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59000727f0 0x7f59001475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43002/0 (socket says 172.21.15.99:43002) 2024-03-04T10:04:04.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 -- 172.21.15.99:0/1268482911 learned_addr learned my addr 172.21.15.99:0/1268482911 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a84156ac0 con 0x7f6a84153990 2024-03-04T10:04:04.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 -- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590015ba50 msgr2=0x7f59000b44c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T10:04:04.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc007560 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.085+0000 7fbdcee44640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc80b62e0 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f590015ba50 0x7f59000b44c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 -- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f590015cc80 msgr2=0x7f59000b2960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f590015cc80 0x7f59000b2960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f590521d640 1 -- 172.21.15.99:0/1268482911 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000b3420 con 0x7f59000727f0 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fe34081d640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3340e31d0 0x7fe3340e0c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe33c06a4e0 tx=0x7fe33c062b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b5070 con 0x7fe3340e31d0 2024-03-04T10:04:04.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c06be50 con 0x7fe3340e31d0 2024-03-04T10:04:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc004a20 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fbdcee44640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc80c8d30 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbda0001030 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2cd401a070 con 0x7f2cd8071890 2024-03-04T10:04:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f2ccdffb640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cbc06b5d0 0x7f2cbc06da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0b2270 con 0x7fe3340e31d0 2024-03-04T10:04:04.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2cd408b1e0 con 0x7f2cd8071890 2024-03-04T10:04:04.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7fcf6affd640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcf5806b5a0 0x7fcf5806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcf6c08a310 con 0x7fcf740cf2e0 2024-03-04T10:04:04.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3340e04a0 con 0x7fe3340e31d0 2024-03-04T10:04:04.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f2ccdffb640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f2cbc074d50 0x7f2cbc077190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2cd400fe50 con 0x7f2cbc074d50 2024-03-04T10:04:04.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cd408b5d0 con 0x7f2cd8071890 2024-03-04T10:04:04.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7fcf6affd640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7fcf58074ca0 0x7fcf580770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7f590521d640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59000727f0 0x7f59001475f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f58fc001ea0 tx=0x7f58fc00cd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0074e0 con 0x7f59000727f0 2024-03-04T10:04:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58fc015030 con 0x7f59000727f0 2024-03-04T10:04:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f2cdd7d0640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cbc06b5d0 0x7f2cbc06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.087+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf580777b0 con 0x7fcf58074ca0 2024-03-04T10:04:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf6c091050 con 0x7fcf740cf2e0 2024-03-04T10:04:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.086+0000 7f3402ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34040cf570 0x7f3404157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f34098d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 0x7f34040c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7fcf7ace1640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7fcf58074ca0 0x7fcf580770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f34098d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340406acd0 con 0x7f340415bc40 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f33fbfff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cfa30 0x7f34040ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f34098d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34040c75a0 con 0x7f34040cf570 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3402ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34040cf570 0x7f3404157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39334/0 (socket says 172.21.15.99:39334) 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3402ffd640 1 -- 172.21.15.99:0/3394423474 learned_addr learned my addr 172.21.15.99:0/3394423474 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f34098d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34040c3f30 con 0x7f34040cfa30 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3913112640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f390404b3c0 0x7f3904035840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f2cddfd1640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f2cbc074d50 0x7f2cbc077190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a84157180 con 0x7f6a84153990 2024-03-04T10:04:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f34037fe640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 0x7f34040c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7fcf7a4e0640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcf5806b5a0 0x7fcf5806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3913112640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f39040cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3913112640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404fe90 0x7f39040d0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3913112640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904054830 con 0x7f390404b3c0 2024-03-04T10:04:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3913112640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390404b7a0 con 0x7f390404fe90 2024-03-04T10:04:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3913112640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040549b0 con 0x7f390404f4d0 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3912110640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f390404b3c0 0x7f3904035840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3912110640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f390404b3c0 0x7f3904035840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38076/0 (socket says 172.21.15.99:38076) 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3912110640 1 -- 172.21.15.99:0/3844762937 learned_addr learned my addr 172.21.15.99:0/3844762937 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f391190f640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f39040cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7f3912911640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404fe90 0x7f39040d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3402ffd640 1 -- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cfa30 msgr2=0x7f34040ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7fcf7a4e0640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcf5806b5a0 0x7fcf5806da60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcf74155880 tx=0x7fcf64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0074e0 con 0x7f59000727f0 2024-03-04T10:04:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59000b3700 con 0x7f59000727f0 2024-03-04T10:04:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7fcf7ace1640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7fcf58074ca0 0x7fcf580770e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3402ffd640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f34040cfa30 0x7f34040ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3402ffd640 1 -- 172.21.15.99:0/3394423474 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 msgr2=0x7f34040c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3402ffd640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f340415bc40 0x7f34040c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3402ffd640 1 -- 172.21.15.99:0/3394423474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34040c9540 con 0x7f34040cf570 2024-03-04T10:04:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f2cdd7d0640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cbc06b5d0 0x7f2cbc06da90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0035000 tx=0x7f2cd002e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f2cddfd1640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f2cbc074d50 0x7f2cbc077190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3402ffd640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34040cf570 0x7f3404157030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33f400c8c0 tx=0x7f33f400cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f391190f640 1 -- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404fe90 msgr2=0x7f39040d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f391190f640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f390404fe90 0x7f39040d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f391190f640 1 -- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f390404b3c0 msgr2=0x7f3904035840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f391190f640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f390404b3c0 0x7f3904035840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f391190f640 1 -- 172.21.15.99:0/3844762937 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904180c10 con 0x7f390404f4d0 2024-03-04T10:04:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59000b3cc0 con 0x7f59000727f0 2024-03-04T10:04:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.088+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe334036a90 con 0x7fe3340e31d0 2024-03-04T10:04:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe3340057d0 con 0x7fe3340e31d0 2024-03-04T10:04:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f391190f640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f39040cfec0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00d620 tx=0x7f38fc00dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbdbc004bc0 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4018070 con 0x7f34040cf570 2024-03-04T10:04:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f40074e0 con 0x7f34040cf570 2024-03-04T10:04:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34040c97e0 con 0x7f34040cf570 2024-03-04T10:04:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.089+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4004a20 con 0x7f34040cf570 2024-03-04T10:04:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcf580777b0 con 0x7fcf58074ca0 2024-03-04T10:04:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00b620 con 0x7f390404f4d0 2024-03-04T10:04:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc019070 con 0x7f390404f4d0 2024-03-04T10:04:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5900074f60 con 0x7f59000727f0 2024-03-04T10:04:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00b9e0 con 0x7f390404f4d0 2024-03-04T10:04:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34040b3010 con 0x7f34040cf570 2024-03-04T10:04:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe33c0c1030 con 0x7fe3340e31d0 2024-03-04T10:04:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904180ef0 con 0x7f390404f4d0 2024-03-04T10:04:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fbdc57fa640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbdac06b4d0 0x7fbdac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbdbc089970 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fbdc57fa640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7fbdac074bd0 0x7fbdac077010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdac0776e0 con 0x7fbdac074bd0 2024-03-04T10:04:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdbc08e280 con 0x7fbdc80cd2d0 2024-03-04T10:04:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fe338ff9640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe32006b5a0 0x7fe32006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3904035d80 con 0x7f390404f4d0 2024-03-04T10:04:04.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7fbdc7fff640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbdac06b4d0 0x7fbdac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe33c12a3e0 con 0x7fe3340e31d0 2024-03-04T10:04:04.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.090+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6a7c03c070 con 0x7f6a84153990 2024-03-04T10:04:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7fe338ff9640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7fe320074ca0 0x7fe3200770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe33c0c4e70 con 0x7fe320074ca0 2024-03-04T10:04:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe33c131050 con 0x7fe3340e31d0 2024-03-04T10:04:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2cd400fe50 con 0x7f2cbc074d50 2024-03-04T10:04:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3404074f60 con 0x7f34040cf570 2024-03-04T10:04:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7fe33b7fe640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7fe320074ca0 0x7fe3200770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7f6a72ffd640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6a6406b5a0 0x7f6a6406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39041809c0 con 0x7f390404f4d0 2024-03-04T10:04:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.091+0000 7fbdcd3e1640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7fbdac074bd0 0x7fbdac077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.092+0000 7fe33b7fe640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7fe320074ca0 0x7fe3200770e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.092+0000 7f6a8259c640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6a6406b5a0 0x7f6a6406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.092+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f33f401b070 con 0x7f34040cf570 2024-03-04T10:04:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.092+0000 7fbdcd3e1640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7fbdac074bd0 0x7fbdac077010 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.092+0000 7f3400ff9640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33e406b520 0x7f33e406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.092+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f33f4089be0 con 0x7f34040cf570 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f3400ff9640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f33e4074c20 0x7f33e4077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f33fbfff640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33e406b520 0x7f33e406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33e4077730 con 0x7f33e4074c20 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f58fc0047e0 con 0x7f59000727f0 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6a7c0ac200 con 0x7f6a84153990 2024-03-04T10:04:04.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f6a8259c640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6a6406b5a0 0x7f6a6406da60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6a74013510 tx=0x7f6a74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f6a72ffd640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f6a64074d20 0x7f6a64077160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7fe33bfff640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe32006b5a0 0x7fe32006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a7c009eb0 con 0x7f6a64074d20 2024-03-04T10:04:04.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a7c0ac5f0 con 0x7f6a84153990 2024-03-04T10:04:04.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== osd.2 v2:172.21.15.99:6816/739656799 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe33c0c4e70 con 0x7fe320074ca0 2024-03-04T10:04:04.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f38fc00bb80 con 0x7f390404f4d0 2024-03-04T10:04:04.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f40544e0 con 0x7f34040cf570 2024-03-04T10:04:04.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f58f5ffb640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58e406b8d0 0x7f58e406dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f58fc0072e0 con 0x7f59000727f0 2024-03-04T10:04:04.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f33fbfff640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33e406b520 0x7f33e406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34040afe10 tx=0x7f33ec016040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbdac0776e0 con 0x7fbdac074bd0 2024-03-04T10:04:04.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f6a81d9b640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f6a64074d20 0x7f6a64077160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.093+0000 7f58f5ffb640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f58e4074fd0 0x7f58e4077410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7fbdc7fff640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbdac06b4d0 0x7fbdac06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbdc81f3f40 tx=0x7fbdb4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58fc018e50 con 0x7f58e4074fd0 2024-03-04T10:04:04.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f34037fe640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f33e4074c20 0x7f33e4077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58fc08e820 con 0x7f59000727f0 2024-03-04T10:04:04.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f3902ffd640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f38f006b520 0x7f38f006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f5905a1e640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f58e4074fd0 0x7f58e4077410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f38fc08a740 con 0x7f390404f4d0 2024-03-04T10:04:04.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f5904a1c640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58e406b8d0 0x7f58e406dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f3902ffd640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f38f0074c20 0x7f38f0077060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f6a81d9b640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f6a64074d20 0x7f6a64077160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38fc001d30 con 0x7f38f0074c20 2024-03-04T10:04:04.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38fc054dc0 con 0x7f390404f4d0 2024-03-04T10:04:04.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7f3912110640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f38f006b520 0x7f38f006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.094+0000 7fe33bfff640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe32006b5a0 0x7fe32006da60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe33404f620 tx=0x7fe330008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f5905a1e640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f58e4074fd0 0x7f58e4077410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f3912911640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f38f0074c20 0x7f38f0077060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f34037fe640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f33e4074c20 0x7f33e4077060 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f3912110640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f38f006b520 0x7f38f006d9e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f390c050300 tx=0x7f390c0ab000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a7c009eb0 con 0x7f6a64074d20 2024-03-04T10:04:04.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f5904a1c640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58e406b8d0 0x7f58e406dd90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58f8036000 tx=0x7f58f8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.095+0000 7f3912911640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f38f0074c20 0x7f38f0077060 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.096+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== osd.3 v2:172.21.15.99:6808/85197782 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f58fc018e50 con 0x7f58e4074fd0 2024-03-04T10:04:04.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.096+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f33e4077730 con 0x7f33e4074c20 2024-03-04T10:04:04.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.097+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f38fc001d30 con 0x7f38f0074c20 2024-03-04T10:04:04.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.121+0000 7fcf7c744640 1 -- 172.21.15.99:0/653243572 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcf5c002dd0 con 0x7fcf58074ca0 2024-03-04T10:04:04.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.122+0000 7fcf6affd640 1 -- 172.21.15.99:0/653243572 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcf5c002dd0 con 0x7fcf58074ca0 2024-03-04T10:04:04.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.122+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2cd806d650 con 0x7f2cbc074d50 2024-03-04T10:04:04.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7f2ccdffb640 1 -- 172.21.15.99:0/382479266 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2cd806d650 con 0x7f2cbc074d50 2024-03-04T10:04:04.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7fcf58074ca0 msgr2=0x7fcf580770e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7fcf58074ca0 0x7fcf580770e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcf5806b5a0 msgr2=0x7fcf5806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcf5806b5a0 0x7fcf5806da60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcf74155880 tx=0x7fcf64008040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 msgr2=0x7fcf7414d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 --2- 172.21.15.99:0/653243572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcf740cf2e0 0x7fcf7414d2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00cb10 tx=0x7fcf6c00a060 comp rx=0 tx=0).stop 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf7ace1640 1 -- 172.21.15.99:0/653243572 reap_dead start 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 shutdown_connections 2024-03-04T10:04:04.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.123+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 >> 172.21.15.99:0/653243572 conn(0x7fcf740d0780 msgr2=0x7fcf74159cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f2cbc074d50 msgr2=0x7f2cbc077190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f2cbc074d50 0x7f2cbc077190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 shutdown_connections 2024-03-04T10:04:04.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cbc06b5d0 msgr2=0x7f2cbc06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cbc06b5d0 0x7f2cbc06da90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0035000 tx=0x7f2cd002e000 comp rx=0 tx=0).stop 2024-03-04T10:04:04.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd8071890 msgr2=0x7f2cd80c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 --2- 172.21.15.99:0/382479266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd8071890 0x7f2cd80c5010 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400e890 tx=0x7f2cd400ed50 comp rx=0 tx=0).stop 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cddfd1640 1 -- 172.21.15.99:0/382479266 reap_dead start 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 shutdown_connections 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 >> 172.21.15.99:0/382479266 conn(0x7f2cd806acd0 msgr2=0x7f2cd80ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7fcf68ff9640 1 -- 172.21.15.99:0/653243572 wait complete. 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 shutdown_connections 2024-03-04T10:04:04.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.124+0000 7f2cdfa34640 1 -- 172.21.15.99:0/382479266 wait complete. 2024-03-04T10:04:04.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.125+0000 7fe34101e640 1 -- 172.21.15.99:0/1874062065 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe334052ca0 con 0x7fe320074ca0 2024-03-04T10:04:04.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.125+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f59000ccac0 con 0x7f58e4074fd0 2024-03-04T10:04:04.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.126+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbda0002dd0 con 0x7fbdac074bd0 2024-03-04T10:04:04.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.126+0000 7fe338ff9640 1 -- 172.21.15.99:0/1874062065 <== osd.2 v2:172.21.15.99:6816/739656799 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe334052ca0 con 0x7fe320074ca0 2024-03-04T10:04:04.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.126+0000 7f58f5ffb640 1 -- 172.21.15.99:0/1268482911 <== osd.3 v2:172.21.15.99:6808/85197782 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f59000ccac0 con 0x7f58e4074fd0 2024-03-04T10:04:04.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdc57fa640 1 -- 172.21.15.99:0/4254248434 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbda0002dd0 con 0x7fbdac074bd0 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7fe320074ca0 msgr2=0x7fe3200770e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7fe320074ca0 0x7fe3200770e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe32006b5a0 msgr2=0x7fe32006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe32006b5a0 0x7fe32006da60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe33404f620 tx=0x7fe330008040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3340e31d0 msgr2=0x7fe3340e0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 --2- 172.21.15.99:0/1874062065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3340e31d0 0x7fe3340e0c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe33c06a4e0 tx=0x7fe33c062b00 comp rx=0 tx=0).stop 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe34081d640 1 -- 172.21.15.99:0/1874062065 reap_dead start 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 shutdown_connections 2024-03-04T10:04:04.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 >> 172.21.15.99:0/1874062065 conn(0x7fe33400a3f0 msgr2=0x7fe3340489a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a84074f60 con 0x7f6a64074d20 2024-03-04T10:04:04.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f58e4074fd0 msgr2=0x7f58e4077410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5907481640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f58e4074fd0 0x7f58e4077410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7fbdac074bd0 msgr2=0x7fbdac077010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdb2ffd640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7fbdac074bd0 0x7fbdac077010 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 shutdown_connections 2024-03-04T10:04:04.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fe3267fc640 1 -- 172.21.15.99:0/1874062065 wait complete. 2024-03-04T10:04:04.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbdac06b4d0 msgr2=0x7fbdac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdb2ffd640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbdac06b4d0 0x7fbdac06d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbdc81f3f40 tx=0x7fbdb4009190 comp rx=0 tx=0).stop 2024-03-04T10:04:04.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbdc80cd2d0 msgr2=0x7fbdc81f3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdb2ffd640 1 --2- 172.21.15.99:0/4254248434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbdc80cd2d0 0x7fbdc81f3a00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00c580 tx=0x7fbdbc00ca40 comp rx=0 tx=0).stop 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7fbdcd3e1640 1 -- 172.21.15.99:0/4254248434 reap_dead start 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58e406b8d0 msgr2=0x7f58e406dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5907481640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58e406b8d0 0x7f58e406dd90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58f8036000 tx=0x7f58f8030000 comp rx=0 tx=0).stop 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59000727f0 msgr2=0x7f59001475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5907481640 1 --2- 172.21.15.99:0/1268482911 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59000727f0 0x7f59001475f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f58fc001ea0 tx=0x7f58fc00cd20 comp rx=0 tx=0).stop 2024-03-04T10:04:04.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.127+0000 7f5905a1e640 1 -- 172.21.15.99:0/1268482911 reap_dead start 2024-03-04T10:04:04.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 shutdown_connections 2024-03-04T10:04:04.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 >> 172.21.15.99:0/4254248434 conn(0x7fbdc80d0780 msgr2=0x7fbdc80b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 shutdown_connections 2024-03-04T10:04:04.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 >> 172.21.15.99:0/1268482911 conn(0x7f59000cc4d0 msgr2=0x7f59000b5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f6a72ffd640 1 -- 172.21.15.99:0/2948435104 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6a84074f60 con 0x7f6a64074d20 2024-03-04T10:04:04.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 shutdown_connections 2024-03-04T10:04:04.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 shutdown_connections 2024-03-04T10:04:04.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f5907481640 1 -- 172.21.15.99:0/1268482911 wait complete. 2024-03-04T10:04:04.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7fbdb2ffd640 1 -- 172.21.15.99:0/4254248434 wait complete. 2024-03-04T10:04:04.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f34040cd420 con 0x7f33e4074c20 2024-03-04T10:04:04.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f6a64074d20 msgr2=0x7f6a64077160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.128+0000 7f6a88ae9640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f6a64074d20 0x7f6a64077160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39040057e0 con 0x7f38f0074c20 2024-03-04T10:04:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6a6406b5a0 msgr2=0x7f6a6406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6a6406b5a0 0x7f6a6406da60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6a74013510 tx=0x7f6a74008040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a84153990 msgr2=0x7f6a841f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 --2- 172.21.15.99:0/2948435104 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6a84153990 0x7f6a841f2c70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c033000 tx=0x7f6a8406dcb0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f3400ff9640 1 -- 172.21.15.99:0/3394423474 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f34040cd420 con 0x7f33e4074c20 2024-03-04T10:04:04.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a82d9d640 1 -- 172.21.15.99:0/2948435104 reap_dead start 2024-03-04T10:04:04.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 shutdown_connections 2024-03-04T10:04:04.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f3902ffd640 1 -- 172.21.15.99:0/3844762937 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f39040057e0 con 0x7f38f0074c20 2024-03-04T10:04:04.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 >> 172.21.15.99:0/2948435104 conn(0x7f6a8406acd0 msgr2=0x7f6a840af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 shutdown_connections 2024-03-04T10:04:04.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f6a88ae9640 1 -- 172.21.15.99:0/2948435104 wait complete. 2024-03-04T10:04:04.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f33e4074c20 msgr2=0x7f33e4077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.129+0000 7f34098d1640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f33e4074c20 0x7f33e4077060 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f38f0074c20 msgr2=0x7f38f0077060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3913112640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f38f0074c20 0x7f38f0077060 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33e406b520 msgr2=0x7f33e406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33e406b520 0x7f33e406d9e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f34040afe10 tx=0x7f33ec016040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34040cf570 msgr2=0x7f3404157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 --2- 172.21.15.99:0/3394423474 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f34040cf570 0x7f3404157030 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f33f400c8c0 tx=0x7f33f400cd80 comp rx=0 tx=0).stop 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34037fe640 1 -- 172.21.15.99:0/3394423474 reap_dead start 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 shutdown_connections 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 >> 172.21.15.99:0/3394423474 conn(0x7f34040d0780 msgr2=0x7f340415ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 shutdown_connections 2024-03-04T10:04:04.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f34098d1640 1 -- 172.21.15.99:0/3394423474 wait complete. 2024-03-04T10:04:04.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f38f006b520 msgr2=0x7f38f006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3913112640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f38f006b520 0x7f38f006d9e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f390c050300 tx=0x7f390c0ab000 comp rx=0 tx=0).stop 2024-03-04T10:04:04.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 msgr2=0x7f39040cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3913112640 1 --2- 172.21.15.99:0/3844762937 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f390404f4d0 0x7f39040cfec0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00d620 tx=0x7f38fc00dae0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.130+0000 7f3912911640 1 -- 172.21.15.99:0/3844762937 reap_dead start 2024-03-04T10:04:04.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.131+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 shutdown_connections 2024-03-04T10:04:04.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.131+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 >> 172.21.15.99:0/3844762937 conn(0x7f390400a400 msgr2=0x7f390400a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.131+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 shutdown_connections 2024-03-04T10:04:04.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.131+0000 7f3913112640 1 -- 172.21.15.99:0/3844762937 wait complete. 2024-03-04T10:04:04.208 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.208 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-04T10:04:04.209 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.209 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-04T10:04:04.210 INFO:teuthology.orchestra.run.smithi099.stdout:55834574860 2024-03-04T10:04:04.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T10:04:04.211 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.211 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T10:04:04.212 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.212 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T10:04:04.213 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-04T10:04:04.214 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.214 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-04T10:04:04.215 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.215 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T10:04:04.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.414+0000 7fd9fc353640 1 Processor -- start 2024-03-04T10:04:04.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.414+0000 7fd9fc353640 1 -- start start 2024-03-04T10:04:04.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7f15f1b45640 1 Processor -- start 2024-03-04T10:04:04.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7f15f1b45640 1 -- start start 2024-03-04T10:04:04.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7fd9fc353640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd9f415c370 0x7fd9f40b2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7fd9fc353640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f406d190 0x7fd9f406d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7fd9fc353640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd9f406e1d0 0x7fd9f406e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7fd9fc353640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f40b34e0 con 0x7fd9f415c370 2024-03-04T10:04:04.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7fd9fc353640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f406eba0 con 0x7fd9f406e1d0 2024-03-04T10:04:04.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.415+0000 7fd9fc353640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41626b0 con 0x7fd9f406d190 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7fd9fa8f0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd9f406e1d0 0x7fd9f406e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7fd9f98ee640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f406d190 0x7fd9f406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7fd9fa8f0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd9f406e1d0 0x7fd9f406e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39356/0 (socket says 172.21.15.99:39356) 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7fd9fa8f0640 1 -- 172.21.15.99:0/126984777 learned_addr learned my addr 172.21.15.99:0/126984777 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7f15f1b45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e40dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7f15f1b45640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f15e4039e00 0x7f15e40eac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7f15f1b45640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e40549d0 con 0x7f15e4050be0 2024-03-04T10:04:04.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7f15f1b45640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e4054cb0 con 0x7f15e4039e00 2024-03-04T10:04:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7f15f1b45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4054f90 con 0x7f15e4050500 2024-03-04T10:04:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.416+0000 7fd9fa0ef640 1 --2- 172.21.15.99:0/126984777 >> v2:172.21.15.99:3300/0 conn(0x7fd9f415c370 0x7fd9f40b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e40dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e40dfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43012/0 (socket says 172.21.15.99:43012) 2024-03-04T10:04:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 -- 172.21.15.99:0/2231258192 learned_addr learned my addr 172.21.15.99:0/2231258192 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7fd9fa8f0640 1 -- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f406d190 msgr2=0x7fd9f406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7fd9fa8f0640 1 --2- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f406d190 0x7fd9f406d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7fd9fa8f0640 1 -- 172.21.15.99:0/126984777 >> v2:172.21.15.99:3300/0 conn(0x7fd9f415c370 msgr2=0x7fd9f40b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7fd9fa8f0640 1 --2- 172.21.15.99:0/126984777 >> v2:172.21.15.99:3300/0 conn(0x7fd9f415c370 0x7fd9f40b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7fd9fa8f0640 1 -- 172.21.15.99:0/126984777 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f4162990 con 0x7fd9f406e1d0 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7fd9f98ee640 1 --2- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f406d190 0x7fd9f406d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f3c643c4640 1 Processor -- start 2024-03-04T10:04:04.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f3c643c4640 1 -- start start 2024-03-04T10:04:04.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 -- 172.21.15.99:0/2231258192 >> v2:172.21.15.162:3300/0 conn(0x7f15e4039e00 msgr2=0x7f15e40eac70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T10:04:04.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 --2- 172.21.15.99:0/2231258192 >> v2:172.21.15.162:3300/0 conn(0x7f15e4039e00 0x7f15e40eac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 -- 172.21.15.99:0/2231258192 >> v1:172.21.15.162:6789/0 conn(0x7f15e4050be0 legacy=0x7f15e4008ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T10:04:04.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f15eb7fe640 1 -- 172.21.15.99:0/2231258192 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e4055270 con 0x7f15e4050500 2024-03-04T10:04:04.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.417+0000 7f230052b640 1 Processor -- start 2024-03-04T10:04:04.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7f230052b640 1 -- start start 2024-03-04T10:04:04.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7fd9fa8f0640 1 --2- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f406e660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4163230 tx=0x7fd9f002b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b59367076a57661c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7f3c643c4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3c5c0d0770 0x7f3c5c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7fd9f90ed640 1 -- 172.21.15.99:0/126984777 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0035020 con 0x7fd9f406e1d0 2024-03-04T10:04:04.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7f15eb7fe640 1 --2- 172.21.15.99:0/2231258192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e40dfd40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009870 tx=0x7f15d802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=452049e981292780 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f15ea7fc640 1 -- 172.21.15.99:0/2231258192 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8035020 con 0x7f15e4050500 2024-03-04T10:04:04.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f15ea7fc640 1 -- 172.21.15.99:0/2231258192 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d8031870 con 0x7f15e4050500 2024-03-04T10:04:04.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f3c62160640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3c5c0d0770 0x7f3c5c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f3c62160640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3c5c0d0770 0x7f3c5c0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38092/0 (socket says 172.21.15.99:38092) 2024-03-04T10:04:04.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f3c62160640 1 -- 172.21.15.99:0/3698624981 learned_addr learned my addr 172.21.15.99:0/3698624981 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7fd9f90ed640 1 -- 172.21.15.99:0/126984777 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f0031880 con 0x7fd9f406e1d0 2024-03-04T10:04:04.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7fd9f90ed640 1 -- 172.21.15.99:0/126984777 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0031c50 con 0x7fd9f406e1d0 2024-03-04T10:04:04.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f230052b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22f0009e20 0x7f22f0056120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7f3c643c4640 1 --2- 172.21.15.99:0/3698624981 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0b9580 0x7f3c5c0b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7fd9fc353640 1 -- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 msgr2=0x7fd9f406e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7fd9fc353640 1 --2- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f406e660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4163230 tx=0x7fd9f002b920 comp rx=0 tx=0).stop 2024-03-04T10:04:04.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f230052b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f003a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f230052b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f0056660 con 0x7f22f000a200 2024-03-04T10:04:04.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f15f1b45640 1 -- 172.21.15.99:0/2231258192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 msgr2=0x7f15e40dfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f15f1b45640 1 --2- 172.21.15.99:0/2231258192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e40dfd40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009870 tx=0x7f15d802b910 comp rx=0 tx=0).stop 2024-03-04T10:04:04.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c62961640 1 --2- 172.21.15.99:0/3698624981 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0b9580 0x7f3c5c0b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fd9fc353640 1 -- 172.21.15.99:0/126984777 shutdown_connections 2024-03-04T10:04:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0d1090 con 0x7f3c5c0b9120 2024-03-04T10:04:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fd9fc353640 1 --2- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f406e660 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fd9fc353640 1 --2- 172.21.15.99:0/126984777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f406d190 0x7fd9f406d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fd9fc353640 1 --2- 172.21.15.99:0/126984777 >> v2:172.21.15.99:3300/0 conn(0x7fd9f415c370 0x7fd9f40b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fd9fc353640 1 -- 172.21.15.99:0/126984777 >> 172.21.15.99:0/126984777 conn(0x7fd9f40b1550 msgr2=0x7fd9f40b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c0b7ee0 con 0x7f3c5c0d0770 2024-03-04T10:04:04.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c0b81c0 con 0x7f3c5c0b9580 2024-03-04T10:04:04.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fe2e6a75640 1 Processor -- start 2024-03-04T10:04:04.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.418+0000 7fcd6fba9640 1 Processor -- start 2024-03-04T10:04:04.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.419+0000 7fcd6fba9640 1 -- start start 2024-03-04T10:04:04.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6fba9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd6815dee0 0x7fcd680d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6fba9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcd680cd640 0x7fcd6806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c6115e640 1 -- 172.21.15.99:0/3698624981 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921071740 0 0) 0x7f3c5c0d1090 con 0x7f3c5c0b9120 2024-03-04T10:04:04.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c6115e640 1 -- 172.21.15.99:0/3698624981 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c48003520 con 0x7f3c5c0b9120 2024-03-04T10:04:04.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6fba9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd680cb360 con 0x7fcd6815f030 2024-03-04T10:04:04.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6fba9640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd680cb640 con 0x7fcd680cd640 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6fba9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680cb920 con 0x7fcd6815dee0 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6e146640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcd680cd640 0x7fcd6806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6d144640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fcd6815f030 0x7fcd680ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34818/0 (socket says 172.21.15.99:34818) 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6d144640 1 -- 172.21.15.99:0/1913728635 learned_addr learned my addr 172.21.15.99:0/1913728635 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c62961640 1 -- 172.21.15.99:0/3698624981 >> v2:172.21.15.99:3300/0 conn(0x7f3c5c0d0770 msgr2=0x7f3c5c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c62961640 1 --2- 172.21.15.99:0/3698624981 >> v2:172.21.15.99:3300/0 conn(0x7f3c5c0d0770 0x7f3c5c0d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c62961640 1 -- 172.21.15.99:0/3698624981 >> v1:172.21.15.99:6789/0 conn(0x7f3c5c0b9120 legacy=0x7f3c5c0d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f3c62961640 1 -- 172.21.15.99:0/3698624981 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0b84a0 con 0x7f3c5c0b9580 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6e146640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcd680cd640 0x7fcd6806e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38110/0 (socket says 172.21.15.99:38110) 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fcd6d945640 1 --2- 172.21.15.99:0/1913728635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd6815dee0 0x7fcd680d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7fcd6e146640 1 -- 172.21.15.99:0/1913728635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd6815dee0 msgr2=0x7fcd680d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7fcd6e146640 1 --2- 172.21.15.99:0/1913728635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd6815dee0 0x7fcd680d0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7fcd6e146640 1 -- 172.21.15.99:0/1913728635 >> v1:172.21.15.99:6789/0 conn(0x7fcd6815f030 legacy=0x7fcd680ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7fcd6e146640 1 -- 172.21.15.99:0/1913728635 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680cbc00 con 0x7fcd680cd640 2024-03-04T10:04:04.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f22fdac6640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f22f000a200 0x7f22f0051cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34810/0 (socket says 172.21.15.99:34810) 2024-03-04T10:04:04.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7fe2e6a75640 1 -- start start 2024-03-04T10:04:04.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f15f1b45640 1 -- 172.21.15.99:0/2231258192 shutdown_connections 2024-03-04T10:04:04.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f15f1b45640 1 --2- 172.21.15.99:0/2231258192 >> v2:172.21.15.162:3300/0 conn(0x7f15e4039e00 0x7f15e40eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f15f1b45640 1 --2- 172.21.15.99:0/2231258192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e40dfd40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f15f1b45640 1 -- 172.21.15.99:0/2231258192 >> 172.21.15.99:0/2231258192 conn(0x7f15e4055bc0 msgr2=0x7f15e4055fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f15f1b45640 1 -- 172.21.15.99:0/2231258192 shutdown_connections 2024-03-04T10:04:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f15f1b45640 1 -- 172.21.15.99:0/2231258192 wait complete. 2024-03-04T10:04:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7fd9fc353640 1 -- 172.21.15.99:0/126984777 shutdown_connections 2024-03-04T10:04:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7fd9fc353640 1 -- 172.21.15.99:0/126984777 wait complete. 2024-03-04T10:04:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f0622140640 1 Processor -- start 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f0622140640 1 -- start start 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6a75640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 0x7fe2e00b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6a75640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe2e0071120 0x7fe2e0162df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e5a73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 0x7fe2e00b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e5a73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 0x7fe2e00b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43030/0 (socket says 172.21.15.99:43030) 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6274640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe2e006acd0 0x7fe2e006d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43626/0 (socket says 172.21.15.99:43626) 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6274640 1 -- 172.21.15.99:0/731934439 learned_addr learned my addr 172.21.15.99:0/731934439 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e015c790 con 0x7fe2e006acd0 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e00715a0 con 0x7fe2e0071120 2024-03-04T10:04:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e006d280 con 0x7fe2e015c3b0 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fcd6e146640 1 --2- 172.21.15.99:0/1913728635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd680cd640 0x7fcd6806e2e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000be30 tx=0x7fcd600302b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add2987b02e9c40b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f22fdac6640 1 -- 172.21.15.99:0/3013026835 learned_addr learned my addr 172.21.15.99:0/3013026835 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f22fe2c7640 1 --2- 172.21.15.99:0/3013026835 >> v2:172.21.15.99:3300/0 conn(0x7f22f0009e20 0x7f22f0056120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.420+0000 7f22feac8640 1 --2- 172.21.15.99:0/3013026835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f003a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f0055c90 con 0x7f22f0009e20 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.421+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f00553a0 con 0x7f22f003a230 2024-03-04T10:04:04.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fd2c5640 1 -- 172.21.15.99:0/3013026835 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918865693 0 0) 0x7f22f0056660 con 0x7f22f000a200 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fd2c5640 1 -- 172.21.15.99:0/3013026835 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e4003520 con 0x7f22f000a200 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fd2c5640 1 -- 172.21.15.99:0/3013026835 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1081932305 0 0) 0x7f22e4003520 con 0x7f22f000a200 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fe2c7640 1 -- 172.21.15.99:0/3013026835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 msgr2=0x7f22f003a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fe2c7640 1 --2- 172.21.15.99:0/3013026835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f003a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fe2c7640 1 -- 172.21.15.99:0/3013026835 >> v1:172.21.15.99:6789/0 conn(0x7f22f000a200 legacy=0x7f22f0051cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f22fe2c7640 1 -- 172.21.15.99:0/3013026835 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0055640 con 0x7f22f0009e20 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fef96ed1640 1 Processor -- start 2024-03-04T10:04:04.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fef96ed1640 1 -- start start 2024-03-04T10:04:04.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7fe2e4a71640 1 -- 172.21.15.99:0/731934439 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097012211 0 0) 0x7fe2e015c790 con 0x7fe2e006acd0 2024-03-04T10:04:04.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e5272640 1 --2- 172.21.15.99:0/731934439 >> v2:172.21.15.99:3300/0 conn(0x7fe2e0071120 0x7fe2e0162df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0622140640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06f250 0x7f061c0ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0622140640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f061c06b3e0 0x7f061c15d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0622140640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061c15e270 con 0x7f061c06e9b0 2024-03-04T10:04:04.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0622140640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061c06ded0 con 0x7f061c06b3e0 2024-03-04T10:04:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0622140640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c06e1b0 con 0x7f061c06f250 2024-03-04T10:04:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.422+0000 7f3c62961640 1 --2- 172.21.15.99:0/3698624981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c0b78b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0b66f0 tx=0x7f3c5402bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f680f3313dbc6fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f15f1b45640 1 Processor -- start 2024-03-04T10:04:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0613fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f061c06e9b0 0x7f061c06acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43640/0 (socket says 172.21.15.99:43640) 2024-03-04T10:04:04.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f0613fff640 1 -- 172.21.15.99:0/3949834232 learned_addr learned my addr 172.21.15.99:0/3949834232 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:04.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f061bfff640 1 --2- 172.21.15.99:0/3949834232 >> v2:172.21.15.162:3300/0 conn(0x7f061c06b3e0 0x7f061c15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fd9fc353640 1 Processor -- start 2024-03-04T10:04:04.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e4a71640 1 -- 172.21.15.99:0/731934439 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2cc003520 con 0x7fe2e006acd0 2024-03-04T10:04:04.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fd9fc353640 1 -- start start 2024-03-04T10:04:04.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f3c6115e640 1 -- 172.21.15.99:0/3698624981 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54039070 con 0x7f3c5c0b9580 2024-03-04T10:04:04.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e5272640 1 -- 172.21.15.99:0/731934439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 msgr2=0x7fe2e00b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e5272640 1 --2- 172.21.15.99:0/731934439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 0x7fe2e00b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f061b7fe640 1 --2- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06f250 0x7f061c0ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e5272640 1 -- 172.21.15.99:0/731934439 >> v1:172.21.15.162:6789/0 conn(0x7fe2e006acd0 legacy=0x7fe2e006d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f3c6115e640 1 -- 172.21.15.99:0/3698624981 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5402cc70 con 0x7f3c5c0b9580 2024-03-04T10:04:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f3c6115e640 1 -- 172.21.15.99:0/3698624981 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54034660 con 0x7f3c5c0b9580 2024-03-04T10:04:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f15f1b45640 1 -- start start 2024-03-04T10:04:04.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e4a71640 1 -- 172.21.15.99:0/731934439 <== mon.1 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2938339051 0 0) 0x7fe2cc003520 con 0x7fe2e006acd0 2024-03-04T10:04:04.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e5272640 1 -- 172.21.15.99:0/731934439 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00717e0 con 0x7fe2e0071120 2024-03-04T10:04:04.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fcd6c943640 1 -- 172.21.15.99:0/1913728635 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60030e60 con 0x7fcd680cd640 2024-03-04T10:04:04.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fcd6c943640 1 -- 172.21.15.99:0/1913728635 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6000b2d0 con 0x7fcd680cd640 2024-03-04T10:04:04.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fef96ed1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7fe2e5a73640 1 --2- 172.21.15.99:0/731934439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 0x7fe2e00b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:04.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef96ed1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fef900d2440 0x7fef9015d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef96ed1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef900ce600 con 0x7fef900d1650 2024-03-04T10:04:04.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef96ed1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9006ced0 con 0x7fef900d2440 2024-03-04T10:04:04.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fef900d2440 0x7fef9015d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fef900d2440 0x7fef9015d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38134/0 (socket says 172.21.15.99:38134) 2024-03-04T10:04:04.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fcd6c943640 1 -- 172.21.15.99:0/1913728635 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60009760 con 0x7fcd680cd640 2024-03-04T10:04:04.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 -- 172.21.15.99:0/2594290976 learned_addr learned my addr 172.21.15.99:0/2594290976 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef96ed1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef900ce8e0 con 0x7fef9006d900 2024-03-04T10:04:04.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 msgr2=0x7f3c5c0b78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f3c643c4640 1 --2- 172.21.15.99:0/3698624981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c0b78b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0b66f0 tx=0x7f3c5402bec0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fcd6fba9640 1 -- 172.21.15.99:0/1913728635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd680cd640 msgr2=0x7fcd6806e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f061bfff640 1 -- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06f250 msgr2=0x7f061c0ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/1913728635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd680cd640 0x7fcd6806e2e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000be30 tx=0x7fcd600302b0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 shutdown_connections 2024-03-04T10:04:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f3c643c4640 1 --2- 172.21.15.99:0/3698624981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c0b78b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f3c643c4640 1 --2- 172.21.15.99:0/3698624981 >> v2:172.21.15.99:3300/0 conn(0x7f3c5c0d0770 0x7f3c5c0d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 >> 172.21.15.99:0/3698624981 conn(0x7f3c5c072820 msgr2=0x7f3c5c072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fd9fc353640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd9f406d190 0x7fd9f41f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f061bfff640 1 --2- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06f250 0x7f061c0ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f061bfff640 1 -- 172.21.15.99:0/3949834232 >> v1:172.21.15.162:6789/0 conn(0x7f061c06e9b0 legacy=0x7f061c06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.423+0000 7f061bfff640 1 -- 172.21.15.99:0/3949834232 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c15da80 con 0x7f061c06b3e0 2024-03-04T10:04:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f15f1b45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15e4039e00 0x7f15e40d73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 -- 172.21.15.99:0/2594290976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 msgr2=0x7fef900d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 --2- 172.21.15.99:0/2594290976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fd9fc353640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f40cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7f22fe2c7640 1 --2- 172.21.15.99:0/3013026835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f0009e20 0x7f22f0056120 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22f804f5b0 tx=0x7f22f808af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3f8d7ae3c7967b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 shutdown_connections 2024-03-04T10:04:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f3c643c4640 1 -- 172.21.15.99:0/3698624981 wait complete. 2024-03-04T10:04:04.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9fc353640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f415c370 0x7fd9f4160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9fc353640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41623a0 con 0x7fd9f406d190 2024-03-04T10:04:04.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fcd6fba9640 1 -- 172.21.15.99:0/1913728635 shutdown_connections 2024-03-04T10:04:04.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/1913728635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd680cd640 0x7fcd6806e2e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/1913728635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd6815dee0 0x7fcd680d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fcd6fba9640 1 -- 172.21.15.99:0/1913728635 >> 172.21.15.99:0/1913728635 conn(0x7fcd68099a00 msgr2=0x7fcd680d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15f1b45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e4180b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9f98ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f40cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f22fd2c5640 1 -- 172.21.15.99:0/3013026835 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4001670 con 0x7f22f0009e20 2024-03-04T10:04:04.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15f1b45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15e4050be0 0x7f15e4181060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9fa8f0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f415c370 0x7fd9f4160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15eb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15e4039e00 0x7f15e40d73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f22fd2c5640 1 -- 172.21.15.99:0/3013026835 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f8071450 con 0x7f22f0009e20 2024-03-04T10:04:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f22fd2c5640 1 -- 172.21.15.99:0/3013026835 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f806ce60 con 0x7f22f0009e20 2024-03-04T10:04:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15eaffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e4180b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15eaffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e4180b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43060/0 (socket says 172.21.15.99:43060) 2024-03-04T10:04:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fe2e5272640 1 --2- 172.21.15.99:0/731934439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e0071120 0x7fe2e0162df0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d802bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63b050000a1cdd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15eb7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15e4039e00 0x7f15e40d73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39388/0 (socket says 172.21.15.99:39388) 2024-03-04T10:04:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15eaffd640 1 -- 172.21.15.99:0/1144255263 learned_addr learned my addr 172.21.15.99:0/1144255263 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9f98ee640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f40cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39384/0 (socket says 172.21.15.99:39384) 2024-03-04T10:04:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9f98ee640 1 -- 172.21.15.99:0/87585617 learned_addr learned my addr 172.21.15.99:0/87585617 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9fa0ef640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd9f406d190 0x7fd9f41f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 -- 172.21.15.99:0/2594290976 >> v1:172.21.15.162:6789/0 conn(0x7fef900d1650 legacy=0x7fef900d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.424+0000 7fef9546e640 1 -- 172.21.15.99:0/2594290976 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef900cebc0 con 0x7fef900d2440 2024-03-04T10:04:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15ebfff640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15e4050be0 0x7f15e4181060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e4051020 con 0x7f15e4050be0 2024-03-04T10:04:04.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40e0280 con 0x7f15e4039e00 2024-03-04T10:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e40511a0 con 0x7f15e4050500 2024-03-04T10:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fcd6fba9640 1 -- 172.21.15.99:0/1913728635 shutdown_connections 2024-03-04T10:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fcd6fba9640 1 -- 172.21.15.99:0/1913728635 wait complete. 2024-03-04T10:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9fc353640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f4162f50 con 0x7fd9f406e1d0 2024-03-04T10:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fd9fc353640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f415c750 con 0x7fd9f415c370 2024-03-04T10:04:04.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7fe2e4a71640 1 -- 172.21.15.99:0/731934439 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8039070 con 0x7fe2e0071120 2024-03-04T10:04:04.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f061bfff640 1 --2- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06b3e0 0x7f061c15d540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0614009d90 tx=0x7f061402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=addb1cd7611b5c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f061affd640 1 -- 172.21.15.99:0/3949834232 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614039070 con 0x7f061c06b3e0 2024-03-04T10:04:04.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f061affd640 1 -- 172.21.15.99:0/3949834232 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061402cd10 con 0x7f061c06b3e0 2024-03-04T10:04:04.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.425+0000 7f061affd640 1 -- 172.21.15.99:0/3949834232 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614034660 con 0x7f061c06b3e0 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fe2e4a71640 1 -- 172.21.15.99:0/731934439 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d802c8a0 con 0x7fe2e0071120 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fe2e4a71640 1 -- 172.21.15.99:0/731934439 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0001670 con 0x7fe2e0071120 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15eaffd640 1 -- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15e4039e00 msgr2=0x7f15e40d73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15eaffd640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15e4039e00 0x7f15e40d73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15eaffd640 1 -- 172.21.15.99:0/1144255263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15e4050be0 msgr2=0x7f15e4181060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15eaffd640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15e4050be0 0x7f15e4181060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15eaffd640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40496b0 con 0x7f15e4050500 2024-03-04T10:04:04.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f0009e20 msgr2=0x7f22f0056120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f230052b640 1 --2- 172.21.15.99:0/3013026835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f0009e20 0x7f22f0056120 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f22f804f5b0 tx=0x7f22f808af90 comp rx=0 tx=0).stop 2024-03-04T10:04:04.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15eaffd640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e4180b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15ec02d440 tx=0x7f15ec073b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f3c643c4640 1 Processor -- start 2024-03-04T10:04:04.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f3c643c4640 1 -- start start 2024-03-04T10:04:04.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec06ce60 con 0x7f15e4050500 2024-03-04T10:04:04.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 shutdown_connections 2024-03-04T10:04:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f230052b640 1 --2- 172.21.15.99:0/3013026835 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f003a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f230052b640 1 --2- 172.21.15.99:0/3013026835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f0009e20 0x7f22f0056120 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 >> 172.21.15.99:0/3013026835 conn(0x7f22f00537a0 msgr2=0x7f22f0053b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e4049990 con 0x7f15e4050500 2024-03-04T10:04:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7f0622140640 1 -- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06b3e0 msgr2=0x7f061c15d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f0622140640 1 --2- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06b3e0 0x7f061c15d540 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0614009d90 tx=0x7f061402bf60 comp rx=0 tx=0).stop 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ec0731c0 con 0x7f15e4050500 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e4049eb0 con 0x7f15e4050500 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec092640 con 0x7f15e4050500 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f0622140640 1 -- 172.21.15.99:0/3949834232 shutdown_connections 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f0622140640 1 --2- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06b3e0 0x7f061c15d540 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f0622140640 1 --2- 172.21.15.99:0/3949834232 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06f250 0x7f061c0ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f0622140640 1 -- 172.21.15.99:0/3949834232 >> 172.21.15.99:0/3949834232 conn(0x7f061c0b1130 msgr2=0x7f061c0b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fd9f98ee640 1 -- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f415c370 msgr2=0x7fd9f4160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fd9f98ee640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f415c370 0x7fd9f4160d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fd9f98ee640 1 -- 172.21.15.99:0/87585617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd9f406d190 msgr2=0x7fd9f41f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fd9f98ee640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd9f406d190 0x7fd9f41f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fd9f98ee640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40c4830 con 0x7fd9f406e1d0 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 shutdown_connections 2024-03-04T10:04:04.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f230052b640 1 -- 172.21.15.99:0/3013026835 wait complete. 2024-03-04T10:04:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.426+0000 7fef9546e640 1 --2- 172.21.15.99:0/2594290976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d2440 0x7fef9015d4c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fef88009e50 tx=0x7fef8802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a706591a8abbf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e0071120 msgr2=0x7fe2e0162df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/731934439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e0071120 0x7fe2e0162df0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d802bd70 comp rx=0 tx=0).stop 2024-03-04T10:04:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fd9f98ee640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f40cb020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec007e40 tx=0x7fd9ec00d4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec01a070 con 0x7fd9f406e1d0 2024-03-04T10:04:04.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 shutdown_connections 2024-03-04T10:04:04.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec003b90 con 0x7fd9f406e1d0 2024-03-04T10:04:04.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00de80 con 0x7fd9f406e1d0 2024-03-04T10:04:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/731934439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e0071120 0x7fe2e0162df0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/731934439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e015c3b0 0x7fe2e00b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 >> 172.21.15.99:0/731934439 conn(0x7fe2e00b10c0 msgr2=0x7fe2e00b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f3c643c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c0b9120 0x7f3c5c1f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fd9fc353640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f40c4b10 con 0x7fd9f406e1d0 2024-03-04T10:04:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7f3c643c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c1f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fd9fc353640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f40c50d0 con 0x7fd9f406e1d0 2024-03-04T10:04:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c62160640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c0b9120 0x7f3c5c1f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c62160640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c0b9120 0x7f3c5c1f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43068/0 (socket says 172.21.15.99:43068) 2024-03-04T10:04:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c62160640 1 -- 172.21.15.99:0/863849072 learned_addr learned my addr 172.21.15.99:0/863849072 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c0d0770 0x7f3c5c15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e4009ad0 con 0x7f15e4050500 2024-03-04T10:04:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c6195f640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c1f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7fcd6fba9640 1 Processor -- start 2024-03-04T10:04:04.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.427+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 shutdown_connections 2024-03-04T10:04:04.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7fe2e6a75640 1 -- 172.21.15.99:0/731934439 wait complete. 2024-03-04T10:04:04.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c62961640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c0d0770 0x7f3c5c15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c1611a0 con 0x7f3c5c0d0770 2024-03-04T10:04:04.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0b8a60 con 0x7f3c5c0b9580 2024-03-04T10:04:04.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c161320 con 0x7f3c5c0b9120 2024-03-04T10:04:04.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f230052b640 1 Processor -- start 2024-03-04T10:04:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f15ec0927e0 con 0x7f15e4050500 2024-03-04T10:04:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f230052b640 1 -- start start 2024-03-04T10:04:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c6195f640 1 -- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c0b9120 msgr2=0x7f3c5c1f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c6195f640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c0b9120 0x7f3c5c1f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c6195f640 1 -- 172.21.15.99:0/863849072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c0d0770 msgr2=0x7f3c5c15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c6195f640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c0d0770 0x7f3c5c15c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c6195f640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c1f0ef0 con 0x7f3c5c0b9580 2024-03-04T10:04:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7f0622140640 1 -- 172.21.15.99:0/3949834232 shutdown_connections 2024-03-04T10:04:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fd9fc353640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f404f090 con 0x7fd9f406e1d0 2024-03-04T10:04:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fe2e6a75640 1 Processor -- start 2024-03-04T10:04:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c6195f640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c1f3f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c580011e0 tx=0x7f3c5800c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f0622140640 1 -- 172.21.15.99:0/3949834232 wait complete. 2024-03-04T10:04:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f15d7fff640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15d006b5a0 0x7f15d006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58017070 con 0x7f3c5c0b9580 2024-03-04T10:04:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c580052f0 con 0x7f3c5c0b9580 2024-03-04T10:04:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c1f4460 con 0x7f3c5c0b9580 2024-03-04T10:04:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fef8f7fe640 1 -- 172.21.15.99:0/2594290976 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8802ca70 con 0x7fef900d2440 2024-03-04T10:04:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fef8f7fe640 1 -- 172.21.15.99:0/2594290976 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8802cc10 con 0x7fef900d2440 2024-03-04T10:04:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fef8f7fe640 1 -- 172.21.15.99:0/2594290976 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef880343f0 con 0x7fef900d2440 2024-03-04T10:04:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f15eb7fe640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15d006b5a0 0x7f15d006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f15ec1049e0 con 0x7f15e4050500 2024-03-04T10:04:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fe2e6a75640 1 -- start start 2024-03-04T10:04:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd9ec01d070 con 0x7fd9f406e1d0 2024-03-04T10:04:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f230052b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 0x7f22f004f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f230052b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f000a200 0x7f22f0177e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fe2e6a75640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e006acd0 0x7fe2e0161520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c0c7770 con 0x7f3c5c0b9580 2024-03-04T10:04:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58003830 con 0x7f3c5c0b9580 2024-03-04T10:04:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fd9eaffd640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd9d806b5a0 0x7fd9d806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7f22fe2c7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 0x7f22f004f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fe2e6a75640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e0071120 0x7fe2e0165c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6a75640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 0x7fe2e0166190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e5a73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e006acd0 0x7fe2e0161520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e5a73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e006acd0 0x7fe2e0161520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38164/0 (socket says 172.21.15.99:38164) 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6a75640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0072080 con 0x7fe2e006acd0 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6a75640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0071da0 con 0x7fe2e015c3b0 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6a75640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e006b160 con 0x7fe2e0071120 2024-03-04T10:04:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f22fe2c7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 0x7f22f004f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39412/0 (socket says 172.21.15.99:39412) 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.428+0000 7fcd6fba9640 1 -- start start 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fcd6fba9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 0x7fcd68154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fcd6fba9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd6815dee0 0x7fcd680c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fcd6fba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd6815f030 0x7fcd68159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fcd6fba9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680d12f0 con 0x7fcd6815dee0 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fcd6fba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680d1170 con 0x7fcd6815f030 2024-03-04T10:04:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.429+0000 7fcd6fba9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd680cc120 con 0x7fcd680cd640 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fcd6d945640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 0x7fcd68154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fcd6d945640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 0x7fcd68154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43070/0 (socket says 172.21.15.99:43070) 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fcd6d945640 1 -- 172.21.15.99:0/616291140 learned_addr learned my addr 172.21.15.99:0/616291140 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fcd6d945640 1 -- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd6815f030 msgr2=0x7fcd68159ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.430+0000 7fcd6d945640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd6815f030 0x7fcd68159ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fcd6d945640 1 -- 172.21.15.99:0/616291140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd6815dee0 msgr2=0x7fcd680c6470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T10:04:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd9ec08a610 con 0x7fd9f406e1d0 2024-03-04T10:04:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f22fe2c7640 1 -- 172.21.15.99:0/2905053745 learned_addr learned my addr 172.21.15.99:0/2905053745 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f22fdac6640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f000a200 0x7f22f0177e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 0x7fe2e0166190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 0x7fe2e0166190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39420/0 (socket says 172.21.15.99:39420) 2024-03-04T10:04:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 -- 172.21.15.99:0/204065182 learned_addr learned my addr 172.21.15.99:0/204065182 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f00e2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f0622140640 1 Processor -- start 2024-03-04T10:04:04.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fd9fa0ef640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd9d806b5a0 0x7fd9d806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fcd6d144640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd6815dee0 0x7fcd680c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f0622140640 1 -- start start 2024-03-04T10:04:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fcd6d945640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd6815dee0 0x7fcd680c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fcd6d945640 1 -- 172.21.15.99:0/616291140 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680c5460 con 0x7fcd680cd640 2024-03-04T10:04:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f000a660 con 0x7f22f000a200 2024-03-04T10:04:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0052790 con 0x7f22f0009e20 2024-03-04T10:04:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f0052400 con 0x7f22f003a230 2024-03-04T10:04:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e5272640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e0071120 0x7fe2e0165c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f22feac8640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f00e2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fcd6d945640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 0x7fcd68154110 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcd6400c770 tx=0x7fcd6400cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7f15eb7fe640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15d006b5a0 0x7f15d006da60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15d8036000 tx=0x7f15d803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 -- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e0071120 msgr2=0x7fe2e0165c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e0071120 0x7fe2e0165c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 -- 172.21.15.99:0/204065182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e006acd0 msgr2=0x7fe2e0161520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e006acd0 0x7fe2e0161520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e6274640 1 -- 172.21.15.99:0/204065182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e0161a60 con 0x7fe2e015c3b0 2024-03-04T10:04:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fe2e5272640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e0071120 0x7fe2e0165c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fef96ed1640 1 -- 172.21.15.99:0/2594290976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d2440 msgr2=0x7fef9015d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.431+0000 7fef96ed1640 1 --2- 172.21.15.99:0/2594290976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d2440 0x7fef9015d4c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fef88009e50 tx=0x7fef8802bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22feac8640 1 -- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 msgr2=0x7f22f004f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22feac8640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 0x7f22f004f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22feac8640 1 -- 172.21.15.99:0/2905053745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f000a200 msgr2=0x7f22f0177e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22feac8640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f000a200 0x7f22f0177e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22feac8640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f004f910 con 0x7f22f003a230 2024-03-04T10:04:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fe2e6274640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 0x7fe2e0166190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000c730 tx=0x7fe2d000cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22fdac6640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f000a200 0x7f22f0177e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fd9fa0ef640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd9d806b5a0 0x7fd9d806da60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4160d10 tx=0x7fd9e4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3c5801a070 con 0x7f3c5c0b9580 2024-03-04T10:04:04.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fef96ed1640 1 -- 172.21.15.99:0/2594290976 shutdown_connections 2024-03-04T10:04:04.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fef96ed1640 1 --2- 172.21.15.99:0/2594290976 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d2440 0x7fef9015d4c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fef96ed1640 1 --2- 172.21.15.99:0/2594290976 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22fe2c7640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 0x7f22f004f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:04.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f22feac8640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f00e2660 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e7f0 tx=0x7f22e800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fef96ed1640 1 -- 172.21.15.99:0/2594290976 >> 172.21.15.99:0/2594290976 conn(0x7fef9009cbd0 msgr2=0x7fef900d3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fef96ed1640 1 -- 172.21.15.99:0/2594290976 shutdown_connections 2024-03-04T10:04:04.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7fef96ed1640 1 -- 172.21.15.99:0/2594290976 wait complete. 2024-03-04T10:04:04.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0622140640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06b3e0 0x7f061c0b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0622140640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06e9b0 0x7f061c153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0622140640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f061c06f250 0x7f061c1588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0622140640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0b2740 con 0x7f061c06f250 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0622140640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061c15dfa0 con 0x7f061c06e9b0 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0622140640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061c0b28c0 con 0x7f061c06b3e0 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f061b7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06b3e0 0x7f061c0b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f0613fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06e9b0 0x7f061c153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f061b7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06b3e0 0x7f061c0b39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43082/0 (socket says 172.21.15.99:43082) 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f061b7fe640 1 -- 172.21.15.99:0/3049572288 learned_addr learned my addr 172.21.15.99:0/3049572288 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f061bfff640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f061c06f250 0x7f061c1588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.432+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c04f090 con 0x7f3c5c0b9580 2024-03-04T10:04:04.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f3c52ffd640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4006b4d0 0x7f3c4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f061bfff640 1 -- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06b3e0 msgr2=0x7f061c0b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f061bfff640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06b3e0 0x7f061c0b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f061bfff640 1 -- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06e9b0 msgr2=0x7f061c153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f061bfff640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06e9b0 0x7f061c153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f061bfff640 1 -- 172.21.15.99:0/3049572288 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061c154770 con 0x7f061c06f250 2024-03-04T10:04:04.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f3c62160640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4006b4d0 0x7f3c4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f061bfff640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f061c06f250 0x7f061c1588e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0614033000 tx=0x7f0614004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614044020 con 0x7f061c06f250 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061c14dad0 con 0x7f061c06f250 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061403f690 con 0x7f061c06f250 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0614039040 con 0x7f061c06f250 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061c14e090 con 0x7f061c06f250 2024-03-04T10:04:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.433+0000 7f3c62160640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4006b4d0 0x7f3c4006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c006a90 tx=0x7f3c4c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.434+0000 7fef96ed1640 1 Processor -- start 2024-03-04T10:04:04.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.434+0000 7fef96ed1640 1 -- start start 2024-03-04T10:04:04.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.434+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061c04f090 con 0x7f061c06f250 2024-03-04T10:04:04.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd64015070 con 0x7fcd680cd640 2024-03-04T10:04:04.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8017070 con 0x7f22f003a230 2024-03-04T10:04:04.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800c940 con 0x7f22f003a230 2024-03-04T10:04:04.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd64004240 con 0x7fcd680cd640 2024-03-04T10:04:04.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd64007620 con 0x7fcd680cd640 2024-03-04T10:04:04.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f004fbf0 con 0x7f22f003a230 2024-03-04T10:04:04.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f15ec08e030 con 0x7f15e4050500 2024-03-04T10:04:04.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f00501b0 con 0x7f22f003a230 2024-03-04T10:04:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef96ed1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef96ed1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d1650 0x7fef900c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef96ed1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef900d2440 0x7fef900c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef96ed1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9006dce0 con 0x7fef900d1650 2024-03-04T10:04:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef96ed1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef900cf180 con 0x7fef900d2440 2024-03-04T10:04:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef96ed1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef900b3020 con 0x7fef9006d900 2024-03-04T10:04:04.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd680c46a0 con 0x7fcd680cd640 2024-03-04T10:04:04.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef94c6d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd680c4c60 con 0x7fcd680cd640 2024-03-04T10:04:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef94c6d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43096/0 (socket says 172.21.15.99:43096) 2024-03-04T10:04:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef94c6d640 1 -- 172.21.15.99:0/3122585933 learned_addr learned my addr 172.21.15.99:0/3122585933 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef9546e640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef900d2440 0x7fef900c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.435+0000 7fef8ffff640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d1650 0x7fef900c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fef9546e640 1 -- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 msgr2=0x7fef900cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800cd10 con 0x7f22f003a230 2024-03-04T10:04:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3c58089e00 con 0x7f3c5c0b9580 2024-03-04T10:04:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0005ad0 con 0x7fe2e015c3b0 2024-03-04T10:04:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fef9546e640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fef9546e640 1 -- 172.21.15.99:0/3122585933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d1650 msgr2=0x7fef900c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fef9546e640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d1650 0x7fef900c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fef9546e640 1 -- 172.21.15.99:0/3122585933 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef900cb800 con 0x7fef900d2440 2024-03-04T10:04:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e0005290 con 0x7f22f003a230 2024-03-04T10:04:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.436+0000 7fef9546e640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef900d2440 0x7fef900c93d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef88032000 tx=0x7fef88004a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd9ec010070 con 0x7fd9f406e1d0 2024-03-04T10:04:04.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e0149d80 con 0x7fe2e015c3b0 2024-03-04T10:04:04.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e014a340 con 0x7fe2e015c3b0 2024-03-04T10:04:04.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0614047020 con 0x7f061c06f250 2024-03-04T10:04:04.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7f0618ff9640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05fc06b500 0x7f05fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6804f090 con 0x7fcd680cd640 2024-03-04T10:04:04.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f22e801a070 con 0x7f22f003a230 2024-03-04T10:04:04.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f06140acd00 con 0x7f061c06f250 2024-03-04T10:04:04.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.437+0000 7f061b7fe640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05fc06b500 0x7f05fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.438+0000 7f22eeffd640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f22dc06b4d0 0x7f22dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.438+0000 7f061b7fe640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05fc06b500 0x7f05fc06d9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f060c00aae0 tx=0x7f060c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.438+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcd640077c0 con 0x7fcd680cd640 2024-03-04T10:04:04.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.438+0000 7f22fe2c7640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f22dc06b4d0 0x7f22dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.438+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f22e808a8a0 con 0x7f22f003a230 2024-03-04T10:04:04.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.438+0000 7fcd5e7fc640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd4c06b5a0 0x7fcd4c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.439+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e004f100 con 0x7fe2e015c3b0 2024-03-04T10:04:04.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.439+0000 7fcd6d144640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd4c06b5a0 0x7fcd4c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.439+0000 7f22fe2c7640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f22dc06b4d0 0x7f22dc06d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22f8073040 tx=0x7f22f80744a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.439+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcd6408a710 con 0x7fcd680cd640 2024-03-04T10:04:04.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.439+0000 7fcd6d144640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd4c06b5a0 0x7fcd4c06da60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd680c6fa0 tx=0x7fcd58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88043030 con 0x7fef900d2440 2024-03-04T10:04:04.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef900cbae0 con 0x7fef900d2440 2024-03-04T10:04:04.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef900cc0a0 con 0x7fef900d2440 2024-03-04T10:04:04.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3c5c1f4460 con 0x7f3c5c0b9580 2024-03-04T10:04:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8803e450 con 0x7fef900d2440 2024-03-04T10:04:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8803e600 con 0x7fef900d2440 2024-03-04T10:04:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0005c70 con 0x7fe2e015c3b0 2024-03-04T10:04:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.441+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0007270 con 0x7fe2e015c3b0 2024-03-04T10:04:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.442+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe2d0007410 con 0x7fe2e015c3b0 2024-03-04T10:04:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.443+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f22e8059340 con 0x7f22f003a230 2024-03-04T10:04:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.443+0000 7fe2d67fc640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2c406b4d0 0x7fe2c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.443+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe2d008ad00 con 0x7fe2e015c3b0 2024-03-04T10:04:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.443+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef70005290 con 0x7fef900d2440 2024-03-04T10:04:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.443+0000 7fe2e5a73640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2c406b4d0 0x7fe2c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.443+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fef88046070 con 0x7fef900d2440 2024-03-04T10:04:04.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.444+0000 7fef8d7fa640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef7406b4d0 0x7fef7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:04.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.444+0000 7fe2e5a73640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2c406b4d0 0x7fe2c406d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009560 tx=0x7fe2dc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.444+0000 7fef94c6d640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef7406b4d0 0x7fef7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:04.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.445+0000 7fef94c6d640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef7406b4d0 0x7fef7406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0099a0 tx=0x7fef7c006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:04.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.446+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f061407b160 con 0x7f061c06f250 2024-03-04T10:04:04.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.447+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcd64059230 con 0x7fcd680cd640 2024-03-04T10:04:04.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.447+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe2d00598a0 con 0x7fe2e015c3b0 2024-03-04T10:04:04.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.447+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fef880aac80 con 0x7fef900d2440 2024-03-04T10:04:04.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.452+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fef8804d070 con 0x7fef900d2440 2024-03-04T10:04:04.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.839+0000 7f15f1b45640 1 -- 172.21.15.99:0/1144255263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f15e4049b60 con 0x7f15e4050500 2024-03-04T10:04:04.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.840+0000 7f15d7fff640 1 -- 172.21.15.99:0/1144255263 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f15e4049b60 con 0x7f15e4050500 2024-03-04T10:04:04.842 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.843+0000 7f15d5ffb640 1 -- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15d006b5a0 msgr2=0x7f15d006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.843+0000 7f15d5ffb640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15d006b5a0 0x7f15d006da60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f15d8036000 tx=0x7f15d803c040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.843+0000 7f15d5ffb640 1 -- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 msgr2=0x7f15e4180b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e4180b20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15ec02d440 tx=0x7f15ec073b70 comp rx=0 tx=0).stop 2024-03-04T10:04:04.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 -- 172.21.15.99:0/1144255263 shutdown_connections 2024-03-04T10:04:04.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15d006b5a0 0x7f15d006da60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15e4050be0 0x7f15e4181060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15e4050500 0x7f15e4180b20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 --2- 172.21.15.99:0/1144255263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15e4039e00 0x7f15e40d73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 -- 172.21.15.99:0/1144255263 >> 172.21.15.99:0/1144255263 conn(0x7f15e4055bc0 msgr2=0x7f15e4037690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 -- 172.21.15.99:0/1144255263 shutdown_connections 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.844+0000 7f15d5ffb640 1 -- 172.21.15.99:0/1144255263 wait complete. 2024-03-04T10:04:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.846+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3c5c0ceb00 con 0x7f3c5c0b9580 2024-03-04T10:04:04.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.846+0000 7f3c52ffd640 1 -- 172.21.15.99:0/863849072 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c58057d50 con 0x7f3c5c0b9580 2024-03-04T10:04:04.849 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.847+0000 7fd9fc353640 1 -- 172.21.15.99:0/87585617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd9f4072230 con 0x7fd9f406e1d0 2024-03-04T10:04:04.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.848+0000 7fd9eaffd640 1 -- 172.21.15.99:0/87585617 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd9ec058b70 con 0x7fd9f406e1d0 2024-03-04T10:04:04.850 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.850+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4006b4d0 msgr2=0x7f3c4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.850+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4006b4d0 0x7f3c4006d990 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c006a90 tx=0x7f3c4c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.850+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 msgr2=0x7f3c5c1f3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.850+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c1f3f20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3c580011e0 tx=0x7f3c5800c890 comp rx=0 tx=0).stop 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.850+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 shutdown_connections 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4006b4d0 0x7f3c4006d990 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c0d0770 0x7f3c5c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0b9580 0x7f3c5c1f3f20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 --2- 172.21.15.99:0/863849072 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c0b9120 0x7f3c5c1f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 >> 172.21.15.99:0/863849072 conn(0x7f3c5c072820 msgr2=0x7f3c5c0b72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 shutdown_connections 2024-03-04T10:04:04.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 -- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd9d806b5a0 msgr2=0x7fd9d806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd9d806b5a0 0x7fd9d806da60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4160d10 tx=0x7fd9e4009190 comp rx=0 tx=0).stop 2024-03-04T10:04:04.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 -- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 msgr2=0x7fd9f40cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f40cb020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec007e40 tx=0x7fd9ec00d4a0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f3c643c4640 1 -- 172.21.15.99:0/863849072 wait complete. 2024-03-04T10:04:04.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 -- 172.21.15.99:0/87585617 shutdown_connections 2024-03-04T10:04:04.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd9d806b5a0 0x7fd9d806da60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd9f415c370 0x7fd9f4160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd9f406e1d0 0x7fd9f40cb020 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 --2- 172.21.15.99:0/87585617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd9f406d190 0x7fd9f41f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7fd9e8ff9640 1 -- 172.21.15.99:0/87585617 >> 172.21.15.99:0/87585617 conn(0x7fd9f40b1550 msgr2=0x7fd9f4164bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.851+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f061c06d750 con 0x7f061c06f250 2024-03-04T10:04:04.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.852+0000 7f0618ff9640 1 -- 172.21.15.99:0/3049572288 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f061407f200 con 0x7f061c06f250 2024-03-04T10:04:04.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.852+0000 7fd9e8ff9640 1 -- 172.21.15.99:0/87585617 shutdown_connections 2024-03-04T10:04:04.857 INFO:teuthology.orchestra.run.smithi099.stdout:55834574857 2024-03-04T10:04:04.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.853+0000 7fd9e8ff9640 1 -- 172.21.15.99:0/87585617 wait complete. 2024-03-04T10:04:04.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.856+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05fc06b500 msgr2=0x7f05fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.856+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f22e0005580 con 0x7f22f003a230 2024-03-04T10:04:04.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.856+0000 7f0622140640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05fc06b500 0x7f05fc06d9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f060c00aae0 tx=0x7f060c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.856+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f061c06f250 msgr2=0x7f061c1588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.856+0000 7f0622140640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f061c06f250 0x7f061c1588e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0614033000 tx=0x7f0614004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:04.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.856+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fef70005850 con 0x7fef900d2440 2024-03-04T10:04:04.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 shutdown_connections 2024-03-04T10:04:04.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05fc06b500 0x7f05fc06d9c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f061c06f250 0x7f061c1588e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f22eeffd640 1 -- 172.21.15.99:0/2905053745 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f22e8058c40 con 0x7f22f003a230 2024-03-04T10:04:04.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f061c06e9b0 0x7f061c153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 --2- 172.21.15.99:0/3049572288 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f061c06b3e0 0x7f061c0b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 >> 172.21.15.99:0/3049572288 conn(0x7f061c0b1130 msgr2=0x7f061c0cf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.861 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 shutdown_connections 2024-03-04T10:04:04.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7f0622140640 1 -- 172.21.15.99:0/3049572288 wait complete. 2024-03-04T10:04:04.861 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.857+0000 7fef8d7fa640 1 -- 172.21.15.99:0/3122585933 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef8807b530 con 0x7fef900d2440 2024-03-04T10:04:04.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.859+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fe2e00ae910 con 0x7fe2e015c3b0 2024-03-04T10:04:04.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fe2d67fc640 1 -- 172.21.15.99:0/204065182 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2d00591a0 con 0x7fe2e015c3b0 2024-03-04T10:04:04.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef7406b4d0 msgr2=0x7fef7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.863 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fef96ed1640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef7406b4d0 0x7fef7406d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0099a0 tx=0x7fef7c006f90 comp rx=0 tx=0).stop 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef900d2440 msgr2=0x7fef900c93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fef96ed1640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef900d2440 0x7fef900c93d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef88032000 tx=0x7fef88004a70 comp rx=0 tx=0).stop 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 shutdown_connections 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.860+0000 7fef96ed1640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef7406b4d0 0x7fef7406d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7fef96ed1640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef900d2440 0x7fef900c93d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7fef96ed1640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef900d1650 0x7fef900c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7fef96ed1640 1 --2- 172.21.15.99:0/3122585933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9006d900 0x7fef900cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 >> 172.21.15.99:0/3122585933 conn(0x7fef9009cbd0 msgr2=0x7fef900b0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 shutdown_connections 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7fef96ed1640 1 -- 172.21.15.99:0/3122585933 wait complete. 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f22dc06b4d0 msgr2=0x7f22dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f22dc06b4d0 0x7f22dc06d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f22f8073040 tx=0x7f22f80744a0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 msgr2=0x7f22f00e2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f00e2660 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e7f0 tx=0x7f22e800ecb0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 shutdown_connections 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f22dc06b4d0 0x7f22dc06d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22f003a230 0x7f22f00e2660 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22f000a200 0x7f22f0177e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 --2- 172.21.15.99:0/2905053745 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22f0009e20 0x7f22f004f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.861+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 >> 172.21.15.99:0/2905053745 conn(0x7f22f00537a0 msgr2=0x7f22f003adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.862+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcd680ccd10 con 0x7fcd680cd640 2024-03-04T10:04:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.862+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 shutdown_connections 2024-03-04T10:04:04.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.862+0000 7f230052b640 1 -- 172.21.15.99:0/2905053745 wait complete. 2024-03-04T10:04:04.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.6 2024-03-04T10:04:04.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.863+0000 7fcd5e7fc640 1 -- 172.21.15.99:0/616291140 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd64058b30 con 0x7fcd680cd640 2024-03-04T10:04:04.868 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:04.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.863+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2c406b4d0 msgr2=0x7fe2c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.863+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2c406b4d0 0x7fe2c406d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009560 tx=0x7fe2dc009190 comp rx=0 tx=0).stop 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.863+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 msgr2=0x7fe2e0166190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.863+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 0x7fe2e0166190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2d000c730 tx=0x7fe2d000cbf0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 shutdown_connections 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2c406b4d0 0x7fe2c406d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2e015c3b0 0x7fe2e0166190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2e0071120 0x7fe2e0165c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 --2- 172.21.15.99:0/204065182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2e006acd0 0x7fe2e0161520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 >> 172.21.15.99:0/204065182 conn(0x7fe2e00b10c0 msgr2=0x7fe2e01651b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 shutdown_connections 2024-03-04T10:04:04.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.864+0000 7fe2e6a75640 1 -- 172.21.15.99:0/204065182 wait complete. 2024-03-04T10:04:04.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.866+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd4c06b5a0 msgr2=0x7fcd4c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.866+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd4c06b5a0 0x7fcd4c06da60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd680c6fa0 tx=0x7fcd58009040 comp rx=0 tx=0).stop 2024-03-04T10:04:04.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.866+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 msgr2=0x7fcd68154110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:04.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.866+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 0x7fcd68154110 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcd6400c770 tx=0x7fcd6400cc30 comp rx=0 tx=0).stop 2024-03-04T10:04:04.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 shutdown_connections 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd4c06b5a0 0x7fcd4c06da60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd6815f030 0x7fcd68159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd6815dee0 0x7fcd680c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 --2- 172.21.15.99:0/616291140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd680cd640 0x7fcd68154110 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 >> 172.21.15.99:0/616291140 conn(0x7fcd68099a00 msgr2=0x7fcd6815b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 shutdown_connections 2024-03-04T10:04:04.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:04.867+0000 7fcd6fba9640 1 -- 172.21.15.99:0/616291140 wait complete. 2024-03-04T10:04:04.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.3 2024-03-04T10:04:04.877 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.1 2024-03-04T10:04:04.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-03-04T10:04:04.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.0 2024-03-04T10:04:04.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.4 2024-03-04T10:04:04.888 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-03-04T10:04:04.889 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.5 2024-03-04T10:04:05.868 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-04T10:04:05.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-04T10:04:05.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-04T10:04:05.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-04T10:04:05.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-04T10:04:05.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-04T10:04:05.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-04T10:04:05.889 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-04T10:04:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.030+0000 7fe9ae31c640 1 Processor -- start 2024-03-04T10:04:06.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.031+0000 7fe9ae31c640 1 -- start start 2024-03-04T10:04:06.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.032+0000 7fe9ae31c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7490 0x7fe9a0006ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.032+0000 7fe9ae31c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe9a00e7870 0x7fe9a0007000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.032+0000 7fe9ae31c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00510b0 con 0x7fe9a0050430 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.032+0000 7fe9ae31c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a00539c0 con 0x7fe9a00e7870 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.032+0000 7fe9ae31c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a0051390 con 0x7fe9a00e7490 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.032+0000 7fe9acb19640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe9a00e7870 0x7fe9a0007000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9adb1b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe9a0050430 0x7fe9a00508a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43666/0 (socket says 172.21.15.99:43666) 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9adb1b640 1 -- 172.21.15.99:0/2709833952 learned_addr learned my addr 172.21.15.99:0/2709833952 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9ad31a640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7490 0x7fe9a0006ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9acb19640 1 -- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7490 msgr2=0x7fe9a0006ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9acb19640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7490 0x7fe9a0006ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9acb19640 1 -- 172.21.15.99:0/2709833952 >> v1:172.21.15.162:6789/0 conn(0x7fe9a0050430 legacy=0x7fe9a00508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.033+0000 7fe9acb19640 1 -- 172.21.15.99:0/2709833952 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a0051670 con 0x7fe9a00e7870 2024-03-04T10:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.034+0000 7fe9ad31a640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7490 0x7fe9a0006ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.035+0000 7fe9acb19640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7870 0x7fe9a0007000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40097a0 tx=0x7fe9a402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5a5406e11c0b7e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.036+0000 7fe99ffff640 1 -- 172.21.15.99:0/2709833952 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4037070 con 0x7fe9a00e7870 2024-03-04T10:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.036+0000 7fe99ffff640 1 -- 172.21.15.99:0/2709833952 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a4030710 con 0x7fe9a00e7870 2024-03-04T10:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7870 msgr2=0x7fe9a0007000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7870 0x7fe9a0007000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40097a0 tx=0x7fe9a402b790 comp rx=0 tx=0).stop 2024-03-04T10:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2709833952 shutdown_connections 2024-03-04T10:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7870 0x7fe9a0007000 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 --2- 172.21.15.99:0/2709833952 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7490 0x7fe9a0006ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2709833952 >> 172.21.15.99:0/2709833952 conn(0x7fe9a0055bc0 msgr2=0x7fe9a0055fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.037+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2709833952 shutdown_connections 2024-03-04T10:04:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.038+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2709833952 wait complete. 2024-03-04T10:04:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.039+0000 7fe9ae31c640 1 Processor -- start 2024-03-04T10:04:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.040+0000 7fe9ae31c640 1 -- start start 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.041+0000 7fe9ae31c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9a0050430 0x7fe9a00cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.041+0000 7fe9ae31c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7490 0x7fe9a004bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.041+0000 7fe9ae31c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 0x7fe9a00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.041+0000 7fe9ae31c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a0054220 con 0x7fe9a0050430 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9ae31c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a00540a0 con 0x7fe9a00e7490 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9ae31c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a00543a0 con 0x7fe9a00e7870 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9adb1b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 0x7fe9a00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9adb1b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 0x7fe9a00cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43106/0 (socket says 172.21.15.99:43106) 2024-03-04T10:04:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9adb1b640 1 -- 172.21.15.99:0/2280456640 learned_addr learned my addr 172.21.15.99:0/2280456640 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9adb1b640 1 -- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7490 msgr2=0x7fe9a004bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9adb1b640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7490 0x7fe9a004bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.042+0000 7fe9adb1b640 1 -- 172.21.15.99:0/2280456640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9a0050430 msgr2=0x7fe9a00cd910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T10:04:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.043+0000 7fe9ad31a640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9a0050430 0x7fe9a00cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.043+0000 7fe9adb1b640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9a0050430 0x7fe9a00cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.043+0000 7fe9adb1b640 1 -- 172.21.15.99:0/2280456640 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a00cdef0 con 0x7fe9a00e7870 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.043+0000 7fe9adb1b640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 0x7fe9a00cf0b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8069430 tx=0x7fe9a8073b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.043+0000 7fe9ad31a640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9a0050430 0x7fe9a00cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.044+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a80731c0 con 0x7fe9a00e7870 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.044+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a808f020 con 0x7fe9a00e7870 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.044+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a806c350 con 0x7fe9a00e7870 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.044+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2280456640 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a00ce1d0 con 0x7fe9a00e7870 2024-03-04T10:04:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.044+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2280456640 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a004c130 con 0x7fe9a00e7870 2024-03-04T10:04:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.047+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2280456640 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe988005290 con 0x7fe9a00e7870 2024-03-04T10:04:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.048+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe9a806c4f0 con 0x7fe9a00e7870 2024-03-04T10:04:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.049+0000 7fe99dffb640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe98c06b5a0 0x7fe98c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.049+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe9a8104d80 con 0x7fe9a00e7870 2024-03-04T10:04:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.049+0000 7fe9ad31a640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe98c06b5a0 0x7fe98c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.050+0000 7fe9ad31a640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe98c06b5a0 0x7fe98c06da60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00d0910 tx=0x7fe998008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.053+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe9a8099030 con 0x7fe9a00e7870 2024-03-04T10:04:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.083+0000 7f3bbbaca640 1 Processor -- start 2024-03-04T10:04:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.084+0000 7f3bbbaca640 1 -- start start 2024-03-04T10:04:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bbbaca640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3bb406d190 0x7f3bb406b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bbbaca640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3bb406b630 0x7f3bb40b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bbbaca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb40b3cb0 0x7f3bb4071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bbbaca640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb416aa70 con 0x7f3bb406d190 2024-03-04T10:04:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bbbaca640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb406ea80 con 0x7f3bb406b630 2024-03-04T10:04:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bbbaca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40b4120 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f2ab212e640 1 Processor -- start 2024-03-04T10:04:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bb9866640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3bb406d190 0x7f3bb406b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f2ab212e640 1 -- start start 2024-03-04T10:04:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bb9866640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3bb406d190 0x7f3bb406b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38216/0 (socket says 172.21.15.99:38216) 2024-03-04T10:04:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.085+0000 7f3bb9866640 1 -- 172.21.15.99:0/1097237537 learned_addr learned my addr 172.21.15.99:0/1097237537 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f3bb9065640 1 --2- 172.21.15.99:0/1097237537 >> v2:172.21.15.162:3300/0 conn(0x7f3bb406b630 0x7f3bb40b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f3bb9866640 1 -- 172.21.15.99:0/1097237537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb40b3cb0 msgr2=0x7f3bb4071160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T10:04:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f3bb9866640 1 --2- 172.21.15.99:0/1097237537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb40b3cb0 0x7f3bb4071160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f3bb9866640 1 -- 172.21.15.99:0/1097237537 >> v2:172.21.15.162:3300/0 conn(0x7f3bb406b630 msgr2=0x7f3bb40b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f3bb9866640 1 --2- 172.21.15.99:0/1097237537 >> v2:172.21.15.162:3300/0 conn(0x7f3bb406b630 0x7f3bb40b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f3bb9866640 1 -- 172.21.15.99:0/1097237537 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb406d9e0 con 0x7f3bb406d190 2024-03-04T10:04:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7fe97711b640 1 Processor -- start 2024-03-04T10:04:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7fe97711b640 1 -- start start 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.086+0000 7f2ab212e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2aa40e0e30 0x7f2aa4048600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab212e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2aa4048b40 0x7f2aa404fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab112c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2aa40e0e30 0x7f2aa4048600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab092b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2aa4048b40 0x7f2aa404fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab092b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2aa4048b40 0x7f2aa404fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38224/0 (socket says 172.21.15.99:38224) 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab092b640 1 -- 172.21.15.99:0/215219434 learned_addr learned my addr 172.21.15.99:0/215219434 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab212e640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa4005e80 con 0x7f2aa4048b40 2024-03-04T10:04:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa40060d0 con 0x7f2aa40e0e30 2024-03-04T10:04:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f3bb9866640 1 --2- 172.21.15.99:0/1097237537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb406d190 0x7f3bb406b0f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40674a0 tx=0x7f3ba402b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c05c52e6db996d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa40063b0 con 0x7f2aa4050180 2024-03-04T10:04:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f2ab192d640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe97711b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe96803c090 0x7fe9680e7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe97711b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 0x7fe968054130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe97711b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe968054670 con 0x7fe9680e3150 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe97711b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe968007b60 con 0x7fe96803c090 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe97711b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe968058800 con 0x7fe968053ca0 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 0x7fe968054130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 0x7fe968054130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43120/0 (socket says 172.21.15.99:43120) 2024-03-04T10:04:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 -- 172.21.15.99:0/2495725874 learned_addr learned my addr 172.21.15.99:0/2495725874 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe96ffff640 1 --2- 172.21.15.99:0/2495725874 >> v2:172.21.15.162:3300/0 conn(0x7fe96803c090 0x7fe9680e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 -- 172.21.15.99:0/2495725874 >> v2:172.21.15.162:3300/0 conn(0x7fe96803c090 msgr2=0x7fe9680e7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 --2- 172.21.15.99:0/2495725874 >> v2:172.21.15.162:3300/0 conn(0x7fe96803c090 0x7fe9680e7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 -- 172.21.15.99:0/2495725874 >> v1:172.21.15.99:6789/0 conn(0x7fe9680e3150 legacy=0x7fe968058040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fe9756b8640 1 -- 172.21.15.99:0/2495725874 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968058ae0 con 0x7fe968053ca0 2024-03-04T10:04:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fc014f7e640 1 Processor -- start 2024-03-04T10:04:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fc014f7e640 1 -- start start 2024-03-04T10:04:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7fc014f7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 0x7fc0100cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7fc014f7e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100cf330 con 0x7fc010161cf0 2024-03-04T10:04:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7fc014f7e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0100cf610 con 0x7fc0100c34f0 2024-03-04T10:04:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7fc014f7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100cf8f0 con 0x7fc0101620d0 2024-03-04T10:04:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.087+0000 7f3bb8864640 1 -- 172.21.15.99:0/1097237537 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4034070 con 0x7f3bb406d190 2024-03-04T10:04:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f3bb8864640 1 -- 172.21.15.99:0/1097237537 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4030880 con 0x7f3bb406d190 2024-03-04T10:04:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f3bbbaca640 1 -- 172.21.15.99:0/1097237537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb406d190 msgr2=0x7f3bb406b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/1097237537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb406d190 0x7f3bb406b0f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3bb40674a0 tx=0x7f3ba402b9c0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f2ab092b640 1 -- 172.21.15.99:0/215219434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 msgr2=0x7f2aa4006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f2ab092b640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa4006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f2ab092b640 1 -- 172.21.15.99:0/215219434 >> v2:172.21.15.162:3300/0 conn(0x7f2aa40e0e30 msgr2=0x7f2aa4048600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f2ab092b640 1 --2- 172.21.15.99:0/215219434 >> v2:172.21.15.162:3300/0 conn(0x7f2aa40e0e30 0x7f2aa4048600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7f2ab092b640 1 -- 172.21.15.99:0/215219434 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa4006690 con 0x7f2aa4048b40 2024-03-04T10:04:06.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fa623b40640 1 Processor -- start 2024-03-04T10:04:06.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.088+0000 7fa623b40640 1 -- start start 2024-03-04T10:04:06.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fc00dd9b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 0x7fc0100cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fc00dd9b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 0x7fc0100cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43130/0 (socket says 172.21.15.99:43130) 2024-03-04T10:04:06.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fc00dd9b640 1 -- 172.21.15.99:0/4031298901 learned_addr learned my addr 172.21.15.99:0/4031298901 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fe9756b8640 1 --2- 172.21.15.99:0/2495725874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 0x7fe968054130 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe97002d440 tx=0x7fe970068aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3469d8ff50e074d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7f7647fff640 1 Processor -- start 2024-03-04T10:04:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fa623b40640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6140df030 0x7fa614038170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7f7647fff640 1 -- start start 2024-03-04T10:04:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 -- 172.21.15.99:0/1097237537 shutdown_connections 2024-03-04T10:04:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/1097237537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb40b3cb0 0x7f3bb4071160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/1097237537 >> v2:172.21.15.162:3300/0 conn(0x7f3bb406b630 0x7f3bb40b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/1097237537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb406d190 0x7f3bb406b0f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 -- 172.21.15.99:0/1097237537 >> 172.21.15.99:0/1097237537 conn(0x7f3bb40b1550 msgr2=0x7f3bb40b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 -- 172.21.15.99:0/1097237537 shutdown_connections 2024-03-04T10:04:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f3bbbaca640 1 -- 172.21.15.99:0/1097237537 wait complete. 2024-03-04T10:04:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.089+0000 7f2ab092b640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa4048b40 0x7f2aa404fc40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4007a50 tx=0x7f2aa802fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f28655525881ac41 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7f2aa3fff640 1 -- 172.21.15.99:0/215219434 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa803d070 con 0x7f2aa4048b40 2024-03-04T10:04:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7f2aa3fff640 1 -- 172.21.15.99:0/215219434 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa8030a70 con 0x7f2aa4048b40 2024-03-04T10:04:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7f2aa3fff640 1 -- 172.21.15.99:0/215219434 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8038440 con 0x7f2aa4048b40 2024-03-04T10:04:06.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa623b40640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6140df410 0x7fa61404fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa623b40640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa6140546c0 0x7fa61404ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa6218dc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6140df030 0x7fa614038170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fc00dd9b640 1 -- 172.21.15.99:0/4031298901 >> v1:172.21.15.162:6789/0 conn(0x7fc0100c34f0 legacy=0x7fc01006e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fc00dd9b640 1 -- 172.21.15.99:0/4031298901 >> v1:172.21.15.99:6789/0 conn(0x7fc010161cf0 legacy=0x7fc010166520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.090+0000 7fc00dd9b640 1 -- 172.21.15.99:0/4031298901 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100cfbd0 con 0x7fc0101620d0 2024-03-04T10:04:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa6218dc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6140df030 0x7fa614038170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38226/0 (socket says 172.21.15.99:38226) 2024-03-04T10:04:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f3bbbaca640 1 Processor -- start 2024-03-04T10:04:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa4048b40 msgr2=0x7f2aa404fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa4048b40 0x7f2aa404fc40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4007a50 tx=0x7f2aa802fcc0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f3bbbaca640 1 -- start start 2024-03-04T10:04:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa6218dc640 1 -- 172.21.15.99:0/3561705181 learned_addr learned my addr 172.21.15.99:0/3561705181 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 shutdown_connections 2024-03-04T10:04:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa4006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 --2- 172.21.15.99:0/215219434 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa4048b40 0x7f2aa404fc40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 --2- 172.21.15.99:0/215219434 >> v2:172.21.15.162:3300/0 conn(0x7f2aa40e0e30 0x7f2aa4048600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 >> 172.21.15.99:0/215219434 conn(0x7f2aa4009fa0 msgr2=0x7f2aa400a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa6210db640 1 --2- 172.21.15.99:0/3561705181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6140df410 0x7fa61404fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa6220dd640 1 --2- 172.21.15.99:0/3561705181 >> v2:172.21.15.162:3300/0 conn(0x7fa6140546c0 0x7fa61404ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 shutdown_connections 2024-03-04T10:04:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa614005f60 con 0x7fa6140df030 2024-03-04T10:04:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f2ab212e640 1 -- 172.21.15.99:0/215219434 wait complete. 2024-03-04T10:04:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6140e38e0 con 0x7fa6140546c0 2024-03-04T10:04:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f7647fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7648161fc0 0x7f764806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa614006200 con 0x7fa6140df410 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f7647fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f76480719b0 0x7f764806b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f7647fff640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764806d720 con 0x7f764806e4e0 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f7647fff640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764806da00 con 0x7f7648161fc0 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f7647fff640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764806dce0 con 0x7f76480719b0 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.091+0000 7f7646ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7648161fc0 0x7f764806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76467fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f764806e4e0 0x7f76480712a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43678/0 (socket says 172.21.15.99:43678) 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76467fc640 1 -- 172.21.15.99:0/997218525 learned_addr learned my addr 172.21.15.99:0/997218525 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76477fe640 1 --2- 172.21.15.99:0/997218525 >> v2:172.21.15.162:3300/0 conn(0x7f76480719b0 0x7f764806b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fe96f7fe640 1 -- 172.21.15.99:0/2495725874 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97006ce60 con 0x7fe968053ca0 2024-03-04T10:04:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fe96f7fe640 1 -- 172.21.15.99:0/2495725874 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9700ac660 con 0x7fe968053ca0 2024-03-04T10:04:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fe96f7fe640 1 -- 172.21.15.99:0/2495725874 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9700ac8d0 con 0x7fe968053ca0 2024-03-04T10:04:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fc00dd9b640 1 --2- 172.21.15.99:0/4031298901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 0x7fc0100cec50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc008009e80 tx=0x7fc00802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47f7a0bc04d5fac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f7645ffb640 1 -- 172.21.15.99:0/997218525 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535772129 0 0) 0x7f764806d720 con 0x7f764806e4e0 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f7645ffb640 1 -- 172.21.15.99:0/997218525 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7630003520 con 0x7f764806e4e0 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76477fe640 1 -- 172.21.15.99:0/997218525 >> v2:172.21.15.99:3300/0 conn(0x7f7648161fc0 msgr2=0x7f764806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76477fe640 1 --2- 172.21.15.99:0/997218525 >> v2:172.21.15.99:3300/0 conn(0x7f7648161fc0 0x7f764806acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76477fe640 1 -- 172.21.15.99:0/997218525 >> v1:172.21.15.162:6789/0 conn(0x7f764806e4e0 legacy=0x7f76480712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f76477fe640 1 -- 172.21.15.99:0/997218525 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764806dfc0 con 0x7f76480719b0 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fa6210db640 1 -- 172.21.15.99:0/3561705181 >> v2:172.21.15.162:3300/0 conn(0x7fa6140546c0 msgr2=0x7fa61404ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fa6210db640 1 --2- 172.21.15.99:0/3561705181 >> v2:172.21.15.162:3300/0 conn(0x7fa6140546c0 0x7fa61404ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fa6210db640 1 -- 172.21.15.99:0/3561705181 >> v2:172.21.15.99:3300/0 conn(0x7fa6140df030 msgr2=0x7fa614038170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fa6210db640 1 --2- 172.21.15.99:0/3561705181 >> v2:172.21.15.99:3300/0 conn(0x7fa6140df030 0x7fa614038170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7fa6210db640 1 -- 172.21.15.99:0/3561705181 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6140064e0 con 0x7fa6140df410 2024-03-04T10:04:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f3bbbaca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb406b630 0x7f3bb40a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f3bbbaca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 0x7f3bb40a9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f3bbbaca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 0x7f3bb4214d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.092+0000 7f3bb9065640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 0x7f3bb40a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bb9065640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 0x7f3bb40a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39508/0 (socket says 172.21.15.99:39508) 2024-03-04T10:04:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 -- 172.21.15.99:0/2495725874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 msgr2=0x7fe968054130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 --2- 172.21.15.99:0/2495725874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 0x7fe968054130 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe97002d440 tx=0x7fe970068aa0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fc00d59a640 1 -- 172.21.15.99:0/4031298901 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00802ca50 con 0x7fc0101620d0 2024-03-04T10:04:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fc00d59a640 1 -- 172.21.15.99:0/4031298901 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00802cbf0 con 0x7fc0101620d0 2024-03-04T10:04:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fc00d59a640 1 -- 172.21.15.99:0/4031298901 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0080343f0 con 0x7fc0101620d0 2024-03-04T10:04:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fa6220dd640 1 --2- 172.21.15.99:0/3561705181 >> v2:172.21.15.162:3300/0 conn(0x7fa6140546c0 0x7fa61404ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 -- 172.21.15.99:0/2495725874 shutdown_connections 2024-03-04T10:04:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 --2- 172.21.15.99:0/2495725874 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe968053ca0 0x7fe968054130 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 --2- 172.21.15.99:0/2495725874 >> v2:172.21.15.162:3300/0 conn(0x7fe96803c090 0x7fe9680e7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 -- 172.21.15.99:0/2495725874 >> 172.21.15.99:0/2495725874 conn(0x7fe968058ed0 msgr2=0x7fe9680592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bb9866640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb406b630 0x7f3bb40a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bb9866640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb406b630 0x7f3bb40a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43144/0 (socket says 172.21.15.99:43144) 2024-03-04T10:04:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bba067640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 0x7f3bb4214d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bba067640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 0x7f3bb4214d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38254/0 (socket says 172.21.15.99:38254) 2024-03-04T10:04:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f6732569640 1 Processor -- start 2024-03-04T10:04:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bb9065640 1 -- 172.21.15.99:0/2867513143 learned_addr learned my addr 172.21.15.99:0/2867513143 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f6732569640 1 -- start start 2024-03-04T10:04:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb406e280 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb406dfa0 con 0x7f3bb406d190 2024-03-04T10:04:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb406e560 con 0x7f3bb406b630 2024-03-04T10:04:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f2ab212e640 1 Processor -- start 2024-03-04T10:04:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f76477fe640 1 --2- 172.21.15.99:0/997218525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f764806b210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f76480d0390 tx=0x7f764002be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c66470493e812c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7fe97711b640 1 -- 172.21.15.99:0/2495725874 shutdown_connections 2024-03-04T10:04:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.093+0000 7f2ab212e640 1 -- start start 2024-03-04T10:04:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fa6210db640 1 --2- 172.21.15.99:0/3561705181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6140df410 0x7fa61404fa30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa6140485c0 tx=0x7fa61802c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=742daae7f3a9de7a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f7645ffb640 1 -- 172.21.15.99:0/997218525 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640039070 con 0x7f76480719b0 2024-03-04T10:04:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f7645ffb640 1 -- 172.21.15.99:0/997218525 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764002cbe0 con 0x7f76480719b0 2024-03-04T10:04:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f7645ffb640 1 -- 172.21.15.99:0/997218525 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640034440 con 0x7f76480719b0 2024-03-04T10:04:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 -- 172.21.15.99:0/4031298901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 msgr2=0x7fc0100cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 --2- 172.21.15.99:0/4031298901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 0x7fc0100cec50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc008009e80 tx=0x7fc00802beb0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f3bba067640 1 -- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb406b630 msgr2=0x7f3bb40a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f3bba067640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb406b630 0x7f3bb40a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f3bba067640 1 -- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 msgr2=0x7f3bb40a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f3bba067640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 0x7f3bb40a9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f3bba067640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40a9ba0 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 -- 172.21.15.99:0/4031298901 shutdown_connections 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fa6208da640 1 -- 172.21.15.99:0/3561705181 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618035070 con 0x7fa6140df410 2024-03-04T10:04:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fa6208da640 1 -- 172.21.15.99:0/3561705181 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618031830 con 0x7fa6140df410 2024-03-04T10:04:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 --2- 172.21.15.99:0/4031298901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0101620d0 0x7fc0100cec50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 -- 172.21.15.99:0/4031298901 >> 172.21.15.99:0/4031298901 conn(0x7fc010099a00 msgr2=0x7fc0100d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f6732569640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 0x7f672c071e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f6732569640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c0acda0 0x7f672c0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f6732569640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f672c0cf930 0x7f672c1617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fa6208da640 1 -- 172.21.15.99:0/3561705181 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618031c00 con 0x7fa6140df410 2024-03-04T10:04:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f6732569640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c0cfda0 con 0x7f672c15db80 2024-03-04T10:04:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f6732569640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c0ad560 con 0x7f672c0cf930 2024-03-04T10:04:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f6732569640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0ad840 con 0x7f672c0acda0 2024-03-04T10:04:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f3bba067640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 0x7f3bb4214d30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00bfa0 tx=0x7f3bac00c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 -- 172.21.15.99:0/997218525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 msgr2=0x7f764806b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 --2- 172.21.15.99:0/997218525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f764806b210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f76480d0390 tx=0x7f764002be30 comp rx=0 tx=0).stop 2024-03-04T10:04:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f672bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 0x7f672c071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f3bb9065640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 0x7f3bb40a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac019070 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 -- 172.21.15.99:0/997218525 shutdown_connections 2024-03-04T10:04:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 --2- 172.21.15.99:0/997218525 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f764806b210 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 --2- 172.21.15.99:0/997218525 >> v2:172.21.15.99:3300/0 conn(0x7f7648161fc0 0x7f764806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 -- 172.21.15.99:0/997218525 >> 172.21.15.99:0/997218525 conn(0x7f76480cf3f0 msgr2=0x7f76480cf7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 -- 172.21.15.99:0/4031298901 shutdown_connections 2024-03-04T10:04:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7fc014f7e640 1 -- 172.21.15.99:0/4031298901 wait complete. 2024-03-04T10:04:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab212e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 0x7f2aa40dd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb40ab1d0 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.094+0000 7f672bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 0x7f672c071e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38260/0 (socket says 172.21.15.99:38260) 2024-03-04T10:04:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 -- 172.21.15.99:0/997218525 shutdown_connections 2024-03-04T10:04:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f672bfff640 1 -- 172.21.15.99:0/3765583345 learned_addr learned my addr 172.21.15.99:0/3765583345 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab212e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa40d2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f672b7fe640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c0acda0 0x7f672c0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f6730b06640 1 --2- 172.21.15.99:0/3765583345 >> v2:172.21.15.162:3300/0 conn(0x7f672c0cf930 0x7f672c1617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab112c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 0x7f2aa40dd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab112c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 0x7f2aa40dd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39524/0 (socket says 172.21.15.99:39524) 2024-03-04T10:04:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab112c640 1 -- 172.21.15.99:0/411733136 learned_addr learned my addr 172.21.15.99:0/411733136 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa40e0e30 0x7f2aa40df6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f6730b06640 1 -- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c0acda0 msgr2=0x7f672c0cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f6730b06640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c0acda0 0x7f672c0cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f6730b06640 1 -- 172.21.15.99:0/3765583345 >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 msgr2=0x7f672c071e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f6730b06640 1 --2- 172.21.15.99:0/3765583345 >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 0x7f672c071e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f6730b06640 1 -- 172.21.15.99:0/3765583345 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c0adb20 con 0x7f672c0cf930 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6140df410 msgr2=0x7fa61404fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 --2- 172.21.15.99:0/3561705181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6140df410 0x7fa61404fa30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa6140485c0 tx=0x7fa61802c970 comp rx=0 tx=0).stop 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 shutdown_connections 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 --2- 172.21.15.99:0/3561705181 >> v2:172.21.15.162:3300/0 conn(0x7fa6140546c0 0x7fa61404ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 --2- 172.21.15.99:0/3561705181 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6140df410 0x7fa61404fa30 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa6140485c0 tx=0x7fa61802c970 comp rx=0 tx=0).stop 2024-03-04T10:04:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 --2- 172.21.15.99:0/3561705181 >> v2:172.21.15.99:3300/0 conn(0x7fa6140df030 0x7fa614038170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 >> 172.21.15.99:0/3561705181 conn(0x7fa614009fe0 msgr2=0x7fa61400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f672b7fe640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c0acda0 0x7f672c0cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7fe97711b640 1 -- 172.21.15.99:0/2495725874 wait complete. 2024-03-04T10:04:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f672bfff640 1 --2- 172.21.15.99:0/3765583345 >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 0x7f672c071e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f7647fff640 1 -- 172.21.15.99:0/997218525 wait complete. 2024-03-04T10:04:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.095+0000 7f2ab092b640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa40d2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa40510c0 con 0x7f2aa40e0e30 2024-03-04T10:04:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4050f40 con 0x7f2aa4048b40 2024-03-04T10:04:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa40e1210 con 0x7f2aa4050180 2024-03-04T10:04:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 shutdown_connections 2024-03-04T10:04:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7fa623b40640 1 -- 172.21.15.99:0/3561705181 wait complete. 2024-03-04T10:04:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac014440 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb40ab790 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac014830 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7f2ab192d640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa40e0e30 0x7f2aa40df6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7fc014f7e640 1 Processor -- start 2024-03-04T10:04:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.096+0000 7fc014f7e640 1 -- start start 2024-03-04T10:04:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab192d640 1 -- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 msgr2=0x7f2aa40d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab192d640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa40d2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab192d640 1 -- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 msgr2=0x7f2aa40dd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab192d640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 0x7f2aa40dd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab192d640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa40d3f40 con 0x7f2aa40e0e30 2024-03-04T10:04:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f7647fff640 1 Processor -- start 2024-03-04T10:04:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab192d640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa40e0e30 0x7f2aa40df6a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00e7c0 tx=0x7f2a9c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f7647fff640 1 -- start start 2024-03-04T10:04:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2ab112c640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 0x7f2aa40dd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c017070 con 0x7f2aa40e0e30 2024-03-04T10:04:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7fe97711b640 1 Processor -- start 2024-03-04T10:04:06.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.097+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c00ee70 con 0x7f2aa40e0e30 2024-03-04T10:04:06.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c00cb40 con 0x7f2aa40e0e30 2024-03-04T10:04:06.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc014f7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 0x7fc0100b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa40d36a0 con 0x7f2aa40e0e30 2024-03-04T10:04:06.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fa623b40640 1 Processor -- start 2024-03-04T10:04:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc014f7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc010161cf0 0x7fc01006f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f6730b06640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c1617b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f672c071110 tx=0x7f672402be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cecf21fbc64ccfc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fa623b40640 1 -- start start 2024-03-04T10:04:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc00e59c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 0x7fc0100b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc00e59c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 0x7fc0100b5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43158/0 (socket says 172.21.15.99:43158) 2024-03-04T10:04:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa40d3c60 con 0x7f2aa40e0e30 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc00e59c640 1 -- 172.21.15.99:0/1332531908 learned_addr learned my addr 172.21.15.99:0/1332531908 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc014f7e640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0101620d0 0x7fc0101f82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc00dd9b640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc010161cf0 0x7fc01006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc00ed9d640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0101620d0 0x7fc0101f82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc014f7e640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100c3ae0 con 0x7fc0101620d0 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc014f7e640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100c3960 con 0x7fc010161cf0 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7fc014f7e640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0100c3c60 con 0x7fc0100c34f0 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7fc00e59c640 1 -- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc010161cf0 msgr2=0x7fc01006f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7fc00e59c640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc010161cf0 0x7fc01006f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7fc00e59c640 1 -- 172.21.15.99:0/1332531908 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0101620d0 msgr2=0x7fc0101f82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7fc00e59c640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0101620d0 0x7fc0101f82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7fc00e59c640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0100b3580 con 0x7fc0100c34f0 2024-03-04T10:04:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7fc00e59c640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 0x7fc0100b5380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc00400c430 tx=0x7fc00400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3bac01c070 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f3baa7fc640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b9806b5a0 0x7f3b9806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f3bb9866640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b9806b5a0 0x7f3b9806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f7647fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f764806e4e0 0x7f76481f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f7647fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f76480b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f764806e4e0 0x7f76481f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f764806e4e0 0x7f76481f6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:43164/0 (socket says 172.21.15.99:43164) 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7646ffd640 1 -- 172.21.15.99:0/3467466333 learned_addr learned my addr 172.21.15.99:0/3467466333 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648161fc0 0x7f7648159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480c9bc0 con 0x7f7648161fc0 2024-03-04T10:04:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480c9a40 con 0x7f76480719b0 2024-03-04T10:04:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480c9d40 con 0x7f764806e4e0 2024-03-04T10:04:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76467fc640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f76480b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76477fe640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648161fc0 0x7f7648159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76467fc640 1 -- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f764806e4e0 msgr2=0x7f76481f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76467fc640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f764806e4e0 0x7f76481f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f672affd640 1 -- 172.21.15.99:0/3765583345 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724039070 con 0x7f672c0cf930 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f672affd640 1 -- 172.21.15.99:0/3765583345 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672402cbe0 con 0x7f672c0cf930 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.098+0000 7f672affd640 1 -- 172.21.15.99:0/3765583345 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724034440 con 0x7f672c0cf930 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 -- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 msgr2=0x7f672c1617b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c1617b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f672c071110 tx=0x7f672402be30 comp rx=0 tx=0).stop 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 -- 172.21.15.99:0/3765583345 shutdown_connections 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c1617b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 --2- 172.21.15.99:0/3765583345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c0acda0 0x7f672c0cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 --2- 172.21.15.99:0/3765583345 >> v2:172.21.15.99:3300/0 conn(0x7f672c15db80 0x7f672c071e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f6732569640 1 -- 172.21.15.99:0/3765583345 >> 172.21.15.99:0/3765583345 conn(0x7f672c06acd0 msgr2=0x7f672c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76467fc640 1 -- 172.21.15.99:0/3467466333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648161fc0 msgr2=0x7f7648159fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76467fc640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648161fc0 0x7f7648159fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76467fc640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7648163ec0 con 0x7f76480719b0 2024-03-04T10:04:06.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f2ab092b640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa40d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:06.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.099+0000 7f76477fe640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648161fc0 0x7f7648159fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:06.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fe97711b640 1 -- start start 2024-03-04T10:04:06.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004011790 con 0x7fc0100c34f0 2024-03-04T10:04:06.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f76467fc640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f76480b2570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f763c00c400 tx=0x7f763c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2a9c01a070 con 0x7f2aa40e0e30 2024-03-04T10:04:06.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f6732569640 1 -- 172.21.15.99:0/3765583345 shutdown_connections 2024-03-04T10:04:06.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f6732569640 1 -- 172.21.15.99:0/3765583345 wait complete. 2024-03-04T10:04:06.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004017070 con 0x7fc0100c34f0 2024-03-04T10:04:06.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fc014f7e640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0100b3860 con 0x7fc0100c34f0 2024-03-04T10:04:06.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c005290 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f3bb9866640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b9806b5a0 0x7f3b9806da60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4036000 tx=0x7f3ba4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c017070 con 0x7f76480719b0 2024-03-04T10:04:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3bac08a3f0 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764815aa80 con 0x7f76480719b0 2024-03-04T10:04:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fc014f7e640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0100b58c0 con 0x7fc0100c34f0 2024-03-04T10:04:06.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f2aa1ffb640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2a9006b520 0x7f2a9006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7648164160 con 0x7f76480719b0 2024-03-04T10:04:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2a9c089f50 con 0x7f2aa40e0e30 2024-03-04T10:04:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c005ce0 con 0x7f76480719b0 2024-03-04T10:04:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0072f0 con 0x7f76480719b0 2024-03-04T10:04:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004011bb0 con 0x7fc0100c34f0 2024-03-04T10:04:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fa623b40640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 0x7fa6140d3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fa623b40640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6140df030 0x7fa614039820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fa623b40640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa614039d60 0x7fa6140d5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fa623b40640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6140507c0 con 0x7fa6140df030 2024-03-04T10:04:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fa623b40640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa614050640 con 0x7fa6140546c0 2024-03-04T10:04:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.100+0000 7fa623b40640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61400c0d0 con 0x7fa614039d60 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c005290 con 0x7f2aa40e0e30 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7f2ab112c640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2a9006b520 0x7f2a9006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7f2ab112c640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2a9006b520 0x7f2a9006d9e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2aac069100 tx=0x7f2aac08af90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fe97711b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe96803c090 0x7fe96816e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fe97711b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe968053ca0 0x7fe9680e56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fe97711b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9680e3150 0x7fe96804eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fe97711b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9680e3530 con 0x7fe968053ca0 2024-03-04T10:04:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fe97711b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe968006630 con 0x7fe96803c090 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.101+0000 7fe97711b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe968006b50 con 0x7fe9680e3150 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7f6732569640 1 Processor -- start 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7f6732569640 1 -- start start 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fa6220dd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa614039d60 0x7fa6140d5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fa6218dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 0x7fa6140d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fa6218dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 0x7fa6140d3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39552/0 (socket says 172.21.15.99:39552) 2024-03-04T10:04:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fa6218dc640 1 -- 172.21.15.99:0/1784135538 learned_addr learned my addr 172.21.15.99:0/1784135538 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fe974eb7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe96803c090 0x7fe96816e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fe974eb7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe96803c090 0x7fe96816e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39556/0 (socket says 172.21.15.99:39556) 2024-03-04T10:04:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fe974eb7640 1 -- 172.21.15.99:0/510119638 learned_addr learned my addr 172.21.15.99:0/510119638 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f763c01a070 con 0x7f76480719b0 2024-03-04T10:04:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628005290 con 0x7f76480719b0 2024-03-04T10:04:06.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.102+0000 7fe9756b8640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9680e3150 0x7fe96804eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fa6218dc640 1 -- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa614039d60 msgr2=0x7fa6140d5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fa6218dc640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa614039d60 0x7fa6140d5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fa6218dc640 1 -- 172.21.15.99:0/1784135538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6140df030 msgr2=0x7fa614039820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fa6218dc640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6140df030 0x7fa614039820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fa6218dc640 1 -- 172.21.15.99:0/1784135538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6140d66b0 con 0x7fa6140546c0 2024-03-04T10:04:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fa6218dc640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 0x7fa6140d3180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00e7b0 tx=0x7fa60c00ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fe96ffff640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe968053ca0 0x7fe9680e56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc00401c460 con 0x7fc0100c34f0 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f762f7fe640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f763406b520 0x7f763406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fe9756b8640 1 -- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe96803c090 msgr2=0x7fe96816e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fe9756b8640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe96803c090 0x7fe96816e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fe9756b8640 1 -- 172.21.15.99:0/510119638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe968053ca0 msgr2=0x7fe9680e56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fe9756b8640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe968053ca0 0x7fe9680e56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7fe9756b8640 1 -- 172.21.15.99:0/510119638 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96804cf30 con 0x7fe9680e3150 2024-03-04T10:04:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f763c089700 con 0x7f76480719b0 2024-03-04T10:04:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f6732569640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 0x7f672c1477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f6732569640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f6732569640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c15db80 0x7f672c158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f672bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 0x7f672c1477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fe9756b8640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9680e3150 0x7fe96804eac0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe97004fe40 tx=0x7fe970062b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f672bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 0x7f672c1477e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38300/0 (socket says 172.21.15.99:38300) 2024-03-04T10:04:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f672bfff640 1 -- 172.21.15.99:0/2644743497 learned_addr learned my addr 172.21.15.99:0/2644743497 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0ad200 con 0x7f672c0acda0 2024-03-04T10:04:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.103+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0ae0e0 con 0x7f672c0cf930 2024-03-04T10:04:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c15df60 con 0x7f672c15db80 2024-03-04T10:04:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f672b7fe640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c15db80 0x7f672c158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f7646ffd640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f763406b520 0x7f763406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fbffeffd640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff006b5a0 0x7fbff006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9700ba020 con 0x7fe9680e3150 2024-03-04T10:04:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc00408a390 con 0x7fc0100c34f0 2024-03-04T10:04:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01004f090 con 0x7fc0100c34f0 2024-03-04T10:04:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97006be50 con 0x7fe9680e3150 2024-03-04T10:04:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9700adc70 con 0x7fe9680e3150 2024-03-04T10:04:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96804d210 con 0x7fe9680e3150 2024-03-04T10:04:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96804d7d0 con 0x7fe9680e3150 2024-03-04T10:04:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7fc00dd9b640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff006b5a0 0x7fbff006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 -- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 msgr2=0x7f672c157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 -- 172.21.15.99:0/2644743497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 msgr2=0x7f672c1477e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 0x7f672c1477e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c158b00 con 0x7f672c15db80 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f672bfff640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 0x7f672c1477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.104+0000 7f6730b06640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c15db80 0x7f672c158130 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6724033000 tx=0x7f6724004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.105+0000 7fc00dd9b640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff006b5a0 0x7fbff006da60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc008035000 tx=0x7fc008030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.105+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe968005380 con 0x7fe9680e3150 2024-03-04T10:04:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.105+0000 7f7646ffd640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f763406b520 0x7f763406d9e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7638009b60 tx=0x7f7638009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.106+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe9700bd020 con 0x7fe9680e3150 2024-03-04T10:04:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.106+0000 7fe96d7fa640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe95406b4d0 0x7fe95406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.106+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe9700aa020 con 0x7fe9680e3150 2024-03-04T10:04:06.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.106+0000 7fe974eb7640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe95406b4d0 0x7fe95406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.107+0000 7fe974eb7640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe95406b4d0 0x7fe95406d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9680e7570 tx=0x7fe95c0090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.108+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2a9c058b20 con 0x7f2aa40e0e30 2024-03-04T10:04:06.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.108+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c018020 con 0x7fa6140546c0 2024-03-04T10:04:06.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.108+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c00c8a0 con 0x7fa6140546c0 2024-03-04T10:04:06.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.108+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c00cc90 con 0x7fa6140546c0 2024-03-04T10:04:06.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.108+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f763c007e20 con 0x7f76480719b0 2024-03-04T10:04:06.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.109+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61404d400 con 0x7fa6140546c0 2024-03-04T10:04:06.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.109+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61404d9c0 con 0x7fa6140546c0 2024-03-04T10:04:06.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.109+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724044020 con 0x7f672c15db80 2024-03-04T10:04:06.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.109+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c158de0 con 0x7f672c15db80 2024-03-04T10:04:06.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.109+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c15b9e0 con 0x7f672c15db80 2024-03-04T10:04:06.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.109+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc004058eb0 con 0x7fc0100c34f0 2024-03-04T10:04:06.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.110+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61403a170 con 0x7fa6140546c0 2024-03-04T10:04:06.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.111+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c074f60 con 0x7f672c15db80 2024-03-04T10:04:06.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.111+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa60c01b020 con 0x7fa6140546c0 2024-03-04T10:04:06.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.111+0000 7fa6127fc640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa60006b4d0 0x7fa60006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.111+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa60c08a6a0 con 0x7fa6140546c0 2024-03-04T10:04:06.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.112+0000 7fa6210db640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa60006b4d0 0x7fa60006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.112+0000 7fa6210db640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa60006b4d0 0x7fa60006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa618037000 tx=0x7fa618031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.113+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672402cbe0 con 0x7f672c15db80 2024-03-04T10:04:06.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.113+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724039040 con 0x7f672c15db80 2024-03-04T10:04:06.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.114+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe9700f5680 con 0x7fe9680e3150 2024-03-04T10:04:06.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.114+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3bac058950 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.115+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6724047020 con 0x7f672c15db80 2024-03-04T10:04:06.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.116+0000 7f6728ff9640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670c06b4d0 0x7f670c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.116+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f67240ad2e0 con 0x7f672c15db80 2024-03-04T10:04:06.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.116+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa61403a170 con 0x7fa6140546c0 2024-03-04T10:04:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.117+0000 7f672bfff640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670c06b4d0 0x7f670c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.118+0000 7f672bfff640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670c06b4d0 0x7f670c06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f671c0113d0 tx=0x7f671c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.118+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f672404e070 con 0x7f672c15db80 2024-03-04T10:04:06.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.448+0000 7fe9ae31c640 1 -- 172.21.15.99:0/2280456640 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe988005580 con 0x7fe9a00e7870 2024-03-04T10:04:06.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.449+0000 7fe99dffb640 1 -- 172.21.15.99:0/2280456640 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe9a80d38a0 con 0x7fe9a00e7870 2024-03-04T10:04:06.450 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.452+0000 7fe9937fe640 1 -- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe98c06b5a0 msgr2=0x7fe98c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.452+0000 7fe9937fe640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe98c06b5a0 0x7fe98c06da60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe9a00d0910 tx=0x7fe998008040 comp rx=0 tx=0).stop 2024-03-04T10:04:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.452+0000 7fe9937fe640 1 -- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 msgr2=0x7fe9a00cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.452+0000 7fe9937fe640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 0x7fe9a00cf0b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8069430 tx=0x7fe9a8073b70 comp rx=0 tx=0).stop 2024-03-04T10:04:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.452+0000 7fe9937fe640 1 -- 172.21.15.99:0/2280456640 shutdown_connections 2024-03-04T10:04:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.452+0000 7fe9937fe640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe98c06b5a0 0x7fe98c06da60 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.453+0000 7fe9937fe640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9a00e7870 0x7fe9a00cf0b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.453+0000 7fe9937fe640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9a00e7490 0x7fe9a004bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.453+0000 7fe9937fe640 1 --2- 172.21.15.99:0/2280456640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9a0050430 0x7fe9a00cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.453+0000 7fe9937fe640 1 -- 172.21.15.99:0/2280456640 >> 172.21.15.99:0/2280456640 conn(0x7fe9a0055bc0 msgr2=0x7fe9a0049bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.453+0000 7fe9937fe640 1 -- 172.21.15.99:0/2280456640 shutdown_connections 2024-03-04T10:04:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.453+0000 7fe9937fe640 1 -- 172.21.15.99:0/2280456640 wait complete. 2024-03-04T10:04:06.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.6 2024-03-04T10:04:06.471 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.489+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe968005380 con 0x7fe9680e3150 2024-03-04T10:04:06.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.490+0000 7fe96d7fa640 1 -- 172.21.15.99:0/510119638 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe9700f4f80 con 0x7fe9680e3150 2024-03-04T10:04:06.491 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.492+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe95406b4d0 msgr2=0x7fe95406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.492+0000 7fe97711b640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe95406b4d0 0x7fe95406d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9680e7570 tx=0x7fe95c0090e0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9680e3150 msgr2=0x7fe96804eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9680e3150 0x7fe96804eac0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe97004fe40 tx=0x7fe970062b00 comp rx=0 tx=0).stop 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 shutdown_connections 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe95406b4d0 0x7fe95406d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9680e3150 0x7fe96804eac0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe968053ca0 0x7fe9680e56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 --2- 172.21.15.99:0/510119638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe96803c090 0x7fe96816e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.493+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 >> 172.21.15.99:0/510119638 conn(0x7fe968058ed0 msgr2=0x7fe968039140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.497+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 shutdown_connections 2024-03-04T10:04:06.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.497+0000 7fe97711b640 1 -- 172.21.15.99:0/510119638 wait complete. 2024-03-04T10:04:06.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2024-03-04T10:04:06.509 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.513+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7628005850 con 0x7f76480719b0 2024-03-04T10:04:06.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.514+0000 7f762f7fe640 1 -- 172.21.15.99:0/3467466333 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f763c057ce0 con 0x7f76480719b0 2024-03-04T10:04:06.515 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f763406b520 msgr2=0x7f763406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f763406b520 0x7f763406d9e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7638009b60 tx=0x7f7638009190 comp rx=0 tx=0).stop 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 msgr2=0x7f76480b2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f76480b2570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f763c00c400 tx=0x7f763c00c8c0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 shutdown_connections 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f763406b520 0x7f763406d9e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648161fc0 0x7f7648159fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480719b0 0x7f76480b2570 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 --2- 172.21.15.99:0/3467466333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f764806e4e0 0x7f76481f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.516+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 >> 172.21.15.99:0/3467466333 conn(0x7f76480cf3f0 msgr2=0x7f764806acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.517+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 shutdown_connections 2024-03-04T10:04:06.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.517+0000 7f7647fff640 1 -- 172.21.15.99:0/3467466333 wait complete. 2024-03-04T10:04:06.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.521+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3b9c005580 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.521+0000 7f3baa7fc640 1 -- 172.21.15.99:0/2867513143 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3bac05c9f0 con 0x7f3bb40b3cb0 2024-03-04T10:04:06.522 INFO:teuthology.orchestra.run.smithi099.stdout:55834574858 2024-03-04T10:04:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.521+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2a8c005580 con 0x7f2aa40e0e30 2024-03-04T10:04:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.522+0000 7f2aa1ffb640 1 -- 172.21.15.99:0/411733136 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2a9c058420 con 0x7f2aa40e0e30 2024-03-04T10:04:06.523 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b9806b5a0 msgr2=0x7f3b9806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b9806b5a0 0x7f3b9806da60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4036000 tx=0x7f3ba4008030 comp rx=0 tx=0).stop 2024-03-04T10:04:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 msgr2=0x7f3bb4214d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 0x7f3bb4214d30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00bfa0 tx=0x7f3bac00c4d0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 shutdown_connections 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b9806b5a0 0x7f3b9806da60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3bb40b3cb0 0x7f3bb4214d30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406d190 0x7f3bb40a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 --2- 172.21.15.99:0/2867513143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3bb406b630 0x7f3bb40a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.523+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 >> 172.21.15.99:0/2867513143 conn(0x7f3bb40b1550 msgr2=0x7f3bb40cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.524+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 shutdown_connections 2024-03-04T10:04:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.524+0000 7f3bbbaca640 1 -- 172.21.15.99:0/2867513143 wait complete. 2024-03-04T10:04:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.525+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc0100b64a0 con 0x7fc0100c34f0 2024-03-04T10:04:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.525+0000 7fbffeffd640 1 -- 172.21.15.99:0/1332531908 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0040587b0 con 0x7fc0100c34f0 2024-03-04T10:04:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.525+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2a9006b520 msgr2=0x7f2a9006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.525+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2a9006b520 0x7f2a9006d9e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2aac069100 tx=0x7f2aac08af90 comp rx=0 tx=0).stop 2024-03-04T10:04:06.527 INFO:teuthology.orchestra.run.smithi099.stdout:55834574860 2024-03-04T10:04:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa40e0e30 msgr2=0x7f2aa40df6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa40e0e30 0x7f2aa40df6a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c00e7c0 tx=0x7f2a9c00ec80 comp rx=0 tx=0).stop 2024-03-04T10:04:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 shutdown_connections 2024-03-04T10:04:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2a9006b520 0x7f2a9006d9e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2aa40e0e30 0x7f2aa40df6a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2aa4050180 0x7f2aa40d2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 --2- 172.21.15.99:0/411733136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2aa4048b40 0x7f2aa40dd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 >> 172.21.15.99:0/411733136 conn(0x7f2aa4009fa0 msgr2=0x7f2aa40deb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 shutdown_connections 2024-03-04T10:04:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.526+0000 7f2ab212e640 1 -- 172.21.15.99:0/411733136 wait complete. 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff006b5a0 msgr2=0x7fbff006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff006b5a0 0x7fbff006da60 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc008035000 tx=0x7fc008030000 comp rx=0 tx=0).stop 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 msgr2=0x7fc0100b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 0x7fc0100b5380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc00400c430 tx=0x7fc00400c8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 shutdown_connections 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff006b5a0 0x7fbff006da60 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0101620d0 0x7fc0101f82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc010161cf0 0x7fc01006f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 --2- 172.21.15.99:0/1332531908 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0100c34f0 0x7fc0100b5380 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc00400c430 tx=0x7fc00400c8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 >> 172.21.15.99:0/1332531908 conn(0x7fc010099a00 msgr2=0x7fc010160fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2024-03-04T10:04:06.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 shutdown_connections 2024-03-04T10:04:06.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.529+0000 7fbffcff9640 1 -- 172.21.15.99:0/1332531908 wait complete. 2024-03-04T10:04:06.532 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.531+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f672c0b1190 con 0x7f672c15db80 2024-03-04T10:04:06.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.531+0000 7f6728ff9640 1 -- 172.21.15.99:0/2644743497 <== mon.2 v2:172.21.15.162:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f672407b7c0 con 0x7f672c15db80 2024-03-04T10:04:06.533 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:06.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.532+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa6140e27b0 con 0x7fa6140546c0 2024-03-04T10:04:06.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.532+0000 7fa6127fc640 1 -- 172.21.15.99:0/1784135538 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa60c022030 con 0x7fa6140546c0 2024-03-04T10:04:06.533 INFO:teuthology.orchestra.run.smithi099.stdout:55834574859 2024-03-04T10:04:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670c06b4d0 msgr2=0x7f670c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670c06b4d0 0x7f670c06d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f671c0113d0 tx=0x7f671c009240 comp rx=0 tx=0).stop 2024-03-04T10:04:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c15db80 msgr2=0x7f672c158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c15db80 0x7f672c158130 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6724033000 tx=0x7f6724004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 shutdown_connections 2024-03-04T10:04:06.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670c06b4d0 0x7f670c06d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f672c15db80 0x7f672c158130 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f672c0cf930 0x7f672c157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 --2- 172.21.15.99:0/2644743497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f672c0acda0 0x7f672c1477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 >> 172.21.15.99:0/2644743497 conn(0x7f672c06acd0 msgr2=0x7f672c06dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.534+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 shutdown_connections 2024-03-04T10:04:06.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.535+0000 7f6732569640 1 -- 172.21.15.99:0/2644743497 wait complete. 2024-03-04T10:04:06.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.535+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa60006b4d0 msgr2=0x7fa60006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.535+0000 7fa623b40640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa60006b4d0 0x7fa60006d990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa618037000 tx=0x7fa618031000 comp rx=0 tx=0).stop 2024-03-04T10:04:06.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.535+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 msgr2=0x7fa6140d3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.535+0000 7fa623b40640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 0x7fa6140d3180 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa60c00e7b0 tx=0x7fa60c00ebb0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 shutdown_connections 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa60006b4d0 0x7fa60006d990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa614039d60 0x7fa6140d5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6140df030 0x7fa614039820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 --2- 172.21.15.99:0/1784135538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6140546c0 0x7fa6140d3180 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 >> 172.21.15.99:0/1784135538 conn(0x7fa614009fe0 msgr2=0x7fa614035930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 shutdown_connections 2024-03-04T10:04:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.536+0000 7fa623b40640 1 -- 172.21.15.99:0/1784135538 wait complete. 2024-03-04T10:04:06.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-03-04T10:04:06.543 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-03-04T10:04:06.544 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.1 2024-03-04T10:04:06.548 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.5 2024-03-04T10:04:06.550 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-03-04T10:04:06.550 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:06.550 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-04T10:04:06.551 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:04:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 Processor -- start 2024-03-04T10:04:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 -- start start 2024-03-04T10:04:06.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8434158410 0x7f84340b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843406b500 con 0x7f8434161ca0 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84340b3f00 con 0x7f8434158410 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.689+0000 7f8439ea0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843406b7e0 con 0x7f843406acd0 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f84337fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8434158410 0x7f84340b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f8433fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8434161ca0 0x7f843415e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34858/0 (socket says 172.21.15.99:34858) 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f8433fff640 1 -- 172.21.15.99:0/3537694519 learned_addr learned my addr 172.21.15.99:0/3537694519 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f8432ffd640 1 --2- 172.21.15.99:0/3537694519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f84337fe640 1 -- 172.21.15.99:0/3537694519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 msgr2=0x7f8434161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f84337fe640 1 --2- 172.21.15.99:0/3537694519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f84337fe640 1 -- 172.21.15.99:0/3537694519 >> v1:172.21.15.99:6789/0 conn(0x7f8434161ca0 legacy=0x7f843415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f84337fe640 1 -- 172.21.15.99:0/3537694519 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843406bac0 con 0x7f8434158410 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.690+0000 7f84327fc640 1 -- 172.21.15.99:0/3537694519 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193529928 0 0) 0x7f843406b500 con 0x7f8434161ca0 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f84337fe640 1 --2- 172.21.15.99:0/3537694519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434158410 0x7f84340b39c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8424009d90 tx=0x7f842402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b71df1f54195410 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f84327fc640 1 -- 172.21.15.99:0/3537694519 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424039070 con 0x7f8434158410 2024-03-04T10:04:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f84327fc640 1 -- 172.21.15.99:0/3537694519 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842402cce0 con 0x7f8434158410 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f84327fc640 1 -- 172.21.15.99:0/3537694519 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8424034660 con 0x7f8434158410 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 -- 172.21.15.99:0/3537694519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434158410 msgr2=0x7f84340b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 --2- 172.21.15.99:0/3537694519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434158410 0x7f84340b39c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8424009d90 tx=0x7f842402bf30 comp rx=0 tx=0).stop 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 -- 172.21.15.99:0/3537694519 shutdown_connections 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 --2- 172.21.15.99:0/3537694519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434161760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 --2- 172.21.15.99:0/3537694519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434158410 0x7f84340b39c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 -- 172.21.15.99:0/3537694519 >> 172.21.15.99:0/3537694519 conn(0x7f84340cfef0 msgr2=0x7f84340d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 -- 172.21.15.99:0/3537694519 shutdown_connections 2024-03-04T10:04:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.691+0000 7f8439ea0640 1 -- 172.21.15.99:0/3537694519 wait complete. 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.692+0000 7f8439ea0640 1 Processor -- start 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 -- start start 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8434158410 0x7f843414a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 0x7f843415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843406c5a0 con 0x7f8434161ca0 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843406c080 con 0x7f8434158410 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8439ea0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843406ee60 con 0x7f843406acd0 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.693+0000 7f8433fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 0x7f843415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 0x7f843415a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38318/0 (socket says 172.21.15.99:38318) 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 -- 172.21.15.99:0/2656569297 learned_addr learned my addr 172.21.15.99:0/2656569297 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f84337fe640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8432ffd640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8434158410 0x7f843414a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 -- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 msgr2=0x7f8434149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 -- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8434158410 msgr2=0x7f843414a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8434158410 0x7f843414a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 -- 172.21.15.99:0/2656569297 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84341594d0 con 0x7f8434161ca0 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8432ffd640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8434158410 0x7f843414a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8433fff640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 0x7f843415a570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f842c00c410 tx=0x7f842c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0074e0 con 0x7f8434161ca0 2024-03-04T10:04:06.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c0047e0 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84341597b0 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c004bd0 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.694+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8434159d70 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.695+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8434074f60 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.695+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f842c011070 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.696+0000 7f841bfff640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f842006b4d0 0x7f842006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.696+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f842c08a040 con 0x7f8434161ca0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.696+0000 7f84337fe640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f842006b4d0 0x7f842006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:06.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.697+0000 7f84337fe640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f842006b4d0 0x7f842006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f8424030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:06.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.700+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f842c01a2b0 con 0x7f8434161ca0 2024-03-04T10:04:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.867+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8434066e10 con 0x7f842006b4d0 2024-03-04T10:04:06.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.876+0000 7f841bfff640 1 -- 172.21.15.99:0/2656569297 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332533 (secure 0 0 0) 0x7f8434066e10 con 0x7f842006b4d0 2024-03-04T10:04:06.880 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:06.882 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f842006b4d0 msgr2=0x7f842006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f842006b4d0 0x7f842006d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8424036000 tx=0x7f8424030000 comp rx=0 tx=0).stop 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 msgr2=0x7f843415a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 0x7f843415a570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f842c00c410 tx=0x7f842c00c8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 shutdown_connections 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f842006b4d0 0x7f842006d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8434161ca0 0x7f843415a570 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8434158410 0x7f843414a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 --2- 172.21.15.99:0/2656569297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f843406acd0 0x7f8434149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 >> 172.21.15.99:0/2656569297 conn(0x7f84340cfef0 msgr2=0x7f84340d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 shutdown_connections 2024-03-04T10:04:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:06.879+0000 7f8439ea0640 1 -- 172.21.15.99:0/2656569297 wait complete. 2024-03-04T10:04:06.891 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-04T10:04:05.442829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219388,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755332,"statfs":{"total":773094113280,"available":772869459968,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003690"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455527+0000","last_change":"2024-03-04T10:03:38.529898+0000","last_active":"2024-03-04T10:03:45.455527+0000","last_peered":"2024-03-04T10:03:45.455527+0000","last_clean":"2024-03-04T10:03:45.455527+0000","last_became_active":"2024-03-04T10:03:38.529499+0000","last_became_peered":"2024-03-04T10:03:38.529499+0000","last_unstale":"2024-03-04T10:03:45.455527+0000","last_undegraded":"2024-03-04T10:03:45.455527+0000","last_fullsized":"2024-03-04T10:03:45.455527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727461+0000","last_change":"2024-03-04T10:03:39.556883+0000","last_active":"2024-03-04T10:03:44.727461+0000","last_peered":"2024-03-04T10:03:44.727461+0000","last_clean":"2024-03-04T10:03:44.727461+0000","last_became_active":"2024-03-04T10:03:39.556616+0000","last_became_peered":"2024-03-04T10:03:39.556616+0000","last_unstale":"2024-03-04T10:03:44.727461+0000","last_undegraded":"2024-03-04T10:03:44.727461+0000","last_fullsized":"2024-03-04T10:03:44.727461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726874+0000","last_change":"2024-03-04T10:03:43.569338+0000","last_active":"2024-03-04T10:03:44.726874+0000","last_peered":"2024-03-04T10:03:44.726874+0000","last_clean":"2024-03-04T10:03:44.726874+0000","last_became_active":"2024-03-04T10:03:43.568332+0000","last_became_peered":"2024-03-04T10:03:43.568332+0000","last_unstale":"2024-03-04T10:03:44.726874+0000","last_undegraded":"2024-03-04T10:03:44.726874+0000","last_fullsized":"2024-03-04T10:03:44.726874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727559+0000","last_change":"2024-03-04T10:03:38.533956+0000","last_active":"2024-03-04T10:03:44.727559+0000","last_peered":"2024-03-04T10:03:44.727559+0000","last_clean":"2024-03-04T10:03:44.727559+0000","last_became_active":"2024-03-04T10:03:38.533665+0000","last_became_peered":"2024-03-04T10:03:38.533665+0000","last_unstale":"2024-03-04T10:03:44.727559+0000","last_undegraded":"2024-03-04T10:03:44.727559+0000","last_fullsized":"2024-03-04T10:03:44.727559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455218+0000","last_change":"2024-03-04T10:03:39.550341+0000","last_active":"2024-03-04T10:03:45.455218+0000","last_peered":"2024-03-04T10:03:45.455218+0000","last_clean":"2024-03-04T10:03:45.455218+0000","last_became_active":"2024-03-04T10:03:39.549960+0000","last_became_peered":"2024-03-04T10:03:39.549960+0000","last_unstale":"2024-03-04T10:03:45.455218+0000","last_undegraded":"2024-03-04T10:03:45.455218+0000","last_fullsized":"2024-03-04T10:03:45.455218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925071+0000","last_change":"2024-03-04T10:03:43.578315+0000","last_active":"2024-03-04T10:03:44.925071+0000","last_peered":"2024-03-04T10:03:44.925071+0000","last_clean":"2024-03-04T10:03:44.925071+0000","last_became_active":"2024-03-04T10:03:43.577864+0000","last_became_peered":"2024-03-04T10:03:43.577864+0000","last_unstale":"2024-03-04T10:03:44.925071+0000","last_undegraded":"2024-03-04T10:03:44.925071+0000","last_fullsized":"2024-03-04T10:03:44.925071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:27.973529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558121+0000","last_change":"2024-03-04T10:03:38.536815+0000","last_active":"2024-03-04T10:03:44.558121+0000","last_peered":"2024-03-04T10:03:44.558121+0000","last_clean":"2024-03-04T10:03:44.558121+0000","last_became_active":"2024-03-04T10:03:38.536526+0000","last_became_peered":"2024-03-04T10:03:38.536526+0000","last_unstale":"2024-03-04T10:03:44.558121+0000","last_undegraded":"2024-03-04T10:03:44.558121+0000","last_fullsized":"2024-03-04T10:03:44.558121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455699+0000","last_change":"2024-03-04T10:03:39.551796+0000","last_active":"2024-03-04T10:03:45.455699+0000","last_peered":"2024-03-04T10:03:45.455699+0000","last_clean":"2024-03-04T10:03:45.455699+0000","last_became_active":"2024-03-04T10:03:39.551527+0000","last_became_peered":"2024-03-04T10:03:39.551527+0000","last_unstale":"2024-03-04T10:03:45.455699+0000","last_undegraded":"2024-03-04T10:03:45.455699+0000","last_fullsized":"2024-03-04T10:03:45.455699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925354+0000","last_change":"2024-03-04T10:03:43.573372+0000","last_active":"2024-03-04T10:03:44.925354+0000","last_peered":"2024-03-04T10:03:44.925354+0000","last_clean":"2024-03-04T10:03:44.925354+0000","last_became_active":"2024-03-04T10:03:43.572926+0000","last_became_peered":"2024-03-04T10:03:43.572926+0000","last_unstale":"2024-03-04T10:03:44.925354+0000","last_undegraded":"2024-03-04T10:03:44.925354+0000","last_fullsized":"2024-03-04T10:03:44.925354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:37.210258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557408+0000","last_change":"2024-03-04T10:03:38.535163+0000","last_active":"2024-03-04T10:03:44.557408+0000","last_peered":"2024-03-04T10:03:44.557408+0000","last_clean":"2024-03-04T10:03:44.557408+0000","last_became_active":"2024-03-04T10:03:38.534751+0000","last_became_peered":"2024-03-04T10:03:38.534751+0000","last_unstale":"2024-03-04T10:03:44.557408+0000","last_undegraded":"2024-03-04T10:03:44.557408+0000","last_fullsized":"2024-03-04T10:03:44.557408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:40.978639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455945+0000","last_change":"2024-03-04T10:03:39.548628+0000","last_active":"2024-03-04T10:03:45.455945+0000","last_peered":"2024-03-04T10:03:45.455945+0000","last_clean":"2024-03-04T10:03:45.455945+0000","last_became_active":"2024-03-04T10:03:39.548297+0000","last_became_peered":"2024-03-04T10:03:39.548297+0000","last_unstale":"2024-03-04T10:03:45.455945+0000","last_undegraded":"2024-03-04T10:03:45.455945+0000","last_fullsized":"2024-03-04T10:03:45.455945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925131+0000","last_change":"2024-03-04T10:03:43.576063+0000","last_active":"2024-03-04T10:03:44.925131+0000","last_peered":"2024-03-04T10:03:44.925131+0000","last_clean":"2024-03-04T10:03:44.925131+0000","last_became_active":"2024-03-04T10:03:43.575658+0000","last_became_peered":"2024-03-04T10:03:43.575658+0000","last_unstale":"2024-03-04T10:03:44.925131+0000","last_undegraded":"2024-03-04T10:03:44.925131+0000","last_fullsized":"2024-03-04T10:03:44.925131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:47.766859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557898+0000","last_change":"2024-03-04T10:03:39.562772+0000","last_active":"2024-03-04T10:03:44.557898+0000","last_peered":"2024-03-04T10:03:44.557898+0000","last_clean":"2024-03-04T10:03:44.557898+0000","last_became_active":"2024-03-04T10:03:39.562479+0000","last_became_peered":"2024-03-04T10:03:39.562479+0000","last_unstale":"2024-03-04T10:03:44.557898+0000","last_undegraded":"2024-03-04T10:03:44.557898+0000","last_fullsized":"2024-03-04T10:03:44.557898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:48.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558549+0000","last_change":"2024-03-04T10:03:38.524487+0000","last_active":"2024-03-04T10:03:44.558549+0000","last_peered":"2024-03-04T10:03:44.558549+0000","last_clean":"2024-03-04T10:03:44.558549+0000","last_became_active":"2024-03-04T10:03:38.524186+0000","last_became_peered":"2024-03-04T10:03:38.524186+0000","last_unstale":"2024-03-04T10:03:44.558549+0000","last_undegraded":"2024-03-04T10:03:44.558549+0000","last_fullsized":"2024-03-04T10:03:44.558549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925560+0000","last_change":"2024-03-04T10:03:43.573621+0000","last_active":"2024-03-04T10:03:44.925560+0000","last_peered":"2024-03-04T10:03:44.925560+0000","last_clean":"2024-03-04T10:03:44.925560+0000","last_became_active":"2024-03-04T10:03:43.573125+0000","last_became_peered":"2024-03-04T10:03:43.573125+0000","last_unstale":"2024-03-04T10:03:44.925560+0000","last_undegraded":"2024-03-04T10:03:44.925560+0000","last_fullsized":"2024-03-04T10:03:44.925560+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727176+0000","last_change":"2024-03-04T10:03:39.553163+0000","last_active":"2024-03-04T10:03:44.727176+0000","last_peered":"2024-03-04T10:03:44.727176+0000","last_clean":"2024-03-04T10:03:44.727176+0000","last_became_active":"2024-03-04T10:03:39.552909+0000","last_became_peered":"2024-03-04T10:03:39.552909+0000","last_unstale":"2024-03-04T10:03:44.727176+0000","last_undegraded":"2024-03-04T10:03:44.727176+0000","last_fullsized":"2024-03-04T10:03:44.727176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455728+0000","last_change":"2024-03-04T10:03:38.534433+0000","last_active":"2024-03-04T10:03:45.455728+0000","last_peered":"2024-03-04T10:03:45.455728+0000","last_clean":"2024-03-04T10:03:45.455728+0000","last_became_active":"2024-03-04T10:03:38.534162+0000","last_became_peered":"2024-03-04T10:03:38.534162+0000","last_unstale":"2024-03-04T10:03:45.455728+0000","last_undegraded":"2024-03-04T10:03:45.455728+0000","last_fullsized":"2024-03-04T10:03:45.455728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925353+0000","last_change":"2024-03-04T10:03:43.572145+0000","last_active":"2024-03-04T10:03:44.925353+0000","last_peered":"2024-03-04T10:03:44.925353+0000","last_clean":"2024-03-04T10:03:44.925353+0000","last_became_active":"2024-03-04T10:03:43.571762+0000","last_became_peered":"2024-03-04T10:03:43.571762+0000","last_unstale":"2024-03-04T10:03:44.925353+0000","last_undegraded":"2024-03-04T10:03:44.925353+0000","last_fullsized":"2024-03-04T10:03:44.925353+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:43.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727042+0000","last_change":"2024-03-04T10:03:39.558092+0000","last_active":"2024-03-04T10:03:44.727042+0000","last_peered":"2024-03-04T10:03:44.727042+0000","last_clean":"2024-03-04T10:03:44.727042+0000","last_became_active":"2024-03-04T10:03:39.557838+0000","last_became_peered":"2024-03-04T10:03:39.557838+0000","last_unstale":"2024-03-04T10:03:44.727042+0000","last_undegraded":"2024-03-04T10:03:44.727042+0000","last_fullsized":"2024-03-04T10:03:44.727042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456127+0000","last_change":"2024-03-04T10:03:38.538667+0000","last_active":"2024-03-04T10:03:45.456127+0000","last_peered":"2024-03-04T10:03:45.456127+0000","last_clean":"2024-03-04T10:03:45.456127+0000","last_became_active":"2024-03-04T10:03:38.538344+0000","last_became_peered":"2024-03-04T10:03:38.538344+0000","last_unstale":"2024-03-04T10:03:45.456127+0000","last_undegraded":"2024-03-04T10:03:45.456127+0000","last_fullsized":"2024-03-04T10:03:45.456127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456240+0000","last_change":"2024-03-04T10:03:43.583702+0000","last_active":"2024-03-04T10:03:45.456240+0000","last_peered":"2024-03-04T10:03:45.456240+0000","last_clean":"2024-03-04T10:03:45.456240+0000","last_became_active":"2024-03-04T10:03:43.583287+0000","last_became_peered":"2024-03-04T10:03:43.583287+0000","last_unstale":"2024-03-04T10:03:45.456240+0000","last_undegraded":"2024-03-04T10:03:45.456240+0000","last_fullsized":"2024-03-04T10:03:45.456240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:26.836809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925694+0000","last_change":"2024-03-04T10:03:39.542379+0000","last_active":"2024-03-04T10:03:44.925694+0000","last_peered":"2024-03-04T10:03:44.925694+0000","last_clean":"2024-03-04T10:03:44.925694+0000","last_became_active":"2024-03-04T10:03:39.541991+0000","last_became_peered":"2024-03-04T10:03:39.541991+0000","last_unstale":"2024-03-04T10:03:44.925694+0000","last_undegraded":"2024-03-04T10:03:44.925694+0000","last_fullsized":"2024-03-04T10:03:44.925694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:48.539804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557756+0000","last_change":"2024-03-04T10:03:38.532133+0000","last_active":"2024-03-04T10:03:44.557756+0000","last_peered":"2024-03-04T10:03:44.557756+0000","last_clean":"2024-03-04T10:03:44.557756+0000","last_became_active":"2024-03-04T10:03:38.531622+0000","last_became_peered":"2024-03-04T10:03:38.531622+0000","last_unstale":"2024-03-04T10:03:44.557756+0000","last_undegraded":"2024-03-04T10:03:44.557756+0000","last_fullsized":"2024-03-04T10:03:44.557756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925652+0000","last_change":"2024-03-04T10:03:43.570872+0000","last_active":"2024-03-04T10:03:44.925652+0000","last_peered":"2024-03-04T10:03:44.925652+0000","last_clean":"2024-03-04T10:03:44.925652+0000","last_became_active":"2024-03-04T10:03:43.570436+0000","last_became_peered":"2024-03-04T10:03:43.570436+0000","last_unstale":"2024-03-04T10:03:44.925652+0000","last_undegraded":"2024-03-04T10:03:44.925652+0000","last_fullsized":"2024-03-04T10:03:44.925652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455648+0000","last_change":"2024-03-04T10:03:39.541376+0000","last_active":"2024-03-04T10:03:45.455648+0000","last_peered":"2024-03-04T10:03:45.455648+0000","last_clean":"2024-03-04T10:03:45.455648+0000","last_became_active":"2024-03-04T10:03:39.541117+0000","last_became_peered":"2024-03-04T10:03:39.541117+0000","last_unstale":"2024-03-04T10:03:45.455648+0000","last_undegraded":"2024-03-04T10:03:45.455648+0000","last_fullsized":"2024-03-04T10:03:45.455648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558575+0000","last_change":"2024-03-04T10:03:38.538684+0000","last_active":"2024-03-04T10:03:44.558575+0000","last_peered":"2024-03-04T10:03:44.558575+0000","last_clean":"2024-03-04T10:03:44.558575+0000","last_became_active":"2024-03-04T10:03:38.538374+0000","last_became_peered":"2024-03-04T10:03:38.538374+0000","last_unstale":"2024-03-04T10:03:44.558575+0000","last_undegraded":"2024-03-04T10:03:44.558575+0000","last_fullsized":"2024-03-04T10:03:44.558575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925011+0000","last_change":"2024-03-04T10:03:43.575117+0000","last_active":"2024-03-04T10:03:44.925011+0000","last_peered":"2024-03-04T10:03:44.925011+0000","last_clean":"2024-03-04T10:03:44.925011+0000","last_became_active":"2024-03-04T10:03:43.574283+0000","last_became_peered":"2024-03-04T10:03:43.574283+0000","last_unstale":"2024-03-04T10:03:44.925011+0000","last_undegraded":"2024-03-04T10:03:44.925011+0000","last_fullsized":"2024-03-04T10:03:44.925011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:09.615407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727535+0000","last_change":"2024-03-04T10:03:39.554004+0000","last_active":"2024-03-04T10:03:44.727535+0000","last_peered":"2024-03-04T10:03:44.727535+0000","last_clean":"2024-03-04T10:03:44.727535+0000","last_became_active":"2024-03-04T10:03:39.553717+0000","last_became_peered":"2024-03-04T10:03:39.553717+0000","last_unstale":"2024-03-04T10:03:44.727535+0000","last_undegraded":"2024-03-04T10:03:44.727535+0000","last_fullsized":"2024-03-04T10:03:44.727535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455963+0000","last_change":"2024-03-04T10:03:38.533053+0000","last_active":"2024-03-04T10:03:45.455963+0000","last_peered":"2024-03-04T10:03:45.455963+0000","last_clean":"2024-03-04T10:03:45.455963+0000","last_became_active":"2024-03-04T10:03:38.532793+0000","last_became_peered":"2024-03-04T10:03:38.532793+0000","last_unstale":"2024-03-04T10:03:45.455963+0000","last_undegraded":"2024-03-04T10:03:45.455963+0000","last_fullsized":"2024-03-04T10:03:45.455963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726840+0000","last_change":"2024-03-04T10:03:43.577818+0000","last_active":"2024-03-04T10:03:44.726840+0000","last_peered":"2024-03-04T10:03:44.726840+0000","last_clean":"2024-03-04T10:03:44.726840+0000","last_became_active":"2024-03-04T10:03:43.577239+0000","last_became_peered":"2024-03-04T10:03:43.577239+0000","last_unstale":"2024-03-04T10:03:44.726840+0000","last_undegraded":"2024-03-04T10:03:44.726840+0000","last_fullsized":"2024-03-04T10:03:44.726840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455944+0000","last_change":"2024-03-04T10:03:39.542820+0000","last_active":"2024-03-04T10:03:45.455944+0000","last_peered":"2024-03-04T10:03:45.455944+0000","last_clean":"2024-03-04T10:03:45.455944+0000","last_became_active":"2024-03-04T10:03:39.542521+0000","last_became_peered":"2024-03-04T10:03:39.542521+0000","last_unstale":"2024-03-04T10:03:45.455944+0000","last_undegraded":"2024-03-04T10:03:45.455944+0000","last_fullsized":"2024-03-04T10:03:45.455944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558219+0000","last_change":"2024-03-04T10:03:38.534337+0000","last_active":"2024-03-04T10:03:44.558219+0000","last_peered":"2024-03-04T10:03:44.558219+0000","last_clean":"2024-03-04T10:03:44.558219+0000","last_became_active":"2024-03-04T10:03:38.534027+0000","last_became_peered":"2024-03-04T10:03:38.534027+0000","last_unstale":"2024-03-04T10:03:44.558219+0000","last_undegraded":"2024-03-04T10:03:44.558219+0000","last_fullsized":"2024-03-04T10:03:44.558219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:18.373547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455176+0000","last_change":"2024-03-04T10:03:43.577959+0000","last_active":"2024-03-04T10:03:45.455176+0000","last_peered":"2024-03-04T10:03:45.455176+0000","last_clean":"2024-03-04T10:03:45.455176+0000","last_became_active":"2024-03-04T10:03:43.576536+0000","last_became_peered":"2024-03-04T10:03:43.576536+0000","last_unstale":"2024-03-04T10:03:45.455176+0000","last_undegraded":"2024-03-04T10:03:45.455176+0000","last_fullsized":"2024-03-04T10:03:45.455176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456091+0000","last_change":"2024-03-04T10:03:39.556129+0000","last_active":"2024-03-04T10:03:45.456091+0000","last_peered":"2024-03-04T10:03:45.456091+0000","last_clean":"2024-03-04T10:03:45.456091+0000","last_became_active":"2024-03-04T10:03:39.555824+0000","last_became_peered":"2024-03-04T10:03:39.555824+0000","last_unstale":"2024-03-04T10:03:45.456091+0000","last_undegraded":"2024-03-04T10:03:45.456091+0000","last_fullsized":"2024-03-04T10:03:45.456091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456030+0000","last_change":"2024-03-04T10:03:38.531643+0000","last_active":"2024-03-04T10:03:45.456030+0000","last_peered":"2024-03-04T10:03:45.456030+0000","last_clean":"2024-03-04T10:03:45.456030+0000","last_became_active":"2024-03-04T10:03:38.531299+0000","last_became_peered":"2024-03-04T10:03:38.531299+0000","last_unstale":"2024-03-04T10:03:45.456030+0000","last_undegraded":"2024-03-04T10:03:45.456030+0000","last_fullsized":"2024-03-04T10:03:45.456030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455431+0000","last_change":"2024-03-04T10:03:43.576642+0000","last_active":"2024-03-04T10:03:45.455431+0000","last_peered":"2024-03-04T10:03:45.455431+0000","last_clean":"2024-03-04T10:03:45.455431+0000","last_became_active":"2024-03-04T10:03:43.576270+0000","last_became_peered":"2024-03-04T10:03:43.576270+0000","last_unstale":"2024-03-04T10:03:45.455431+0000","last_undegraded":"2024-03-04T10:03:45.455431+0000","last_fullsized":"2024-03-04T10:03:45.455431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925763+0000","last_change":"2024-03-04T10:03:39.546721+0000","last_active":"2024-03-04T10:03:44.925763+0000","last_peered":"2024-03-04T10:03:44.925763+0000","last_clean":"2024-03-04T10:03:44.925763+0000","last_became_active":"2024-03-04T10:03:39.546438+0000","last_became_peered":"2024-03-04T10:03:39.546438+0000","last_unstale":"2024-03-04T10:03:44.925763+0000","last_undegraded":"2024-03-04T10:03:44.925763+0000","last_fullsized":"2024-03-04T10:03:44.925763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455730+0000","last_change":"2024-03-04T10:03:38.534806+0000","last_active":"2024-03-04T10:03:45.455730+0000","last_peered":"2024-03-04T10:03:45.455730+0000","last_clean":"2024-03-04T10:03:45.455730+0000","last_became_active":"2024-03-04T10:03:38.534555+0000","last_became_peered":"2024-03-04T10:03:38.534555+0000","last_unstale":"2024-03-04T10:03:45.455730+0000","last_undegraded":"2024-03-04T10:03:45.455730+0000","last_fullsized":"2024-03-04T10:03:45.455730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455898+0000","last_change":"2024-03-04T10:03:43.583916+0000","last_active":"2024-03-04T10:03:45.455898+0000","last_peered":"2024-03-04T10:03:45.455898+0000","last_clean":"2024-03-04T10:03:45.455898+0000","last_became_active":"2024-03-04T10:03:43.583537+0000","last_became_peered":"2024-03-04T10:03:43.583537+0000","last_unstale":"2024-03-04T10:03:45.455898+0000","last_undegraded":"2024-03-04T10:03:45.455898+0000","last_fullsized":"2024-03-04T10:03:45.455898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:46.674144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454949+0000","last_change":"2024-03-04T10:03:39.539248+0000","last_active":"2024-03-04T10:03:45.454949+0000","last_peered":"2024-03-04T10:03:45.454949+0000","last_clean":"2024-03-04T10:03:45.454949+0000","last_became_active":"2024-03-04T10:03:39.538978+0000","last_became_peered":"2024-03-04T10:03:39.538978+0000","last_unstale":"2024-03-04T10:03:45.454949+0000","last_undegraded":"2024-03-04T10:03:45.454949+0000","last_fullsized":"2024-03-04T10:03:45.454949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556029+0000","last_change":"2024-03-04T10:03:38.534970+0000","last_active":"2024-03-04T10:03:44.556029+0000","last_peered":"2024-03-04T10:03:44.556029+0000","last_clean":"2024-03-04T10:03:44.556029+0000","last_became_active":"2024-03-04T10:03:38.534694+0000","last_became_peered":"2024-03-04T10:03:38.534694+0000","last_unstale":"2024-03-04T10:03:44.556029+0000","last_undegraded":"2024-03-04T10:03:44.556029+0000","last_fullsized":"2024-03-04T10:03:44.556029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456270+0000","last_change":"2024-03-04T10:03:43.582855+0000","last_active":"2024-03-04T10:03:45.456270+0000","last_peered":"2024-03-04T10:03:45.456270+0000","last_clean":"2024-03-04T10:03:45.456270+0000","last_became_active":"2024-03-04T10:03:43.580626+0000","last_became_peered":"2024-03-04T10:03:43.580626+0000","last_unstale":"2024-03-04T10:03:45.456270+0000","last_undegraded":"2024-03-04T10:03:45.456270+0000","last_fullsized":"2024-03-04T10:03:45.456270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:39.012948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454952+0000","last_change":"2024-03-04T10:03:39.538900+0000","last_active":"2024-03-04T10:03:45.454952+0000","last_peered":"2024-03-04T10:03:45.454952+0000","last_clean":"2024-03-04T10:03:45.454952+0000","last_became_active":"2024-03-04T10:03:39.538624+0000","last_became_peered":"2024-03-04T10:03:39.538624+0000","last_unstale":"2024-03-04T10:03:45.454952+0000","last_undegraded":"2024-03-04T10:03:45.454952+0000","last_fullsized":"2024-03-04T10:03:45.454952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557795+0000","last_change":"2024-03-04T10:03:38.534850+0000","last_active":"2024-03-04T10:03:44.557795+0000","last_peered":"2024-03-04T10:03:44.557795+0000","last_clean":"2024-03-04T10:03:44.557795+0000","last_became_active":"2024-03-04T10:03:38.534604+0000","last_became_peered":"2024-03-04T10:03:38.534604+0000","last_unstale":"2024-03-04T10:03:44.557795+0000","last_undegraded":"2024-03-04T10:03:44.557795+0000","last_fullsized":"2024-03-04T10:03:44.557795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455383+0000","last_change":"2024-03-04T10:03:43.576525+0000","last_active":"2024-03-04T10:03:45.455383+0000","last_peered":"2024-03-04T10:03:45.455383+0000","last_clean":"2024-03-04T10:03:45.455383+0000","last_became_active":"2024-03-04T10:03:43.576149+0000","last_became_peered":"2024-03-04T10:03:43.576149+0000","last_unstale":"2024-03-04T10:03:45.455383+0000","last_undegraded":"2024-03-04T10:03:45.455383+0000","last_fullsized":"2024-03-04T10:03:45.455383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727079+0000","last_change":"2024-03-04T10:03:39.557088+0000","last_active":"2024-03-04T10:03:44.727079+0000","last_peered":"2024-03-04T10:03:44.727079+0000","last_clean":"2024-03-04T10:03:44.727079+0000","last_became_active":"2024-03-04T10:03:39.556715+0000","last_became_peered":"2024-03-04T10:03:39.556715+0000","last_unstale":"2024-03-04T10:03:44.727079+0000","last_undegraded":"2024-03-04T10:03:44.727079+0000","last_fullsized":"2024-03-04T10:03:44.727079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455694+0000","last_change":"2024-03-04T10:03:38.537571+0000","last_active":"2024-03-04T10:03:45.455694+0000","last_peered":"2024-03-04T10:03:45.455694+0000","last_clean":"2024-03-04T10:03:45.455694+0000","last_became_active":"2024-03-04T10:03:38.536646+0000","last_became_peered":"2024-03-04T10:03:38.536646+0000","last_unstale":"2024-03-04T10:03:45.455694+0000","last_undegraded":"2024-03-04T10:03:45.455694+0000","last_fullsized":"2024-03-04T10:03:45.455694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557928+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:03:44.557928+0000","last_peered":"2024-03-04T10:03:44.557928+0000","last_clean":"2024-03-04T10:03:44.557928+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:03:44.557928+0000","last_undegraded":"2024-03-04T10:03:44.557928+0000","last_fullsized":"2024-03-04T10:03:44.557928+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558259+0000","last_change":"2024-03-04T10:03:43.579804+0000","last_active":"2024-03-04T10:03:44.558259+0000","last_peered":"2024-03-04T10:03:44.558259+0000","last_clean":"2024-03-04T10:03:44.558259+0000","last_became_active":"2024-03-04T10:03:43.579420+0000","last_became_peered":"2024-03-04T10:03:43.579420+0000","last_unstale":"2024-03-04T10:03:44.558259+0000","last_undegraded":"2024-03-04T10:03:44.558259+0000","last_fullsized":"2024-03-04T10:03:44.558259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557993+0000","last_change":"2024-03-04T10:03:39.550717+0000","last_active":"2024-03-04T10:03:44.557993+0000","last_peered":"2024-03-04T10:03:44.557993+0000","last_clean":"2024-03-04T10:03:44.557993+0000","last_became_active":"2024-03-04T10:03:39.550331+0000","last_became_peered":"2024-03-04T10:03:39.550331+0000","last_unstale":"2024-03-04T10:03:44.557993+0000","last_undegraded":"2024-03-04T10:03:44.557993+0000","last_fullsized":"2024-03-04T10:03:44.557993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556242+0000","last_change":"2024-03-04T10:03:38.529200+0000","last_active":"2024-03-04T10:03:44.556242+0000","last_peered":"2024-03-04T10:03:44.556242+0000","last_clean":"2024-03-04T10:03:44.556242+0000","last_became_active":"2024-03-04T10:03:38.528943+0000","last_became_peered":"2024-03-04T10:03:38.528943+0000","last_unstale":"2024-03-04T10:03:44.556242+0000","last_undegraded":"2024-03-04T10:03:44.556242+0000","last_fullsized":"2024-03-04T10:03:44.556242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557839+0000","last_change":"2024-03-04T10:03:43.578553+0000","last_active":"2024-03-04T10:03:44.557839+0000","last_peered":"2024-03-04T10:03:44.557839+0000","last_clean":"2024-03-04T10:03:44.557839+0000","last_became_active":"2024-03-04T10:03:43.577929+0000","last_became_peered":"2024-03-04T10:03:43.577929+0000","last_unstale":"2024-03-04T10:03:44.557839+0000","last_undegraded":"2024-03-04T10:03:44.557839+0000","last_fullsized":"2024-03-04T10:03:44.557839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455715+0000","last_change":"2024-03-04T10:03:39.556135+0000","last_active":"2024-03-04T10:03:45.455715+0000","last_peered":"2024-03-04T10:03:45.455715+0000","last_clean":"2024-03-04T10:03:45.455715+0000","last_became_active":"2024-03-04T10:03:39.555846+0000","last_became_peered":"2024-03-04T10:03:39.555846+0000","last_unstale":"2024-03-04T10:03:45.455715+0000","last_undegraded":"2024-03-04T10:03:45.455715+0000","last_fullsized":"2024-03-04T10:03:45.455715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455561+0000","last_change":"2024-03-04T10:03:38.530735+0000","last_active":"2024-03-04T10:03:45.455561+0000","last_peered":"2024-03-04T10:03:45.455561+0000","last_clean":"2024-03-04T10:03:45.455561+0000","last_became_active":"2024-03-04T10:03:38.530463+0000","last_became_peered":"2024-03-04T10:03:38.530463+0000","last_unstale":"2024-03-04T10:03:45.455561+0000","last_undegraded":"2024-03-04T10:03:45.455561+0000","last_fullsized":"2024-03-04T10:03:45.455561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455391+0000","last_change":"2024-03-04T10:03:43.577622+0000","last_active":"2024-03-04T10:03:45.455391+0000","last_peered":"2024-03-04T10:03:45.455391+0000","last_clean":"2024-03-04T10:03:45.455391+0000","last_became_active":"2024-03-04T10:03:43.575914+0000","last_became_peered":"2024-03-04T10:03:43.575914+0000","last_unstale":"2024-03-04T10:03:45.455391+0000","last_undegraded":"2024-03-04T10:03:45.455391+0000","last_fullsized":"2024-03-04T10:03:45.455391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558308+0000","last_change":"2024-03-04T10:03:39.551071+0000","last_active":"2024-03-04T10:03:44.558308+0000","last_peered":"2024-03-04T10:03:44.558308+0000","last_clean":"2024-03-04T10:03:44.558308+0000","last_became_active":"2024-03-04T10:03:39.550739+0000","last_became_peered":"2024-03-04T10:03:39.550739+0000","last_unstale":"2024-03-04T10:03:44.558308+0000","last_undegraded":"2024-03-04T10:03:44.558308+0000","last_fullsized":"2024-03-04T10:03:44.558308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:14.185049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556247+0000","last_change":"2024-03-04T10:03:38.534772+0000","last_active":"2024-03-04T10:03:44.556247+0000","last_peered":"2024-03-04T10:03:44.556247+0000","last_clean":"2024-03-04T10:03:44.556247+0000","last_became_active":"2024-03-04T10:03:38.534520+0000","last_became_peered":"2024-03-04T10:03:38.534520+0000","last_unstale":"2024-03-04T10:03:44.556247+0000","last_undegraded":"2024-03-04T10:03:44.556247+0000","last_fullsized":"2024-03-04T10:03:44.556247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455292+0000","last_change":"2024-03-04T10:03:43.570591+0000","last_active":"2024-03-04T10:03:45.455292+0000","last_peered":"2024-03-04T10:03:45.455292+0000","last_clean":"2024-03-04T10:03:45.455292+0000","last_became_active":"2024-03-04T10:03:43.570154+0000","last_became_peered":"2024-03-04T10:03:43.570154+0000","last_unstale":"2024-03-04T10:03:45.455292+0000","last_undegraded":"2024-03-04T10:03:45.455292+0000","last_fullsized":"2024-03-04T10:03:45.455292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558471+0000","last_change":"2024-03-04T10:03:39.547823+0000","last_active":"2024-03-04T10:03:44.558471+0000","last_peered":"2024-03-04T10:03:44.558471+0000","last_clean":"2024-03-04T10:03:44.558471+0000","last_became_active":"2024-03-04T10:03:39.547553+0000","last_became_peered":"2024-03-04T10:03:39.547553+0000","last_unstale":"2024-03-04T10:03:44.558471+0000","last_undegraded":"2024-03-04T10:03:44.558471+0000","last_fullsized":"2024-03-04T10:03:44.558471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925780+0000","last_change":"2024-03-04T10:03:38.530675+0000","last_active":"2024-03-04T10:03:44.925780+0000","last_peered":"2024-03-04T10:03:44.925780+0000","last_clean":"2024-03-04T10:03:44.925780+0000","last_became_active":"2024-03-04T10:03:38.530406+0000","last_became_peered":"2024-03-04T10:03:38.530406+0000","last_unstale":"2024-03-04T10:03:44.925780+0000","last_undegraded":"2024-03-04T10:03:44.925780+0000","last_fullsized":"2024-03-04T10:03:44.925780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:29.050824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925002+0000","last_change":"2024-03-04T10:03:43.577882+0000","last_active":"2024-03-04T10:03:44.925002+0000","last_peered":"2024-03-04T10:03:44.925002+0000","last_clean":"2024-03-04T10:03:44.925002+0000","last_became_active":"2024-03-04T10:03:43.577500+0000","last_became_peered":"2024-03-04T10:03:43.577500+0000","last_unstale":"2024-03-04T10:03:44.925002+0000","last_undegraded":"2024-03-04T10:03:44.925002+0000","last_fullsized":"2024-03-04T10:03:44.925002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:38.890692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557934+0000","last_change":"2024-03-04T10:03:39.546483+0000","last_active":"2024-03-04T10:03:44.557934+0000","last_peered":"2024-03-04T10:03:44.557934+0000","last_clean":"2024-03-04T10:03:44.557934+0000","last_became_active":"2024-03-04T10:03:39.546211+0000","last_became_peered":"2024-03-04T10:03:39.546211+0000","last_unstale":"2024-03-04T10:03:44.557934+0000","last_undegraded":"2024-03-04T10:03:44.557934+0000","last_fullsized":"2024-03-04T10:03:44.557934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555858+0000","last_change":"2024-03-04T10:03:38.532190+0000","last_active":"2024-03-04T10:03:44.555858+0000","last_peered":"2024-03-04T10:03:44.555858+0000","last_clean":"2024-03-04T10:03:44.555858+0000","last_became_active":"2024-03-04T10:03:38.531859+0000","last_became_peered":"2024-03-04T10:03:38.531859+0000","last_unstale":"2024-03-04T10:03:44.555858+0000","last_undegraded":"2024-03-04T10:03:44.555858+0000","last_fullsized":"2024-03-04T10:03:44.555858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:30:57.662758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455223+0000","last_change":"2024-03-04T10:03:43.575386+0000","last_active":"2024-03-04T10:03:45.455223+0000","last_peered":"2024-03-04T10:03:45.455223+0000","last_clean":"2024-03-04T10:03:45.455223+0000","last_became_active":"2024-03-04T10:03:43.575009+0000","last_became_peered":"2024-03-04T10:03:43.575009+0000","last_unstale":"2024-03-04T10:03:45.455223+0000","last_undegraded":"2024-03-04T10:03:45.455223+0000","last_fullsized":"2024-03-04T10:03:45.455223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:05.911238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556867+0000","last_change":"2024-03-04T10:03:39.546825+0000","last_active":"2024-03-04T10:03:44.556867+0000","last_peered":"2024-03-04T10:03:44.556867+0000","last_clean":"2024-03-04T10:03:44.556867+0000","last_became_active":"2024-03-04T10:03:39.546567+0000","last_became_peered":"2024-03-04T10:03:39.546567+0000","last_unstale":"2024-03-04T10:03:44.556867+0000","last_undegraded":"2024-03-04T10:03:44.556867+0000","last_fullsized":"2024-03-04T10:03:44.556867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557579+0000","last_change":"2024-03-04T10:03:38.529048+0000","last_active":"2024-03-04T10:03:44.557579+0000","last_peered":"2024-03-04T10:03:44.557579+0000","last_clean":"2024-03-04T10:03:44.557579+0000","last_became_active":"2024-03-04T10:03:38.528276+0000","last_became_peered":"2024-03-04T10:03:38.528276+0000","last_unstale":"2024-03-04T10:03:44.557579+0000","last_undegraded":"2024-03-04T10:03:44.557579+0000","last_fullsized":"2024-03-04T10:03:44.557579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727557+0000","last_change":"2024-03-04T10:03:43.578301+0000","last_active":"2024-03-04T10:03:44.727557+0000","last_peered":"2024-03-04T10:03:44.727557+0000","last_clean":"2024-03-04T10:03:44.727557+0000","last_became_active":"2024-03-04T10:03:43.577909+0000","last_became_peered":"2024-03-04T10:03:43.577909+0000","last_unstale":"2024-03-04T10:03:44.727557+0000","last_undegraded":"2024-03-04T10:03:44.727557+0000","last_fullsized":"2024-03-04T10:03:44.727557+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:37.260262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558519+0000","last_change":"2024-03-04T10:03:39.547091+0000","last_active":"2024-03-04T10:03:44.558519+0000","last_peered":"2024-03-04T10:03:44.558519+0000","last_clean":"2024-03-04T10:03:44.558519+0000","last_became_active":"2024-03-04T10:03:39.546836+0000","last_became_peered":"2024-03-04T10:03:39.546836+0000","last_unstale":"2024-03-04T10:03:44.558519+0000","last_undegraded":"2024-03-04T10:03:44.558519+0000","last_fullsized":"2024-03-04T10:03:44.558519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455647+0000","last_change":"2024-03-04T10:03:38.535462+0000","last_active":"2024-03-04T10:03:45.455647+0000","last_peered":"2024-03-04T10:03:45.455647+0000","last_clean":"2024-03-04T10:03:45.455647+0000","last_became_active":"2024-03-04T10:03:38.535165+0000","last_became_peered":"2024-03-04T10:03:38.535165+0000","last_unstale":"2024-03-04T10:03:45.455647+0000","last_undegraded":"2024-03-04T10:03:45.455647+0000","last_fullsized":"2024-03-04T10:03:45.455647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727307+0000","last_change":"2024-03-04T10:03:43.576692+0000","last_active":"2024-03-04T10:03:44.727307+0000","last_peered":"2024-03-04T10:03:44.727307+0000","last_clean":"2024-03-04T10:03:44.727307+0000","last_became_active":"2024-03-04T10:03:43.576321+0000","last_became_peered":"2024-03-04T10:03:43.576321+0000","last_unstale":"2024-03-04T10:03:44.727307+0000","last_undegraded":"2024-03-04T10:03:44.727307+0000","last_fullsized":"2024-03-04T10:03:44.727307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:14:32.702691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455942+0000","last_change":"2024-03-04T10:03:39.547184+0000","last_active":"2024-03-04T10:03:45.455942+0000","last_peered":"2024-03-04T10:03:45.455942+0000","last_clean":"2024-03-04T10:03:45.455942+0000","last_became_active":"2024-03-04T10:03:39.546899+0000","last_became_peered":"2024-03-04T10:03:39.546899+0000","last_unstale":"2024-03-04T10:03:45.455942+0000","last_undegraded":"2024-03-04T10:03:45.455942+0000","last_fullsized":"2024-03-04T10:03:45.455942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455444+0000","last_change":"2024-03-04T10:03:38.529853+0000","last_active":"2024-03-04T10:03:45.455444+0000","last_peered":"2024-03-04T10:03:45.455444+0000","last_clean":"2024-03-04T10:03:45.455444+0000","last_became_active":"2024-03-04T10:03:38.529259+0000","last_became_peered":"2024-03-04T10:03:38.529259+0000","last_unstale":"2024-03-04T10:03:45.455444+0000","last_undegraded":"2024-03-04T10:03:45.455444+0000","last_fullsized":"2024-03-04T10:03:45.455444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456026+0000","last_change":"2024-03-04T10:03:43.572841+0000","last_active":"2024-03-04T10:03:45.456026+0000","last_peered":"2024-03-04T10:03:45.456026+0000","last_clean":"2024-03-04T10:03:45.456026+0000","last_became_active":"2024-03-04T10:03:43.572414+0000","last_became_peered":"2024-03-04T10:03:43.572414+0000","last_unstale":"2024-03-04T10:03:45.456026+0000","last_undegraded":"2024-03-04T10:03:45.456026+0000","last_fullsized":"2024-03-04T10:03:45.456026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557911+0000","last_change":"2024-03-04T10:03:39.552501+0000","last_active":"2024-03-04T10:03:44.557911+0000","last_peered":"2024-03-04T10:03:44.557911+0000","last_clean":"2024-03-04T10:03:44.557911+0000","last_became_active":"2024-03-04T10:03:39.551870+0000","last_became_peered":"2024-03-04T10:03:39.551870+0000","last_unstale":"2024-03-04T10:03:44.557911+0000","last_undegraded":"2024-03-04T10:03:44.557911+0000","last_fullsized":"2024-03-04T10:03:44.557911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:48.793130+0000","last_change":"2024-03-04T10:03:38.534457+0000","last_active":"2024-03-04T10:03:48.793130+0000","last_peered":"2024-03-04T10:03:48.793130+0000","last_clean":"2024-03-04T10:03:48.793130+0000","last_became_active":"2024-03-04T10:03:38.534185+0000","last_became_peered":"2024-03-04T10:03:38.534185+0000","last_unstale":"2024-03-04T10:03:48.793130+0000","last_undegraded":"2024-03-04T10:03:48.793130+0000","last_fullsized":"2024-03-04T10:03:48.793130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455693+0000","last_change":"2024-03-04T10:03:43.572170+0000","last_active":"2024-03-04T10:03:45.455693+0000","last_peered":"2024-03-04T10:03:45.455693+0000","last_clean":"2024-03-04T10:03:45.455693+0000","last_became_active":"2024-03-04T10:03:43.571770+0000","last_became_peered":"2024-03-04T10:03:43.571770+0000","last_unstale":"2024-03-04T10:03:45.455693+0000","last_undegraded":"2024-03-04T10:03:45.455693+0000","last_fullsized":"2024-03-04T10:03:45.455693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:19.867931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925655+0000","last_change":"2024-03-04T10:03:39.548855+0000","last_active":"2024-03-04T10:03:44.925655+0000","last_peered":"2024-03-04T10:03:44.925655+0000","last_clean":"2024-03-04T10:03:44.925655+0000","last_became_active":"2024-03-04T10:03:39.548568+0000","last_became_peered":"2024-03-04T10:03:39.548568+0000","last_unstale":"2024-03-04T10:03:44.925655+0000","last_undegraded":"2024-03-04T10:03:44.925655+0000","last_fullsized":"2024-03-04T10:03:44.925655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:19.378207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455946+0000","last_change":"2024-03-04T10:03:38.537010+0000","last_active":"2024-03-04T10:03:45.455946+0000","last_peered":"2024-03-04T10:03:45.455946+0000","last_clean":"2024-03-04T10:03:45.455946+0000","last_became_active":"2024-03-04T10:03:38.536784+0000","last_became_peered":"2024-03-04T10:03:38.536784+0000","last_unstale":"2024-03-04T10:03:45.455946+0000","last_undegraded":"2024-03-04T10:03:45.455946+0000","last_fullsized":"2024-03-04T10:03:45.455946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925807+0000","last_change":"2024-03-04T10:03:43.575162+0000","last_active":"2024-03-04T10:03:44.925807+0000","last_peered":"2024-03-04T10:03:44.925807+0000","last_clean":"2024-03-04T10:03:44.925807+0000","last_became_active":"2024-03-04T10:03:43.574407+0000","last_became_peered":"2024-03-04T10:03:43.574407+0000","last_unstale":"2024-03-04T10:03:44.925807+0000","last_undegraded":"2024-03-04T10:03:44.925807+0000","last_fullsized":"2024-03-04T10:03:44.925807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455373+0000","last_change":"2024-03-04T10:03:39.550513+0000","last_active":"2024-03-04T10:03:45.455373+0000","last_peered":"2024-03-04T10:03:45.455373+0000","last_clean":"2024-03-04T10:03:45.455373+0000","last_became_active":"2024-03-04T10:03:39.549951+0000","last_became_peered":"2024-03-04T10:03:39.549951+0000","last_unstale":"2024-03-04T10:03:45.455373+0000","last_undegraded":"2024-03-04T10:03:45.455373+0000","last_fullsized":"2024-03-04T10:03:45.455373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455931+0000","last_change":"2024-03-04T10:03:38.538520+0000","last_active":"2024-03-04T10:03:45.455931+0000","last_peered":"2024-03-04T10:03:45.455931+0000","last_clean":"2024-03-04T10:03:45.455931+0000","last_became_active":"2024-03-04T10:03:38.538251+0000","last_became_peered":"2024-03-04T10:03:38.538251+0000","last_unstale":"2024-03-04T10:03:45.455931+0000","last_undegraded":"2024-03-04T10:03:45.455931+0000","last_fullsized":"2024-03-04T10:03:45.455931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727165+0000","last_change":"2024-03-04T10:03:43.573784+0000","last_active":"2024-03-04T10:03:44.727165+0000","last_peered":"2024-03-04T10:03:44.727165+0000","last_clean":"2024-03-04T10:03:44.727165+0000","last_became_active":"2024-03-04T10:03:43.573040+0000","last_became_peered":"2024-03-04T10:03:43.573040+0000","last_unstale":"2024-03-04T10:03:44.727165+0000","last_undegraded":"2024-03-04T10:03:44.727165+0000","last_fullsized":"2024-03-04T10:03:44.727165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:31.435305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455930+0000","last_change":"2024-03-04T10:03:39.559601+0000","last_active":"2024-03-04T10:03:45.455930+0000","last_peered":"2024-03-04T10:03:45.455930+0000","last_clean":"2024-03-04T10:03:45.455930+0000","last_became_active":"2024-03-04T10:03:39.559321+0000","last_became_peered":"2024-03-04T10:03:39.559321+0000","last_unstale":"2024-03-04T10:03:45.455930+0000","last_undegraded":"2024-03-04T10:03:45.455930+0000","last_fullsized":"2024-03-04T10:03:45.455930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558393+0000","last_change":"2024-03-04T10:03:38.536855+0000","last_active":"2024-03-04T10:03:44.558393+0000","last_peered":"2024-03-04T10:03:44.558393+0000","last_clean":"2024-03-04T10:03:44.558393+0000","last_became_active":"2024-03-04T10:03:38.536551+0000","last_became_peered":"2024-03-04T10:03:38.536551+0000","last_unstale":"2024-03-04T10:03:44.558393+0000","last_undegraded":"2024-03-04T10:03:44.558393+0000","last_fullsized":"2024-03-04T10:03:44.558393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727003+0000","last_change":"2024-03-04T10:03:43.569782+0000","last_active":"2024-03-04T10:03:44.727003+0000","last_peered":"2024-03-04T10:03:44.727003+0000","last_clean":"2024-03-04T10:03:44.727003+0000","last_became_active":"2024-03-04T10:03:43.568928+0000","last_became_peered":"2024-03-04T10:03:43.568928+0000","last_unstale":"2024-03-04T10:03:44.727003+0000","last_undegraded":"2024-03-04T10:03:44.727003+0000","last_fullsized":"2024-03-04T10:03:44.727003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:33.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455065+0000","last_change":"2024-03-04T10:03:43.577832+0000","last_active":"2024-03-04T10:03:45.455065+0000","last_peered":"2024-03-04T10:03:45.455065+0000","last_clean":"2024-03-04T10:03:45.455065+0000","last_became_active":"2024-03-04T10:03:43.577460+0000","last_became_peered":"2024-03-04T10:03:43.577460+0000","last_unstale":"2024-03-04T10:03:45.455065+0000","last_undegraded":"2024-03-04T10:03:45.455065+0000","last_fullsized":"2024-03-04T10:03:45.455065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455950+0000","last_change":"2024-03-04T10:03:39.547155+0000","last_active":"2024-03-04T10:03:45.455950+0000","last_peered":"2024-03-04T10:03:45.455950+0000","last_clean":"2024-03-04T10:03:45.455950+0000","last_became_active":"2024-03-04T10:03:39.546882+0000","last_became_peered":"2024-03-04T10:03:39.546882+0000","last_unstale":"2024-03-04T10:03:45.455950+0000","last_undegraded":"2024-03-04T10:03:45.455950+0000","last_fullsized":"2024-03-04T10:03:45.455950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455427+0000","last_change":"2024-03-04T10:03:38.533042+0000","last_active":"2024-03-04T10:03:45.455427+0000","last_peered":"2024-03-04T10:03:45.455427+0000","last_clean":"2024-03-04T10:03:45.455427+0000","last_became_active":"2024-03-04T10:03:38.532609+0000","last_became_peered":"2024-03-04T10:03:38.532609+0000","last_unstale":"2024-03-04T10:03:45.455427+0000","last_undegraded":"2024-03-04T10:03:45.455427+0000","last_fullsized":"2024-03-04T10:03:45.455427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558268+0000","last_change":"2024-03-04T10:03:43.577800+0000","last_active":"2024-03-04T10:03:44.558268+0000","last_peered":"2024-03-04T10:03:44.558268+0000","last_clean":"2024-03-04T10:03:44.558268+0000","last_became_active":"2024-03-04T10:03:43.577379+0000","last_became_peered":"2024-03-04T10:03:43.577379+0000","last_unstale":"2024-03-04T10:03:44.558268+0000","last_undegraded":"2024-03-04T10:03:44.558268+0000","last_fullsized":"2024-03-04T10:03:44.558268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726957+0000","last_change":"2024-03-04T10:03:39.555669+0000","last_active":"2024-03-04T10:03:44.726957+0000","last_peered":"2024-03-04T10:03:44.726957+0000","last_clean":"2024-03-04T10:03:44.726957+0000","last_became_active":"2024-03-04T10:03:39.555415+0000","last_became_peered":"2024-03-04T10:03:39.555415+0000","last_unstale":"2024-03-04T10:03:44.726957+0000","last_undegraded":"2024-03-04T10:03:44.726957+0000","last_fullsized":"2024-03-04T10:03:44.726957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727092+0000","last_change":"2024-03-04T10:03:38.537206+0000","last_active":"2024-03-04T10:03:44.727092+0000","last_peered":"2024-03-04T10:03:44.727092+0000","last_clean":"2024-03-04T10:03:44.727092+0000","last_became_active":"2024-03-04T10:03:38.536938+0000","last_became_peered":"2024-03-04T10:03:38.536938+0000","last_unstale":"2024-03-04T10:03:44.727092+0000","last_undegraded":"2024-03-04T10:03:44.727092+0000","last_fullsized":"2024-03-04T10:03:44.727092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558028+0000","last_change":"2024-03-04T10:03:38.528419+0000","last_active":"2024-03-04T10:03:44.558028+0000","last_peered":"2024-03-04T10:03:44.558028+0000","last_clean":"2024-03-04T10:03:44.558028+0000","last_became_active":"2024-03-04T10:03:38.528143+0000","last_became_peered":"2024-03-04T10:03:38.528143+0000","last_unstale":"2024-03-04T10:03:44.558028+0000","last_undegraded":"2024-03-04T10:03:44.558028+0000","last_fullsized":"2024-03-04T10:03:44.558028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455342+0000","last_change":"2024-03-04T10:03:39.537531+0000","last_active":"2024-03-04T10:03:45.455342+0000","last_peered":"2024-03-04T10:03:45.455342+0000","last_clean":"2024-03-04T10:03:45.455342+0000","last_became_active":"2024-03-04T10:03:39.537282+0000","last_became_peered":"2024-03-04T10:03:39.537282+0000","last_unstale":"2024-03-04T10:03:45.455342+0000","last_undegraded":"2024-03-04T10:03:45.455342+0000","last_fullsized":"2024-03-04T10:03:45.455342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556191+0000","last_change":"2024-03-04T10:03:43.576190+0000","last_active":"2024-03-04T10:03:44.556191+0000","last_peered":"2024-03-04T10:03:44.556191+0000","last_clean":"2024-03-04T10:03:44.556191+0000","last_became_active":"2024-03-04T10:03:43.575758+0000","last_became_peered":"2024-03-04T10:03:43.575758+0000","last_unstale":"2024-03-04T10:03:44.556191+0000","last_undegraded":"2024-03-04T10:03:44.556191+0000","last_fullsized":"2024-03-04T10:03:44.556191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:27:39.111875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727248+0000","last_change":"2024-03-04T10:03:38.538563+0000","last_active":"2024-03-04T10:03:44.727248+0000","last_peered":"2024-03-04T10:03:44.727248+0000","last_clean":"2024-03-04T10:03:44.727248+0000","last_became_active":"2024-03-04T10:03:38.538284+0000","last_became_peered":"2024-03-04T10:03:38.538284+0000","last_unstale":"2024-03-04T10:03:44.727248+0000","last_undegraded":"2024-03-04T10:03:44.727248+0000","last_fullsized":"2024-03-04T10:03:44.727248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925234+0000","last_change":"2024-03-04T10:03:39.543278+0000","last_active":"2024-03-04T10:03:44.925234+0000","last_peered":"2024-03-04T10:03:44.925234+0000","last_clean":"2024-03-04T10:03:44.925234+0000","last_became_active":"2024-03-04T10:03:39.542993+0000","last_became_peered":"2024-03-04T10:03:39.542993+0000","last_unstale":"2024-03-04T10:03:44.925234+0000","last_undegraded":"2024-03-04T10:03:44.925234+0000","last_fullsized":"2024-03-04T10:03:44.925234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:44.723658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727569+0000","last_change":"2024-03-04T10:03:43.569930+0000","last_active":"2024-03-04T10:03:44.727569+0000","last_peered":"2024-03-04T10:03:44.727569+0000","last_clean":"2024-03-04T10:03:44.727569+0000","last_became_active":"2024-03-04T10:03:43.569524+0000","last_became_peered":"2024-03-04T10:03:43.569524+0000","last_unstale":"2024-03-04T10:03:44.727569+0000","last_undegraded":"2024-03-04T10:03:44.727569+0000","last_fullsized":"2024-03-04T10:03:44.727569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:28.002170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556756+0000","last_change":"2024-03-04T10:03:39.547564+0000","last_active":"2024-03-04T10:03:44.556756+0000","last_peered":"2024-03-04T10:03:44.556756+0000","last_clean":"2024-03-04T10:03:44.556756+0000","last_became_active":"2024-03-04T10:03:39.547245+0000","last_became_peered":"2024-03-04T10:03:39.547245+0000","last_unstale":"2024-03-04T10:03:44.556756+0000","last_undegraded":"2024-03-04T10:03:44.556756+0000","last_fullsized":"2024-03-04T10:03:44.556756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:15:14.213025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555940+0000","last_change":"2024-03-04T10:03:39.549137+0000","last_active":"2024-03-04T10:03:44.555940+0000","last_peered":"2024-03-04T10:03:44.555940+0000","last_clean":"2024-03-04T10:03:44.555940+0000","last_became_active":"2024-03-04T10:03:39.548867+0000","last_became_peered":"2024-03-04T10:03:39.548867+0000","last_unstale":"2024-03-04T10:03:44.555940+0000","last_undegraded":"2024-03-04T10:03:44.555940+0000","last_fullsized":"2024-03-04T10:03:44.555940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455603+0000","last_change":"2024-03-04T10:03:39.555564+0000","last_active":"2024-03-04T10:03:45.455603+0000","last_peered":"2024-03-04T10:03:45.455603+0000","last_clean":"2024-03-04T10:03:45.455603+0000","last_became_active":"2024-03-04T10:03:39.555310+0000","last_became_peered":"2024-03-04T10:03:39.555310+0000","last_unstale":"2024-03-04T10:03:45.455603+0000","last_undegraded":"2024-03-04T10:03:45.455603+0000","last_fullsized":"2024-03-04T10:03:45.455603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455247+0000","last_change":"2024-03-04T10:03:39.550377+0000","last_active":"2024-03-04T10:03:45.455247+0000","last_peered":"2024-03-04T10:03:45.455247+0000","last_clean":"2024-03-04T10:03:45.455247+0000","last_became_active":"2024-03-04T10:03:39.550028+0000","last_became_peered":"2024-03-04T10:03:39.550028+0000","last_unstale":"2024-03-04T10:03:45.455247+0000","last_undegraded":"2024-03-04T10:03:45.455247+0000","last_fullsized":"2024-03-04T10:03:45.455247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727357+0000","last_change":"2024-03-04T10:03:39.553769+0000","last_active":"2024-03-04T10:03:44.727357+0000","last_peered":"2024-03-04T10:03:44.727357+0000","last_clean":"2024-03-04T10:03:44.727357+0000","last_became_active":"2024-03-04T10:03:39.553517+0000","last_became_peered":"2024-03-04T10:03:39.553517+0000","last_unstale":"2024-03-04T10:03:44.727357+0000","last_undegraded":"2024-03-04T10:03:44.727357+0000","last_fullsized":"2024-03-04T10:03:44.727357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455401+0000","last_change":"2024-03-04T10:03:39.551104+0000","last_active":"2024-03-04T10:03:45.455401+0000","last_peered":"2024-03-04T10:03:45.455401+0000","last_clean":"2024-03-04T10:03:45.455401+0000","last_became_active":"2024-03-04T10:03:39.550823+0000","last_became_peered":"2024-03-04T10:03:39.550823+0000","last_unstale":"2024-03-04T10:03:45.455401+0000","last_undegraded":"2024-03-04T10:03:45.455401+0000","last_fullsized":"2024-03-04T10:03:45.455401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558684+0000","last_change":"2024-03-04T10:03:39.563672+0000","last_active":"2024-03-04T10:03:44.558684+0000","last_peered":"2024-03-04T10:03:44.558684+0000","last_clean":"2024-03-04T10:03:44.558684+0000","last_became_active":"2024-03-04T10:03:39.563394+0000","last_became_peered":"2024-03-04T10:03:39.563394+0000","last_unstale":"2024-03-04T10:03:44.558684+0000","last_undegraded":"2024-03-04T10:03:44.558684+0000","last_fullsized":"2024-03-04T10:03:44.558684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727442+0000","last_change":"2024-03-04T10:03:39.549746+0000","last_active":"2024-03-04T10:03:44.727442+0000","last_peered":"2024-03-04T10:03:44.727442+0000","last_clean":"2024-03-04T10:03:44.727442+0000","last_became_active":"2024-03-04T10:03:39.549349+0000","last_became_peered":"2024-03-04T10:03:39.549349+0000","last_unstale":"2024-03-04T10:03:44.727442+0000","last_undegraded":"2024-03-04T10:03:44.727442+0000","last_fullsized":"2024-03-04T10:03:44.727442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455405+0000","last_change":"2024-03-04T10:03:39.556217+0000","last_active":"2024-03-04T10:03:45.455405+0000","last_peered":"2024-03-04T10:03:45.455405+0000","last_clean":"2024-03-04T10:03:45.455405+0000","last_became_active":"2024-03-04T10:03:39.555896+0000","last_became_peered":"2024-03-04T10:03:39.555896+0000","last_unstale":"2024-03-04T10:03:45.455405+0000","last_undegraded":"2024-03-04T10:03:45.455405+0000","last_fullsized":"2024-03-04T10:03:45.455405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726990+0000","last_change":"2024-03-04T10:03:39.546403+0000","last_active":"2024-03-04T10:03:44.726990+0000","last_peered":"2024-03-04T10:03:44.726990+0000","last_clean":"2024-03-04T10:03:44.726990+0000","last_became_active":"2024-03-04T10:03:39.546136+0000","last_became_peered":"2024-03-04T10:03:39.546136+0000","last_unstale":"2024-03-04T10:03:44.726990+0000","last_undegraded":"2024-03-04T10:03:44.726990+0000","last_fullsized":"2024-03-04T10:03:44.726990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455552+0000","last_change":"2024-03-04T10:03:39.553616+0000","last_active":"2024-03-04T10:03:45.455552+0000","last_peered":"2024-03-04T10:03:45.455552+0000","last_clean":"2024-03-04T10:03:45.455552+0000","last_became_active":"2024-03-04T10:03:39.553356+0000","last_became_peered":"2024-03-04T10:03:39.553356+0000","last_unstale":"2024-03-04T10:03:45.455552+0000","last_undegraded":"2024-03-04T10:03:45.455552+0000","last_fullsized":"2024-03-04T10:03:45.455552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455120+0000","last_change":"2024-03-04T10:03:39.539406+0000","last_active":"2024-03-04T10:03:45.455120+0000","last_peered":"2024-03-04T10:03:45.455120+0000","last_clean":"2024-03-04T10:03:45.455120+0000","last_became_active":"2024-03-04T10:03:39.539139+0000","last_became_peered":"2024-03-04T10:03:39.539139+0000","last_unstale":"2024-03-04T10:03:45.455120+0000","last_undegraded":"2024-03-04T10:03:45.455120+0000","last_fullsized":"2024-03-04T10:03:45.455120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558323+0000","last_change":"2024-03-04T10:03:39.554026+0000","last_active":"2024-03-04T10:03:44.558323+0000","last_peered":"2024-03-04T10:03:44.558323+0000","last_clean":"2024-03-04T10:03:44.558323+0000","last_became_active":"2024-03-04T10:03:39.553711+0000","last_became_peered":"2024-03-04T10:03:39.553711+0000","last_unstale":"2024-03-04T10:03:44.558323+0000","last_undegraded":"2024-03-04T10:03:44.558323+0000","last_fullsized":"2024-03-04T10:03:44.558323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558201+0000","last_change":"2024-03-04T10:03:39.545468+0000","last_active":"2024-03-04T10:03:44.558201+0000","last_peered":"2024-03-04T10:03:44.558201+0000","last_clean":"2024-03-04T10:03:44.558201+0000","last_became_active":"2024-03-04T10:03:39.545202+0000","last_became_peered":"2024-03-04T10:03:39.545202+0000","last_unstale":"2024-03-04T10:03:44.558201+0000","last_undegraded":"2024-03-04T10:03:44.558201+0000","last_fullsized":"2024-03-04T10:03:44.558201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:42.641276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556175+0000","last_change":"2024-03-04T10:03:39.549637+0000","last_active":"2024-03-04T10:03:44.556175+0000","last_peered":"2024-03-04T10:03:44.556175+0000","last_clean":"2024-03-04T10:03:44.556175+0000","last_became_active":"2024-03-04T10:03:39.549384+0000","last_became_peered":"2024-03-04T10:03:39.549384+0000","last_unstale":"2024-03-04T10:03:44.556175+0000","last_undegraded":"2024-03-04T10:03:44.556175+0000","last_fullsized":"2024-03-04T10:03:44.556175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456190+0000","last_change":"2024-03-04T10:03:39.559786+0000","last_active":"2024-03-04T10:03:45.456190+0000","last_peered":"2024-03-04T10:03:45.456190+0000","last_clean":"2024-03-04T10:03:45.456190+0000","last_became_active":"2024-03-04T10:03:39.559507+0000","last_became_peered":"2024-03-04T10:03:39.559507+0000","last_unstale":"2024-03-04T10:03:45.456190+0000","last_undegraded":"2024-03-04T10:03:45.456190+0000","last_fullsized":"2024-03-04T10:03:45.456190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455259+0000","last_change":"2024-03-04T10:03:39.547601+0000","last_active":"2024-03-04T10:03:45.455259+0000","last_peered":"2024-03-04T10:03:45.455259+0000","last_clean":"2024-03-04T10:03:45.455259+0000","last_became_active":"2024-03-04T10:03:39.547309+0000","last_became_peered":"2024-03-04T10:03:39.547309+0000","last_unstale":"2024-03-04T10:03:45.455259+0000","last_undegraded":"2024-03-04T10:03:45.455259+0000","last_fullsized":"2024-03-04T10:03:45.455259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456059+0000","last_change":"2024-03-04T10:03:39.543308+0000","last_active":"2024-03-04T10:03:45.456059+0000","last_peered":"2024-03-04T10:03:45.456059+0000","last_clean":"2024-03-04T10:03:45.456059+0000","last_became_active":"2024-03-04T10:03:39.543064+0000","last_became_peered":"2024-03-04T10:03:39.543064+0000","last_unstale":"2024-03-04T10:03:45.456059+0000","last_undegraded":"2024-03-04T10:03:45.456059+0000","last_fullsized":"2024-03-04T10:03:45.456059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455862+0000","last_change":"2024-03-04T10:03:39.540533+0000","last_active":"2024-03-04T10:03:45.455862+0000","last_peered":"2024-03-04T10:03:45.455862+0000","last_clean":"2024-03-04T10:03:45.455862+0000","last_became_active":"2024-03-04T10:03:39.540274+0000","last_became_peered":"2024-03-04T10:03:39.540274+0000","last_unstale":"2024-03-04T10:03:45.455862+0000","last_undegraded":"2024-03-04T10:03:45.455862+0000","last_fullsized":"2024-03-04T10:03:45.455862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727155+0000","last_change":"2024-03-04T10:03:39.556252+0000","last_active":"2024-03-04T10:03:44.727155+0000","last_peered":"2024-03-04T10:03:44.727155+0000","last_clean":"2024-03-04T10:03:44.727155+0000","last_became_active":"2024-03-04T10:03:39.556009+0000","last_became_peered":"2024-03-04T10:03:39.556009+0000","last_unstale":"2024-03-04T10:03:44.727155+0000","last_undegraded":"2024-03-04T10:03:44.727155+0000","last_fullsized":"2024-03-04T10:03:44.727155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925508+0000","last_change":"2024-03-04T10:03:39.552666+0000","last_active":"2024-03-04T10:03:44.925508+0000","last_peered":"2024-03-04T10:03:44.925508+0000","last_clean":"2024-03-04T10:03:44.925508+0000","last_became_active":"2024-03-04T10:03:39.552178+0000","last_became_peered":"2024-03-04T10:03:39.552178+0000","last_unstale":"2024-03-04T10:03:44.925508+0000","last_undegraded":"2024-03-04T10:03:44.925508+0000","last_fullsized":"2024-03-04T10:03:44.925508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558388+0000","last_change":"2024-03-04T10:03:39.557130+0000","last_active":"2024-03-04T10:03:44.558388+0000","last_peered":"2024-03-04T10:03:44.558388+0000","last_clean":"2024-03-04T10:03:44.558388+0000","last_became_active":"2024-03-04T10:03:39.556736+0000","last_became_peered":"2024-03-04T10:03:39.556736+0000","last_unstale":"2024-03-04T10:03:44.558388+0000","last_undegraded":"2024-03-04T10:03:44.558388+0000","last_fullsized":"2024-03-04T10:03:44.558388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558631+0000","last_change":"2024-03-04T10:03:39.556929+0000","last_active":"2024-03-04T10:03:44.558631+0000","last_peered":"2024-03-04T10:03:44.558631+0000","last_clean":"2024-03-04T10:03:44.558631+0000","last_became_active":"2024-03-04T10:03:39.556672+0000","last_became_peered":"2024-03-04T10:03:39.556672+0000","last_unstale":"2024-03-04T10:03:44.558631+0000","last_undegraded":"2024-03-04T10:03:44.558631+0000","last_fullsized":"2024-03-04T10:03:44.558631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558062+0000","last_change":"2024-03-04T10:03:39.553116+0000","last_active":"2024-03-04T10:03:44.558062+0000","last_peered":"2024-03-04T10:03:44.558062+0000","last_clean":"2024-03-04T10:03:44.558062+0000","last_became_active":"2024-03-04T10:03:39.551779+0000","last_became_peered":"2024-03-04T10:03:39.551779+0000","last_unstale":"2024-03-04T10:03:44.558062+0000","last_undegraded":"2024-03-04T10:03:44.558062+0000","last_fullsized":"2024-03-04T10:03:44.558062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:34.980080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726862+0000","last_change":"2024-03-04T10:03:39.557751+0000","last_active":"2024-03-04T10:03:44.726862+0000","last_peered":"2024-03-04T10:03:44.726862+0000","last_clean":"2024-03-04T10:03:44.726862+0000","last_became_active":"2024-03-04T10:03:39.557443+0000","last_became_peered":"2024-03-04T10:03:39.557443+0000","last_unstale":"2024-03-04T10:03:44.726862+0000","last_undegraded":"2024-03-04T10:03:44.726862+0000","last_fullsized":"2024-03-04T10:03:44.726862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557825+0000","last_change":"2024-03-04T10:03:39.547100+0000","last_active":"2024-03-04T10:03:44.557825+0000","last_peered":"2024-03-04T10:03:44.557825+0000","last_clean":"2024-03-04T10:03:44.557825+0000","last_became_active":"2024-03-04T10:03:39.546794+0000","last_became_peered":"2024-03-04T10:03:39.546794+0000","last_unstale":"2024-03-04T10:03:44.557825+0000","last_undegraded":"2024-03-04T10:03:44.557825+0000","last_fullsized":"2024-03-04T10:03:44.557825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455496+0000","last_change":"2024-03-04T10:03:39.548756+0000","last_active":"2024-03-04T10:03:45.455496+0000","last_peered":"2024-03-04T10:03:45.455496+0000","last_clean":"2024-03-04T10:03:45.455496+0000","last_became_active":"2024-03-04T10:03:39.548474+0000","last_became_peered":"2024-03-04T10:03:39.548474+0000","last_unstale":"2024-03-04T10:03:45.455496+0000","last_undegraded":"2024-03-04T10:03:45.455496+0000","last_fullsized":"2024-03-04T10:03:45.455496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557997+0000","last_change":"2024-03-04T10:03:39.544100+0000","last_active":"2024-03-04T10:03:44.557997+0000","last_peered":"2024-03-04T10:03:44.557997+0000","last_clean":"2024-03-04T10:03:44.557997+0000","last_became_active":"2024-03-04T10:03:39.543581+0000","last_became_peered":"2024-03-04T10:03:39.543581+0000","last_unstale":"2024-03-04T10:03:44.557997+0000","last_undegraded":"2024-03-04T10:03:44.557997+0000","last_fullsized":"2024-03-04T10:03:44.557997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455591+0000","last_change":"2024-03-04T10:03:39.556280+0000","last_active":"2024-03-04T10:03:45.455591+0000","last_peered":"2024-03-04T10:03:45.455591+0000","last_clean":"2024-03-04T10:03:45.455591+0000","last_became_active":"2024-03-04T10:03:39.555867+0000","last_became_peered":"2024-03-04T10:03:39.555867+0000","last_unstale":"2024-03-04T10:03:45.455591+0000","last_undegraded":"2024-03-04T10:03:45.455591+0000","last_fullsized":"2024-03-04T10:03:45.455591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455074+0000","last_change":"2024-03-04T10:03:39.538030+0000","last_active":"2024-03-04T10:03:45.455074+0000","last_peered":"2024-03-04T10:03:45.455074+0000","last_clean":"2024-03-04T10:03:45.455074+0000","last_became_active":"2024-03-04T10:03:39.537783+0000","last_became_peered":"2024-03-04T10:03:39.537783+0000","last_unstale":"2024-03-04T10:03:45.455074+0000","last_undegraded":"2024-03-04T10:03:45.455074+0000","last_fullsized":"2024-03-04T10:03:45.455074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455845+0000","last_change":"2024-03-04T10:03:39.545505+0000","last_active":"2024-03-04T10:03:45.455845+0000","last_peered":"2024-03-04T10:03:45.455845+0000","last_clean":"2024-03-04T10:03:45.455845+0000","last_became_active":"2024-03-04T10:03:39.545176+0000","last_became_peered":"2024-03-04T10:03:39.545176+0000","last_unstale":"2024-03-04T10:03:45.455845+0000","last_undegraded":"2024-03-04T10:03:45.455845+0000","last_fullsized":"2024-03-04T10:03:45.455845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557636+0000","last_change":"2024-03-04T10:03:39.551482+0000","last_active":"2024-03-04T10:03:44.557636+0000","last_peered":"2024-03-04T10:03:44.557636+0000","last_clean":"2024-03-04T10:03:44.557636+0000","last_became_active":"2024-03-04T10:03:39.551234+0000","last_became_peered":"2024-03-04T10:03:39.551234+0000","last_unstale":"2024-03-04T10:03:44.557636+0000","last_undegraded":"2024-03-04T10:03:44.557636+0000","last_fullsized":"2024-03-04T10:03:44.557636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:22.803941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:04:06.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:04:07.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.030+0000 7fa262acb640 1 Processor -- start 2024-03-04T10:04:07.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.030+0000 7fa262acb640 1 -- start start 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa262acb640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 0x7fa25c15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa262acb640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c1624e0 con 0x7fa25c0cfab0 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa262acb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25c0cd010 con 0x7fa25c0cf650 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa262acb640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c071df0 con 0x7fa25c154a80 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa261068640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa25c0cfab0 0x7fa25c161dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34862/0 (socket says 172.21.15.99:34862) 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa261068640 1 -- 172.21.15.99:0/2894469263 learned_addr learned my addr 172.21.15.99:0/2894469263 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554435886 0 0) 0x7fa25c1624e0 con 0x7fa25c0cfab0 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa260867640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 0x7fa25c15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa24c003520 con 0x7fa25c0cfab0 2024-03-04T10:04:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2224161553 0 0) 0x7fa24c003520 con 0x7fa25c0cfab0 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 msgr2=0x7fa25c15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 0x7fa25c15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 >> v1:172.21.15.162:6789/0 conn(0x7fa25c0cf650 legacy=0x7fa25c1616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0cdaa0 con 0x7fa25c0cfab0 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fa2540019f0 con 0x7fa25c0cfab0 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 >> v1:172.21.15.99:6789/0 conn(0x7fa25c0cfab0 legacy=0x7fa25c161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 0x7fa24c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24c003f60 0x7fa24c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa24c004d40 0x7fa24c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c071df0 con 0x7fa24c004d40 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c00baa0 con 0x7fa24c003f60 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa24c00bca0 con 0x7fa25c154a80 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa261068640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa24c004d40 0x7fa24c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa25bfff640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24c003f60 0x7fa24c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.031+0000 7fa260867640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 0x7fa24c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.032+0000 7fa261068640 1 -- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 msgr2=0x7fa24c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.032+0000 7fa261068640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c154a80 0x7fa24c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.032+0000 7fa261068640 1 -- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24c003f60 msgr2=0x7fa24c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.032+0000 7fa261068640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa24c003f60 0x7fa24c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.032+0000 7fa261068640 1 -- 172.21.15.99:0/2894469263 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c003520 con 0x7fa24c004d40 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.032+0000 7fa261068640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa24c004d40 0x7fa24c0051c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa25400a840 tx=0x7fa24c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8c32cfac8723c44 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25400a460 con 0x7fa24c004d40 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa254034630 con 0x7fa24c004d40 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa25b7fe640 1 -- 172.21.15.99:0/2894469263 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254034a60 con 0x7fa24c004d40 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa262acb640 1 -- 172.21.15.99:0/2894469263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa24c004d40 msgr2=0x7fa24c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa262acb640 1 --2- 172.21.15.99:0/2894469263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa24c004d40 0x7fa24c0051c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa25400a840 tx=0x7fa24c005830 comp rx=0 tx=0).stop 2024-03-04T10:04:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa261068640 1 -- 172.21.15.99:0/2894469263 reap_dead start 2024-03-04T10:04:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa262acb640 1 -- 172.21.15.99:0/2894469263 shutdown_connections 2024-03-04T10:04:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa262acb640 1 -- 172.21.15.99:0/2894469263 >> 172.21.15.99:0/2894469263 conn(0x7fa25c06f5f0 msgr2=0x7fa25c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa262acb640 1 -- 172.21.15.99:0/2894469263 shutdown_connections 2024-03-04T10:04:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.033+0000 7fa262acb640 1 -- 172.21.15.99:0/2894469263 wait complete. 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.034+0000 7fa262acb640 1 Processor -- start 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.034+0000 7fa262acb640 1 -- start start 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa262acb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 0x7fa25c0b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa262acb640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c0b4650 0x7fa25c0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa262acb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa25c15e860 0x7fa25c15ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa262acb640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0ce340 con 0x7fa25c0b5440 2024-03-04T10:04:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa262acb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0ce060 con 0x7fa25c15e860 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa262acb640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0ce620 con 0x7fa25c0b4650 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa260867640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 0x7fa25c0b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa260867640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 0x7fa25c0b5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38330/0 (socket says 172.21.15.99:38330) 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa260867640 1 -- 172.21.15.99:0/3690784659 learned_addr learned my addr 172.21.15.99:0/3690784659 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa25bfff640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c0b4650 0x7fa25c0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.035+0000 7fa261068640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa25c15e860 0x7fa25c15ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa260867640 1 -- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c0b4650 msgr2=0x7fa25c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa260867640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c0b4650 0x7fa25c0b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa260867640 1 -- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa25c15e860 msgr2=0x7fa25c15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa260867640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa25c15e860 0x7fa25c15ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa260867640 1 -- 172.21.15.99:0/3690784659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c160b10 con 0x7fa25c0b5440 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa260867640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 0x7fa25c0b5820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa250011a00 tx=0x7fa250011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000fb50 con 0x7fa25c0b5440 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25000fcf0 con 0x7fa25c0b5440 2024-03-04T10:04:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c160df0 con 0x7fa25c0b5440 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000fea0 con 0x7fa25c0b5440 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.036+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c1613b0 con 0x7fa25c0b5440 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.037+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c074f60 con 0x7fa25c0b5440 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.037+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa250012040 con 0x7fa25c0b5440 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.038+0000 7fa2597fa640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa24006b500 0x7fa24006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.038+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa25008acd0 con 0x7fa25c0b5440 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.038+0000 7fa25bfff640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa24006b500 0x7fa24006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.039+0000 7fa25bfff640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa24006b500 0x7fa24006d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0c51a0 tx=0x7fa248009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.042+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa250019030 con 0x7fa25c0b5440 2024-03-04T10:04:07.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.211+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa25c0d1b40 con 0x7fa24006b500 2024-03-04T10:04:07.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.219+0000 7fa2597fa640 1 -- 172.21.15.99:0/3690784659 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332533 (secure 0 0 0) 0x7fa25c0d1b40 con 0x7fa24006b500 2024-03-04T10:04:07.224 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:07.229 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:04:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa24006b500 msgr2=0x7fa24006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa24006b500 0x7fa24006d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0c51a0 tx=0x7fa248009040 comp rx=0 tx=0).stop 2024-03-04T10:04:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 msgr2=0x7fa25c0b5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 0x7fa25c0b5820 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa250011a00 tx=0x7fa250011ec0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 shutdown_connections 2024-03-04T10:04:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa24006b500 0x7fa24006d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa25c15e860 0x7fa25c15ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa25c0b4650 0x7fa25c0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 --2- 172.21.15.99:0/3690784659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa25c0b5440 0x7fa25c0b5820 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 >> 172.21.15.99:0/3690784659 conn(0x7fa25c06f5f0 msgr2=0x7fa25c0b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:07.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.222+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 shutdown_connections 2024-03-04T10:04:07.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.223+0000 7fa262acb640 1 -- 172.21.15.99:0/3690784659 wait complete. 2024-03-04T10:04:07.235 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-04T10:04:05.442829+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219388,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755332,"statfs":{"total":773094113280,"available":772869459968,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003690"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455527+0000","last_change":"2024-03-04T10:03:38.529898+0000","last_active":"2024-03-04T10:03:45.455527+0000","last_peered":"2024-03-04T10:03:45.455527+0000","last_clean":"2024-03-04T10:03:45.455527+0000","last_became_active":"2024-03-04T10:03:38.529499+0000","last_became_peered":"2024-03-04T10:03:38.529499+0000","last_unstale":"2024-03-04T10:03:45.455527+0000","last_undegraded":"2024-03-04T10:03:45.455527+0000","last_fullsized":"2024-03-04T10:03:45.455527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727461+0000","last_change":"2024-03-04T10:03:39.556883+0000","last_active":"2024-03-04T10:03:44.727461+0000","last_peered":"2024-03-04T10:03:44.727461+0000","last_clean":"2024-03-04T10:03:44.727461+0000","last_became_active":"2024-03-04T10:03:39.556616+0000","last_became_peered":"2024-03-04T10:03:39.556616+0000","last_unstale":"2024-03-04T10:03:44.727461+0000","last_undegraded":"2024-03-04T10:03:44.727461+0000","last_fullsized":"2024-03-04T10:03:44.727461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726874+0000","last_change":"2024-03-04T10:03:43.569338+0000","last_active":"2024-03-04T10:03:44.726874+0000","last_peered":"2024-03-04T10:03:44.726874+0000","last_clean":"2024-03-04T10:03:44.726874+0000","last_became_active":"2024-03-04T10:03:43.568332+0000","last_became_peered":"2024-03-04T10:03:43.568332+0000","last_unstale":"2024-03-04T10:03:44.726874+0000","last_undegraded":"2024-03-04T10:03:44.726874+0000","last_fullsized":"2024-03-04T10:03:44.726874+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727559+0000","last_change":"2024-03-04T10:03:38.533956+0000","last_active":"2024-03-04T10:03:44.727559+0000","last_peered":"2024-03-04T10:03:44.727559+0000","last_clean":"2024-03-04T10:03:44.727559+0000","last_became_active":"2024-03-04T10:03:38.533665+0000","last_became_peered":"2024-03-04T10:03:38.533665+0000","last_unstale":"2024-03-04T10:03:44.727559+0000","last_undegraded":"2024-03-04T10:03:44.727559+0000","last_fullsized":"2024-03-04T10:03:44.727559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455218+0000","last_change":"2024-03-04T10:03:39.550341+0000","last_active":"2024-03-04T10:03:45.455218+0000","last_peered":"2024-03-04T10:03:45.455218+0000","last_clean":"2024-03-04T10:03:45.455218+0000","last_became_active":"2024-03-04T10:03:39.549960+0000","last_became_peered":"2024-03-04T10:03:39.549960+0000","last_unstale":"2024-03-04T10:03:45.455218+0000","last_undegraded":"2024-03-04T10:03:45.455218+0000","last_fullsized":"2024-03-04T10:03:45.455218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925071+0000","last_change":"2024-03-04T10:03:43.578315+0000","last_active":"2024-03-04T10:03:44.925071+0000","last_peered":"2024-03-04T10:03:44.925071+0000","last_clean":"2024-03-04T10:03:44.925071+0000","last_became_active":"2024-03-04T10:03:43.577864+0000","last_became_peered":"2024-03-04T10:03:43.577864+0000","last_unstale":"2024-03-04T10:03:44.925071+0000","last_undegraded":"2024-03-04T10:03:44.925071+0000","last_fullsized":"2024-03-04T10:03:44.925071+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:27.973529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558121+0000","last_change":"2024-03-04T10:03:38.536815+0000","last_active":"2024-03-04T10:03:44.558121+0000","last_peered":"2024-03-04T10:03:44.558121+0000","last_clean":"2024-03-04T10:03:44.558121+0000","last_became_active":"2024-03-04T10:03:38.536526+0000","last_became_peered":"2024-03-04T10:03:38.536526+0000","last_unstale":"2024-03-04T10:03:44.558121+0000","last_undegraded":"2024-03-04T10:03:44.558121+0000","last_fullsized":"2024-03-04T10:03:44.558121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455699+0000","last_change":"2024-03-04T10:03:39.551796+0000","last_active":"2024-03-04T10:03:45.455699+0000","last_peered":"2024-03-04T10:03:45.455699+0000","last_clean":"2024-03-04T10:03:45.455699+0000","last_became_active":"2024-03-04T10:03:39.551527+0000","last_became_peered":"2024-03-04T10:03:39.551527+0000","last_unstale":"2024-03-04T10:03:45.455699+0000","last_undegraded":"2024-03-04T10:03:45.455699+0000","last_fullsized":"2024-03-04T10:03:45.455699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925354+0000","last_change":"2024-03-04T10:03:43.573372+0000","last_active":"2024-03-04T10:03:44.925354+0000","last_peered":"2024-03-04T10:03:44.925354+0000","last_clean":"2024-03-04T10:03:44.925354+0000","last_became_active":"2024-03-04T10:03:43.572926+0000","last_became_peered":"2024-03-04T10:03:43.572926+0000","last_unstale":"2024-03-04T10:03:44.925354+0000","last_undegraded":"2024-03-04T10:03:44.925354+0000","last_fullsized":"2024-03-04T10:03:44.925354+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:37.210258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557408+0000","last_change":"2024-03-04T10:03:38.535163+0000","last_active":"2024-03-04T10:03:44.557408+0000","last_peered":"2024-03-04T10:03:44.557408+0000","last_clean":"2024-03-04T10:03:44.557408+0000","last_became_active":"2024-03-04T10:03:38.534751+0000","last_became_peered":"2024-03-04T10:03:38.534751+0000","last_unstale":"2024-03-04T10:03:44.557408+0000","last_undegraded":"2024-03-04T10:03:44.557408+0000","last_fullsized":"2024-03-04T10:03:44.557408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:40.978639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455945+0000","last_change":"2024-03-04T10:03:39.548628+0000","last_active":"2024-03-04T10:03:45.455945+0000","last_peered":"2024-03-04T10:03:45.455945+0000","last_clean":"2024-03-04T10:03:45.455945+0000","last_became_active":"2024-03-04T10:03:39.548297+0000","last_became_peered":"2024-03-04T10:03:39.548297+0000","last_unstale":"2024-03-04T10:03:45.455945+0000","last_undegraded":"2024-03-04T10:03:45.455945+0000","last_fullsized":"2024-03-04T10:03:45.455945+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925131+0000","last_change":"2024-03-04T10:03:43.576063+0000","last_active":"2024-03-04T10:03:44.925131+0000","last_peered":"2024-03-04T10:03:44.925131+0000","last_clean":"2024-03-04T10:03:44.925131+0000","last_became_active":"2024-03-04T10:03:43.575658+0000","last_became_peered":"2024-03-04T10:03:43.575658+0000","last_unstale":"2024-03-04T10:03:44.925131+0000","last_undegraded":"2024-03-04T10:03:44.925131+0000","last_fullsized":"2024-03-04T10:03:44.925131+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:47.766859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557898+0000","last_change":"2024-03-04T10:03:39.562772+0000","last_active":"2024-03-04T10:03:44.557898+0000","last_peered":"2024-03-04T10:03:44.557898+0000","last_clean":"2024-03-04T10:03:44.557898+0000","last_became_active":"2024-03-04T10:03:39.562479+0000","last_became_peered":"2024-03-04T10:03:39.562479+0000","last_unstale":"2024-03-04T10:03:44.557898+0000","last_undegraded":"2024-03-04T10:03:44.557898+0000","last_fullsized":"2024-03-04T10:03:44.557898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:48.563637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558549+0000","last_change":"2024-03-04T10:03:38.524487+0000","last_active":"2024-03-04T10:03:44.558549+0000","last_peered":"2024-03-04T10:03:44.558549+0000","last_clean":"2024-03-04T10:03:44.558549+0000","last_became_active":"2024-03-04T10:03:38.524186+0000","last_became_peered":"2024-03-04T10:03:38.524186+0000","last_unstale":"2024-03-04T10:03:44.558549+0000","last_undegraded":"2024-03-04T10:03:44.558549+0000","last_fullsized":"2024-03-04T10:03:44.558549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925560+0000","last_change":"2024-03-04T10:03:43.573621+0000","last_active":"2024-03-04T10:03:44.925560+0000","last_peered":"2024-03-04T10:03:44.925560+0000","last_clean":"2024-03-04T10:03:44.925560+0000","last_became_active":"2024-03-04T10:03:43.573125+0000","last_became_peered":"2024-03-04T10:03:43.573125+0000","last_unstale":"2024-03-04T10:03:44.925560+0000","last_undegraded":"2024-03-04T10:03:44.925560+0000","last_fullsized":"2024-03-04T10:03:44.925560+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727176+0000","last_change":"2024-03-04T10:03:39.553163+0000","last_active":"2024-03-04T10:03:44.727176+0000","last_peered":"2024-03-04T10:03:44.727176+0000","last_clean":"2024-03-04T10:03:44.727176+0000","last_became_active":"2024-03-04T10:03:39.552909+0000","last_became_peered":"2024-03-04T10:03:39.552909+0000","last_unstale":"2024-03-04T10:03:44.727176+0000","last_undegraded":"2024-03-04T10:03:44.727176+0000","last_fullsized":"2024-03-04T10:03:44.727176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455728+0000","last_change":"2024-03-04T10:03:38.534433+0000","last_active":"2024-03-04T10:03:45.455728+0000","last_peered":"2024-03-04T10:03:45.455728+0000","last_clean":"2024-03-04T10:03:45.455728+0000","last_became_active":"2024-03-04T10:03:38.534162+0000","last_became_peered":"2024-03-04T10:03:38.534162+0000","last_unstale":"2024-03-04T10:03:45.455728+0000","last_undegraded":"2024-03-04T10:03:45.455728+0000","last_fullsized":"2024-03-04T10:03:45.455728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925353+0000","last_change":"2024-03-04T10:03:43.572145+0000","last_active":"2024-03-04T10:03:44.925353+0000","last_peered":"2024-03-04T10:03:44.925353+0000","last_clean":"2024-03-04T10:03:44.925353+0000","last_became_active":"2024-03-04T10:03:43.571762+0000","last_became_peered":"2024-03-04T10:03:43.571762+0000","last_unstale":"2024-03-04T10:03:44.925353+0000","last_undegraded":"2024-03-04T10:03:44.925353+0000","last_fullsized":"2024-03-04T10:03:44.925353+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:43.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727042+0000","last_change":"2024-03-04T10:03:39.558092+0000","last_active":"2024-03-04T10:03:44.727042+0000","last_peered":"2024-03-04T10:03:44.727042+0000","last_clean":"2024-03-04T10:03:44.727042+0000","last_became_active":"2024-03-04T10:03:39.557838+0000","last_became_peered":"2024-03-04T10:03:39.557838+0000","last_unstale":"2024-03-04T10:03:44.727042+0000","last_undegraded":"2024-03-04T10:03:44.727042+0000","last_fullsized":"2024-03-04T10:03:44.727042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456127+0000","last_change":"2024-03-04T10:03:38.538667+0000","last_active":"2024-03-04T10:03:45.456127+0000","last_peered":"2024-03-04T10:03:45.456127+0000","last_clean":"2024-03-04T10:03:45.456127+0000","last_became_active":"2024-03-04T10:03:38.538344+0000","last_became_peered":"2024-03-04T10:03:38.538344+0000","last_unstale":"2024-03-04T10:03:45.456127+0000","last_undegraded":"2024-03-04T10:03:45.456127+0000","last_fullsized":"2024-03-04T10:03:45.456127+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456240+0000","last_change":"2024-03-04T10:03:43.583702+0000","last_active":"2024-03-04T10:03:45.456240+0000","last_peered":"2024-03-04T10:03:45.456240+0000","last_clean":"2024-03-04T10:03:45.456240+0000","last_became_active":"2024-03-04T10:03:43.583287+0000","last_became_peered":"2024-03-04T10:03:43.583287+0000","last_unstale":"2024-03-04T10:03:45.456240+0000","last_undegraded":"2024-03-04T10:03:45.456240+0000","last_fullsized":"2024-03-04T10:03:45.456240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:26.836809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925694+0000","last_change":"2024-03-04T10:03:39.542379+0000","last_active":"2024-03-04T10:03:44.925694+0000","last_peered":"2024-03-04T10:03:44.925694+0000","last_clean":"2024-03-04T10:03:44.925694+0000","last_became_active":"2024-03-04T10:03:39.541991+0000","last_became_peered":"2024-03-04T10:03:39.541991+0000","last_unstale":"2024-03-04T10:03:44.925694+0000","last_undegraded":"2024-03-04T10:03:44.925694+0000","last_fullsized":"2024-03-04T10:03:44.925694+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:48.539804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557756+0000","last_change":"2024-03-04T10:03:38.532133+0000","last_active":"2024-03-04T10:03:44.557756+0000","last_peered":"2024-03-04T10:03:44.557756+0000","last_clean":"2024-03-04T10:03:44.557756+0000","last_became_active":"2024-03-04T10:03:38.531622+0000","last_became_peered":"2024-03-04T10:03:38.531622+0000","last_unstale":"2024-03-04T10:03:44.557756+0000","last_undegraded":"2024-03-04T10:03:44.557756+0000","last_fullsized":"2024-03-04T10:03:44.557756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925652+0000","last_change":"2024-03-04T10:03:43.570872+0000","last_active":"2024-03-04T10:03:44.925652+0000","last_peered":"2024-03-04T10:03:44.925652+0000","last_clean":"2024-03-04T10:03:44.925652+0000","last_became_active":"2024-03-04T10:03:43.570436+0000","last_became_peered":"2024-03-04T10:03:43.570436+0000","last_unstale":"2024-03-04T10:03:44.925652+0000","last_undegraded":"2024-03-04T10:03:44.925652+0000","last_fullsized":"2024-03-04T10:03:44.925652+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455648+0000","last_change":"2024-03-04T10:03:39.541376+0000","last_active":"2024-03-04T10:03:45.455648+0000","last_peered":"2024-03-04T10:03:45.455648+0000","last_clean":"2024-03-04T10:03:45.455648+0000","last_became_active":"2024-03-04T10:03:39.541117+0000","last_became_peered":"2024-03-04T10:03:39.541117+0000","last_unstale":"2024-03-04T10:03:45.455648+0000","last_undegraded":"2024-03-04T10:03:45.455648+0000","last_fullsized":"2024-03-04T10:03:45.455648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558575+0000","last_change":"2024-03-04T10:03:38.538684+0000","last_active":"2024-03-04T10:03:44.558575+0000","last_peered":"2024-03-04T10:03:44.558575+0000","last_clean":"2024-03-04T10:03:44.558575+0000","last_became_active":"2024-03-04T10:03:38.538374+0000","last_became_peered":"2024-03-04T10:03:38.538374+0000","last_unstale":"2024-03-04T10:03:44.558575+0000","last_undegraded":"2024-03-04T10:03:44.558575+0000","last_fullsized":"2024-03-04T10:03:44.558575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925011+0000","last_change":"2024-03-04T10:03:43.575117+0000","last_active":"2024-03-04T10:03:44.925011+0000","last_peered":"2024-03-04T10:03:44.925011+0000","last_clean":"2024-03-04T10:03:44.925011+0000","last_became_active":"2024-03-04T10:03:43.574283+0000","last_became_peered":"2024-03-04T10:03:43.574283+0000","last_unstale":"2024-03-04T10:03:44.925011+0000","last_undegraded":"2024-03-04T10:03:44.925011+0000","last_fullsized":"2024-03-04T10:03:44.925011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:09.615407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727535+0000","last_change":"2024-03-04T10:03:39.554004+0000","last_active":"2024-03-04T10:03:44.727535+0000","last_peered":"2024-03-04T10:03:44.727535+0000","last_clean":"2024-03-04T10:03:44.727535+0000","last_became_active":"2024-03-04T10:03:39.553717+0000","last_became_peered":"2024-03-04T10:03:39.553717+0000","last_unstale":"2024-03-04T10:03:44.727535+0000","last_undegraded":"2024-03-04T10:03:44.727535+0000","last_fullsized":"2024-03-04T10:03:44.727535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455963+0000","last_change":"2024-03-04T10:03:38.533053+0000","last_active":"2024-03-04T10:03:45.455963+0000","last_peered":"2024-03-04T10:03:45.455963+0000","last_clean":"2024-03-04T10:03:45.455963+0000","last_became_active":"2024-03-04T10:03:38.532793+0000","last_became_peered":"2024-03-04T10:03:38.532793+0000","last_unstale":"2024-03-04T10:03:45.455963+0000","last_undegraded":"2024-03-04T10:03:45.455963+0000","last_fullsized":"2024-03-04T10:03:45.455963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:40.697210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726840+0000","last_change":"2024-03-04T10:03:43.577818+0000","last_active":"2024-03-04T10:03:44.726840+0000","last_peered":"2024-03-04T10:03:44.726840+0000","last_clean":"2024-03-04T10:03:44.726840+0000","last_became_active":"2024-03-04T10:03:43.577239+0000","last_became_peered":"2024-03-04T10:03:43.577239+0000","last_unstale":"2024-03-04T10:03:44.726840+0000","last_undegraded":"2024-03-04T10:03:44.726840+0000","last_fullsized":"2024-03-04T10:03:44.726840+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455944+0000","last_change":"2024-03-04T10:03:39.542820+0000","last_active":"2024-03-04T10:03:45.455944+0000","last_peered":"2024-03-04T10:03:45.455944+0000","last_clean":"2024-03-04T10:03:45.455944+0000","last_became_active":"2024-03-04T10:03:39.542521+0000","last_became_peered":"2024-03-04T10:03:39.542521+0000","last_unstale":"2024-03-04T10:03:45.455944+0000","last_undegraded":"2024-03-04T10:03:45.455944+0000","last_fullsized":"2024-03-04T10:03:45.455944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558219+0000","last_change":"2024-03-04T10:03:38.534337+0000","last_active":"2024-03-04T10:03:44.558219+0000","last_peered":"2024-03-04T10:03:44.558219+0000","last_clean":"2024-03-04T10:03:44.558219+0000","last_became_active":"2024-03-04T10:03:38.534027+0000","last_became_peered":"2024-03-04T10:03:38.534027+0000","last_unstale":"2024-03-04T10:03:44.558219+0000","last_undegraded":"2024-03-04T10:03:44.558219+0000","last_fullsized":"2024-03-04T10:03:44.558219+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:18.373547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455176+0000","last_change":"2024-03-04T10:03:43.577959+0000","last_active":"2024-03-04T10:03:45.455176+0000","last_peered":"2024-03-04T10:03:45.455176+0000","last_clean":"2024-03-04T10:03:45.455176+0000","last_became_active":"2024-03-04T10:03:43.576536+0000","last_became_peered":"2024-03-04T10:03:43.576536+0000","last_unstale":"2024-03-04T10:03:45.455176+0000","last_undegraded":"2024-03-04T10:03:45.455176+0000","last_fullsized":"2024-03-04T10:03:45.455176+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456091+0000","last_change":"2024-03-04T10:03:39.556129+0000","last_active":"2024-03-04T10:03:45.456091+0000","last_peered":"2024-03-04T10:03:45.456091+0000","last_clean":"2024-03-04T10:03:45.456091+0000","last_became_active":"2024-03-04T10:03:39.555824+0000","last_became_peered":"2024-03-04T10:03:39.555824+0000","last_unstale":"2024-03-04T10:03:45.456091+0000","last_undegraded":"2024-03-04T10:03:45.456091+0000","last_fullsized":"2024-03-04T10:03:45.456091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456030+0000","last_change":"2024-03-04T10:03:38.531643+0000","last_active":"2024-03-04T10:03:45.456030+0000","last_peered":"2024-03-04T10:03:45.456030+0000","last_clean":"2024-03-04T10:03:45.456030+0000","last_became_active":"2024-03-04T10:03:38.531299+0000","last_became_peered":"2024-03-04T10:03:38.531299+0000","last_unstale":"2024-03-04T10:03:45.456030+0000","last_undegraded":"2024-03-04T10:03:45.456030+0000","last_fullsized":"2024-03-04T10:03:45.456030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455431+0000","last_change":"2024-03-04T10:03:43.576642+0000","last_active":"2024-03-04T10:03:45.455431+0000","last_peered":"2024-03-04T10:03:45.455431+0000","last_clean":"2024-03-04T10:03:45.455431+0000","last_became_active":"2024-03-04T10:03:43.576270+0000","last_became_peered":"2024-03-04T10:03:43.576270+0000","last_unstale":"2024-03-04T10:03:45.455431+0000","last_undegraded":"2024-03-04T10:03:45.455431+0000","last_fullsized":"2024-03-04T10:03:45.455431+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925763+0000","last_change":"2024-03-04T10:03:39.546721+0000","last_active":"2024-03-04T10:03:44.925763+0000","last_peered":"2024-03-04T10:03:44.925763+0000","last_clean":"2024-03-04T10:03:44.925763+0000","last_became_active":"2024-03-04T10:03:39.546438+0000","last_became_peered":"2024-03-04T10:03:39.546438+0000","last_unstale":"2024-03-04T10:03:44.925763+0000","last_undegraded":"2024-03-04T10:03:44.925763+0000","last_fullsized":"2024-03-04T10:03:44.925763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455730+0000","last_change":"2024-03-04T10:03:38.534806+0000","last_active":"2024-03-04T10:03:45.455730+0000","last_peered":"2024-03-04T10:03:45.455730+0000","last_clean":"2024-03-04T10:03:45.455730+0000","last_became_active":"2024-03-04T10:03:38.534555+0000","last_became_peered":"2024-03-04T10:03:38.534555+0000","last_unstale":"2024-03-04T10:03:45.455730+0000","last_undegraded":"2024-03-04T10:03:45.455730+0000","last_fullsized":"2024-03-04T10:03:45.455730+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455898+0000","last_change":"2024-03-04T10:03:43.583916+0000","last_active":"2024-03-04T10:03:45.455898+0000","last_peered":"2024-03-04T10:03:45.455898+0000","last_clean":"2024-03-04T10:03:45.455898+0000","last_became_active":"2024-03-04T10:03:43.583537+0000","last_became_peered":"2024-03-04T10:03:43.583537+0000","last_unstale":"2024-03-04T10:03:45.455898+0000","last_undegraded":"2024-03-04T10:03:45.455898+0000","last_fullsized":"2024-03-04T10:03:45.455898+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:46.674144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454949+0000","last_change":"2024-03-04T10:03:39.539248+0000","last_active":"2024-03-04T10:03:45.454949+0000","last_peered":"2024-03-04T10:03:45.454949+0000","last_clean":"2024-03-04T10:03:45.454949+0000","last_became_active":"2024-03-04T10:03:39.538978+0000","last_became_peered":"2024-03-04T10:03:39.538978+0000","last_unstale":"2024-03-04T10:03:45.454949+0000","last_undegraded":"2024-03-04T10:03:45.454949+0000","last_fullsized":"2024-03-04T10:03:45.454949+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556029+0000","last_change":"2024-03-04T10:03:38.534970+0000","last_active":"2024-03-04T10:03:44.556029+0000","last_peered":"2024-03-04T10:03:44.556029+0000","last_clean":"2024-03-04T10:03:44.556029+0000","last_became_active":"2024-03-04T10:03:38.534694+0000","last_became_peered":"2024-03-04T10:03:38.534694+0000","last_unstale":"2024-03-04T10:03:44.556029+0000","last_undegraded":"2024-03-04T10:03:44.556029+0000","last_fullsized":"2024-03-04T10:03:44.556029+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456270+0000","last_change":"2024-03-04T10:03:43.582855+0000","last_active":"2024-03-04T10:03:45.456270+0000","last_peered":"2024-03-04T10:03:45.456270+0000","last_clean":"2024-03-04T10:03:45.456270+0000","last_became_active":"2024-03-04T10:03:43.580626+0000","last_became_peered":"2024-03-04T10:03:43.580626+0000","last_unstale":"2024-03-04T10:03:45.456270+0000","last_undegraded":"2024-03-04T10:03:45.456270+0000","last_fullsized":"2024-03-04T10:03:45.456270+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:39.012948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.454952+0000","last_change":"2024-03-04T10:03:39.538900+0000","last_active":"2024-03-04T10:03:45.454952+0000","last_peered":"2024-03-04T10:03:45.454952+0000","last_clean":"2024-03-04T10:03:45.454952+0000","last_became_active":"2024-03-04T10:03:39.538624+0000","last_became_peered":"2024-03-04T10:03:39.538624+0000","last_unstale":"2024-03-04T10:03:45.454952+0000","last_undegraded":"2024-03-04T10:03:45.454952+0000","last_fullsized":"2024-03-04T10:03:45.454952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557795+0000","last_change":"2024-03-04T10:03:38.534850+0000","last_active":"2024-03-04T10:03:44.557795+0000","last_peered":"2024-03-04T10:03:44.557795+0000","last_clean":"2024-03-04T10:03:44.557795+0000","last_became_active":"2024-03-04T10:03:38.534604+0000","last_became_peered":"2024-03-04T10:03:38.534604+0000","last_unstale":"2024-03-04T10:03:44.557795+0000","last_undegraded":"2024-03-04T10:03:44.557795+0000","last_fullsized":"2024-03-04T10:03:44.557795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455383+0000","last_change":"2024-03-04T10:03:43.576525+0000","last_active":"2024-03-04T10:03:45.455383+0000","last_peered":"2024-03-04T10:03:45.455383+0000","last_clean":"2024-03-04T10:03:45.455383+0000","last_became_active":"2024-03-04T10:03:43.576149+0000","last_became_peered":"2024-03-04T10:03:43.576149+0000","last_unstale":"2024-03-04T10:03:45.455383+0000","last_undegraded":"2024-03-04T10:03:45.455383+0000","last_fullsized":"2024-03-04T10:03:45.455383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:51.836606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727079+0000","last_change":"2024-03-04T10:03:39.557088+0000","last_active":"2024-03-04T10:03:44.727079+0000","last_peered":"2024-03-04T10:03:44.727079+0000","last_clean":"2024-03-04T10:03:44.727079+0000","last_became_active":"2024-03-04T10:03:39.556715+0000","last_became_peered":"2024-03-04T10:03:39.556715+0000","last_unstale":"2024-03-04T10:03:44.727079+0000","last_undegraded":"2024-03-04T10:03:44.727079+0000","last_fullsized":"2024-03-04T10:03:44.727079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455694+0000","last_change":"2024-03-04T10:03:38.537571+0000","last_active":"2024-03-04T10:03:45.455694+0000","last_peered":"2024-03-04T10:03:45.455694+0000","last_clean":"2024-03-04T10:03:45.455694+0000","last_became_active":"2024-03-04T10:03:38.536646+0000","last_became_peered":"2024-03-04T10:03:38.536646+0000","last_unstale":"2024-03-04T10:03:45.455694+0000","last_undegraded":"2024-03-04T10:03:45.455694+0000","last_fullsized":"2024-03-04T10:03:45.455694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557928+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:03:44.557928+0000","last_peered":"2024-03-04T10:03:44.557928+0000","last_clean":"2024-03-04T10:03:44.557928+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:03:44.557928+0000","last_undegraded":"2024-03-04T10:03:44.557928+0000","last_fullsized":"2024-03-04T10:03:44.557928+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558259+0000","last_change":"2024-03-04T10:03:43.579804+0000","last_active":"2024-03-04T10:03:44.558259+0000","last_peered":"2024-03-04T10:03:44.558259+0000","last_clean":"2024-03-04T10:03:44.558259+0000","last_became_active":"2024-03-04T10:03:43.579420+0000","last_became_peered":"2024-03-04T10:03:43.579420+0000","last_unstale":"2024-03-04T10:03:44.558259+0000","last_undegraded":"2024-03-04T10:03:44.558259+0000","last_fullsized":"2024-03-04T10:03:44.558259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:16.559608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557993+0000","last_change":"2024-03-04T10:03:39.550717+0000","last_active":"2024-03-04T10:03:44.557993+0000","last_peered":"2024-03-04T10:03:44.557993+0000","last_clean":"2024-03-04T10:03:44.557993+0000","last_became_active":"2024-03-04T10:03:39.550331+0000","last_became_peered":"2024-03-04T10:03:39.550331+0000","last_unstale":"2024-03-04T10:03:44.557993+0000","last_undegraded":"2024-03-04T10:03:44.557993+0000","last_fullsized":"2024-03-04T10:03:44.557993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:12.526741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556242+0000","last_change":"2024-03-04T10:03:38.529200+0000","last_active":"2024-03-04T10:03:44.556242+0000","last_peered":"2024-03-04T10:03:44.556242+0000","last_clean":"2024-03-04T10:03:44.556242+0000","last_became_active":"2024-03-04T10:03:38.528943+0000","last_became_peered":"2024-03-04T10:03:38.528943+0000","last_unstale":"2024-03-04T10:03:44.556242+0000","last_undegraded":"2024-03-04T10:03:44.556242+0000","last_fullsized":"2024-03-04T10:03:44.556242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557839+0000","last_change":"2024-03-04T10:03:43.578553+0000","last_active":"2024-03-04T10:03:44.557839+0000","last_peered":"2024-03-04T10:03:44.557839+0000","last_clean":"2024-03-04T10:03:44.557839+0000","last_became_active":"2024-03-04T10:03:43.577929+0000","last_became_peered":"2024-03-04T10:03:43.577929+0000","last_unstale":"2024-03-04T10:03:44.557839+0000","last_undegraded":"2024-03-04T10:03:44.557839+0000","last_fullsized":"2024-03-04T10:03:44.557839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:28.324570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455715+0000","last_change":"2024-03-04T10:03:39.556135+0000","last_active":"2024-03-04T10:03:45.455715+0000","last_peered":"2024-03-04T10:03:45.455715+0000","last_clean":"2024-03-04T10:03:45.455715+0000","last_became_active":"2024-03-04T10:03:39.555846+0000","last_became_peered":"2024-03-04T10:03:39.555846+0000","last_unstale":"2024-03-04T10:03:45.455715+0000","last_undegraded":"2024-03-04T10:03:45.455715+0000","last_fullsized":"2024-03-04T10:03:45.455715+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455561+0000","last_change":"2024-03-04T10:03:38.530735+0000","last_active":"2024-03-04T10:03:45.455561+0000","last_peered":"2024-03-04T10:03:45.455561+0000","last_clean":"2024-03-04T10:03:45.455561+0000","last_became_active":"2024-03-04T10:03:38.530463+0000","last_became_peered":"2024-03-04T10:03:38.530463+0000","last_unstale":"2024-03-04T10:03:45.455561+0000","last_undegraded":"2024-03-04T10:03:45.455561+0000","last_fullsized":"2024-03-04T10:03:45.455561+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:05.385312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455391+0000","last_change":"2024-03-04T10:03:43.577622+0000","last_active":"2024-03-04T10:03:45.455391+0000","last_peered":"2024-03-04T10:03:45.455391+0000","last_clean":"2024-03-04T10:03:45.455391+0000","last_became_active":"2024-03-04T10:03:43.575914+0000","last_became_peered":"2024-03-04T10:03:43.575914+0000","last_unstale":"2024-03-04T10:03:45.455391+0000","last_undegraded":"2024-03-04T10:03:45.455391+0000","last_fullsized":"2024-03-04T10:03:45.455391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:57:28.245578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558308+0000","last_change":"2024-03-04T10:03:39.551071+0000","last_active":"2024-03-04T10:03:44.558308+0000","last_peered":"2024-03-04T10:03:44.558308+0000","last_clean":"2024-03-04T10:03:44.558308+0000","last_became_active":"2024-03-04T10:03:39.550739+0000","last_became_peered":"2024-03-04T10:03:39.550739+0000","last_unstale":"2024-03-04T10:03:44.558308+0000","last_undegraded":"2024-03-04T10:03:44.558308+0000","last_fullsized":"2024-03-04T10:03:44.558308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:14.185049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556247+0000","last_change":"2024-03-04T10:03:38.534772+0000","last_active":"2024-03-04T10:03:44.556247+0000","last_peered":"2024-03-04T10:03:44.556247+0000","last_clean":"2024-03-04T10:03:44.556247+0000","last_became_active":"2024-03-04T10:03:38.534520+0000","last_became_peered":"2024-03-04T10:03:38.534520+0000","last_unstale":"2024-03-04T10:03:44.556247+0000","last_undegraded":"2024-03-04T10:03:44.556247+0000","last_fullsized":"2024-03-04T10:03:44.556247+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455292+0000","last_change":"2024-03-04T10:03:43.570591+0000","last_active":"2024-03-04T10:03:45.455292+0000","last_peered":"2024-03-04T10:03:45.455292+0000","last_clean":"2024-03-04T10:03:45.455292+0000","last_became_active":"2024-03-04T10:03:43.570154+0000","last_became_peered":"2024-03-04T10:03:43.570154+0000","last_unstale":"2024-03-04T10:03:45.455292+0000","last_undegraded":"2024-03-04T10:03:45.455292+0000","last_fullsized":"2024-03-04T10:03:45.455292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558471+0000","last_change":"2024-03-04T10:03:39.547823+0000","last_active":"2024-03-04T10:03:44.558471+0000","last_peered":"2024-03-04T10:03:44.558471+0000","last_clean":"2024-03-04T10:03:44.558471+0000","last_became_active":"2024-03-04T10:03:39.547553+0000","last_became_peered":"2024-03-04T10:03:39.547553+0000","last_unstale":"2024-03-04T10:03:44.558471+0000","last_undegraded":"2024-03-04T10:03:44.558471+0000","last_fullsized":"2024-03-04T10:03:44.558471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925780+0000","last_change":"2024-03-04T10:03:38.530675+0000","last_active":"2024-03-04T10:03:44.925780+0000","last_peered":"2024-03-04T10:03:44.925780+0000","last_clean":"2024-03-04T10:03:44.925780+0000","last_became_active":"2024-03-04T10:03:38.530406+0000","last_became_peered":"2024-03-04T10:03:38.530406+0000","last_unstale":"2024-03-04T10:03:44.925780+0000","last_undegraded":"2024-03-04T10:03:44.925780+0000","last_fullsized":"2024-03-04T10:03:44.925780+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:29.050824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925002+0000","last_change":"2024-03-04T10:03:43.577882+0000","last_active":"2024-03-04T10:03:44.925002+0000","last_peered":"2024-03-04T10:03:44.925002+0000","last_clean":"2024-03-04T10:03:44.925002+0000","last_became_active":"2024-03-04T10:03:43.577500+0000","last_became_peered":"2024-03-04T10:03:43.577500+0000","last_unstale":"2024-03-04T10:03:44.925002+0000","last_undegraded":"2024-03-04T10:03:44.925002+0000","last_fullsized":"2024-03-04T10:03:44.925002+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:38.890692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557934+0000","last_change":"2024-03-04T10:03:39.546483+0000","last_active":"2024-03-04T10:03:44.557934+0000","last_peered":"2024-03-04T10:03:44.557934+0000","last_clean":"2024-03-04T10:03:44.557934+0000","last_became_active":"2024-03-04T10:03:39.546211+0000","last_became_peered":"2024-03-04T10:03:39.546211+0000","last_unstale":"2024-03-04T10:03:44.557934+0000","last_undegraded":"2024-03-04T10:03:44.557934+0000","last_fullsized":"2024-03-04T10:03:44.557934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555858+0000","last_change":"2024-03-04T10:03:38.532190+0000","last_active":"2024-03-04T10:03:44.555858+0000","last_peered":"2024-03-04T10:03:44.555858+0000","last_clean":"2024-03-04T10:03:44.555858+0000","last_became_active":"2024-03-04T10:03:38.531859+0000","last_became_peered":"2024-03-04T10:03:38.531859+0000","last_unstale":"2024-03-04T10:03:44.555858+0000","last_undegraded":"2024-03-04T10:03:44.555858+0000","last_fullsized":"2024-03-04T10:03:44.555858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:30:57.662758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455223+0000","last_change":"2024-03-04T10:03:43.575386+0000","last_active":"2024-03-04T10:03:45.455223+0000","last_peered":"2024-03-04T10:03:45.455223+0000","last_clean":"2024-03-04T10:03:45.455223+0000","last_became_active":"2024-03-04T10:03:43.575009+0000","last_became_peered":"2024-03-04T10:03:43.575009+0000","last_unstale":"2024-03-04T10:03:45.455223+0000","last_undegraded":"2024-03-04T10:03:45.455223+0000","last_fullsized":"2024-03-04T10:03:45.455223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:48:05.911238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556867+0000","last_change":"2024-03-04T10:03:39.546825+0000","last_active":"2024-03-04T10:03:44.556867+0000","last_peered":"2024-03-04T10:03:44.556867+0000","last_clean":"2024-03-04T10:03:44.556867+0000","last_became_active":"2024-03-04T10:03:39.546567+0000","last_became_peered":"2024-03-04T10:03:39.546567+0000","last_unstale":"2024-03-04T10:03:44.556867+0000","last_undegraded":"2024-03-04T10:03:44.556867+0000","last_fullsized":"2024-03-04T10:03:44.556867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:14:48.498223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557579+0000","last_change":"2024-03-04T10:03:38.529048+0000","last_active":"2024-03-04T10:03:44.557579+0000","last_peered":"2024-03-04T10:03:44.557579+0000","last_clean":"2024-03-04T10:03:44.557579+0000","last_became_active":"2024-03-04T10:03:38.528276+0000","last_became_peered":"2024-03-04T10:03:38.528276+0000","last_unstale":"2024-03-04T10:03:44.557579+0000","last_undegraded":"2024-03-04T10:03:44.557579+0000","last_fullsized":"2024-03-04T10:03:44.557579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727557+0000","last_change":"2024-03-04T10:03:43.578301+0000","last_active":"2024-03-04T10:03:44.727557+0000","last_peered":"2024-03-04T10:03:44.727557+0000","last_clean":"2024-03-04T10:03:44.727557+0000","last_became_active":"2024-03-04T10:03:43.577909+0000","last_became_peered":"2024-03-04T10:03:43.577909+0000","last_unstale":"2024-03-04T10:03:44.727557+0000","last_undegraded":"2024-03-04T10:03:44.727557+0000","last_fullsized":"2024-03-04T10:03:44.727557+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:37.260262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558519+0000","last_change":"2024-03-04T10:03:39.547091+0000","last_active":"2024-03-04T10:03:44.558519+0000","last_peered":"2024-03-04T10:03:44.558519+0000","last_clean":"2024-03-04T10:03:44.558519+0000","last_became_active":"2024-03-04T10:03:39.546836+0000","last_became_peered":"2024-03-04T10:03:39.546836+0000","last_unstale":"2024-03-04T10:03:44.558519+0000","last_undegraded":"2024-03-04T10:03:44.558519+0000","last_fullsized":"2024-03-04T10:03:44.558519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455647+0000","last_change":"2024-03-04T10:03:38.535462+0000","last_active":"2024-03-04T10:03:45.455647+0000","last_peered":"2024-03-04T10:03:45.455647+0000","last_clean":"2024-03-04T10:03:45.455647+0000","last_became_active":"2024-03-04T10:03:38.535165+0000","last_became_peered":"2024-03-04T10:03:38.535165+0000","last_unstale":"2024-03-04T10:03:45.455647+0000","last_undegraded":"2024-03-04T10:03:45.455647+0000","last_fullsized":"2024-03-04T10:03:45.455647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727307+0000","last_change":"2024-03-04T10:03:43.576692+0000","last_active":"2024-03-04T10:03:44.727307+0000","last_peered":"2024-03-04T10:03:44.727307+0000","last_clean":"2024-03-04T10:03:44.727307+0000","last_became_active":"2024-03-04T10:03:43.576321+0000","last_became_peered":"2024-03-04T10:03:43.576321+0000","last_unstale":"2024-03-04T10:03:44.727307+0000","last_undegraded":"2024-03-04T10:03:44.727307+0000","last_fullsized":"2024-03-04T10:03:44.727307+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:14:32.702691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455942+0000","last_change":"2024-03-04T10:03:39.547184+0000","last_active":"2024-03-04T10:03:45.455942+0000","last_peered":"2024-03-04T10:03:45.455942+0000","last_clean":"2024-03-04T10:03:45.455942+0000","last_became_active":"2024-03-04T10:03:39.546899+0000","last_became_peered":"2024-03-04T10:03:39.546899+0000","last_unstale":"2024-03-04T10:03:45.455942+0000","last_undegraded":"2024-03-04T10:03:45.455942+0000","last_fullsized":"2024-03-04T10:03:45.455942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455444+0000","last_change":"2024-03-04T10:03:38.529853+0000","last_active":"2024-03-04T10:03:45.455444+0000","last_peered":"2024-03-04T10:03:45.455444+0000","last_clean":"2024-03-04T10:03:45.455444+0000","last_became_active":"2024-03-04T10:03:38.529259+0000","last_became_peered":"2024-03-04T10:03:38.529259+0000","last_unstale":"2024-03-04T10:03:45.455444+0000","last_undegraded":"2024-03-04T10:03:45.455444+0000","last_fullsized":"2024-03-04T10:03:45.455444+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:56.652500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456026+0000","last_change":"2024-03-04T10:03:43.572841+0000","last_active":"2024-03-04T10:03:45.456026+0000","last_peered":"2024-03-04T10:03:45.456026+0000","last_clean":"2024-03-04T10:03:45.456026+0000","last_became_active":"2024-03-04T10:03:43.572414+0000","last_became_peered":"2024-03-04T10:03:43.572414+0000","last_unstale":"2024-03-04T10:03:45.456026+0000","last_undegraded":"2024-03-04T10:03:45.456026+0000","last_fullsized":"2024-03-04T10:03:45.456026+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:45.625071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557911+0000","last_change":"2024-03-04T10:03:39.552501+0000","last_active":"2024-03-04T10:03:44.557911+0000","last_peered":"2024-03-04T10:03:44.557911+0000","last_clean":"2024-03-04T10:03:44.557911+0000","last_became_active":"2024-03-04T10:03:39.551870+0000","last_became_peered":"2024-03-04T10:03:39.551870+0000","last_unstale":"2024-03-04T10:03:44.557911+0000","last_undegraded":"2024-03-04T10:03:44.557911+0000","last_fullsized":"2024-03-04T10:03:44.557911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:48.793130+0000","last_change":"2024-03-04T10:03:38.534457+0000","last_active":"2024-03-04T10:03:48.793130+0000","last_peered":"2024-03-04T10:03:48.793130+0000","last_clean":"2024-03-04T10:03:48.793130+0000","last_became_active":"2024-03-04T10:03:38.534185+0000","last_became_peered":"2024-03-04T10:03:38.534185+0000","last_unstale":"2024-03-04T10:03:48.793130+0000","last_undegraded":"2024-03-04T10:03:48.793130+0000","last_fullsized":"2024-03-04T10:03:48.793130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455693+0000","last_change":"2024-03-04T10:03:43.572170+0000","last_active":"2024-03-04T10:03:45.455693+0000","last_peered":"2024-03-04T10:03:45.455693+0000","last_clean":"2024-03-04T10:03:45.455693+0000","last_became_active":"2024-03-04T10:03:43.571770+0000","last_became_peered":"2024-03-04T10:03:43.571770+0000","last_unstale":"2024-03-04T10:03:45.455693+0000","last_undegraded":"2024-03-04T10:03:45.455693+0000","last_fullsized":"2024-03-04T10:03:45.455693+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:19.867931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925655+0000","last_change":"2024-03-04T10:03:39.548855+0000","last_active":"2024-03-04T10:03:44.925655+0000","last_peered":"2024-03-04T10:03:44.925655+0000","last_clean":"2024-03-04T10:03:44.925655+0000","last_became_active":"2024-03-04T10:03:39.548568+0000","last_became_peered":"2024-03-04T10:03:39.548568+0000","last_unstale":"2024-03-04T10:03:44.925655+0000","last_undegraded":"2024-03-04T10:03:44.925655+0000","last_fullsized":"2024-03-04T10:03:44.925655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:19.378207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455946+0000","last_change":"2024-03-04T10:03:38.537010+0000","last_active":"2024-03-04T10:03:45.455946+0000","last_peered":"2024-03-04T10:03:45.455946+0000","last_clean":"2024-03-04T10:03:45.455946+0000","last_became_active":"2024-03-04T10:03:38.536784+0000","last_became_peered":"2024-03-04T10:03:38.536784+0000","last_unstale":"2024-03-04T10:03:45.455946+0000","last_undegraded":"2024-03-04T10:03:45.455946+0000","last_fullsized":"2024-03-04T10:03:45.455946+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:43.718998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925807+0000","last_change":"2024-03-04T10:03:43.575162+0000","last_active":"2024-03-04T10:03:44.925807+0000","last_peered":"2024-03-04T10:03:44.925807+0000","last_clean":"2024-03-04T10:03:44.925807+0000","last_became_active":"2024-03-04T10:03:43.574407+0000","last_became_peered":"2024-03-04T10:03:43.574407+0000","last_unstale":"2024-03-04T10:03:44.925807+0000","last_undegraded":"2024-03-04T10:03:44.925807+0000","last_fullsized":"2024-03-04T10:03:44.925807+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455373+0000","last_change":"2024-03-04T10:03:39.550513+0000","last_active":"2024-03-04T10:03:45.455373+0000","last_peered":"2024-03-04T10:03:45.455373+0000","last_clean":"2024-03-04T10:03:45.455373+0000","last_became_active":"2024-03-04T10:03:39.549951+0000","last_became_peered":"2024-03-04T10:03:39.549951+0000","last_unstale":"2024-03-04T10:03:45.455373+0000","last_undegraded":"2024-03-04T10:03:45.455373+0000","last_fullsized":"2024-03-04T10:03:45.455373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455931+0000","last_change":"2024-03-04T10:03:38.538520+0000","last_active":"2024-03-04T10:03:45.455931+0000","last_peered":"2024-03-04T10:03:45.455931+0000","last_clean":"2024-03-04T10:03:45.455931+0000","last_became_active":"2024-03-04T10:03:38.538251+0000","last_became_peered":"2024-03-04T10:03:38.538251+0000","last_unstale":"2024-03-04T10:03:45.455931+0000","last_undegraded":"2024-03-04T10:03:45.455931+0000","last_fullsized":"2024-03-04T10:03:45.455931+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727165+0000","last_change":"2024-03-04T10:03:43.573784+0000","last_active":"2024-03-04T10:03:44.727165+0000","last_peered":"2024-03-04T10:03:44.727165+0000","last_clean":"2024-03-04T10:03:44.727165+0000","last_became_active":"2024-03-04T10:03:43.573040+0000","last_became_peered":"2024-03-04T10:03:43.573040+0000","last_unstale":"2024-03-04T10:03:44.727165+0000","last_undegraded":"2024-03-04T10:03:44.727165+0000","last_fullsized":"2024-03-04T10:03:44.727165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:31.435305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455930+0000","last_change":"2024-03-04T10:03:39.559601+0000","last_active":"2024-03-04T10:03:45.455930+0000","last_peered":"2024-03-04T10:03:45.455930+0000","last_clean":"2024-03-04T10:03:45.455930+0000","last_became_active":"2024-03-04T10:03:39.559321+0000","last_became_peered":"2024-03-04T10:03:39.559321+0000","last_unstale":"2024-03-04T10:03:45.455930+0000","last_undegraded":"2024-03-04T10:03:45.455930+0000","last_fullsized":"2024-03-04T10:03:45.455930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:15.835064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558393+0000","last_change":"2024-03-04T10:03:38.536855+0000","last_active":"2024-03-04T10:03:44.558393+0000","last_peered":"2024-03-04T10:03:44.558393+0000","last_clean":"2024-03-04T10:03:44.558393+0000","last_became_active":"2024-03-04T10:03:38.536551+0000","last_became_peered":"2024-03-04T10:03:38.536551+0000","last_unstale":"2024-03-04T10:03:44.558393+0000","last_undegraded":"2024-03-04T10:03:44.558393+0000","last_fullsized":"2024-03-04T10:03:44.558393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:20:47.535145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727003+0000","last_change":"2024-03-04T10:03:43.569782+0000","last_active":"2024-03-04T10:03:44.727003+0000","last_peered":"2024-03-04T10:03:44.727003+0000","last_clean":"2024-03-04T10:03:44.727003+0000","last_became_active":"2024-03-04T10:03:43.568928+0000","last_became_peered":"2024-03-04T10:03:43.568928+0000","last_unstale":"2024-03-04T10:03:44.727003+0000","last_undegraded":"2024-03-04T10:03:44.727003+0000","last_fullsized":"2024-03-04T10:03:44.727003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:33.789086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455065+0000","last_change":"2024-03-04T10:03:43.577832+0000","last_active":"2024-03-04T10:03:45.455065+0000","last_peered":"2024-03-04T10:03:45.455065+0000","last_clean":"2024-03-04T10:03:45.455065+0000","last_became_active":"2024-03-04T10:03:43.577460+0000","last_became_peered":"2024-03-04T10:03:43.577460+0000","last_unstale":"2024-03-04T10:03:45.455065+0000","last_undegraded":"2024-03-04T10:03:45.455065+0000","last_fullsized":"2024-03-04T10:03:45.455065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:40:18.217917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455950+0000","last_change":"2024-03-04T10:03:39.547155+0000","last_active":"2024-03-04T10:03:45.455950+0000","last_peered":"2024-03-04T10:03:45.455950+0000","last_clean":"2024-03-04T10:03:45.455950+0000","last_became_active":"2024-03-04T10:03:39.546882+0000","last_became_peered":"2024-03-04T10:03:39.546882+0000","last_unstale":"2024-03-04T10:03:45.455950+0000","last_undegraded":"2024-03-04T10:03:45.455950+0000","last_fullsized":"2024-03-04T10:03:45.455950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455427+0000","last_change":"2024-03-04T10:03:38.533042+0000","last_active":"2024-03-04T10:03:45.455427+0000","last_peered":"2024-03-04T10:03:45.455427+0000","last_clean":"2024-03-04T10:03:45.455427+0000","last_became_active":"2024-03-04T10:03:38.532609+0000","last_became_peered":"2024-03-04T10:03:38.532609+0000","last_unstale":"2024-03-04T10:03:45.455427+0000","last_undegraded":"2024-03-04T10:03:45.455427+0000","last_fullsized":"2024-03-04T10:03:45.455427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:09.051257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558268+0000","last_change":"2024-03-04T10:03:43.577800+0000","last_active":"2024-03-04T10:03:44.558268+0000","last_peered":"2024-03-04T10:03:44.558268+0000","last_clean":"2024-03-04T10:03:44.558268+0000","last_became_active":"2024-03-04T10:03:43.577379+0000","last_became_peered":"2024-03-04T10:03:43.577379+0000","last_unstale":"2024-03-04T10:03:44.558268+0000","last_undegraded":"2024-03-04T10:03:44.558268+0000","last_fullsized":"2024-03-04T10:03:44.558268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:06.681534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726957+0000","last_change":"2024-03-04T10:03:39.555669+0000","last_active":"2024-03-04T10:03:44.726957+0000","last_peered":"2024-03-04T10:03:44.726957+0000","last_clean":"2024-03-04T10:03:44.726957+0000","last_became_active":"2024-03-04T10:03:39.555415+0000","last_became_peered":"2024-03-04T10:03:39.555415+0000","last_unstale":"2024-03-04T10:03:44.726957+0000","last_undegraded":"2024-03-04T10:03:44.726957+0000","last_fullsized":"2024-03-04T10:03:44.726957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727092+0000","last_change":"2024-03-04T10:03:38.537206+0000","last_active":"2024-03-04T10:03:44.727092+0000","last_peered":"2024-03-04T10:03:44.727092+0000","last_clean":"2024-03-04T10:03:44.727092+0000","last_became_active":"2024-03-04T10:03:38.536938+0000","last_became_peered":"2024-03-04T10:03:38.536938+0000","last_unstale":"2024-03-04T10:03:44.727092+0000","last_undegraded":"2024-03-04T10:03:44.727092+0000","last_fullsized":"2024-03-04T10:03:44.727092+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:04.405754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558028+0000","last_change":"2024-03-04T10:03:38.528419+0000","last_active":"2024-03-04T10:03:44.558028+0000","last_peered":"2024-03-04T10:03:44.558028+0000","last_clean":"2024-03-04T10:03:44.558028+0000","last_became_active":"2024-03-04T10:03:38.528143+0000","last_became_peered":"2024-03-04T10:03:38.528143+0000","last_unstale":"2024-03-04T10:03:44.558028+0000","last_undegraded":"2024-03-04T10:03:44.558028+0000","last_fullsized":"2024-03-04T10:03:44.558028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455342+0000","last_change":"2024-03-04T10:03:39.537531+0000","last_active":"2024-03-04T10:03:45.455342+0000","last_peered":"2024-03-04T10:03:45.455342+0000","last_clean":"2024-03-04T10:03:45.455342+0000","last_became_active":"2024-03-04T10:03:39.537282+0000","last_became_peered":"2024-03-04T10:03:39.537282+0000","last_unstale":"2024-03-04T10:03:45.455342+0000","last_undegraded":"2024-03-04T10:03:45.455342+0000","last_fullsized":"2024-03-04T10:03:45.455342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556191+0000","last_change":"2024-03-04T10:03:43.576190+0000","last_active":"2024-03-04T10:03:44.556191+0000","last_peered":"2024-03-04T10:03:44.556191+0000","last_clean":"2024-03-04T10:03:44.556191+0000","last_became_active":"2024-03-04T10:03:43.575758+0000","last_became_peered":"2024-03-04T10:03:43.575758+0000","last_unstale":"2024-03-04T10:03:44.556191+0000","last_undegraded":"2024-03-04T10:03:44.556191+0000","last_fullsized":"2024-03-04T10:03:44.556191+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:27:39.111875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727248+0000","last_change":"2024-03-04T10:03:38.538563+0000","last_active":"2024-03-04T10:03:44.727248+0000","last_peered":"2024-03-04T10:03:44.727248+0000","last_clean":"2024-03-04T10:03:44.727248+0000","last_became_active":"2024-03-04T10:03:38.538284+0000","last_became_peered":"2024-03-04T10:03:38.538284+0000","last_unstale":"2024-03-04T10:03:44.727248+0000","last_undegraded":"2024-03-04T10:03:44.727248+0000","last_fullsized":"2024-03-04T10:03:44.727248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:37.507463+0000","last_clean_scrub_stamp":"2024-03-04T10:03:37.507463+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:01:56.295706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925234+0000","last_change":"2024-03-04T10:03:39.543278+0000","last_active":"2024-03-04T10:03:44.925234+0000","last_peered":"2024-03-04T10:03:44.925234+0000","last_clean":"2024-03-04T10:03:44.925234+0000","last_became_active":"2024-03-04T10:03:39.542993+0000","last_became_peered":"2024-03-04T10:03:39.542993+0000","last_unstale":"2024-03-04T10:03:44.925234+0000","last_undegraded":"2024-03-04T10:03:44.925234+0000","last_fullsized":"2024-03-04T10:03:44.925234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:44.723658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727569+0000","last_change":"2024-03-04T10:03:43.569930+0000","last_active":"2024-03-04T10:03:44.727569+0000","last_peered":"2024-03-04T10:03:44.727569+0000","last_clean":"2024-03-04T10:03:44.727569+0000","last_became_active":"2024-03-04T10:03:43.569524+0000","last_became_peered":"2024-03-04T10:03:43.569524+0000","last_unstale":"2024-03-04T10:03:44.727569+0000","last_undegraded":"2024-03-04T10:03:44.727569+0000","last_fullsized":"2024-03-04T10:03:44.727569+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:42.544990+0000","last_clean_scrub_stamp":"2024-03-04T10:03:42.544990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:28.002170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556756+0000","last_change":"2024-03-04T10:03:39.547564+0000","last_active":"2024-03-04T10:03:44.556756+0000","last_peered":"2024-03-04T10:03:44.556756+0000","last_clean":"2024-03-04T10:03:44.556756+0000","last_became_active":"2024-03-04T10:03:39.547245+0000","last_became_peered":"2024-03-04T10:03:39.547245+0000","last_unstale":"2024-03-04T10:03:44.556756+0000","last_undegraded":"2024-03-04T10:03:44.556756+0000","last_fullsized":"2024-03-04T10:03:44.556756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:15:14.213025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.555940+0000","last_change":"2024-03-04T10:03:39.549137+0000","last_active":"2024-03-04T10:03:44.555940+0000","last_peered":"2024-03-04T10:03:44.555940+0000","last_clean":"2024-03-04T10:03:44.555940+0000","last_became_active":"2024-03-04T10:03:39.548867+0000","last_became_peered":"2024-03-04T10:03:39.548867+0000","last_unstale":"2024-03-04T10:03:44.555940+0000","last_undegraded":"2024-03-04T10:03:44.555940+0000","last_fullsized":"2024-03-04T10:03:44.555940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455603+0000","last_change":"2024-03-04T10:03:39.555564+0000","last_active":"2024-03-04T10:03:45.455603+0000","last_peered":"2024-03-04T10:03:45.455603+0000","last_clean":"2024-03-04T10:03:45.455603+0000","last_became_active":"2024-03-04T10:03:39.555310+0000","last_became_peered":"2024-03-04T10:03:39.555310+0000","last_unstale":"2024-03-04T10:03:45.455603+0000","last_undegraded":"2024-03-04T10:03:45.455603+0000","last_fullsized":"2024-03-04T10:03:45.455603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455247+0000","last_change":"2024-03-04T10:03:39.550377+0000","last_active":"2024-03-04T10:03:45.455247+0000","last_peered":"2024-03-04T10:03:45.455247+0000","last_clean":"2024-03-04T10:03:45.455247+0000","last_became_active":"2024-03-04T10:03:39.550028+0000","last_became_peered":"2024-03-04T10:03:39.550028+0000","last_unstale":"2024-03-04T10:03:45.455247+0000","last_undegraded":"2024-03-04T10:03:45.455247+0000","last_fullsized":"2024-03-04T10:03:45.455247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:33.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727357+0000","last_change":"2024-03-04T10:03:39.553769+0000","last_active":"2024-03-04T10:03:44.727357+0000","last_peered":"2024-03-04T10:03:44.727357+0000","last_clean":"2024-03-04T10:03:44.727357+0000","last_became_active":"2024-03-04T10:03:39.553517+0000","last_became_peered":"2024-03-04T10:03:39.553517+0000","last_unstale":"2024-03-04T10:03:44.727357+0000","last_undegraded":"2024-03-04T10:03:44.727357+0000","last_fullsized":"2024-03-04T10:03:44.727357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455401+0000","last_change":"2024-03-04T10:03:39.551104+0000","last_active":"2024-03-04T10:03:45.455401+0000","last_peered":"2024-03-04T10:03:45.455401+0000","last_clean":"2024-03-04T10:03:45.455401+0000","last_became_active":"2024-03-04T10:03:39.550823+0000","last_became_peered":"2024-03-04T10:03:39.550823+0000","last_unstale":"2024-03-04T10:03:45.455401+0000","last_undegraded":"2024-03-04T10:03:45.455401+0000","last_fullsized":"2024-03-04T10:03:45.455401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558684+0000","last_change":"2024-03-04T10:03:39.563672+0000","last_active":"2024-03-04T10:03:44.558684+0000","last_peered":"2024-03-04T10:03:44.558684+0000","last_clean":"2024-03-04T10:03:44.558684+0000","last_became_active":"2024-03-04T10:03:39.563394+0000","last_became_peered":"2024-03-04T10:03:39.563394+0000","last_unstale":"2024-03-04T10:03:44.558684+0000","last_undegraded":"2024-03-04T10:03:44.558684+0000","last_fullsized":"2024-03-04T10:03:44.558684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727442+0000","last_change":"2024-03-04T10:03:39.549746+0000","last_active":"2024-03-04T10:03:44.727442+0000","last_peered":"2024-03-04T10:03:44.727442+0000","last_clean":"2024-03-04T10:03:44.727442+0000","last_became_active":"2024-03-04T10:03:39.549349+0000","last_became_peered":"2024-03-04T10:03:39.549349+0000","last_unstale":"2024-03-04T10:03:44.727442+0000","last_undegraded":"2024-03-04T10:03:44.727442+0000","last_fullsized":"2024-03-04T10:03:44.727442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:36:23.940662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455405+0000","last_change":"2024-03-04T10:03:39.556217+0000","last_active":"2024-03-04T10:03:45.455405+0000","last_peered":"2024-03-04T10:03:45.455405+0000","last_clean":"2024-03-04T10:03:45.455405+0000","last_became_active":"2024-03-04T10:03:39.555896+0000","last_became_peered":"2024-03-04T10:03:39.555896+0000","last_unstale":"2024-03-04T10:03:45.455405+0000","last_undegraded":"2024-03-04T10:03:45.455405+0000","last_fullsized":"2024-03-04T10:03:45.455405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726990+0000","last_change":"2024-03-04T10:03:39.546403+0000","last_active":"2024-03-04T10:03:44.726990+0000","last_peered":"2024-03-04T10:03:44.726990+0000","last_clean":"2024-03-04T10:03:44.726990+0000","last_became_active":"2024-03-04T10:03:39.546136+0000","last_became_peered":"2024-03-04T10:03:39.546136+0000","last_unstale":"2024-03-04T10:03:44.726990+0000","last_undegraded":"2024-03-04T10:03:44.726990+0000","last_fullsized":"2024-03-04T10:03:44.726990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455552+0000","last_change":"2024-03-04T10:03:39.553616+0000","last_active":"2024-03-04T10:03:45.455552+0000","last_peered":"2024-03-04T10:03:45.455552+0000","last_clean":"2024-03-04T10:03:45.455552+0000","last_became_active":"2024-03-04T10:03:39.553356+0000","last_became_peered":"2024-03-04T10:03:39.553356+0000","last_unstale":"2024-03-04T10:03:45.455552+0000","last_undegraded":"2024-03-04T10:03:45.455552+0000","last_fullsized":"2024-03-04T10:03:45.455552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:40:05.582540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455120+0000","last_change":"2024-03-04T10:03:39.539406+0000","last_active":"2024-03-04T10:03:45.455120+0000","last_peered":"2024-03-04T10:03:45.455120+0000","last_clean":"2024-03-04T10:03:45.455120+0000","last_became_active":"2024-03-04T10:03:39.539139+0000","last_became_peered":"2024-03-04T10:03:39.539139+0000","last_unstale":"2024-03-04T10:03:45.455120+0000","last_undegraded":"2024-03-04T10:03:45.455120+0000","last_fullsized":"2024-03-04T10:03:45.455120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558323+0000","last_change":"2024-03-04T10:03:39.554026+0000","last_active":"2024-03-04T10:03:44.558323+0000","last_peered":"2024-03-04T10:03:44.558323+0000","last_clean":"2024-03-04T10:03:44.558323+0000","last_became_active":"2024-03-04T10:03:39.553711+0000","last_became_peered":"2024-03-04T10:03:39.553711+0000","last_unstale":"2024-03-04T10:03:44.558323+0000","last_undegraded":"2024-03-04T10:03:44.558323+0000","last_fullsized":"2024-03-04T10:03:44.558323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558201+0000","last_change":"2024-03-04T10:03:39.545468+0000","last_active":"2024-03-04T10:03:44.558201+0000","last_peered":"2024-03-04T10:03:44.558201+0000","last_clean":"2024-03-04T10:03:44.558201+0000","last_became_active":"2024-03-04T10:03:39.545202+0000","last_became_peered":"2024-03-04T10:03:39.545202+0000","last_unstale":"2024-03-04T10:03:44.558201+0000","last_undegraded":"2024-03-04T10:03:44.558201+0000","last_fullsized":"2024-03-04T10:03:44.558201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:42.641276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.556175+0000","last_change":"2024-03-04T10:03:39.549637+0000","last_active":"2024-03-04T10:03:44.556175+0000","last_peered":"2024-03-04T10:03:44.556175+0000","last_clean":"2024-03-04T10:03:44.556175+0000","last_became_active":"2024-03-04T10:03:39.549384+0000","last_became_peered":"2024-03-04T10:03:39.549384+0000","last_unstale":"2024-03-04T10:03:44.556175+0000","last_undegraded":"2024-03-04T10:03:44.556175+0000","last_fullsized":"2024-03-04T10:03:44.556175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:10:06.389971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456190+0000","last_change":"2024-03-04T10:03:39.559786+0000","last_active":"2024-03-04T10:03:45.456190+0000","last_peered":"2024-03-04T10:03:45.456190+0000","last_clean":"2024-03-04T10:03:45.456190+0000","last_became_active":"2024-03-04T10:03:39.559507+0000","last_became_peered":"2024-03-04T10:03:39.559507+0000","last_unstale":"2024-03-04T10:03:45.456190+0000","last_undegraded":"2024-03-04T10:03:45.456190+0000","last_fullsized":"2024-03-04T10:03:45.456190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:54:24.291703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455259+0000","last_change":"2024-03-04T10:03:39.547601+0000","last_active":"2024-03-04T10:03:45.455259+0000","last_peered":"2024-03-04T10:03:45.455259+0000","last_clean":"2024-03-04T10:03:45.455259+0000","last_became_active":"2024-03-04T10:03:39.547309+0000","last_became_peered":"2024-03-04T10:03:39.547309+0000","last_unstale":"2024-03-04T10:03:45.455259+0000","last_undegraded":"2024-03-04T10:03:45.455259+0000","last_fullsized":"2024-03-04T10:03:45.455259+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.456059+0000","last_change":"2024-03-04T10:03:39.543308+0000","last_active":"2024-03-04T10:03:45.456059+0000","last_peered":"2024-03-04T10:03:45.456059+0000","last_clean":"2024-03-04T10:03:45.456059+0000","last_became_active":"2024-03-04T10:03:39.543064+0000","last_became_peered":"2024-03-04T10:03:39.543064+0000","last_unstale":"2024-03-04T10:03:45.456059+0000","last_undegraded":"2024-03-04T10:03:45.456059+0000","last_fullsized":"2024-03-04T10:03:45.456059+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455862+0000","last_change":"2024-03-04T10:03:39.540533+0000","last_active":"2024-03-04T10:03:45.455862+0000","last_peered":"2024-03-04T10:03:45.455862+0000","last_clean":"2024-03-04T10:03:45.455862+0000","last_became_active":"2024-03-04T10:03:39.540274+0000","last_became_peered":"2024-03-04T10:03:39.540274+0000","last_unstale":"2024-03-04T10:03:45.455862+0000","last_undegraded":"2024-03-04T10:03:45.455862+0000","last_fullsized":"2024-03-04T10:03:45.455862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.727155+0000","last_change":"2024-03-04T10:03:39.556252+0000","last_active":"2024-03-04T10:03:44.727155+0000","last_peered":"2024-03-04T10:03:44.727155+0000","last_clean":"2024-03-04T10:03:44.727155+0000","last_became_active":"2024-03-04T10:03:39.556009+0000","last_became_peered":"2024-03-04T10:03:39.556009+0000","last_unstale":"2024-03-04T10:03:44.727155+0000","last_undegraded":"2024-03-04T10:03:44.727155+0000","last_fullsized":"2024-03-04T10:03:44.727155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.925508+0000","last_change":"2024-03-04T10:03:39.552666+0000","last_active":"2024-03-04T10:03:44.925508+0000","last_peered":"2024-03-04T10:03:44.925508+0000","last_clean":"2024-03-04T10:03:44.925508+0000","last_became_active":"2024-03-04T10:03:39.552178+0000","last_became_peered":"2024-03-04T10:03:39.552178+0000","last_unstale":"2024-03-04T10:03:44.925508+0000","last_undegraded":"2024-03-04T10:03:44.925508+0000","last_fullsized":"2024-03-04T10:03:44.925508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:41.701869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558388+0000","last_change":"2024-03-04T10:03:39.557130+0000","last_active":"2024-03-04T10:03:44.558388+0000","last_peered":"2024-03-04T10:03:44.558388+0000","last_clean":"2024-03-04T10:03:44.558388+0000","last_became_active":"2024-03-04T10:03:39.556736+0000","last_became_peered":"2024-03-04T10:03:39.556736+0000","last_unstale":"2024-03-04T10:03:44.558388+0000","last_undegraded":"2024-03-04T10:03:44.558388+0000","last_fullsized":"2024-03-04T10:03:44.558388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:37:23.631825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558631+0000","last_change":"2024-03-04T10:03:39.556929+0000","last_active":"2024-03-04T10:03:44.558631+0000","last_peered":"2024-03-04T10:03:44.558631+0000","last_clean":"2024-03-04T10:03:44.558631+0000","last_became_active":"2024-03-04T10:03:39.556672+0000","last_became_peered":"2024-03-04T10:03:39.556672+0000","last_unstale":"2024-03-04T10:03:44.558631+0000","last_undegraded":"2024-03-04T10:03:44.558631+0000","last_fullsized":"2024-03-04T10:03:44.558631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:50.987119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.558062+0000","last_change":"2024-03-04T10:03:39.553116+0000","last_active":"2024-03-04T10:03:44.558062+0000","last_peered":"2024-03-04T10:03:44.558062+0000","last_clean":"2024-03-04T10:03:44.558062+0000","last_became_active":"2024-03-04T10:03:39.551779+0000","last_became_peered":"2024-03-04T10:03:39.551779+0000","last_unstale":"2024-03-04T10:03:44.558062+0000","last_undegraded":"2024-03-04T10:03:44.558062+0000","last_fullsized":"2024-03-04T10:03:44.558062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:06:34.980080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.726862+0000","last_change":"2024-03-04T10:03:39.557751+0000","last_active":"2024-03-04T10:03:44.726862+0000","last_peered":"2024-03-04T10:03:44.726862+0000","last_clean":"2024-03-04T10:03:44.726862+0000","last_became_active":"2024-03-04T10:03:39.557443+0000","last_became_peered":"2024-03-04T10:03:39.557443+0000","last_unstale":"2024-03-04T10:03:44.726862+0000","last_undegraded":"2024-03-04T10:03:44.726862+0000","last_fullsized":"2024-03-04T10:03:44.726862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:57.273477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557825+0000","last_change":"2024-03-04T10:03:39.547100+0000","last_active":"2024-03-04T10:03:44.557825+0000","last_peered":"2024-03-04T10:03:44.557825+0000","last_clean":"2024-03-04T10:03:44.557825+0000","last_became_active":"2024-03-04T10:03:39.546794+0000","last_became_peered":"2024-03-04T10:03:39.546794+0000","last_unstale":"2024-03-04T10:03:44.557825+0000","last_undegraded":"2024-03-04T10:03:44.557825+0000","last_fullsized":"2024-03-04T10:03:44.557825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.733992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455496+0000","last_change":"2024-03-04T10:03:39.548756+0000","last_active":"2024-03-04T10:03:45.455496+0000","last_peered":"2024-03-04T10:03:45.455496+0000","last_clean":"2024-03-04T10:03:45.455496+0000","last_became_active":"2024-03-04T10:03:39.548474+0000","last_became_peered":"2024-03-04T10:03:39.548474+0000","last_unstale":"2024-03-04T10:03:45.455496+0000","last_undegraded":"2024-03-04T10:03:45.455496+0000","last_fullsized":"2024-03-04T10:03:45.455496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557997+0000","last_change":"2024-03-04T10:03:39.544100+0000","last_active":"2024-03-04T10:03:44.557997+0000","last_peered":"2024-03-04T10:03:44.557997+0000","last_clean":"2024-03-04T10:03:44.557997+0000","last_became_active":"2024-03-04T10:03:39.543581+0000","last_became_peered":"2024-03-04T10:03:39.543581+0000","last_unstale":"2024-03-04T10:03:44.557997+0000","last_undegraded":"2024-03-04T10:03:44.557997+0000","last_fullsized":"2024-03-04T10:03:44.557997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:39.318927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455591+0000","last_change":"2024-03-04T10:03:39.556280+0000","last_active":"2024-03-04T10:03:45.455591+0000","last_peered":"2024-03-04T10:03:45.455591+0000","last_clean":"2024-03-04T10:03:45.455591+0000","last_became_active":"2024-03-04T10:03:39.555867+0000","last_became_peered":"2024-03-04T10:03:39.555867+0000","last_unstale":"2024-03-04T10:03:45.455591+0000","last_undegraded":"2024-03-04T10:03:45.455591+0000","last_fullsized":"2024-03-04T10:03:45.455591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:02.648666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455074+0000","last_change":"2024-03-04T10:03:39.538030+0000","last_active":"2024-03-04T10:03:45.455074+0000","last_peered":"2024-03-04T10:03:45.455074+0000","last_clean":"2024-03-04T10:03:45.455074+0000","last_became_active":"2024-03-04T10:03:39.537783+0000","last_became_peered":"2024-03-04T10:03:39.537783+0000","last_unstale":"2024-03-04T10:03:45.455074+0000","last_undegraded":"2024-03-04T10:03:45.455074+0000","last_fullsized":"2024-03-04T10:03:45.455074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:38.878147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:45.455845+0000","last_change":"2024-03-04T10:03:39.545505+0000","last_active":"2024-03-04T10:03:45.455845+0000","last_peered":"2024-03-04T10:03:45.455845+0000","last_clean":"2024-03-04T10:03:45.455845+0000","last_became_active":"2024-03-04T10:03:39.545176+0000","last_became_peered":"2024-03-04T10:03:39.545176+0000","last_unstale":"2024-03-04T10:03:45.455845+0000","last_undegraded":"2024-03-04T10:03:45.455845+0000","last_fullsized":"2024-03-04T10:03:45.455845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:54:34.857824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-04T10:03:44.557636+0000","last_change":"2024-03-04T10:03:39.551482+0000","last_active":"2024-03-04T10:03:44.557636+0000","last_peered":"2024-03-04T10:03:44.557636+0000","last_clean":"2024-03-04T10:03:44.557636+0000","last_became_active":"2024-03-04T10:03:39.551234+0000","last_became_peered":"2024-03-04T10:03:39.551234+0000","last_unstale":"2024-03-04T10:03:44.557636+0000","last_undegraded":"2024-03-04T10:03:44.557636+0000","last_fullsized":"2024-03-04T10:03:44.557636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:38.512123+0000","last_clean_scrub_stamp":"2024-03-04T10:03:38.512123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:04:22.803941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:04:07.243 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-04T10:04:07.244 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T10:04:07.244 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-04T10:04:07.244 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-04T10:04:07.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.372+0000 7f1d0c196640 1 Processor -- start 2024-03-04T10:04:07.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.372+0000 7f1d0c196640 1 -- start start 2024-03-04T10:04:07.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0c196640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1d040cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0c196640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1d0406d3b0 0x7f1d040cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0c196640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040d0140 con 0x7f1d040ce9c0 2024-03-04T10:04:07.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0c196640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0415cbb0 con 0x7f1d0406d3b0 2024-03-04T10:04:07.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0c196640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0415c280 con 0x7f1d040cfd60 2024-03-04T10:04:07.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0a733640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1d040ce9c0 0x7f1d040cee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34864/0 (socket says 172.21.15.99:34864) 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d0a733640 1 -- 172.21.15.99:0/4215108246 learned_addr learned my addr 172.21.15.99:0/4215108246 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d09f32640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1d040cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d09731640 1 --2- 172.21.15.99:0/4215108246 >> v2:172.21.15.162:3300/0 conn(0x7f1d0406d3b0 0x7f1d040cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928001499 0 0) 0x7f1d040d0140 con 0x7f1d040ce9c0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.373+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf0003520 con 0x7f1d040ce9c0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041389414 0 0) 0x7f1cf0003520 con 0x7f1d040ce9c0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 msgr2=0x7f1d040cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1d040cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 >> v2:172.21.15.162:3300/0 conn(0x7f1d0406d3b0 msgr2=0x7f1d040cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 --2- 172.21.15.99:0/4215108246 >> v2:172.21.15.162:3300/0 conn(0x7f1d0406d3b0 0x7f1d040cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0415c560 con 0x7f1d040ce9c0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f1cfc001670 con 0x7f1d040ce9c0 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1d040cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:07.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09731640 1 --2- 172.21.15.99:0/4215108246 >> v2:172.21.15.162:3300/0 conn(0x7f1d0406d3b0 0x7f1d040cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 >> v1:172.21.15.99:6789/0 conn(0x7f1d040ce9c0 legacy=0x7f1d040cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1cf00039b0 0x7f1cf0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf0004800 0x7f1cf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1cf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0415cbb0 con 0x7f1cf00039b0 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0415c280 con 0x7f1cf0004800 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf000b9c0 con 0x7f1d040cfd60 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1cf00039b0 0x7f1cf0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d0a733640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1cf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09731640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf0004800 0x7f1cf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 -- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 msgr2=0x7f1cf00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d040cfd60 0x7f1cf00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 -- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf0004800 msgr2=0x7f1cf0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf0004800 0x7f1cf0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09f32640 1 -- 172.21.15.99:0/4215108246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf0003520 con 0x7f1cf00039b0 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.374+0000 7f1d09731640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf0004800 0x7f1cf0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.375+0000 7f1d09f32640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1cf00039b0 0x7f1cf0003df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000e350 tx=0x7f1cf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52002788c847e350 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.375+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00039070 con 0x7f1cf00039b0 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.375+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0002cd70 con 0x7f1cf00039b0 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.375+0000 7f1d08f30640 1 -- 172.21.15.99:0/4215108246 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00009de0 con 0x7f1cf00039b0 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0c196640 1 -- 172.21.15.99:0/4215108246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1cf00039b0 msgr2=0x7f1cf0003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0c196640 1 --2- 172.21.15.99:0/4215108246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1cf00039b0 0x7f1cf0003df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000e350 tx=0x7f1cf0005830 comp rx=0 tx=0).stop 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0a733640 1 -- 172.21.15.99:0/4215108246 reap_dead start 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0c196640 1 -- 172.21.15.99:0/4215108246 shutdown_connections 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0c196640 1 -- 172.21.15.99:0/4215108246 >> 172.21.15.99:0/4215108246 conn(0x7f1d0406acd0 msgr2=0x7f1d0406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0c196640 1 -- 172.21.15.99:0/4215108246 shutdown_connections 2024-03-04T10:04:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.376+0000 7f1d0c196640 1 -- 172.21.15.99:0/4215108246 wait complete. 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.377+0000 7f1d0c196640 1 Processor -- start 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.377+0000 7f1d0c196640 1 -- start start 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf00039b0 0x7f1d04155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 0x7f1d04153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d0415a030 0x7f1d0415a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040cbdf0 con 0x7f1d04155810 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0415bf30 con 0x7f1cf00039b0 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040cbf70 con 0x7f1d0415a030 2024-03-04T10:04:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 0x7f1d04153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 0x7f1d04153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38352/0 (socket says 172.21.15.99:38352) 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 -- 172.21.15.99:0/560340333 learned_addr learned my addr 172.21.15.99:0/560340333 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09f32640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf00039b0 0x7f1d04155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0a733640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d0415a030 0x7f1d0415a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 -- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d0415a030 msgr2=0x7f1d0415a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d0415a030 0x7f1d0415a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 -- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf00039b0 msgr2=0x7f1d04155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf00039b0 0x7f1d04155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 -- 172.21.15.99:0/560340333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d041565e0 con 0x7f1d04155810 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d09731640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 0x7f1d04153350 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4011900 tx=0x7f1cf4011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf400fbd0 con 0x7f1d04155810 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf400fd70 con 0x7f1d04155810 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d04155c90 con 0x7f1d04155810 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf4015660 con 0x7f1d04155810 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.378+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d040a8850 con 0x7f1d04155810 2024-03-04T10:04:07.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.379+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0415a9f0 con 0x7f1d04155810 2024-03-04T10:04:07.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.380+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1cf4015800 con 0x7f1d04155810 2024-03-04T10:04:07.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.380+0000 7f1cfa7fc640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ce806b500 0x7f1ce806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.380+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1cf408afb0 con 0x7f1d04155810 2024-03-04T10:04:07.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.380+0000 7f1d09f32640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ce806b500 0x7f1ce806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.381+0000 7f1d09f32640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ce806b500 0x7f1ce806d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1d00036000 tx=0x7f1d0002e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.384+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1d04155c90 con 0x7f1d04155810 2024-03-04T10:04:07.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.598+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d04074f60 con 0x7f1d04155810 2024-03-04T10:04:07.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.598+0000 7f1cfa7fc640 1 -- 172.21.15.99:0/560340333 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1cf4019030 con 0x7f1d04155810 2024-03-04T10:04:07.599 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:07.600 INFO:teuthology.orchestra.run.smithi099.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T10:04:07.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ce806b500 msgr2=0x7f1ce806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ce806b500 0x7f1ce806d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1d00036000 tx=0x7f1d0002e000 comp rx=0 tx=0).stop 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 msgr2=0x7f1d04153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 0x7f1d04153350 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4011900 tx=0x7f1cf4011dc0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 shutdown_connections 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ce806b500 0x7f1ce806d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d0415a030 0x7f1d0415a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d04155810 0x7f1d04153350 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 --2- 172.21.15.99:0/560340333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf00039b0 0x7f1d04155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 >> 172.21.15.99:0/560340333 conn(0x7f1d0406acd0 msgr2=0x7f1d041592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 shutdown_connections 2024-03-04T10:04:07.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.600+0000 7f1d0c196640 1 -- 172.21.15.99:0/560340333 wait complete. 2024-03-04T10:04:07.612 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-04T10:04:07.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:07.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.713+0000 7ff39cb9d640 1 Processor -- start 2024-03-04T10:04:07.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.713+0000 7ff39cb9d640 1 -- start start 2024-03-04T10:04:07.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39cb9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff3980cfd60 0x7ff3980cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39cb9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff39806d3b0 0x7ff3980cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39cb9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff3980ce9c0 0x7ff3980cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39cb9d640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff39815cbb0 con 0x7ff3980ce9c0 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39cb9d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3980d0140 con 0x7ff3980cfd60 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39cb9d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39815c280 con 0x7ff39806d3b0 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff3980ce9c0 0x7ff3980cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff3980ce9c0 0x7ff3980cee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38358/0 (socket says 172.21.15.99:38358) 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 -- 172.21.15.99:0/2396842654 learned_addr learned my addr 172.21.15.99:0/2396842654 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff39659c640 1 --2- 172.21.15.99:0/2396842654 >> v2:172.21.15.162:3300/0 conn(0x7ff3980cfd60 0x7ff3980cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff395d9b640 1 --2- 172.21.15.99:0/2396842654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff39806d3b0 0x7ff3980cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 -- 172.21.15.99:0/2396842654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff39806d3b0 msgr2=0x7ff3980cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 --2- 172.21.15.99:0/2396842654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff39806d3b0 0x7ff3980cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 -- 172.21.15.99:0/2396842654 >> v2:172.21.15.162:3300/0 conn(0x7ff3980cfd60 msgr2=0x7ff3980cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:07.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 --2- 172.21.15.99:0/2396842654 >> v2:172.21.15.162:3300/0 conn(0x7ff3980cfd60 0x7ff3980cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.714+0000 7ff396d9d640 1 -- 172.21.15.99:0/2396842654 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39815c560 con 0x7ff3980ce9c0 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.715+0000 7ff396d9d640 1 --2- 172.21.15.99:0/2396842654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980cee50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff3980b4cd0 tx=0x7ff39002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490051b8c3ea5a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.715+0000 7ff39559a640 1 -- 172.21.15.99:0/2396842654 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390035020 con 0x7ff3980ce9c0 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.715+0000 7ff39559a640 1 -- 172.21.15.99:0/2396842654 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff390031870 con 0x7ff3980ce9c0 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.715+0000 7ff39559a640 1 -- 172.21.15.99:0/2396842654 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390031c40 con 0x7ff3980ce9c0 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 -- 172.21.15.99:0/2396842654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 msgr2=0x7ff3980cee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/2396842654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980cee50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff3980b4cd0 tx=0x7ff39002b910 comp rx=0 tx=0).stop 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 -- 172.21.15.99:0/2396842654 shutdown_connections 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/2396842654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980cee50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/2396842654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff39806d3b0 0x7ff3980cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/2396842654 >> v2:172.21.15.162:3300/0 conn(0x7ff3980cfd60 0x7ff3980cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 -- 172.21.15.99:0/2396842654 >> 172.21.15.99:0/2396842654 conn(0x7ff39806acd0 msgr2=0x7ff39806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:07.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 -- 172.21.15.99:0/2396842654 shutdown_connections 2024-03-04T10:04:07.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.716+0000 7ff39cb9d640 1 -- 172.21.15.99:0/2396842654 wait complete. 2024-03-04T10:04:07.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.717+0000 7ff39cb9d640 1 Processor -- start 2024-03-04T10:04:07.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.717+0000 7ff39cb9d640 1 -- start start 2024-03-04T10:04:07.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39cb9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39806d3b0 0x7ff3980aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39cb9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39cb9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3980cfd60 0x7ff3980b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39cb9d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39815bf30 con 0x7ff3980ce9c0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39cb9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980cf390 con 0x7ff39806d3b0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39cb9d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff398071350 con 0x7ff3980cfd60 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38362/0 (socket says 172.21.15.99:38362) 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 -- 172.21.15.99:0/628861839 learned_addr learned my addr 172.21.15.99:0/628861839 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff39659c640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39806d3b0 0x7ff3980aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff396d9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3980cfd60 0x7ff3980b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 -- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3980cfd60 msgr2=0x7ff3980b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3980cfd60 0x7ff3980b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 -- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39806d3b0 msgr2=0x7ff3980aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39806d3b0 0x7ff3980aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 -- 172.21.15.99:0/628861839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980a9ed0 con 0x7ff3980ce9c0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.718+0000 7ff395d9b640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980ac940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00d7d0 tx=0x7ff38c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.719+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0096a0 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.719+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c019070 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.719+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c009a60 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.719+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3980aa1b0 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.719+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3980aa770 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.720+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff398067680 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.720+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff38c009c00 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.720+0000 7ff386ffd640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.721+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff38c08a770 con 0x7ff3980ce9c0 2024-03-04T10:04:07.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.721+0000 7ff39659c640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:07.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.721+0000 7ff39659c640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff37806b4d0 0x7ff37806d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff388003aa0 tx=0x7ff388009830 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:07.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.725+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff398067680 con 0x7ff3980ce9c0 2024-03-04T10:04:07.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.894+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff398074f60 con 0x7ff3980ce9c0 2024-03-04T10:04:07.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.896+0000 7ff386ffd640 1 -- 172.21.15.99:0/628861839 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7ff38c058cf0 con 0x7ff3980ce9c0 2024-03-04T10:04:07.897 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:07.897 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff37806b4d0 msgr2=0x7ff37806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff37806b4d0 0x7ff37806d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff388003aa0 tx=0x7ff388009830 comp rx=0 tx=0).stop 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 msgr2=0x7ff3980ac940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980ac940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00d7d0 tx=0x7ff38c00dc90 comp rx=0 tx=0).stop 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 shutdown_connections 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff37806b4d0 0x7ff37806d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3980cfd60 0x7ff3980b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3980ce9c0 0x7ff3980ac940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 --2- 172.21.15.99:0/628861839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff39806d3b0 0x7ff3980aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:07.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 >> 172.21.15.99:0/628861839 conn(0x7ff39806acd0 msgr2=0x7ff39806eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:07.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 shutdown_connections 2024-03-04T10:04:07.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:07.898+0000 7ff39cb9d640 1 -- 172.21.15.99:0/628861839 wait complete. 2024-03-04T10:04:07.910 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:07.910 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:08.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.012+0000 7f2d312fe640 1 Processor -- start 2024-03-04T10:04:08.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.012+0000 7f2d312fe640 1 -- start start 2024-03-04T10:04:08.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.012+0000 7f2d312fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2d2c0b9410 0x7f2d2c1622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.012+0000 7f2d312fe640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0d2400 con 0x7f2d2c0b5700 2024-03-04T10:04:08.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.012+0000 7f2d312fe640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0d26e0 con 0x7f2d2c0d34a0 2024-03-04T10:04:08.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.012+0000 7f2d312fe640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d29c0 con 0x7f2d2c0b9410 2024-03-04T10:04:08.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2d2c0b5700 0x7f2d2c0d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34870/0 (socket says 172.21.15.99:34870) 2024-03-04T10:04:08.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 -- 172.21.15.99:0/4228953324 learned_addr learned my addr 172.21.15.99:0/4228953324 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:08.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2b7fe640 1 --2- 172.21.15.99:0/4228953324 >> v2:172.21.15.162:3300/0 conn(0x7f2d2c0b9410 0x7f2d2c1622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716837177 0 0) 0x7f2d2c0d2400 con 0x7f2d2c0b5700 2024-03-04T10:04:08.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d14003520 con 0x7f2d2c0b5700 2024-03-04T10:04:08.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603197250 0 0) 0x7f2d14003520 con 0x7f2d2c0b5700 2024-03-04T10:04:08.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 >> v2:172.21.15.162:3300/0 conn(0x7f2d2c0b9410 msgr2=0x7f2d2c1622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.018 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 --2- 172.21.15.99:0/4228953324 >> v2:172.21.15.162:3300/0 conn(0x7f2d2c0b9410 0x7f2d2c1622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.018 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 >> v1:172.21.15.162:6789/0 conn(0x7f2d2c0d34a0 legacy=0x7f2d2c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0d2ca0 con 0x7f2d2c0b5700 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f2d1c001670 con 0x7f2d2c0b5700 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 >> v1:172.21.15.99:6789/0 conn(0x7f2d2c0b5700 legacy=0x7f2d2c0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d29c0 con 0x7f2d140039b0 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1400c160 con 0x7f2d14004800 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1400c360 con 0x7f2d140051c0 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2a7fc640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2b7fe640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 -- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d140051c0 msgr2=0x7f2d1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 -- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d14004800 msgr2=0x7f2d14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.013+0000 7f2d2affd640 1 -- 172.21.15.99:0/4228953324 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14003520 con 0x7f2d140039b0 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.014+0000 7f2d2b7fe640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.014+0000 7f2d2b7fe640 1 -- 172.21.15.99:0/4228953324 reap_dead start 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.014+0000 7f2d2affd640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c240 tx=0x7f2d14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ba02188cd871ad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.014+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c007cf0 con 0x7f2d140039b0 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.014+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c030070 con 0x7f2d140039b0 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.014+0000 7f2d29ffb640 1 -- 172.21.15.99:0/4228953324 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c005d30 con 0x7f2d140039b0 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 -- 172.21.15.99:0/4228953324 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 msgr2=0x7f2d14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c240 tx=0x7f2d14005830 comp rx=0 tx=0).stop 2024-03-04T10:04:08.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 -- 172.21.15.99:0/4228953324 shutdown_connections 2024-03-04T10:04:08.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 --2- 172.21.15.99:0/4228953324 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 -- 172.21.15.99:0/4228953324 >> 172.21.15.99:0/4228953324 conn(0x7f2d2c0d0490 msgr2=0x7f2d2c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 -- 172.21.15.99:0/4228953324 shutdown_connections 2024-03-04T10:04:08.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.015+0000 7f2d312fe640 1 -- 172.21.15.99:0/4228953324 wait complete. 2024-03-04T10:04:08.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.016+0000 7f2d312fe640 1 Processor -- start 2024-03-04T10:04:08.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.016+0000 7f2d312fe640 1 -- start start 2024-03-04T10:04:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d312fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d2c0c9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d312fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d2c15e7c0 0x7f2d2c0aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d312fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d2c15ec00 0x7f2d2c15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d312fe640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d1b00 con 0x7f2d140039b0 2024-03-04T10:04:08.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d312fe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0b5cf0 con 0x7f2d2c15ec00 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d312fe640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0d1c80 con 0x7f2d2c15e7c0 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d2c0c9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d2c0c9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38380/0 (socket says 172.21.15.99:38380) 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 -- 172.21.15.99:0/3520884245 learned_addr learned my addr 172.21.15.99:0/3520884245 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2b7fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d2c15ec00 0x7f2d2c15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2a7fc640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d2c15e7c0 0x7f2d2c0aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 -- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d2c15e7c0 msgr2=0x7f2d2c0aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d2c15e7c0 0x7f2d2c0aabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 -- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d2c15ec00 msgr2=0x7f2d2c15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d2c15ec00 0x7f2d2c15f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 -- 172.21.15.99:0/3520884245 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0c8f30 con 0x7f2d140039b0 2024-03-04T10:04:08.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.017+0000 7f2d2affd640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d2c0c9fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c240 tx=0x7f2d2c0d25d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.018+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c001c30 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.018+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c004bf0 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.018+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c0c9210 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.018+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c007070 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.018+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c0c97d0 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.019+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c074f60 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.019+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2d1c048020 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.019+0000 7f2d137fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d1806b500 0x7f2d1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.020+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2d1c0ab6a0 con 0x7f2d140039b0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.020+0000 7f2d2a7fc640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d1806b500 0x7f2d1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.020+0000 7f2d2a7fc640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d1806b500 0x7f2d1806d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d20000c00 tx=0x7f2d20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.024+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2d1c079b90 con 0x7f2d140039b0 2024-03-04T10:04:08.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.195+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d2c0cedc0 con 0x7f2d140039b0 2024-03-04T10:04:08.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.196+0000 7f2d137fe640 1 -- 172.21.15.99:0/3520884245 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d1c07dc30 con 0x7f2d140039b0 2024-03-04T10:04:08.197 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:08.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d1806b500 msgr2=0x7f2d1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d1806b500 0x7f2d1806d9c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d20000c00 tx=0x7f2d20009040 comp rx=0 tx=0).stop 2024-03-04T10:04:08.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 msgr2=0x7f2d2c0c9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d2c0c9fd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c240 tx=0x7f2d2c0d25d0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 shutdown_connections 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2d1806b500 0x7f2d1806d9c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d2c15ec00 0x7f2d2c15f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d2c15e7c0 0x7f2d2c0aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 --2- 172.21.15.99:0/3520884245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d140039b0 0x7f2d2c0c9fd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 >> 172.21.15.99:0/3520884245 conn(0x7f2d2c0d0490 msgr2=0x7f2d2c0ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 shutdown_connections 2024-03-04T10:04:08.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.198+0000 7f2d312fe640 1 -- 172.21.15.99:0/3520884245 wait complete. 2024-03-04T10:04:08.210 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:08.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.313+0000 7f5e9628c640 1 Processor -- start 2024-03-04T10:04:08.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.313+0000 7f5e9628c640 1 -- start start 2024-03-04T10:04:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.313+0000 7f5e9628c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5e9006bce0 0x7f5e9006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.313+0000 7f5e9628c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9015bae0 con 0x7f5e9015e100 2024-03-04T10:04:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.313+0000 7f5e9628c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9015ee60 con 0x7f5e9015b700 2024-03-04T10:04:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.313+0000 7f5e9628c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e9006c680 con 0x7f5e9006bce0 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e94829640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5e9015e100 0x7f5e9015e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34880/0 (socket says 172.21.15.99:34880) 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e94829640 1 -- 172.21.15.99:0/2743648360 learned_addr learned my addr 172.21.15.99:0/2743648360 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8f7fe640 1 --2- 172.21.15.99:0/2743648360 >> v2:172.21.15.162:3300/0 conn(0x7f5e9006bce0 0x7f5e9006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044538682 0 0) 0x7f5e9015bae0 con 0x7f5e9015e100 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e78003520 con 0x7f5e9015e100 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608009827 0 0) 0x7f5e9015ee60 con 0x7f5e9015b700 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9015bae0 con 0x7f5e9015b700 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/2743648360 >> v1:172.21.15.162:6789/0 conn(0x7f5e9015b700 legacy=0x7f5e900ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/2743648360 >> v1:172.21.15.99:6789/0 conn(0x7f5e9015e100 legacy=0x7f5e9015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.314+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/2743648360 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900ad540 con 0x7f5e9006bce0 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.315+0000 7f5e8f7fe640 1 --2- 172.21.15.99:0/2743648360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e9006c140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e840097a0 tx=0x7f5e8402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=707071528a7fb4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.315+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84034070 con 0x7f5e9006bce0 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.315+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e84030610 con 0x7f5e9006bce0 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.315+0000 7f5e8effd640 1 -- 172.21.15.99:0/2743648360 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e840309e0 con 0x7f5e9006bce0 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 -- 172.21.15.99:0/2743648360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 msgr2=0x7f5e9006c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 --2- 172.21.15.99:0/2743648360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e9006c140 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e840097a0 tx=0x7f5e8402b960 comp rx=0 tx=0).stop 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 -- 172.21.15.99:0/2743648360 shutdown_connections 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 --2- 172.21.15.99:0/2743648360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e9006c140 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 -- 172.21.15.99:0/2743648360 >> 172.21.15.99:0/2743648360 conn(0x7f5e90071bc0 msgr2=0x7f5e90071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 -- 172.21.15.99:0/2743648360 shutdown_connections 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.316+0000 7f5e9628c640 1 -- 172.21.15.99:0/2743648360 wait complete. 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.317+0000 7f5e9628c640 1 Processor -- start 2024-03-04T10:04:08.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.317+0000 7f5e9628c640 1 -- start start 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e901f6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 0x7f5e900b06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e9015e100 0x7f5e900b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9015f2c0 con 0x7f5e9015b700 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e900adb90 con 0x7f5e9006bce0 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9015f440 con 0x7f5e9015e100 2024-03-04T10:04:08.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 0x7f5e900b06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 0x7f5e900b06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38392/0 (socket says 172.21.15.99:38392) 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/3824364611 learned_addr learned my addr 172.21.15.99:0/3824364611 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e94829640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e9015e100 0x7f5e900b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8ffff640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e901f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e9015e100 msgr2=0x7f5e900b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e9015e100 0x7f5e900b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 msgr2=0x7f5e901f6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e901f6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 -- 172.21.15.99:0/3824364611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900c5a00 con 0x7f5e9015b700 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8f7fe640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 0x7f5e900b06d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5e8402bf70 tx=0x7f5e840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84034040 con 0x7f5e9015b700 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e84030410 con 0x7f5e9015b700 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900c5ce0 con 0x7f5e9015b700 2024-03-04T10:04:08.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84006e10 con 0x7f5e9015b700 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.318+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e900c62a0 con 0x7f5e9015b700 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.319+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e90074f60 con 0x7f5e9015b700 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.320+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5e84037070 con 0x7f5e9015b700 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.320+0000 7f5e8cff9640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.320+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5e840ac380 con 0x7f5e9015b700 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.320+0000 7f5e8ffff640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.321+0000 7f5e8ffff640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5e80001650 tx=0x7f5e80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.324+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5e8407afe0 con 0x7f5e9015b700 2024-03-04T10:04:08.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.492+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e900b01b0 con 0x7f5e9015b700 2024-03-04T10:04:08.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.492+0000 7f5e8cff9640 1 -- 172.21.15.99:0/3824364611 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e8407a8e0 con 0x7f5e9015b700 2024-03-04T10:04:08.493 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:08.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e7c06b4d0 msgr2=0x7f5e7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5e80001650 tx=0x7f5e80009040 comp rx=0 tx=0).stop 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 msgr2=0x7f5e900b06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 0x7f5e900b06d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5e8402bf70 tx=0x7f5e840046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 shutdown_connections 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e7c06b4d0 0x7f5e7c06d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e9015e100 0x7f5e900b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e9015b700 0x7f5e900b06d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 --2- 172.21.15.99:0/3824364611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9006bce0 0x7f5e901f6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 >> 172.21.15.99:0/3824364611 conn(0x7f5e90071bc0 msgr2=0x7f5e9015d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 shutdown_connections 2024-03-04T10:04:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.494+0000 7f5e9628c640 1 -- 172.21.15.99:0/3824364611 wait complete. 2024-03-04T10:04:08.506 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:08.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.609+0000 7f72643c4640 1 Processor -- start 2024-03-04T10:04:08.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.609+0000 7f72643c4640 1 -- start start 2024-03-04T10:04:08.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f72643c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f72643c4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f725c0700a0 0x7f725c15d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f72643c4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f725c15cec0 con 0x7f725c15bba0 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f72643c4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725c15c100 con 0x7f725c0700a0 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f72643c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c15c3e0 con 0x7f725c06bce0 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f725c0700a0 0x7f725c15d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f725c0700a0 0x7f725c15d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38400/0 (socket says 172.21.15.99:38400) 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 -- 172.21.15.99:0/2915563168 learned_addr learned my addr 172.21.15.99:0/2915563168 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262160640 1 --2- 172.21.15.99:0/2915563168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 -- 172.21.15.99:0/2915563168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 msgr2=0x7f725c0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 --2- 172.21.15.99:0/2915563168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 -- 172.21.15.99:0/2915563168 >> v1:172.21.15.162:6789/0 conn(0x7f725c15bba0 legacy=0x7f725c06c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.610+0000 7f7262961640 1 -- 172.21.15.99:0/2915563168 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c15c6c0 con 0x7f725c0700a0 2024-03-04T10:04:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.611+0000 7f7262961640 1 --2- 172.21.15.99:0/2915563168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c15d0b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7254009e50 tx=0x7f725402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44641aeafeb8dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.611+0000 7f726115e640 1 -- 172.21.15.99:0/2915563168 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725402c9e0 con 0x7f725c0700a0 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.611+0000 7f726115e640 1 -- 172.21.15.99:0/2915563168 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725402cb80 con 0x7f725c0700a0 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.611+0000 7f726115e640 1 -- 172.21.15.99:0/2915563168 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72540343f0 con 0x7f725c0700a0 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 -- 172.21.15.99:0/2915563168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 msgr2=0x7f725c15d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 --2- 172.21.15.99:0/2915563168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c15d0b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7254009e50 tx=0x7f725402be40 comp rx=0 tx=0).stop 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 -- 172.21.15.99:0/2915563168 shutdown_connections 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 --2- 172.21.15.99:0/2915563168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c15d0b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 --2- 172.21.15.99:0/2915563168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 -- 172.21.15.99:0/2915563168 >> 172.21.15.99:0/2915563168 conn(0x7f725c071bc0 msgr2=0x7f725c071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 -- 172.21.15.99:0/2915563168 shutdown_connections 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.612+0000 7f72643c4640 1 -- 172.21.15.99:0/2915563168 wait complete. 2024-03-04T10:04:08.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.613+0000 7f72643c4640 1 Processor -- start 2024-03-04T10:04:08.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.613+0000 7f72643c4640 1 -- start start 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f725c15bba0 0x7f725c14cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c15d5f0 con 0x7f725c0700a0 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c0cc3e0 con 0x7f725c15bba0 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c15d770 con 0x7f725c06bce0 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c13d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38404/0 (socket says 172.21.15.99:38404) 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 -- 172.21.15.99:0/3537709881 learned_addr learned my addr 172.21.15.99:0/3537709881 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 -- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 msgr2=0x7f725c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 -- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f725c15bba0 msgr2=0x7f725c14cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f725c15bba0 0x7f725c14cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 -- 172.21.15.99:0/3537709881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c0cd1e0 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f726195f640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c13d7b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b420 tx=0x7f724c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011830 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c016070 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725c14dab0 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011c10 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.614+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725c14e070 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.615+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725c074f60 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.616+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f724c01b460 con 0x7f725c0700a0 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.616+0000 7f7252ffd640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.616+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f724c089f50 con 0x7f725c0700a0 2024-03-04T10:04:08.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.616+0000 7f7262160640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.617+0000 7f7262160640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f724006b4d0 0x7f724006d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72580098f0 tx=0x7f7258008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.620+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f724c020020 con 0x7f725c0700a0 2024-03-04T10:04:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.790+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f725c0af560 con 0x7f725c0700a0 2024-03-04T10:04:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.791+0000 7f7252ffd640 1 -- 172.21.15.99:0/3537709881 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f724c058b20 con 0x7f725c0700a0 2024-03-04T10:04:08.792 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f724006b4d0 msgr2=0x7f724006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f724006b4d0 0x7f724006d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f72580098f0 tx=0x7f7258008040 comp rx=0 tx=0).stop 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 msgr2=0x7f725c13d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c13d7b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f724c00b420 tx=0x7f724c00b8e0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 shutdown_connections 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f725c15bba0 0x7f725c14cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.792+0000 7f72643c4640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f725c0700a0 0x7f725c13d7b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.793+0000 7f72643c4640 1 --2- 172.21.15.99:0/3537709881 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f725c06bce0 0x7f725c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.793+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 >> 172.21.15.99:0/3537709881 conn(0x7f725c071bc0 msgr2=0x7f725c15eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.793+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 shutdown_connections 2024-03-04T10:04:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.793+0000 7f72643c4640 1 -- 172.21.15.99:0/3537709881 wait complete. 2024-03-04T10:04:08.804 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:08.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.906+0000 7f91420a4640 1 Processor -- start 2024-03-04T10:04:08.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.906+0000 7f91420a4640 1 -- start start 2024-03-04T10:04:08.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f91420a4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f913c15da20 0x7f913c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f91420a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 0x7f913c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f91420a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f913c1616e0 0x7f913c161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f91420a4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c0d0000 con 0x7f913c15da20 2024-03-04T10:04:08.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f91420a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c0cc920 con 0x7f913c1616e0 2024-03-04T10:04:08.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f91420a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c162450 con 0x7f913c0cf660 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f913c15da20 0x7f913c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f913c15da20 0x7f913c0ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38420/0 (socket says 172.21.15.99:38420) 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 -- 172.21.15.99:0/841575750 learned_addr learned my addr 172.21.15.99:0/841575750 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913bfff640 1 --2- 172.21.15.99:0/841575750 >> v2:172.21.15.162:3300/0 conn(0x7f913c1616e0 0x7f913c161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 -- 172.21.15.99:0/841575750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 msgr2=0x7f913c0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 --2- 172.21.15.99:0/841575750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 0x7f913c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 -- 172.21.15.99:0/841575750 >> v2:172.21.15.162:3300/0 conn(0x7f913c1616e0 msgr2=0x7f913c161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 --2- 172.21.15.99:0/841575750 >> v2:172.21.15.162:3300/0 conn(0x7f913c1616e0 0x7f913c161b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.907+0000 7f913b7fe640 1 -- 172.21.15.99:0/841575750 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0cc2d0 con 0x7f913c15da20 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.908+0000 7f913b7fe640 1 --2- 172.21.15.99:0/841575750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c0ccdb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f913c0d1340 tx=0x7f912c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dbd634bc2a66f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.908+0000 7f913a7fc640 1 -- 172.21.15.99:0/841575750 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c035020 con 0x7f913c15da20 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.908+0000 7f913a7fc640 1 -- 172.21.15.99:0/841575750 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c031870 con 0x7f913c15da20 2024-03-04T10:04:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.908+0000 7f913a7fc640 1 -- 172.21.15.99:0/841575750 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c031c40 con 0x7f913c15da20 2024-03-04T10:04:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 -- 172.21.15.99:0/841575750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 msgr2=0x7f913c0ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 --2- 172.21.15.99:0/841575750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c0ccdb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f913c0d1340 tx=0x7f912c02b910 comp rx=0 tx=0).stop 2024-03-04T10:04:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 -- 172.21.15.99:0/841575750 shutdown_connections 2024-03-04T10:04:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 --2- 172.21.15.99:0/841575750 >> v2:172.21.15.162:3300/0 conn(0x7f913c1616e0 0x7f913c161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 --2- 172.21.15.99:0/841575750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 0x7f913c0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 --2- 172.21.15.99:0/841575750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c0ccdb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 -- 172.21.15.99:0/841575750 >> 172.21.15.99:0/841575750 conn(0x7f913c0b33a0 msgr2=0x7f913c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 -- 172.21.15.99:0/841575750 shutdown_connections 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.909+0000 7f91420a4640 1 -- 172.21.15.99:0/841575750 wait complete. 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.910+0000 7f91420a4640 1 Processor -- start 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.910+0000 7f91420a4640 1 -- start start 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 0x7f913c15f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f913c1616e0 0x7f913c1f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0cbf80 con 0x7f913c15da20 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0cd2f0 con 0x7f913c1616e0 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15de00 con 0x7f913c0cf660 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c146a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38434/0 (socket says 172.21.15.99:38434) 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 -- 172.21.15.99:0/3046459380 learned_addr learned my addr 172.21.15.99:0/3046459380 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913bfff640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f913c1616e0 0x7f913c1f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 -- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 msgr2=0x7f913c15f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 0x7f913c15f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 -- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f913c1616e0 msgr2=0x7f913c1f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f913c1616e0 0x7f913c1f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 -- 172.21.15.99:0/3046459380 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c15f9a0 con 0x7f913c15da20 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f913affd640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c146a90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f913000b4a0 tx=0x7f913000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130014070 con 0x7f913c15da20 2024-03-04T10:04:08.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9130010610 con 0x7f913c15da20 2024-03-04T10:04:08.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c15fc80 con 0x7f913c15da20 2024-03-04T10:04:08.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130010a00 con 0x7f913c15da20 2024-03-04T10:04:08.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.911+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c160240 con 0x7f913c15da20 2024-03-04T10:04:08.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.912+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c074f60 con 0x7f913c15da20 2024-03-04T10:04:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.913+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9130017070 con 0x7f913c15da20 2024-03-04T10:04:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.913+0000 7f9123fff640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f912806b500 0x7f912806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.913+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9130089f60 con 0x7f913c15da20 2024-03-04T10:04:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.913+0000 7f913b7fe640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f912806b500 0x7f912806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.914+0000 7f913b7fe640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f912806b500 0x7f912806d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f912c036000 tx=0x7f912c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:08.917+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9130058aa0 con 0x7f913c15da20 2024-03-04T10:04:08.970 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:08.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.091+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f913c15d070 con 0x7f913c15da20 2024-03-04T10:04:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.091+0000 7f9123fff640 1 -- 172.21.15.99:0/3046459380 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f91300583a0 con 0x7f913c15da20 2024-03-04T10:04:09.093 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f912806b500 msgr2=0x7f912806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f912806b500 0x7f912806d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f912c036000 tx=0x7f912c030000 comp rx=0 tx=0).stop 2024-03-04T10:04:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 msgr2=0x7f913c146a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c146a90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f913000b4a0 tx=0x7f913000b960 comp rx=0 tx=0).stop 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 shutdown_connections 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f912806b500 0x7f912806d9c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f913c1616e0 0x7f913c1f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f913c15da20 0x7f913c146a90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 --2- 172.21.15.99:0/3046459380 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f913c0cf660 0x7f913c15f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 >> 172.21.15.99:0/3046459380 conn(0x7f913c0b33a0 msgr2=0x7f913c0b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 shutdown_connections 2024-03-04T10:04:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.093+0000 7f91420a4640 1 -- 172.21.15.99:0/3046459380 wait complete. 2024-03-04T10:04:09.105 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:09.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.205+0000 7f745bc80640 1 Processor -- start 2024-03-04T10:04:09.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.205+0000 7f745bc80640 1 -- start start 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745bc80640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f745406d3b0 0x7f745415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745bc80640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745415b410 0x7f745415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745bc80640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f745415be00 con 0x7f745415b030 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745bc80640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74540b2720 con 0x7f745406d3b0 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745bc80640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745406d8b0 con 0x7f745415b410 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f745406d3b0 0x7f745415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f745406d3b0 0x7f745415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38436/0 (socket says 172.21.15.99:38436) 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 -- 172.21.15.99:0/2325834917 learned_addr learned my addr 172.21.15.99:0/2325834917 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745a21d640 1 --2- 172.21.15.99:0/2325834917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745415b410 0x7f745415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 -- 172.21.15.99:0/2325834917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745415b410 msgr2=0x7f745415b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:09.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 --2- 172.21.15.99:0/2325834917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745415b410 0x7f745415b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 -- 172.21.15.99:0/2325834917 >> v1:172.21.15.162:6789/0 conn(0x7f745415b030 legacy=0x7f745415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.206+0000 7f745921b640 1 -- 172.21.15.99:0/2325834917 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f745406db90 con 0x7f745406d3b0 2024-03-04T10:04:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745921b640 1 --2- 172.21.15.99:0/2325834917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745406d3b0 0x7f745415c120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f74540b28a0 tx=0x7f744402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387b512a6e2ef305 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f7458a1a640 1 -- 172.21.15.99:0/2325834917 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7444035020 con 0x7f745406d3b0 2024-03-04T10:04:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f7458a1a640 1 -- 172.21.15.99:0/2325834917 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7444031640 con 0x7f745406d3b0 2024-03-04T10:04:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f7458a1a640 1 -- 172.21.15.99:0/2325834917 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744402d040 con 0x7f745406d3b0 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745bc80640 1 -- 172.21.15.99:0/2325834917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745406d3b0 msgr2=0x7f745415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745bc80640 1 --2- 172.21.15.99:0/2325834917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745406d3b0 0x7f745415c120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f74540b28a0 tx=0x7f744402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745bc80640 1 -- 172.21.15.99:0/2325834917 shutdown_connections 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745bc80640 1 --2- 172.21.15.99:0/2325834917 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745415b410 0x7f745415b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745bc80640 1 --2- 172.21.15.99:0/2325834917 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745406d3b0 0x7f745415c120 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.207+0000 7f745bc80640 1 -- 172.21.15.99:0/2325834917 >> 172.21.15.99:0/2325834917 conn(0x7f745406acd0 msgr2=0x7f745406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.208+0000 7f745bc80640 1 -- 172.21.15.99:0/2325834917 shutdown_connections 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.208+0000 7f745bc80640 1 -- 172.21.15.99:0/2325834917 wait complete. 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 Processor -- start 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 -- start start 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745406d3b0 0x7f74540a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f745415b030 0x7f74541513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 0x7f74540a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745406e690 con 0x7f745415b410 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745406e150 con 0x7f745415b030 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745bc80640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745406e810 con 0x7f745406d3b0 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745a21d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 0x7f74540a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745a21d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 0x7f74540a8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38444/0 (socket says 172.21.15.99:38444) 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.209+0000 7f745a21d640 1 -- 172.21.15.99:0/1166096751 learned_addr learned my addr 172.21.15.99:0/1166096751 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745921b640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f745415b030 0x7f74541513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f7459a1c640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745406d3b0 0x7f74540a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745a21d640 1 -- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745406d3b0 msgr2=0x7f74540a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745a21d640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745406d3b0 0x7f74540a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745a21d640 1 -- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f745415b030 msgr2=0x7f74541513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745a21d640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f745415b030 0x7f74541513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745a21d640 1 -- 172.21.15.99:0/1166096751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454151920 con 0x7f745415b410 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f7459a1c640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745406d3b0 0x7f74540a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745921b640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f745415b030 0x7f74541513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745a21d640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 0x7f74540a8460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f744c007e60 tx=0x7f744c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c014420 con 0x7f745415b410 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c019070 con 0x7f745415b410 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74540a8c90 con 0x7f745415b410 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c014820 con 0x7f745415b410 2024-03-04T10:04:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.210+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74540a9250 con 0x7f745415b410 2024-03-04T10:04:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.211+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7454074f60 con 0x7f745415b410 2024-03-04T10:04:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.212+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f744c0149c0 con 0x7f745415b410 2024-03-04T10:04:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.212+0000 7f744a7fc640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f743806b4d0 0x7f743806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.212+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f744c08ada0 con 0x7f745415b410 2024-03-04T10:04:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.212+0000 7f7459a1c640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f743806b4d0 0x7f743806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.213+0000 7f7459a1c640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f743806b4d0 0x7f743806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7450001680 tx=0x7f7450009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.216+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7454151920 con 0x7f745415b410 2024-03-04T10:04:09.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.421+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74540aec30 con 0x7f745415b410 2024-03-04T10:04:09.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.421+0000 7f744a7fc640 1 -- 172.21.15.99:0/1166096751 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f744c0598e0 con 0x7f745415b410 2024-03-04T10:04:09.422 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:09.422 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:09.423 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f743806b4d0 msgr2=0x7f743806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f743806b4d0 0x7f743806d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7450001680 tx=0x7f7450009040 comp rx=0 tx=0).stop 2024-03-04T10:04:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 msgr2=0x7f74540a8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 0x7f74540a8460 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f744c007e60 tx=0x7f744c00e4b0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 shutdown_connections 2024-03-04T10:04:09.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f743806b4d0 0x7f743806d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f745415b410 0x7f74540a8460 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f745415b030 0x7f74541513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 --2- 172.21.15.99:0/1166096751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f745406d3b0 0x7f74540a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 >> 172.21.15.99:0/1166096751 conn(0x7f745406acd0 msgr2=0x7f74541555f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 shutdown_connections 2024-03-04T10:04:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.423+0000 7f745bc80640 1 -- 172.21.15.99:0/1166096751 wait complete. 2024-03-04T10:04:09.435 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:09.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.534+0000 7f6749ae8640 1 Processor -- start 2024-03-04T10:04:09.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.534+0000 7f6749ae8640 1 -- start start 2024-03-04T10:04:09.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6749ae8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f674406b7c0 0x7f674415c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6749ae8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f67440cc330 0x7f67440cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6749ae8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67440cf700 con 0x7f674406acd0 2024-03-04T10:04:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6749ae8640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440d02d0 con 0x7f67440cc330 2024-03-04T10:04:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6749ae8640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67440cf9a0 con 0x7f674406b7c0 2024-03-04T10:04:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f67440cc330 0x7f67440cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f67440cc330 0x7f67440cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38456/0 (socket says 172.21.15.99:38456) 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 -- 172.21.15.99:0/3770845579 learned_addr learned my addr 172.21.15.99:0/3770845579 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f673bfff640 1 --2- 172.21.15.99:0/3770845579 >> v2:172.21.15.162:3300/0 conn(0x7f674406b7c0 0x7f674415c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 -- 172.21.15.99:0/3770845579 >> v2:172.21.15.162:3300/0 conn(0x7f674406b7c0 msgr2=0x7f674415c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 --2- 172.21.15.99:0/3770845579 >> v2:172.21.15.162:3300/0 conn(0x7f674406b7c0 0x7f674415c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 -- 172.21.15.99:0/3770845579 >> v1:172.21.15.162:6789/0 conn(0x7f674406acd0 legacy=0x7f674406b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f6743fff640 1 -- 172.21.15.99:0/3770845579 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67440cfc80 con 0x7f67440cc330 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.535+0000 7f673bfff640 1 --2- 172.21.15.99:0/3770845579 >> v2:172.21.15.162:3300/0 conn(0x7f674406b7c0 0x7f674415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6743fff640 1 --2- 172.21.15.99:0/3770845579 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67440cc330 0x7f67440cc7e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f67440afc30 tx=0x7f673c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1bc1d454383099 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6742ffd640 1 -- 172.21.15.99:0/3770845579 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c02ca80 con 0x7f67440cc330 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6742ffd640 1 -- 172.21.15.99:0/3770845579 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673c02cc20 con 0x7f67440cc330 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6742ffd640 1 -- 172.21.15.99:0/3770845579 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c0343f0 con 0x7f67440cc330 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6749ae8640 1 -- 172.21.15.99:0/3770845579 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67440cc330 msgr2=0x7f67440cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3770845579 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67440cc330 0x7f67440cc7e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f67440afc30 tx=0x7f673c02bee0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6749ae8640 1 -- 172.21.15.99:0/3770845579 shutdown_connections 2024-03-04T10:04:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3770845579 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f67440cc330 0x7f67440cc7e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3770845579 >> v2:172.21.15.162:3300/0 conn(0x7f674406b7c0 0x7f674415c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.536+0000 7f6749ae8640 1 -- 172.21.15.99:0/3770845579 >> 172.21.15.99:0/3770845579 conn(0x7f67440d0770 msgr2=0x7f67440d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.537+0000 7f6749ae8640 1 -- 172.21.15.99:0/3770845579 shutdown_connections 2024-03-04T10:04:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.537+0000 7f6749ae8640 1 -- 172.21.15.99:0/3770845579 wait complete. 2024-03-04T10:04:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 Processor -- start 2024-03-04T10:04:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 -- start start 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f674406acd0 0x7f67440c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 0x7f67440cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67440cc330 0x7f67440cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744159850 con 0x7f674406b7c0 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67440d13e0 con 0x7f67440cc330 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f6749ae8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67441599d0 con 0x7f674406acd0 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f673bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 0x7f67440cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f673bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 0x7f67440cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38464/0 (socket says 172.21.15.99:38464) 2024-03-04T10:04:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.538+0000 7f673bfff640 1 -- 172.21.15.99:0/3533878330 learned_addr learned my addr 172.21.15.99:0/3533878330 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f6743fff640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67440cc330 0x7f67440cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f67437fe640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f674406acd0 0x7f67440c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f673bfff640 1 -- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f674406acd0 msgr2=0x7f67440c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f673bfff640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f674406acd0 0x7f67440c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f673bfff640 1 -- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67440cc330 msgr2=0x7f67440cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f673bfff640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67440cc330 0x7f67440cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f673bfff640 1 -- 172.21.15.99:0/3533878330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67440c82c0 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f673bfff640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 0x7f67440cd430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f672c00e7e0 tx=0x7f672c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c017070 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f672c00c6d0 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67440c85a0 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c00f040 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.539+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67440c8b60 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.540+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6744074f60 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.541+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f672c01a070 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.541+0000 7f6740ff9640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f672406b500 0x7f672406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.541+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f672c08a3c0 con 0x7f674406b7c0 2024-03-04T10:04:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.541+0000 7f67437fe640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f672406b500 0x7f672406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.542+0000 7f67437fe640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f672406b500 0x7f672406d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6734007fd0 tx=0x7f6734009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.545+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f672c058f90 con 0x7f674406b7c0 2024-03-04T10:04:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.719+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67440cca00 con 0x7f674406b7c0 2024-03-04T10:04:09.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.720+0000 7f6740ff9640 1 -- 172.21.15.99:0/3533878330 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7f672c058890 con 0x7f674406b7c0 2024-03-04T10:04:09.722 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:09.722 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f672406b500 msgr2=0x7f672406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f672406b500 0x7f672406d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6734007fd0 tx=0x7f6734009040 comp rx=0 tx=0).stop 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 msgr2=0x7f67440cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 0x7f67440cd430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f672c00e7e0 tx=0x7f672c00eca0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 shutdown_connections 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f672406b500 0x7f672406d9c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67440cc330 0x7f67440cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f674406b7c0 0x7f67440cd430 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 --2- 172.21.15.99:0/3533878330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f674406acd0 0x7f67440c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 >> 172.21.15.99:0/3533878330 conn(0x7f67440d0770 msgr2=0x7f674415d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 shutdown_connections 2024-03-04T10:04:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.723+0000 7f6749ae8640 1 -- 172.21.15.99:0/3533878330 wait complete. 2024-03-04T10:04:09.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:09.735 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.840+0000 7f3a4770a640 1 Processor -- start 2024-03-04T10:04:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.840+0000 7f3a4770a640 1 -- start start 2024-03-04T10:04:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.840+0000 7f3a4770a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a4015c7f0 0x7f3a4006acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a4770a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a400c3e20 0x7f3a4006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a4770a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4015cf50 con 0x7f3a400b0150 2024-03-04T10:04:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a4770a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400cc390 con 0x7f3a400c3e20 2024-03-04T10:04:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a4770a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400cc670 con 0x7f3a4015c7f0 2024-03-04T10:04:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a400c3e20 0x7f3a4006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a400c3e20 0x7f3a4006b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38474/0 (socket says 172.21.15.99:38474) 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 -- 172.21.15.99:0/2834247836 learned_addr learned my addr 172.21.15.99:0/2834247836 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a454a6640 1 --2- 172.21.15.99:0/2834247836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a4015c7f0 0x7f3a4006acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 -- 172.21.15.99:0/2834247836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a4015c7f0 msgr2=0x7f3a4006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 --2- 172.21.15.99:0/2834247836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a4015c7f0 0x7f3a4006acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 -- 172.21.15.99:0/2834247836 >> v1:172.21.15.162:6789/0 conn(0x7f3a400b0150 legacy=0x7f3a4015f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:09.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.841+0000 7f3a45ca7640 1 -- 172.21.15.99:0/2834247836 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400cc950 con 0x7f3a400c3e20 2024-03-04T10:04:09.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a45ca7640 1 --2- 172.21.15.99:0/2834247836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400c3e20 0x7f3a4006b210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009e50 tx=0x7f3a3802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e4d831facbd05a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a37fff640 1 -- 172.21.15.99:0/2834247836 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3802c9e0 con 0x7f3a400c3e20 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a37fff640 1 -- 172.21.15.99:0/2834247836 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3802cb80 con 0x7f3a400c3e20 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a37fff640 1 -- 172.21.15.99:0/2834247836 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380343f0 con 0x7f3a400c3e20 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a4770a640 1 -- 172.21.15.99:0/2834247836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400c3e20 msgr2=0x7f3a4006b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2834247836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400c3e20 0x7f3a4006b210 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009e50 tx=0x7f3a3802be40 comp rx=0 tx=0).stop 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a4770a640 1 -- 172.21.15.99:0/2834247836 shutdown_connections 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2834247836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400c3e20 0x7f3a4006b210 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2834247836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a4015c7f0 0x7f3a4006acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.842+0000 7f3a4770a640 1 -- 172.21.15.99:0/2834247836 >> 172.21.15.99:0/2834247836 conn(0x7f3a400b22e0 msgr2=0x7f3a400b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.843+0000 7f3a4770a640 1 -- 172.21.15.99:0/2834247836 shutdown_connections 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.843+0000 7f3a4770a640 1 -- 172.21.15.99:0/2834247836 wait complete. 2024-03-04T10:04:09.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 Processor -- start 2024-03-04T10:04:09.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 -- start start 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 0x7f3a400cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a400c3e20 0x7f3a400b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a400b6090 0x7f3a400cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400b2e00 con 0x7f3a400b0150 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4015cbe0 con 0x7f3a400b6090 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a4770a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400b2f80 con 0x7f3a400c3e20 2024-03-04T10:04:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a454a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 0x7f3a400cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a454a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 0x7f3a400cfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38478/0 (socket says 172.21.15.99:38478) 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.844+0000 7f3a454a6640 1 -- 172.21.15.99:0/2125358459 learned_addr learned my addr 172.21.15.99:0/2125358459 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a454a6640 1 -- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a400c3e20 msgr2=0x7f3a400b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a44ca5640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a400c3e20 0x7f3a400b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a45ca7640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a400b6090 0x7f3a400cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a454a6640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a400c3e20 0x7f3a400b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a454a6640 1 -- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a400b6090 msgr2=0x7f3a400cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a454a6640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a400b6090 0x7f3a400cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a454a6640 1 -- 172.21.15.99:0/2125358459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400ce5b0 con 0x7f3a400b0150 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a454a6640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 0x7f3a400cfbd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00d800 tx=0x7f3a3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c00b6c0 con 0x7f3a400b0150 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c019070 con 0x7f3a400b0150 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a400ce890 con 0x7f3a400b0150 2024-03-04T10:04:09.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c00ba80 con 0x7f3a400b0150 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.845+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a400cee50 con 0x7f3a400b0150 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.846+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40074f60 con 0x7f3a400b0150 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.847+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3a3c00bc20 con 0x7f3a400b0150 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.847+0000 7f3a35ffb640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a2406b4d0 0x7f3a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.847+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3a3c08a9f0 con 0x7f3a400b0150 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.847+0000 7f3a44ca5640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a2406b4d0 0x7f3a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.848+0000 7f3a44ca5640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a2406b4d0 0x7f3a2406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a400704e0 tx=0x7f3a30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:09.851+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a3c059530 con 0x7f3a400b0150 2024-03-04T10:04:10.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.022+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a40072f90 con 0x7f3a400b0150 2024-03-04T10:04:10.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.023+0000 7f3a35ffb640 1 -- 172.21.15.99:0/2125358459 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a3c058e30 con 0x7f3a400b0150 2024-03-04T10:04:10.024 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.024+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a2406b4d0 msgr2=0x7f3a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.024+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a2406b4d0 0x7f3a2406d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a400704e0 tx=0x7f3a30009040 comp rx=0 tx=0).stop 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.024+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 msgr2=0x7f3a400cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.024+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 0x7f3a400cfbd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00d800 tx=0x7f3a3c00dcc0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 shutdown_connections 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a2406b4d0 0x7f3a2406d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a400b6090 0x7f3a400cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a400c3e20 0x7f3a400b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 --2- 172.21.15.99:0/2125358459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a400b0150 0x7f3a400cfbd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 >> 172.21.15.99:0/2125358459 conn(0x7f3a400b22e0 msgr2=0x7f3a40155a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 shutdown_connections 2024-03-04T10:04:10.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.025+0000 7f3a4770a640 1 -- 172.21.15.99:0/2125358459 wait complete. 2024-03-04T10:04:10.037 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 Processor -- start 2024-03-04T10:04:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 -- start start 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040b5700 0x7fe0040d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0040b9410 0x7fe00406c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0040d2400 con 0x7fe0040d34a0 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0040d26e0 con 0x7fe0040b9410 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.139+0000 7fe008c06640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040d29c0 con 0x7fe0040b5700 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe002d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0040b9410 0x7fe00406c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe00259c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040b5700 0x7fe0040d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fdffbfff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe0040d34a0 0x7fe00406c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43758/0 (socket says 172.21.15.99:43758) 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe002d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0040b9410 0x7fe00406c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:39742/0 (socket says 172.21.15.99:39742) 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fdffbfff640 1 -- 172.21.15.99:0/4051117492 learned_addr learned my addr 172.21.15.99:0/4051117492 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe001d9b640 1 -- 172.21.15.99:0/4051117492 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753432044 0 0) 0x7fe0040d2400 con 0x7fe0040d34a0 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe001d9b640 1 -- 172.21.15.99:0/4051117492 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff0003520 con 0x7fe0040d34a0 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe002d9d640 1 -- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040b5700 msgr2=0x7fe0040d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe002d9d640 1 --2- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040b5700 0x7fe0040d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe002d9d640 1 -- 172.21.15.99:0/4051117492 >> v1:172.21.15.162:6789/0 conn(0x7fe0040d34a0 legacy=0x7fe00406c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe002d9d640 1 -- 172.21.15.99:0/4051117492 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0040d2ca0 con 0x7fe0040b9410 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.140+0000 7fe00259c640 1 --2- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040b5700 0x7fe0040d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.141+0000 7fe002d9d640 1 --2- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b9410 0x7fe00406c7f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d90 tx=0x7fdffc02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df8b060ac2561e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.141+0000 7fe001d9b640 1 -- 172.21.15.99:0/4051117492 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc039070 con 0x7fe0040b9410 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.141+0000 7fe001d9b640 1 -- 172.21.15.99:0/4051117492 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdffc02cbd0 con 0x7fe0040b9410 2024-03-04T10:04:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.141+0000 7fe001d9b640 1 -- 172.21.15.99:0/4051117492 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdffc034440 con 0x7fe0040b9410 2024-03-04T10:04:10.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 -- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b9410 msgr2=0x7fe00406c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 --2- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b9410 0x7fe00406c7f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdffc009d90 tx=0x7fdffc02be20 comp rx=0 tx=0).stop 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 -- 172.21.15.99:0/4051117492 shutdown_connections 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 --2- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b9410 0x7fe00406c7f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 --2- 172.21.15.99:0/4051117492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040b5700 0x7fe0040d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 -- 172.21.15.99:0/4051117492 >> 172.21.15.99:0/4051117492 conn(0x7fe0040d0490 msgr2=0x7fe0040d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 -- 172.21.15.99:0/4051117492 shutdown_connections 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.142+0000 7fe008c06640 1 -- 172.21.15.99:0/4051117492 wait complete. 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.143+0000 7fe008c06640 1 Processor -- start 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.143+0000 7fe008c06640 1 -- start start 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe008c06640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b5700 0x7fe0040c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe008c06640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 0x7fe0040c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe008c06640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040d34a0 0x7fe0040b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe008c06640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00406ceb0 con 0x7fe0040b9410 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe008c06640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00406cd30 con 0x7fe0040b5700 2024-03-04T10:04:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe008c06640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0040b5ae0 con 0x7fe0040d34a0 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 0x7fe0040c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 0x7fe0040c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38482/0 (socket says 172.21.15.99:38482) 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 -- 172.21.15.99:0/3656970928 learned_addr learned my addr 172.21.15.99:0/3656970928 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe00259c640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b5700 0x7fe0040c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe002d9d640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040d34a0 0x7fe0040b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 -- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040d34a0 msgr2=0x7fe0040b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040d34a0 0x7fe0040b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 -- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b5700 msgr2=0x7fe0040c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b5700 0x7fe0040c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fdffbfff640 1 -- 172.21.15.99:0/3656970928 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0040c94b0 con 0x7fe0040b9410 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.144+0000 7fe00259c640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b5700 0x7fe0040c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.145+0000 7fdffbfff640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 0x7fe0040c8f70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00c390 tx=0x7fdfec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.145+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec017070 con 0x7fe0040b9410 2024-03-04T10:04:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.145+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec005ce0 con 0x7fe0040b9410 2024-03-04T10:04:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.145+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe004152960 con 0x7fe0040b9410 2024-03-04T10:04:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.145+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec0072f0 con 0x7fe0040b9410 2024-03-04T10:04:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.145+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe004152f20 con 0x7fe0040b9410 2024-03-04T10:04:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.146+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe004074f60 con 0x7fe0040b9410 2024-03-04T10:04:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.146+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdfec01a070 con 0x7fe0040b9410 2024-03-04T10:04:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.147+0000 7fdffaffd640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfe406b4d0 0x7fdfe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.147+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdfec089940 con 0x7fe0040b9410 2024-03-04T10:04:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.147+0000 7fe00259c640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfe406b4d0 0x7fdfe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.147+0000 7fe00259c640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfe406b4d0 0x7fdfe406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009ba0 tx=0x7fdff4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.151+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdfec058480 con 0x7fe0040b9410 2024-03-04T10:04:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.321+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0040b41f0 con 0x7fe0040b9410 2024-03-04T10:04:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.321+0000 7fdffaffd640 1 -- 172.21.15.99:0/3656970928 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdfec057d80 con 0x7fe0040b9410 2024-03-04T10:04:10.322 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfe406b4d0 msgr2=0x7fdfe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfe406b4d0 0x7fdfe406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009ba0 tx=0x7fdff4008040 comp rx=0 tx=0).stop 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 msgr2=0x7fe0040c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 0x7fe0040c8f70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00c390 tx=0x7fdfec00c850 comp rx=0 tx=0).stop 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 shutdown_connections 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfe406b4d0 0x7fdfe406d990 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009ba0 tx=0x7fdff4008040 comp rx=0 tx=0).stop 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040d34a0 0x7fe0040b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040b9410 0x7fe0040c8f70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 --2- 172.21.15.99:0/3656970928 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe0040b5700 0x7fe0040c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 >> 172.21.15.99:0/3656970928 conn(0x7fe0040d0490 msgr2=0x7fe00415fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 shutdown_connections 2024-03-04T10:04:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.323+0000 7fe008c06640 1 -- 172.21.15.99:0/3656970928 wait complete. 2024-03-04T10:04:10.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:10.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.438+0000 7f92476d1640 1 Processor -- start 2024-03-04T10:04:10.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.439+0000 7f92476d1640 1 -- start start 2024-03-04T10:04:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.439+0000 7f92476d1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f924015de40 0x7f92400b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.439+0000 7f92476d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cf020 0x7f92400d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f92476d1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92400ca970 con 0x7f92400cec40 2024-03-04T10:04:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f92476d1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9240161d90 con 0x7f924015de40 2024-03-04T10:04:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f92476d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9240162070 con 0x7f92400cf020 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f924546d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f92400cec40 0x7f924015d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34890/0 (socket says 172.21.15.99:34890) 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f924546d640 1 -- 172.21.15.99:0/3228264550 learned_addr learned my addr 172.21.15.99:0/3228264550 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9245c6e640 1 --2- 172.21.15.99:0/3228264550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cf020 0x7f92400d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9244c6c640 1 --2- 172.21.15.99:0/3228264550 >> v2:172.21.15.162:3300/0 conn(0x7f924015de40 0x7f92400b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9237fff640 1 -- 172.21.15.99:0/3228264550 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446418669 0 0) 0x7f92400ca970 con 0x7f92400cec40 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9237fff640 1 -- 172.21.15.99:0/3228264550 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f922c003520 con 0x7f92400cec40 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9245c6e640 1 -- 172.21.15.99:0/3228264550 >> v2:172.21.15.162:3300/0 conn(0x7f924015de40 msgr2=0x7f92400b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9245c6e640 1 --2- 172.21.15.99:0/3228264550 >> v2:172.21.15.162:3300/0 conn(0x7f924015de40 0x7f92400b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9245c6e640 1 -- 172.21.15.99:0/3228264550 >> v1:172.21.15.99:6789/0 conn(0x7f92400cec40 legacy=0x7f924015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9245c6e640 1 -- 172.21.15.99:0/3228264550 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240162350 con 0x7f92400cf020 2024-03-04T10:04:10.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.440+0000 7f9244c6c640 1 --2- 172.21.15.99:0/3228264550 >> v2:172.21.15.162:3300/0 conn(0x7f924015de40 0x7f92400b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.441+0000 7f9245c6e640 1 --2- 172.21.15.99:0/3228264550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cf020 0x7f92400d2110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92380097a0 tx=0x7f923802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9465a3cc68356260 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.441+0000 7f9237fff640 1 -- 172.21.15.99:0/3228264550 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238035020 con 0x7f92400cf020 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.441+0000 7f9237fff640 1 -- 172.21.15.99:0/3228264550 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9238031830 con 0x7f92400cf020 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.441+0000 7f9237fff640 1 -- 172.21.15.99:0/3228264550 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238031c00 con 0x7f92400cf020 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.441+0000 7f92476d1640 1 -- 172.21.15.99:0/3228264550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cf020 msgr2=0x7f92400d2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.441+0000 7f92476d1640 1 --2- 172.21.15.99:0/3228264550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cf020 0x7f92400d2110 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f92380097a0 tx=0x7f923802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.442+0000 7f92476d1640 1 -- 172.21.15.99:0/3228264550 shutdown_connections 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.442+0000 7f92476d1640 1 --2- 172.21.15.99:0/3228264550 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cf020 0x7f92400d2110 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.442+0000 7f92476d1640 1 --2- 172.21.15.99:0/3228264550 >> v2:172.21.15.162:3300/0 conn(0x7f924015de40 0x7f92400b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.442+0000 7f92476d1640 1 -- 172.21.15.99:0/3228264550 >> 172.21.15.99:0/3228264550 conn(0x7f92400d0580 msgr2=0x7f92400d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.442+0000 7f92476d1640 1 -- 172.21.15.99:0/3228264550 shutdown_connections 2024-03-04T10:04:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.442+0000 7f92476d1640 1 -- 172.21.15.99:0/3228264550 wait complete. 2024-03-04T10:04:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 Processor -- start 2024-03-04T10:04:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 -- start start 2024-03-04T10:04:10.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cec40 0x7f9240152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 0x7f92400cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f924015de40 0x7f924014c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92400d3300 con 0x7f92400cf020 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92400d1bf0 con 0x7f924015de40 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.443+0000 7f92476d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92400d3480 con 0x7f92400cec40 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 0x7f92400cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 0x7f92400cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38492/0 (socket says 172.21.15.99:38492) 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 -- 172.21.15.99:0/2505550761 learned_addr learned my addr 172.21.15.99:0/2505550761 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9245c6e640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f924015de40 0x7f924014c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f924546d640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cec40 0x7f9240152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 -- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cec40 msgr2=0x7f9240152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cec40 0x7f9240152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 -- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f924015de40 msgr2=0x7f924014c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f924015de40 0x7f924014c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 -- 172.21.15.99:0/2505550761 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924015f720 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9245c6e640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f924015de40 0x7f924014c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9244c6c640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 0x7f92400cba40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f923000ebe0 tx=0x7f923000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230017070 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923000cc30 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230013470 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924015f9a0 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.444+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f924015ff60 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.445+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9240074f60 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.446+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f923001a070 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.446+0000 7f9235ffb640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.446+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f923008a780 con 0x7f92400cf020 2024-03-04T10:04:10.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.447+0000 7f924546d640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.447+0000 7f924546d640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f922406b4d0 0x7f922406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f923c0012e0 tx=0x7f923c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.451+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f924015f720 con 0x7f92400cf020 2024-03-04T10:04:10.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.622+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92400d1930 con 0x7f92400cf020 2024-03-04T10:04:10.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.623+0000 7f9235ffb640 1 -- 172.21.15.99:0/2505550761 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9230059350 con 0x7f92400cf020 2024-03-04T10:04:10.624 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f922406b4d0 msgr2=0x7f922406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f922406b4d0 0x7f922406d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f923c0012e0 tx=0x7f923c009040 comp rx=0 tx=0).stop 2024-03-04T10:04:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 msgr2=0x7f92400cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 0x7f92400cba40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f923000ebe0 tx=0x7f923000c520 comp rx=0 tx=0).stop 2024-03-04T10:04:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 shutdown_connections 2024-03-04T10:04:10.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f922406b4d0 0x7f922406d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f924015de40 0x7f924014c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92400cf020 0x7f92400cba40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 --2- 172.21.15.99:0/2505550761 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92400cec40 0x7f9240152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 >> 172.21.15.99:0/2505550761 conn(0x7f92400d0580 msgr2=0x7f924006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 shutdown_connections 2024-03-04T10:04:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.625+0000 7f92476d1640 1 -- 172.21.15.99:0/2505550761 wait complete. 2024-03-04T10:04:10.637 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:10.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.738+0000 7f668dbb9640 1 Processor -- start 2024-03-04T10:04:10.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.738+0000 7f668dbb9640 1 -- start start 2024-03-04T10:04:10.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f668dbb9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66880ccbd0 0x7f66880b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f668dbb9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f66880ceec0 0x7f66880b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f668dbb9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668815ca50 con 0x7f668815c2e0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f668dbb9640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668815d2f0 con 0x7f66880ceec0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f668dbb9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668815d6d0 con 0x7f66880ccbd0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f6687fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f66880ceec0 0x7f66880b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f667effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f668815c2e0 0x7f66880ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34900/0 (socket says 172.21.15.99:34900) 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.739+0000 7f667effd640 1 -- 172.21.15.99:0/3465552897 learned_addr learned my addr 172.21.15.99:0/3465552897 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f66877fe640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66880ccbd0 0x7f66880b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f6687fff640 1 -- 172.21.15.99:0/3465552897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66880ccbd0 msgr2=0x7f66880b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f6687fff640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66880ccbd0 0x7f66880b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f6687fff640 1 -- 172.21.15.99:0/3465552897 >> v1:172.21.15.99:6789/0 conn(0x7f668815c2e0 legacy=0x7f66880ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f6687fff640 1 -- 172.21.15.99:0/3465552897 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668815cd30 con 0x7f66880ceec0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f6686ffd640 1 -- 172.21.15.99:0/3465552897 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327235248 0 0) 0x7f668815ca50 con 0x7f668815c2e0 2024-03-04T10:04:10.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f66877fe640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66880ccbd0 0x7f66880b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.740+0000 7f6687fff640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880b0560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6680009d90 tx=0x7f668002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b5534de60b65da server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f6686ffd640 1 -- 172.21.15.99:0/3465552897 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680039070 con 0x7f66880ceec0 2024-03-04T10:04:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f6686ffd640 1 -- 172.21.15.99:0/3465552897 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668002cc50 con 0x7f66880ceec0 2024-03-04T10:04:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f6686ffd640 1 -- 172.21.15.99:0/3465552897 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680034440 con 0x7f66880ceec0 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 -- 172.21.15.99:0/3465552897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 msgr2=0x7f66880b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880b0560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6680009d90 tx=0x7f668002bea0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 -- 172.21.15.99:0/3465552897 shutdown_connections 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880b0560 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 --2- 172.21.15.99:0/3465552897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f66880ccbd0 0x7f66880b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 -- 172.21.15.99:0/3465552897 >> 172.21.15.99:0/3465552897 conn(0x7f6688066dc0 msgr2=0x7f66880671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 -- 172.21.15.99:0/3465552897 shutdown_connections 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.741+0000 7f668dbb9640 1 -- 172.21.15.99:0/3465552897 wait complete. 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.742+0000 7f668dbb9640 1 Processor -- start 2024-03-04T10:04:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 -- start start 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66880ccbd0 0x7f66880c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f668815c2e0 0x7f6688149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880ccfb0 con 0x7f66880ceec0 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668815c760 con 0x7f66880ccbd0 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f668dbb9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66880b0aa0 con 0x7f668815c2e0 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f667effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f667effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880cb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38512/0 (socket says 172.21.15.99:38512) 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f667effd640 1 -- 172.21.15.99:0/2815478129 learned_addr learned my addr 172.21.15.99:0/2815478129 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f66877fe640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66880ccbd0 0x7f66880c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.743+0000 7f6687fff640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f668815c2e0 0x7f6688149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f667effd640 1 -- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f668815c2e0 msgr2=0x7f6688149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f667effd640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f668815c2e0 0x7f6688149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f667effd640 1 -- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66880ccbd0 msgr2=0x7f66880c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f667effd640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66880ccbd0 0x7f66880c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f667effd640 1 -- 172.21.15.99:0/2815478129 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688156a00 con 0x7f66880ceec0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f667effd640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880cb2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f667000c460 tx=0x7f667000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670018070 con 0x7f66880ceec0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670007500 con 0x7f66880ceec0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6688156ce0 con 0x7f66880ceec0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670004a20 con 0x7f66880ceec0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.744+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6688156660 con 0x7f66880ceec0 2024-03-04T10:04:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.745+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66880c7d60 con 0x7f66880ceec0 2024-03-04T10:04:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.745+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f667001b070 con 0x7f66880ceec0 2024-03-04T10:04:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.746+0000 7f6684ff9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.746+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6670089810 con 0x7f66880ceec0 2024-03-04T10:04:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.746+0000 7f66877fe640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.747+0000 7f66877fe640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f666806b4d0 0x7f666806d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f667800a990 tx=0x7f6678008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.750+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6670058350 con 0x7f66880ceec0 2024-03-04T10:04:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.919+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6688074f60 con 0x7f66880ceec0 2024-03-04T10:04:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.919+0000 7f6684ff9640 1 -- 172.21.15.99:0/2815478129 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6670087e20 con 0x7f66880ceec0 2024-03-04T10:04:10.920 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f666806b4d0 msgr2=0x7f666806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f666806b4d0 0x7f666806d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f667800a990 tx=0x7f6678008040 comp rx=0 tx=0).stop 2024-03-04T10:04:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 msgr2=0x7f66880cb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880cb2c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f667000c460 tx=0x7f667000c920 comp rx=0 tx=0).stop 2024-03-04T10:04:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 shutdown_connections 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f666806b4d0 0x7f666806d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f668815c2e0 0x7f6688149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f66880ceec0 0x7f66880cb2c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 --2- 172.21.15.99:0/2815478129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f66880ccbd0 0x7f66880c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 >> 172.21.15.99:0/2815478129 conn(0x7f6688066dc0 msgr2=0x7f66880cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 shutdown_connections 2024-03-04T10:04:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:10.921+0000 7f668dbb9640 1 -- 172.21.15.99:0/2815478129 wait complete. 2024-03-04T10:04:10.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.032+0000 7f6e54390640 1 Processor -- start 2024-03-04T10:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.032+0000 7f6e54390640 1 -- start start 2024-03-04T10:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e54390640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6e4c0b0700 0x7f6e4c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e54390640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6e4c0b47c0 0x7f6e4c0b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e54390640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c0ccad0 con 0x7f6e4c15b960 2024-03-04T10:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e54390640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0b5190 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e54390640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c0cf860 con 0x7f6e4c0b0700 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6e4c0b47c0 0x7f6e4c0b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6e4c0b47c0 0x7f6e4c0b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38520/0 (socket says 172.21.15.99:38520) 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 -- 172.21.15.99:0/2338097693 learned_addr learned my addr 172.21.15.99:0/2338097693 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5192b640 1 --2- 172.21.15.99:0/2338097693 >> v2:172.21.15.162:3300/0 conn(0x7f6e4c0b0700 0x7f6e4c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 -- 172.21.15.99:0/2338097693 >> v2:172.21.15.162:3300/0 conn(0x7f6e4c0b0700 msgr2=0x7f6e4c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 --2- 172.21.15.99:0/2338097693 >> v2:172.21.15.162:3300/0 conn(0x7f6e4c0b0700 0x7f6e4c0b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 -- 172.21.15.99:0/2338097693 >> v1:172.21.15.162:6789/0 conn(0x7f6e4c15b960 legacy=0x7f6e4c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.033+0000 7f6e5292d640 1 -- 172.21.15.99:0/2338097693 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0cfab0 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.034+0000 7f6e5292d640 1 --2- 172.21.15.99:0/2338097693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0b4c50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0b73c0 tx=0x7f6e4402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1632a2eed403be82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.034+0000 7f6e5112a640 1 -- 172.21.15.99:0/2338097693 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4402ca80 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.034+0000 7f6e5112a640 1 -- 172.21.15.99:0/2338097693 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4402cc20 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.034+0000 7f6e5112a640 1 -- 172.21.15.99:0/2338097693 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e440343f0 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.034+0000 7f6e54390640 1 -- 172.21.15.99:0/2338097693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 msgr2=0x7f6e4c0b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.034+0000 7f6e54390640 1 --2- 172.21.15.99:0/2338097693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0b4c50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0b73c0 tx=0x7f6e4402bee0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.035+0000 7f6e54390640 1 -- 172.21.15.99:0/2338097693 shutdown_connections 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.035+0000 7f6e54390640 1 --2- 172.21.15.99:0/2338097693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0b4c50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.035+0000 7f6e54390640 1 --2- 172.21.15.99:0/2338097693 >> v2:172.21.15.162:3300/0 conn(0x7f6e4c0b0700 0x7f6e4c0b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.035+0000 7f6e54390640 1 -- 172.21.15.99:0/2338097693 >> 172.21.15.99:0/2338097693 conn(0x7f6e4c06f5f0 msgr2=0x7f6e4c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.035+0000 7f6e54390640 1 -- 172.21.15.99:0/2338097693 shutdown_connections 2024-03-04T10:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.035+0000 7f6e54390640 1 -- 172.21.15.99:0/2338097693 wait complete. 2024-03-04T10:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 Processor -- start 2024-03-04T10:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 -- start start 2024-03-04T10:04:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0b0700 0x7f6e4c1f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c15b960 0x7f6e4c14d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c15bd40 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0d0100 con 0x7f6e4c15b960 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.036+0000 7f6e54390640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c15bec0 con 0x7f6e4c0b0700 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0a9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38528/0 (socket says 172.21.15.99:38528) 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 -- 172.21.15.99:0/3086445635 learned_addr learned my addr 172.21.15.99:0/3086445635 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5292d640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c15b960 0x7f6e4c14d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5212c640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0b0700 0x7f6e4c1f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 -- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0b0700 msgr2=0x7f6e4c1f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0b0700 0x7f6e4c1f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 -- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c15b960 msgr2=0x7f6e4c14d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c15b960 0x7f6e4c14d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 -- 172.21.15.99:0/3086445635 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c0cc820 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5292d640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c15b960 0x7f6e4c14d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5212c640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0b0700 0x7f6e4c1f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e5192b640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0a9360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00d830 tx=0x7f6e3c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c00b6c0 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c019070 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c00ba80 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c0a85a0 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.037+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c0a8b60 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.038+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c074f60 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.039+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6e3c01e460 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.039+0000 7f6e42ffd640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e3006b4d0 0x7f6e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.039+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6e3c08ab60 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.039+0000 7f6e5212c640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e3006b4d0 0x7f6e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.040+0000 7f6e5212c640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e3006b4d0 0x7f6e3006d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6e480015e0 tx=0x7f6e48009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.043+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6e3c059730 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.246+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e4c15b160 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.246+0000 7f6e42ffd640 1 -- 172.21.15.99:0/3086445635 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f6e3c059030 con 0x7f6e4c0b47c0 2024-03-04T10:04:11.247 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:11.247 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:11.248 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e3006b4d0 msgr2=0x7f6e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e3006b4d0 0x7f6e3006d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6e480015e0 tx=0x7f6e48009040 comp rx=0 tx=0).stop 2024-03-04T10:04:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 msgr2=0x7f6e4c0a9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0a9360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00d830 tx=0x7f6e3c00dcf0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 shutdown_connections 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e3006b4d0 0x7f6e3006d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c15b960 0x7f6e4c14d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0b47c0 0x7f6e4c0a9360 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 --2- 172.21.15.99:0/3086445635 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0b0700 0x7f6e4c1f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 >> 172.21.15.99:0/3086445635 conn(0x7f6e4c06f5f0 msgr2=0x7f6e4c0d0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 shutdown_connections 2024-03-04T10:04:11.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.248+0000 7f6e54390640 1 -- 172.21.15.99:0/3086445635 wait complete. 2024-03-04T10:04:11.260 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.359+0000 7fc0de124640 1 Processor -- start 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.359+0000 7fc0de124640 1 -- start start 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0de124640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 0x7fc0d80d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0de124640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc0d80cc690 0x7fc0d80d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0de124640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80cfb40 con 0x7fc0d80d0e40 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0de124640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d80d2880 con 0x7fc0d80cc690 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0de124640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80cfe20 con 0x7fc0d815bbb0 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d6ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc0d80d0e40 0x7fc0d80cbf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34906/0 (socket says 172.21.15.99:34906) 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d6ffd640 1 -- 172.21.15.99:0/13207804 learned_addr learned my addr 172.21.15.99:0/13207804 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/13207804 >> v2:172.21.15.162:3300/0 conn(0x7fc0d80cc690 0x7fc0d80d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646775090 0 0) 0x7fc0d80cfb40 con 0x7fc0d80d0e40 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d77fe640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 0x7fc0d80d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c0003520 con 0x7fc0d80d0e40 2024-03-04T10:04:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310404565 0 0) 0x7fc0c0003520 con 0x7fc0d80d0e40 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 msgr2=0x7fc0d80d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 0x7fc0d80d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 >> v2:172.21.15.162:3300/0 conn(0x7fc0d80cc690 msgr2=0x7fc0d80d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 --2- 172.21.15.99:0/13207804 >> v2:172.21.15.162:3300/0 conn(0x7fc0d80cc690 0x7fc0d80d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80d0100 con 0x7fc0d80d0e40 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fc0cc001670 con 0x7fc0d80d0e40 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.360+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 >> v1:172.21.15.99:6789/0 conn(0x7fc0d80d0e40 legacy=0x7fc0d80cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d67fc640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c00039b0 0x7fc0c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d67fc640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 0x7fc0c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d67fc640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0c0004d40 0x7fc0c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80d2880 con 0x7fc0c0004d40 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80cfe20 con 0x7fc0c00039b0 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c000b9c0 con 0x7fc0d815bbb0 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0c0004d40 0x7fc0c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d77fe640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c00039b0 0x7fc0c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d6ffd640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 0x7fc0c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d7fff640 1 -- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 msgr2=0x7fc0c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d815bbb0 0x7fc0c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d7fff640 1 -- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c00039b0 msgr2=0x7fc0c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c00039b0 0x7fc0c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.361+0000 7fc0d7fff640 1 -- 172.21.15.99:0/13207804 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0003520 con 0x7fc0c0004d40 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0c0004d40 0x7fc0c00051c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80b13c0 tx=0x7fc0c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982366e617e92d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0039070 con 0x7fc0c0004d40 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d002cbd0 con 0x7fc0c0004d40 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0d67fc640 1 -- 172.21.15.99:0/13207804 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0009c10 con 0x7fc0c0004d40 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0de124640 1 -- 172.21.15.99:0/13207804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0c0004d40 msgr2=0x7fc0c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0de124640 1 --2- 172.21.15.99:0/13207804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0c0004d40 0x7fc0c00051c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80b13c0 tx=0x7fc0c0005830 comp rx=0 tx=0).stop 2024-03-04T10:04:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0d7fff640 1 -- 172.21.15.99:0/13207804 reap_dead start 2024-03-04T10:04:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0de124640 1 -- 172.21.15.99:0/13207804 shutdown_connections 2024-03-04T10:04:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0de124640 1 -- 172.21.15.99:0/13207804 >> 172.21.15.99:0/13207804 conn(0x7fc0d80b3480 msgr2=0x7fc0d80b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0de124640 1 -- 172.21.15.99:0/13207804 shutdown_connections 2024-03-04T10:04:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.362+0000 7fc0de124640 1 -- 172.21.15.99:0/13207804 wait complete. 2024-03-04T10:04:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 Processor -- start 2024-03-04T10:04:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 -- start start 2024-03-04T10:04:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c0004d40 0x7fc0d8151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d8150c60 0x7fc0d80c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 0x7fc0d8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d815bf90 con 0x7fc0d81504d0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80d0900 con 0x7fc0c0004d40 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.364+0000 7fc0de124640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d8152a00 con 0x7fc0d8150c60 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 0x7fc0d8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 0x7fc0d8153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38544/0 (socket says 172.21.15.99:38544) 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 -- 172.21.15.99:0/898580901 learned_addr learned my addr 172.21.15.99:0/898580901 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d77fe640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c0004d40 0x7fc0d8151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d6ffd640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d8150c60 0x7fc0d80c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 -- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d8150c60 msgr2=0x7fc0d80c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d8150c60 0x7fc0d80c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 -- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c0004d40 msgr2=0x7fc0d8151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c0004d40 0x7fc0d8151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 -- 172.21.15.99:0/898580901 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8152720 con 0x7fc0d81504d0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0d7fff640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 0x7fc0d8153e30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000e230 tx=0x7fc0d0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0039030 con 0x7fc0d81504d0 2024-03-04T10:04:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d0043070 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d815d490 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0009c10 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.365+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d815da50 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.366+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d815dc50 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.367+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc0d0005d10 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.367+0000 7fc0bffff640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0c406b500 0x7fc0c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.367+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc0d00ac680 con 0x7fc0d81504d0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.367+0000 7fc0d77fe640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0c406b500 0x7fc0c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.368+0000 7fc0d77fe640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0c406b500 0x7fc0c406d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8157420 tx=0x7fc0c8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.371+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc0d815dc50 con 0x7fc0d81504d0 2024-03-04T10:04:11.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.542+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d8074f60 con 0x7fc0d81504d0 2024-03-04T10:04:11.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.543+0000 7fc0bffff640 1 -- 172.21.15.99:0/898580901 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18478 (secure 0 0 0) 0x7fc0d007b280 con 0x7fc0d81504d0 2024-03-04T10:04:11.544 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:11.545 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":24,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:03:43.916123+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0c406b500 msgr2=0x7fc0c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0c406b500 0x7fc0c406d9c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8157420 tx=0x7fc0c8009190 comp rx=0 tx=0).stop 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 msgr2=0x7fc0d8153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 0x7fc0d8153e30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000e230 tx=0x7fc0d0009750 comp rx=0 tx=0).stop 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 shutdown_connections 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0c406b500 0x7fc0c406d9c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d81504d0 0x7fc0d8153e30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d8150c60 0x7fc0d80c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 --2- 172.21.15.99:0/898580901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0c0004d40 0x7fc0d8151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 >> 172.21.15.99:0/898580901 conn(0x7fc0d80b3480 msgr2=0x7fc0d80d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.545+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 shutdown_connections 2024-03-04T10:04:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.546+0000 7fc0de124640 1 -- 172.21.15.99:0/898580901 wait complete. 2024-03-04T10:04:11.558 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.657+0000 7f58556f1640 1 Processor -- start 2024-03-04T10:04:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.657+0000 7f58556f1640 1 -- start start 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.658+0000 7f58556f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58500d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.658+0000 7f58556f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f58500cc690 0x7f58500d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.658+0000 7f58556f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58500cfb40 con 0x7f58500d0e40 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.658+0000 7f58556f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58500d2880 con 0x7f58500cc690 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.658+0000 7f58556f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58500cfe20 con 0x7f585015bbb0 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584f7fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f58500cc690 0x7f58500d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584effd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58500d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584e7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f58500d0e40 0x7f58500cbf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:43780/0 (socket says 172.21.15.99:43780) 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584e7fc640 1 -- 172.21.15.99:0/1911163326 learned_addr learned my addr 172.21.15.99:0/1911163326 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584dffb640 1 -- 172.21.15.99:0/1911163326 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482367122 0 0) 0x7f58500cfb40 con 0x7f58500d0e40 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584dffb640 1 -- 172.21.15.99:0/1911163326 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5838003520 con 0x7f58500d0e40 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584effd640 1 -- 172.21.15.99:0/1911163326 >> v2:172.21.15.162:3300/0 conn(0x7f58500cc690 msgr2=0x7f58500d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584effd640 1 --2- 172.21.15.99:0/1911163326 >> v2:172.21.15.162:3300/0 conn(0x7f58500cc690 0x7f58500d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584effd640 1 -- 172.21.15.99:0/1911163326 >> v1:172.21.15.162:6789/0 conn(0x7f58500d0e40 legacy=0x7f58500cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.659+0000 7f584effd640 1 -- 172.21.15.99:0/1911163326 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58500d0100 con 0x7f585015bbb0 2024-03-04T10:04:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.660+0000 7f584effd640 1 --2- 172.21.15.99:0/1911163326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58500d0900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5840009d90 tx=0x7f584002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5524216619b1320 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.660+0000 7f584dffb640 1 -- 172.21.15.99:0/1911163326 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840039070 con 0x7f585015bbb0 2024-03-04T10:04:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.660+0000 7f584dffb640 1 -- 172.21.15.99:0/1911163326 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584002cc50 con 0x7f585015bbb0 2024-03-04T10:04:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.660+0000 7f584dffb640 1 -- 172.21.15.99:0/1911163326 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840034440 con 0x7f585015bbb0 2024-03-04T10:04:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.660+0000 7f58556f1640 1 -- 172.21.15.99:0/1911163326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 msgr2=0x7f58500d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.660+0000 7f58556f1640 1 --2- 172.21.15.99:0/1911163326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58500d0900 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5840009d90 tx=0x7f584002bea0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.661+0000 7f58556f1640 1 -- 172.21.15.99:0/1911163326 shutdown_connections 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.661+0000 7f58556f1640 1 --2- 172.21.15.99:0/1911163326 >> v2:172.21.15.162:3300/0 conn(0x7f58500cc690 0x7f58500d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.661+0000 7f58556f1640 1 --2- 172.21.15.99:0/1911163326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58500d0900 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.661+0000 7f58556f1640 1 -- 172.21.15.99:0/1911163326 >> 172.21.15.99:0/1911163326 conn(0x7f58500b3480 msgr2=0x7f58500b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.661+0000 7f58556f1640 1 -- 172.21.15.99:0/1911163326 shutdown_connections 2024-03-04T10:04:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.661+0000 7f58556f1640 1 -- 172.21.15.99:0/1911163326 wait complete. 2024-03-04T10:04:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.662+0000 7f58556f1640 1 Processor -- start 2024-03-04T10:04:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.662+0000 7f58556f1640 1 -- start start 2024-03-04T10:04:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58500cc690 0x7f58501f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 0x7f58501f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58501518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585015bf90 con 0x7f58500d0e40 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58500d21e0 con 0x7f58500cc690 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5850154c50 con 0x7f585015bbb0 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 0x7f58501f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 0x7f58501f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38552/0 (socket says 172.21.15.99:38552) 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 -- 172.21.15.99:0/3050122720 learned_addr learned my addr 172.21.15.99:0/3050122720 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584effd640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58500cc690 0x7f58501f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584f7fe640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58501518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 -- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 msgr2=0x7f58501518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58501518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 -- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58500cc690 msgr2=0x7f58501f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58500cc690 0x7f58501f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 -- 172.21.15.99:0/3050122720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5850154970 con 0x7f58500d0e40 2024-03-04T10:04:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f584e7fc640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 0x7f58501f4110 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f584400c400 tx=0x7f584400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844017070 con 0x7f58500d0e40 2024-03-04T10:04:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5844005d00 con 0x7f58500d0e40 2024-03-04T10:04:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.663+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585015d420 con 0x7f58500d0e40 2024-03-04T10:04:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.664+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844007310 con 0x7f58500d0e40 2024-03-04T10:04:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.664+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585015d9e0 con 0x7f58500d0e40 2024-03-04T10:04:11.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.664+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585015dbe0 con 0x7f58500d0e40 2024-03-04T10:04:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.665+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f584401a070 con 0x7f58500d0e40 2024-03-04T10:04:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.665+0000 7f58377fe640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f583c06b4d0 0x7f583c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.665+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5844089890 con 0x7f58500d0e40 2024-03-04T10:04:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.666+0000 7f584effd640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f583c06b4d0 0x7f583c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:11.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.666+0000 7f584effd640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f583c06b4d0 0x7f583c06d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5840036000 tx=0x7f5840030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:11.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.670+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f585015dbe0 con 0x7f58500d0e40 2024-03-04T10:04:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.876+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5850074f60 con 0x7f58500d0e40 2024-03-04T10:04:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.877+0000 7f58377fe640 1 -- 172.21.15.99:0/3050122720 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f5844057e10 con 0x7f58500d0e40 2024-03-04T10:04:11.878 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:11.878 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f583c06b4d0 msgr2=0x7f583c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f583c06b4d0 0x7f583c06d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5840036000 tx=0x7f5840030000 comp rx=0 tx=0).stop 2024-03-04T10:04:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 msgr2=0x7f58501f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 0x7f58501f4110 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f584400c400 tx=0x7f584400c8c0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 shutdown_connections 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f583c06b4d0 0x7f583c06d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f585015bbb0 0x7f58501518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58500d0e40 0x7f58501f4110 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 --2- 172.21.15.99:0/3050122720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58500cc690 0x7f58501f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 >> 172.21.15.99:0/3050122720 conn(0x7f58500b3480 msgr2=0x7f58500cea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 shutdown_connections 2024-03-04T10:04:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:11.879+0000 7f58556f1640 1 -- 172.21.15.99:0/3050122720 wait complete. 2024-03-04T10:04:12.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:03:38.859750+0000', 'modified': '2024-03-04T10:04:00.946856+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6835/593641651', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 593641651}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 593641651}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:04:12.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:04:12.294 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T10:04:12.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.396+0000 7f5802d45640 1 Processor -- start 2024-03-04T10:04:12.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.396+0000 7f5802d45640 1 -- start start 2024-03-04T10:04:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f5802d45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d01d0 0x7f57fc06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f5802d45640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f57fc0d12a0 0x7f57fc162430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f5802d45640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f57fc06acd0 0x7f57fc06b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f5802d45640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc0d05b0 con 0x7f57fc0d12a0 2024-03-04T10:04:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f5802d45640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc06bbd0 con 0x7f57fc06acd0 2024-03-04T10:04:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f5802d45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc070180 con 0x7f57fc0d01d0 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f57fbfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f57fc0d12a0 0x7f57fc162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f57fbfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f57fc0d12a0 0x7f57fc162430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38568/0 (socket says 172.21.15.99:38568) 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.397+0000 7f57fbfff640 1 -- 172.21.15.99:0/111643685 learned_addr learned my addr 172.21.15.99:0/111643685 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f5800ae1640 1 --2- 172.21.15.99:0/111643685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d01d0 0x7f57fc06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f58012e2640 1 --2- 172.21.15.99:0/111643685 >> v2:172.21.15.162:3300/0 conn(0x7f57fc06acd0 0x7f57fc06b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f57fbfff640 1 -- 172.21.15.99:0/111643685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d01d0 msgr2=0x7f57fc06faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f57fbfff640 1 --2- 172.21.15.99:0/111643685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d01d0 0x7f57fc06faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f57fbfff640 1 -- 172.21.15.99:0/111643685 >> v2:172.21.15.162:3300/0 conn(0x7f57fc06acd0 msgr2=0x7f57fc06b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f57fbfff640 1 --2- 172.21.15.99:0/111643685 >> v2:172.21.15.162:3300/0 conn(0x7f57fc06acd0 0x7f57fc06b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f57fbfff640 1 -- 172.21.15.99:0/111643685 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc070460 con 0x7f57fc0d12a0 2024-03-04T10:04:12.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f58012e2640 1 --2- 172.21.15.99:0/111643685 >> v2:172.21.15.162:3300/0 conn(0x7f57fc06acd0 0x7f57fc06b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.398+0000 7f57fbfff640 1 --2- 172.21.15.99:0/111643685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc0d12a0 0x7f57fc162430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57fc073510 tx=0x7f57f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf120d5018244f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f57fb7fe640 1 -- 172.21.15.99:0/111643685 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0035020 con 0x7f57fc0d12a0 2024-03-04T10:04:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f57fb7fe640 1 -- 172.21.15.99:0/111643685 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f0031870 con 0x7f57fc0d12a0 2024-03-04T10:04:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f57fb7fe640 1 -- 172.21.15.99:0/111643685 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0031c40 con 0x7f57fc0d12a0 2024-03-04T10:04:12.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 -- 172.21.15.99:0/111643685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc0d12a0 msgr2=0x7f57fc162430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 --2- 172.21.15.99:0/111643685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc0d12a0 0x7f57fc162430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57fc073510 tx=0x7f57f002b910 comp rx=0 tx=0).stop 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 -- 172.21.15.99:0/111643685 shutdown_connections 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 --2- 172.21.15.99:0/111643685 >> v2:172.21.15.162:3300/0 conn(0x7f57fc06acd0 0x7f57fc06b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 --2- 172.21.15.99:0/111643685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc0d12a0 0x7f57fc162430 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 --2- 172.21.15.99:0/111643685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d01d0 0x7f57fc06faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 -- 172.21.15.99:0/111643685 >> 172.21.15.99:0/111643685 conn(0x7f57fc0d0780 msgr2=0x7f57fc0d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 -- 172.21.15.99:0/111643685 shutdown_connections 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.399+0000 7f5802d45640 1 -- 172.21.15.99:0/111643685 wait complete. 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.400+0000 7f5802d45640 1 Processor -- start 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.400+0000 7f5802d45640 1 -- start start 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5802d45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 0x7f57fc1f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5802d45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57fc0d01d0 0x7f57fc0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5802d45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d12a0 0x7f57fc0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5802d45640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc161f10 con 0x7f57fc06acd0 2024-03-04T10:04:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5802d45640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc161c30 con 0x7f57fc0d01d0 2024-03-04T10:04:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5802d45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0cadd0 con 0x7f57fc0d12a0 2024-03-04T10:04:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 0x7f57fc1f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 0x7f57fc1f3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38570/0 (socket says 172.21.15.99:38570) 2024-03-04T10:04:12.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.401+0000 7f5800ae1640 1 -- 172.21.15.99:0/137986898 learned_addr learned my addr 172.21.15.99:0/137986898 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f57fbfff640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57fc0d01d0 0x7f57fc0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f58012e2640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d12a0 0x7f57fc0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5800ae1640 1 -- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d12a0 msgr2=0x7f57fc0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5800ae1640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d12a0 0x7f57fc0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5800ae1640 1 -- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57fc0d01d0 msgr2=0x7f57fc0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5800ae1640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57fc0d01d0 0x7f57fc0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5800ae1640 1 -- 172.21.15.99:0/137986898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc1f4c40 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5800ae1640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 0x7f57fc1f3ca0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00e9b0 tx=0x7f57ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f57f97fa640 1 -- 172.21.15.99:0/137986898 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec017070 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f57f97fa640 1 -- 172.21.15.99:0/137986898 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec00c9a0 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f57f97fa640 1 -- 172.21.15.99:0/137986898 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec00cd90 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc1f4680 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.402+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc0c8980 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.403+0000 7f57f97fa640 1 -- 172.21.15.99:0/137986898 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f57ec01a070 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.404+0000 7f57f97fa640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57e806b4d0 0x7f57e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.404+0000 7f57f97fa640 1 -- 172.21.15.99:0/137986898 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57ec08a890 con 0x7f57fc06acd0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.404+0000 7f57fbfff640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57e806b4d0 0x7f57e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.405+0000 7f57fbfff640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57e806b4d0 0x7f57e806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f57f0036000 tx=0x7f57f003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.405+0000 7f5805a01740 1 Processor -- start 2024-03-04T10:04:12.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.405+0000 7f5805a01740 1 -- start start 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1f9f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5805a01740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 0x560a1fa24c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1f9f5380 0x560a1fa25170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5805a01740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1f95f7c0 con 0x560a1fa22280 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1f908b50 con 0x560a1f9f5380 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a1f95b860 con 0x560a1fa21ea0 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 0x560a1fa24c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 0x560a1fa24c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38574/0 (socket says 172.21.15.99:38574) 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 -- 172.21.15.99:0/4106733278 learned_addr learned my addr 172.21.15.99:0/4106733278 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5800ae1640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1f9f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f58012e2640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1f9f5380 0x560a1fa25170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 -- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 msgr2=0x560a1f9f4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1f9f4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 -- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1f9f5380 msgr2=0x560a1fa25170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1f9f5380 0x560a1fa25170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 -- 172.21.15.99:0/4106733278 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a1f91c9a0 con 0x560a1fa22280 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f5800ae1640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1f9f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f58012e2640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1f9f5380 0x560a1fa25170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57fbfff640 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 0x560a1fa24c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009930 tx=0x7f57f0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfd3c838d2d23a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57e27fc640 1 -- 172.21.15.99:0/4106733278 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0054070 con 0x560a1fa22280 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57e27fc640 1 -- 172.21.15.99:0/4106733278 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f003a960 con 0x560a1fa22280 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.406+0000 7f57e27fc640 1 -- 172.21.15.99:0/4106733278 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f003ad30 con 0x560a1fa22280 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 -- 172.21.15.99:0/4106733278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 msgr2=0x560a1fa24c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 0x560a1fa24c30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009930 tx=0x7f57f0030000 comp rx=0 tx=0).stop 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 -- 172.21.15.99:0/4106733278 shutdown_connections 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1f9f5380 0x560a1fa25170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa22280 0x560a1fa24c30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 --2- 172.21.15.99:0/4106733278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1f9f4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 -- 172.21.15.99:0/4106733278 >> 172.21.15.99:0/4106733278 conn(0x560a1f9f70b0 msgr2=0x560a1fa3fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 -- 172.21.15.99:0/4106733278 shutdown_connections 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 -- 172.21.15.99:0/4106733278 wait complete. 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.407+0000 7f5805a01740 1 -- start start 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 Processor -- start 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1fa514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fa3ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1f9f9750 con 0x560a1fa1b510 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1f91bf90 con 0x560a1fa22280 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a1f9fbf90 con 0x560a1fa21ea0 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.408+0000 7f5805a01740 10 client.0 _ll_register_callbacks cb 0x560a1f93dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38578/0 (socket says 172.21.15.99:38578) 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 -- 172.21.15.99:0/817324594 learned_addr learned my addr 172.21.15.99:0/817324594 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57fbfff640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fa3ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f5800ae1640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1fa514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 -- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 msgr2=0x560a1fa514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1fa514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 -- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 msgr2=0x560a1fa3ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fa3ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 -- 172.21.15.99:0/817324594 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a1f6742d0 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f58012e2640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57f4004340 tx=0x7f57f4004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8eba7136aa5634 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57db7fe640 1 -- 172.21.15.99:0/817324594 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f4019020 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57db7fe640 1 -- 172.21.15.99:0/817324594 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57f4007b20 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57f003a960 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57db7fe640 1 -- 172.21.15.99:0/817324594 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57f4023050 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f5805a01740 10 client.4818 fetch_fsmap learned FSMap version 8 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560a1f73e6b0 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57db7fe640 1 -- 172.21.15.99:0/817324594 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f003a960 con 0x560a1fa1b510 2024-03-04T10:04:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.409+0000 7f57db7fe640 1 -- 172.21.15.99:0/817324594 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 8) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f57f4026020 con 0x560a1fa1b510 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5805a01740 10 client.4818 fetch_fsmap finished waiting for FSMap version 8 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5805a01740 10 client.4818 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5805a01740 20 client.4818 populate_metadata read hostname 'smithi099' 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5805a01740 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa1d330 0x560a1fa53ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5805a01740 4 client.4818 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 --> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a1f96cd00 con 0x560a1fa1d330 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5800ae1640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa1d330 0x560a1fa53ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f57db7fe640 10 client.4818 ms_handle_connect on v2:172.21.15.99:6833/593641651 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f5800ae1640 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa1d330 0x560a1fa53ed0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40977eb0319ff7ba server_cookie=fdbf4bdf3d232386 in_seq=0 out_seq=0 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f57db7fe640 1 -- 172.21.15.99:0/817324594 <== mds.0 v2:172.21.15.99:6833/593641651 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560a1f96cd00 con 0x560a1fa1d330 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.410+0000 7f57db7fe640 10 client.4818 handle_command_reply: tid=0 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 client.4818 shutdown 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 msgr2=0x560a1fa244f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f57f4004340 tx=0x7f57f4004800 comp rx=0 tx=0).stop 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 shutdown_connections 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa1d330 0x560a1fa53ed0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fa3ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f57db7fe640 0 client.4818 ms_handle_reset on v2:172.21.15.99:6833/593641651 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 --2- 172.21.15.99:0/817324594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa21ea0 0x560a1fa514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 >> 172.21.15.99:0/817324594 conn(0x560a1f9f5380 msgr2=0x560a1f9f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 shutdown_connections 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 1 -- 172.21.15.99:0/817324594 wait complete. 2024-03-04T10:04:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.411+0000 7f5805a01740 20 client.4818 trim_cache size 0 max 16384 2024-03-04T10:04:12.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.426+0000 7f5805a01740 1 Processor -- start 2024-03-04T10:04:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 -- start start 2024-03-04T10:04:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa21ea0 0x560a1fa53360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa22280 0x560a1fa5a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc2006a0 con 0x560a1fa1b510 2024-03-04T10:04:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0ca2f0 con 0x560a1fa21ea0 2024-03-04T10:04:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc2000a0 con 0x560a1fa22280 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5800ae1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38586/0 (socket says 172.21.15.99:38586) 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.427+0000 7f5800ae1640 1 -- 172.21.15.99:0/3454915099 learned_addr learned my addr 172.21.15.99:0/3454915099 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f57fbfff640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa21ea0 0x560a1fa53360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5800ae1640 1 -- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa22280 msgr2=0x560a1fa5a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f58012e2640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa22280 0x560a1fa5a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5800ae1640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa22280 0x560a1fa5a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5800ae1640 1 -- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa21ea0 msgr2=0x560a1fa53360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5800ae1640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa21ea0 0x560a1fa53360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5800ae1640 1 -- 172.21.15.99:0/3454915099 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a1f9f5b90 con 0x560a1fa1b510 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f57fbfff640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa21ea0 0x560a1fa53360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f58012e2640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa22280 0x560a1fa5a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:12.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5800ae1640 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0551c0 tx=0x7f57ec08a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368f284686fe7324 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f57e1ffb640 1 -- 172.21.15.99:0/3454915099 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec021040 con 0x560a1fa1b510 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f57e1ffb640 1 -- 172.21.15.99:0/3454915099 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec087070 con 0x560a1fa1b510 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f57e1ffb640 1 -- 172.21.15.99:0/3454915099 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec05ceb0 con 0x560a1fa1b510 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 -- 172.21.15.99:0/3454915099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 msgr2=0x560a1fa244d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0551c0 tx=0x7f57ec08a040 comp rx=0 tx=0).stop 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 -- 172.21.15.99:0/3454915099 shutdown_connections 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa22280 0x560a1fa5a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa21ea0 0x560a1fa53360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 --2- 172.21.15.99:0/3454915099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa1b510 0x560a1fa244d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.428+0000 7f5805a01740 1 -- 172.21.15.99:0/3454915099 >> 172.21.15.99:0/3454915099 conn(0x560a1f9f5380 msgr2=0x560a1fa3fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.429+0000 7f5805a01740 1 -- 172.21.15.99:0/3454915099 shutdown_connections 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.429+0000 7f5805a01740 1 -- 172.21.15.99:0/3454915099 wait complete. 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.429+0000 7f5805a01740 1 -- start start 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.429+0000 7f5805a01740 1 Processor -- start 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa1b510 0x560a1fa1c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 0x560a1fab2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fab2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1fa512b0 con 0x560a1fa21ea0 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1fa24090 con 0x560a1fa22280 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a1fa1d730 con 0x560a1fa1b510 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5805a01740 10 client.0 _ll_register_callbacks cb 0x7f57ec009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:04:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 0x560a1fab2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 0x560a1fab2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38590/0 (socket says 172.21.15.99:38590) 2024-03-04T10:04:12.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 -- 172.21.15.99:0/5901687 learned_addr learned my addr 172.21.15.99:0/5901687 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:12.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f58012e2640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fab2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f5800ae1640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa1b510 0x560a1fa1c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 -- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa1b510 msgr2=0x560a1fa1c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa1b510 0x560a1fa1c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 -- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 msgr2=0x560a1fab2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fab2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.430+0000 7f57fbfff640 1 -- 172.21.15.99:0/5901687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a1fa22ef0 con 0x560a1fa21ea0 2024-03-04T10:04:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57fbfff640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 0x560a1fab2a50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f57f0000c00 tx=0x7f57f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513323d824213e53 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:12.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57db7fe640 1 -- 172.21.15.99:0/5901687 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f005d060 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57db7fe640 1 -- 172.21.15.99:0/5901687 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f57f00317b0 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57ec087070 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57db7fe640 1 -- 172.21.15.99:0/5901687 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57f0063050 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 10 client.4824 fetch_fsmap learned FSMap version 8 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560a1f9faad0 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57db7fe640 1 -- 172.21.15.99:0/5901687 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec087070 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f57db7fe640 1 -- 172.21.15.99:0/5901687 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 8) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f57f0066020 con 0x560a1fa21ea0 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 10 client.4824 fetch_fsmap finished waiting for FSMap version 8 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 10 client.4824 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 20 client.4824 populate_metadata read hostname 'smithi099' 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa4f9e0 0x560a1fa5cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 4 client.4824 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:04:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 --> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560a1f967050 con 0x560a1fa4f9e0 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.431+0000 7f5800ae1640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa4f9e0 0x560a1fa5cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f57db7fe640 10 client.4824 ms_handle_connect on v2:172.21.15.99:6833/593641651 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5800ae1640 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa4f9e0 0x560a1fa5cda0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc73c9daf9c6dcec server_cookie=be93c82a36286c76 in_seq=0 out_seq=0 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f57db7fe640 1 -- 172.21.15.99:0/5901687 <== mds.0 v2:172.21.15.99:6833/593641651 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x560a1f967050 con 0x560a1fa4f9e0 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f57db7fe640 10 client.4824 handle_command_reply: tid=0 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 client.4824 shutdown 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 msgr2=0x560a1fab2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 0x560a1fab2a50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f57f0000c00 tx=0x7f57f0004730 comp rx=0 tx=0).stop 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 shutdown_connections 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:6833/593641651,v1:172.21.15.99:6835/593641651] conn(0x560a1fa4f9e0 0x560a1fa5cda0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560a1fa22280 0x560a1fab2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560a1fa21ea0 0x560a1fab2a50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f57db7fe640 0 client.4824 ms_handle_reset on v2:172.21.15.99:6833/593641651 2024-03-04T10:04:12.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 --2- 172.21.15.99:0/5901687 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560a1fa1b510 0x560a1fa1c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 >> 172.21.15.99:0/5901687 conn(0x560a1f9f5380 msgr2=0x560a1fa5a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 shutdown_connections 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 1 -- 172.21.15.99:0/5901687 wait complete. 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.432+0000 7f5805a01740 20 client.4824 trim_cache size 0 max 16384 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 4275, 2024-03-04T10:04:12.451 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 32.893180381000001, 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 8, 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 24, 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 24, 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 39.398153379999997 2024-03-04T10:04:12.452 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57e806b4d0 msgr2=0x7f57e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57e806b4d0 0x7f57e806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f57f0036000 tx=0x7f57f003c040 comp rx=0 tx=0).stop 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 msgr2=0x7f57fc1f3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 0x7f57fc1f3ca0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00e9b0 tx=0x7f57ec00ee70 comp rx=0 tx=0).stop 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 shutdown_connections 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57e806b4d0 0x7f57e806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57fc0d12a0 0x7f57fc0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f57fc0d01d0 0x7f57fc0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 --2- 172.21.15.99:0/137986898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f57fc06acd0 0x7f57fc1f3ca0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 >> 172.21.15.99:0/137986898 conn(0x7f57fc0d0780 msgr2=0x7f57fc156540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 shutdown_connections 2024-03-04T10:04:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:12.433+0000 7f5802d45640 1 -- 172.21.15.99:0/137986898 wait complete. 2024-03-04T10:04:12.455 INFO:teuthology.run_tasks:Running task kclient... 2024-03-04T10:04:12.479 INFO:tasks.kclient:Mounting kernel clients... 2024-03-04T10:04:12.479 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-04T10:04:12.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.479 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:04:12.525 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-04T10:04:12.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.526 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T10:04:12.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.597 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link delete ceph-brx 2024-03-04T10:04:12.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.697 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:04:12.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.712 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link delete ceph-brx 2024-03-04T10:04:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:Cannot find device "ceph-brx" 2024-03-04T10:04:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:04:12.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.787 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list 2024-03-04T10:04:12.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link delete ceph-brx 2024-03-04T10:04:12.882 INFO:teuthology.orchestra.run.smithi162.stderr:Cannot find device "ceph-brx" 2024-03-04T10:04:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:04:12.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.884 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list 2024-03-04T10:04:12.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:12.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link delete ceph-brx 2024-03-04T10:04:13.014 INFO:teuthology.orchestra.run.smithi162.stderr:Cannot find device "ceph-brx" 2024-03-04T10:04:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:04:13.016 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-03-04T10:04:13.016 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T10:04:13.016 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:13.017 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:04:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.118+0000 7f3b52bd4640 1 Processor -- start 2024-03-04T10:04:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.118+0000 7f3b52bd4640 1 -- start start 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.118+0000 7f3b52bd4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3b4c15b030 0x7f3b4c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.118+0000 7f3b52bd4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3b4c06d3b0 0x7f3b4c15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b52bd4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c15e2d0 con 0x7f3b4c15b410 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b52bd4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c0b2720 con 0x7f3b4c06d3b0 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b52bd4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c06d8b0 con 0x7f3b4c15b030 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3b4c06d3b0 0x7f3b4c15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3b4c06d3b0 0x7f3b4c15dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38596/0 (socket says 172.21.15.99:38596) 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 -- 172.21.15.99:0/2946720195 learned_addr learned my addr 172.21.15.99:0/2946720195 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 -- 172.21.15.99:0/2946720195 >> v2:172.21.15.162:3300/0 conn(0x7f3b4c15b030 msgr2=0x7f3b4c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b50970640 1 --2- 172.21.15.99:0/2946720195 >> v2:172.21.15.162:3300/0 conn(0x7f3b4c15b030 0x7f3b4c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 --2- 172.21.15.99:0/2946720195 >> v2:172.21.15.162:3300/0 conn(0x7f3b4c15b030 0x7f3b4c15c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 -- 172.21.15.99:0/2946720195 >> v1:172.21.15.162:6789/0 conn(0x7f3b4c15b410 legacy=0x7f3b4c15b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.119+0000 7f3b4bfff640 1 -- 172.21.15.99:0/2946720195 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c06db90 con 0x7f3b4c06d3b0 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b4bfff640 1 --2- 172.21.15.99:0/2946720195 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c15dd90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0725c0 tx=0x7f3b3802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4643aa52efb903aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b4b7fe640 1 -- 172.21.15.99:0/2946720195 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38038070 con 0x7f3b4c06d3b0 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b4b7fe640 1 -- 172.21.15.99:0/2946720195 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3802f050 con 0x7f3b4c06d3b0 2024-03-04T10:04:13.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b4b7fe640 1 -- 172.21.15.99:0/2946720195 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38031970 con 0x7f3b4c06d3b0 2024-03-04T10:04:13.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2946720195 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c06d3b0 msgr2=0x7f3b4c15dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:13.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2946720195 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c15dd90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0725c0 tx=0x7f3b3802b7f0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2946720195 shutdown_connections 2024-03-04T10:04:13.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2946720195 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c15dd90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2946720195 >> v2:172.21.15.162:3300/0 conn(0x7f3b4c15b030 0x7f3b4c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2946720195 >> 172.21.15.99:0/2946720195 conn(0x7f3b4c06acd0 msgr2=0x7f3b4c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2946720195 shutdown_connections 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.120+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2946720195 wait complete. 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 Processor -- start 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 -- start start 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c14d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b4c15b030 0x7f3b4c0acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 0x7f3b4c158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c06e690 con 0x7f3b4c15b410 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c06e150 con 0x7f3b4c06d3b0 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.122+0000 7f3b52bd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c06e810 con 0x7f3b4c15b030 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 0x7f3b4c158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 0x7f3b4c158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38600/0 (socket says 172.21.15.99:38600) 2024-03-04T10:04:13.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 -- 172.21.15.99:0/2560923188 learned_addr learned my addr 172.21.15.99:0/2560923188 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b50970640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c14d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b4bfff640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b4c15b030 0x7f3b4c0acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 -- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b4c15b030 msgr2=0x7f3b4c0acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b4c15b030 0x7f3b4c0acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 -- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b4c06d3b0 msgr2=0x7f3b4c14d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c14d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 -- 172.21.15.99:0/2560923188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c14d750 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b50970640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c14d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b51171640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 0x7f3b4c158060 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400c780 tx=0x7f3b4400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44004b60 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b44004d00 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c0aac30 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400aa90 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.123+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c0ab1f0 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.124+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4c074f60 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.125+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3b4400ac30 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.125+0000 7f3b497fa640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b3006b4d0 0x7f3b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.125+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3b4408a890 con 0x7f3b4c15b410 2024-03-04T10:04:13.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.125+0000 7f3b50970640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b3006b4d0 0x7f3b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:13.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.126+0000 7f3b50970640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b3006b4d0 0x7f3b3006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b40006fd0 tx=0x7f3b40008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:13.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.129+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3b44020070 con 0x7f3b4c15b410 2024-03-04T10:04:13.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.336+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3b4c06c310 con 0x7f3b4c15b410 2024-03-04T10:04:13.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.336+0000 7f3b497fa640 1 -- 172.21.15.99:0/2560923188 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3b440593d0 con 0x7f3b4c15b410 2024-03-04T10:04:13.337 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b3006b4d0 msgr2=0x7f3b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b3006b4d0 0x7f3b3006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3b40006fd0 tx=0x7f3b40008040 comp rx=0 tx=0).stop 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 msgr2=0x7f3b4c158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 0x7f3b4c158060 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400c780 tx=0x7f3b4400cc40 comp rx=0 tx=0).stop 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 shutdown_connections 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3b3006b4d0 0x7f3b3006d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3b4c15b410 0x7f3b4c158060 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3b4c15b030 0x7f3b4c0acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 --2- 172.21.15.99:0/2560923188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b4c06d3b0 0x7f3b4c14d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 >> 172.21.15.99:0/2560923188 conn(0x7f3b4c06acd0 msgr2=0x7f3b4c0b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 shutdown_connections 2024-03-04T10:04:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:13.338+0000 7f3b52bd4640 1 -- 172.21.15.99:0/2560923188 wait complete. 2024-03-04T10:04:13.349 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:04:13.350 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:04:13.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.350 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:13.365 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42732sec preferred_lft 42732sec 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:04:13.366 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:13.367 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T10:04:13.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link add name ceph-brx type bridge 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip addr flush dev ceph-brx 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set ceph-brx up 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T10:04:13.367 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:13.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.474 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:04:13.500 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:04:13.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.502 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:04:13.558 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:04:13.558 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:04:13.558 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:04:13.558 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:04:13.558 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:04:13.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.559 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:13.559 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:13.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:04:13.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:04:13.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:04:13.559 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:13.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.656 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:04:13.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.671 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list-id 2024-03-04T10:04:13.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.726 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:13.726 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:13.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:04:13.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-04T10:04:13.727 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:13.809 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-04T10:04:13.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-04T10:04:13.810 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:13.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:13.937 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:13.937 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:13.937 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:04:13.937 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T10:04:13.937 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:13.978 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:13.979 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:14.037 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:04:14.037 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:14.037 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:14.092 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:04:14.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:14.093 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:14.177 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:14.177 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:14.273 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:14.273 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:14.273 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:04:14.274 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:04:14.276 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:04:14.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:14.276 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:14.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:14.343 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:14.399 INFO:teuthology.orchestra.run.smithi099.stdout:client4610 2024-03-04T10:04:14.399 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:14.399 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:14.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:14.454 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:14.511 INFO:teuthology.orchestra.run.smithi099.stdout:client4610 2024-03-04T10:04:14.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:14.511 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:14.566 INFO:teuthology.orchestra.run.smithi099.stdout:client4610 2024-03-04T10:04:14.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:14.567 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.client4610/status') 2024-03-04T10:04:14.633 INFO:teuthology.orchestra.run.smithi099.stdout:instance: client.4610 (3)172.21.15.99:0/3520821074 2024-03-04T10:04:14.633 INFO:teuthology.orchestra.run.smithi099.stdout:blocklisted: false 2024-03-04T10:04:14.633 INFO:teuthology.orchestra.run.smithi099.stderr:0+1 records in 2024-03-04T10:04:14.634 INFO:teuthology.orchestra.run.smithi099.stderr:0+1 records out 2024-03-04T10:04:14.634 INFO:teuthology.orchestra.run.smithi099.stderr:70 bytes copied, 2.0567e-05 s, 3.4 MB/s 2024-03-04T10:04:14.635 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-03-04T10:04:14.635 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T10:04:14.635 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-04T10:04:14.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:04:14.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.774+0000 7f31b3dc4640 1 Processor -- start 2024-03-04T10:04:14.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.774+0000 7f31b3dc4640 1 -- start start 2024-03-04T10:04:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b3dc4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f31ac06d3b0 0x7f31ac15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b3dc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31ac15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b3dc4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ac15be00 con 0x7f31ac15b030 2024-03-04T10:04:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b3dc4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac0b2720 con 0x7f31ac06d3b0 2024-03-04T10:04:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b3dc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06d8b0 con 0x7f31ac15b410 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b1b60640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f31ac15b030 0x7f31ac15dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56270/0 (socket says 172.21.15.99:56270) 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b1b60640 1 -- 172.21.15.99:0/1043223164 learned_addr learned my addr 172.21.15.99:0/1043223164 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514271647 0 0) 0x7f31ac15be00 con 0x7f31ac15b030 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b135f640 1 --2- 172.21.15.99:0/1043223164 >> v2:172.21.15.162:3300/0 conn(0x7f31ac06d3b0 0x7f31ac15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3198003520 con 0x7f31ac15b030 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.775+0000 7f31b2361640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31ac15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051401561 0 0) 0x7f3198003520 con 0x7f31ac15b030 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 msgr2=0x7f31ac15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31ac15b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 >> v2:172.21.15.162:3300/0 conn(0x7f31ac06d3b0 msgr2=0x7f31ac15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:14.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 --2- 172.21.15.99:0/1043223164 >> v2:172.21.15.162:3300/0 conn(0x7f31ac06d3b0 0x7f31ac15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac06db90 con 0x7f31ac15b030 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f31a80019f0 con 0x7f31ac15b030 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 >> v1:172.21.15.99:6789/0 conn(0x7f31ac15b030 legacy=0x7f31ac15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f3198003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3198004800 0x7f3198004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0b2720 con 0x7f31980039b0 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06d8b0 con 0x7f3198004800 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319800b9f0 con 0x7f31ac15b410 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b1b60640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f3198003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b135f640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3198004800 0x7f3198004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b2361640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b1b60640 1 -- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 msgr2=0x7f31980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b1b60640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b1b60640 1 -- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3198004800 msgr2=0x7f3198004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b1b60640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3198004800 0x7f3198004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b1b60640 1 -- 172.21.15.99:0/1043223164 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3198003520 con 0x7f31980039b0 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.776+0000 7f31b2361640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15b410 0x7f31980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b1b60640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f3198003df0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31a800a870 tx=0x7f3198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f06cc05b25ecfaf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a800a4c0 con 0x7f31980039b0 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a8034630 con 0x7f31980039b0 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b0b5e640 1 -- 172.21.15.99:0/1043223164 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8034a20 con 0x7f31980039b0 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b3dc4640 1 -- 172.21.15.99:0/1043223164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 msgr2=0x7f3198003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:14.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/1043223164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f3198003df0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f31a800a870 tx=0x7f3198005830 comp rx=0 tx=0).stop 2024-03-04T10:04:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b2361640 1 -- 172.21.15.99:0/1043223164 reap_dead start 2024-03-04T10:04:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b3dc4640 1 -- 172.21.15.99:0/1043223164 shutdown_connections 2024-03-04T10:04:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.777+0000 7f31b3dc4640 1 -- 172.21.15.99:0/1043223164 >> 172.21.15.99:0/1043223164 conn(0x7f31ac06acd0 msgr2=0x7f31ac06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.778+0000 7f31b3dc4640 1 -- 172.21.15.99:0/1043223164 shutdown_connections 2024-03-04T10:04:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.778+0000 7f31b3dc4640 1 -- 172.21.15.99:0/1043223164 wait complete. 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 Processor -- start 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 -- start start 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f31ac166850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15a4e0 0x7f31ac151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31ac0cee20 0x7f31ac0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06e690 con 0x7f31980039b0 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06e150 con 0x7f31ac0cee20 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.779+0000 7f31b3dc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac06e810 con 0x7f31ac15a4e0 2024-03-04T10:04:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f31ac166850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f31ac166850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52152/0 (socket says 172.21.15.99:52152) 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 -- 172.21.15.99:0/3101677945 learned_addr learned my addr 172.21.15.99:0/3101677945 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b2361640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31ac0cee20 0x7f31ac0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b135f640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15a4e0 0x7f31ac151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 -- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15a4e0 msgr2=0x7f31ac151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15a4e0 0x7f31ac151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 -- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31ac0cee20 msgr2=0x7f31ac0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31ac0cee20 0x7f31ac0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 -- 172.21.15.99:0/3101677945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac151870 con 0x7f31980039b0 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b135f640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15a4e0 0x7f31ac151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b1b60640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f31ac166850 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f31a80012e0 tx=0x7f31ac0cd490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8043020 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a803a040 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac167ad0 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8002030 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.780+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac167230 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.781+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac0cf7e0 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.782+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f31a8046020 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.782+0000 7f31a27fc640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f319006b500 0x7f319006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.782+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f31a80ac7a0 con 0x7f31980039b0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.782+0000 7f31b135f640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f319006b500 0x7f319006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.783+0000 7f31b135f640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f319006b500 0x7f319006d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31ac1659b0 tx=0x7f319c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:14.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.786+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f31a804d030 con 0x7f31980039b0 2024-03-04T10:04:14.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.993+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f31ac074f60 con 0x7f31980039b0 2024-03-04T10:04:14.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.993+0000 7f31a27fc640 1 -- 172.21.15.99:0/3101677945 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f31a800a560 con 0x7f31980039b0 2024-03-04T10:04:14.995 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f319006b500 msgr2=0x7f319006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f319006b500 0x7f319006d9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f31ac1659b0 tx=0x7f319c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 msgr2=0x7f31ac166850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f31ac166850 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f31a80012e0 tx=0x7f31ac0cd490 comp rx=0 tx=0).stop 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 shutdown_connections 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f319006b500 0x7f319006d9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31ac0cee20 0x7f31ac0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31ac15a4e0 0x7f31ac151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 --2- 172.21.15.99:0/3101677945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31980039b0 0x7f31ac166850 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:14.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 >> 172.21.15.99:0/3101677945 conn(0x7f31ac06acd0 msgr2=0x7f31ac0b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:14.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 shutdown_connections 2024-03-04T10:04:14.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:14.995+0000 7f31b3dc4640 1 -- 172.21.15.99:0/3101677945 wait complete. 2024-03-04T10:04:15.007 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T10:04:15.007 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:04:15.007 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:04:15.007 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-04T10:04:15.008 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-04T10:04:15.008 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T10:04:15.008 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T10:04:15.008 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:04:15.008 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-04T10:04:15.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.008 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:04:15.023 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:04:15.024 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42730sec preferred_lft 42730sec 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:15.025 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link tentative 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link tentative 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.026 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:04:15.089 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:04:15.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.091 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:04:15.146 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:04:15.146 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:04:15.147 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:04:15.147 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:04:15.147 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:04:15.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.147 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:15.147 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:15.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:04:15.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:04:15.147 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:04:15.147 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:15.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.242 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:04:15.257 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:04:15.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.258 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list-id 2024-03-04T10:04:15.313 INFO:teuthology.orchestra.run.smithi099.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-04T10:04:15.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.313 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:15.314 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:15.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:04:15.314 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-04T10:04:15.314 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:15.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.395 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 42:8c:15:b1:7c:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T10:04:15.462 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.463 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::408c:15ff:feb1:7c51/64 scope link 2024-03-04T10:04:15.463 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.464 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:15.531 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 42:8c:15:b1:7c:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T10:04:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T10:04:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::408c:15ff:feb1:7c51/64 scope link 2024-03-04T10:04:15.532 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:15.533 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-04T10:04:15.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.533 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-04T10:04:15.534 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:15.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.662 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:15.662 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:15.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.1 up 2024-03-04T10:04:15.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set dev brx.1 master ceph-brx 2024-03-04T10:04:15.662 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:15.703 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:15.703 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-04T10:04:15.762 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-04T10:04:15.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:15.762 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-04T10:04:15.816 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-03-04T10:04:15.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.817 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:15.901 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:15.901 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:15.907 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 2@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:04:15.908 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:04:15.909 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:04:15.910 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:04:15.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.910 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-04T10:04:15.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:15.936 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:15.993 INFO:teuthology.orchestra.run.smithi099.stdout:client4619 2024-03-04T10:04:15.993 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:15.993 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:16.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.048 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:16.105 INFO:teuthology.orchestra.run.smithi099.stdout:client4619 2024-03-04T10:04:16.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.105 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:16.160 INFO:teuthology.orchestra.run.smithi099.stdout:client4619 2024-03-04T10:04:16.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.161 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.client4619/status') 2024-03-04T10:04:16.227 INFO:teuthology.orchestra.run.smithi099.stdout:instance: client.4619 (3)172.21.15.99:0/2520136844 2024-03-04T10:04:16.227 INFO:teuthology.orchestra.run.smithi099.stdout:blocklisted: false 2024-03-04T10:04:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:0+1 records in 2024-03-04T10:04:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:0+1 records out 2024-03-04T10:04:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:70 bytes copied, 2.1373e-05 s, 3.3 MB/s 2024-03-04T10:04:16.229 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-03-04T10:04:16.229 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T10:04:16.229 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-04T10:04:16.229 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:04:16.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.367+0000 7fab2471b640 1 Processor -- start 2024-03-04T10:04:16.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.368+0000 7fab2471b640 1 -- start start 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.368+0000 7fab2471b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fab1c0b2e60 0x7fab1c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.368+0000 7fab2471b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1c15e1e0 con 0x7fab1c0b5470 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.368+0000 7fab2471b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1c0d1c10 con 0x7fab1c15de00 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.368+0000 7fab2471b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1c0b98c0 con 0x7fab1c0b2e60 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab22cb8640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fab1c0b5470 0x7fab1c0d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56282/0 (socket says 172.21.15.99:56282) 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab22cb8640 1 -- 172.21.15.99:0/4271814612 learned_addr learned my addr 172.21.15.99:0/4271814612 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab21cb6640 1 --2- 172.21.15.99:0/4271814612 >> v2:172.21.15.162:3300/0 conn(0x7fab1c0b2e60 0x7fab1c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394519151 0 0) 0x7fab1c15e1e0 con 0x7fab1c0b5470 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab08003520 con 0x7fab1c0b5470 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877909094 0 0) 0x7fab1c0d1c10 con 0x7fab1c15de00 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab1c15e1e0 con 0x7fab1c15de00 2024-03-04T10:04:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1932175128 0 0) 0x7fab08003520 con 0x7fab1c0b5470 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 >> v2:172.21.15.162:3300/0 conn(0x7fab1c0b2e60 msgr2=0x7fab1c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 --2- 172.21.15.99:0/4271814612 >> v2:172.21.15.162:3300/0 conn(0x7fab1c0b2e60 0x7fab1c0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 >> v1:172.21.15.162:6789/0 conn(0x7fab1c15de00 legacy=0x7fab1c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c0d1ef0 con 0x7fab1c0b5470 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fab140019f0 con 0x7fab1c0b5470 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 >> v1:172.21.15.99:6789/0 conn(0x7fab1c0b5470 legacy=0x7fab1c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab08003b40 0x7fab08003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab08004800 0x7fab08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab080051c0 0x7fab0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0b98c0 con 0x7fab080051c0 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0800c040 con 0x7fab08003b40 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0800c240 con 0x7fab08004800 2024-03-04T10:04:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab22cb8640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab080051c0 0x7fab0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab224b7640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab08003b40 0x7fab08003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.369+0000 7fab21cb6640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab08004800 0x7fab08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 -- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab08004800 msgr2=0x7fab08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab08004800 0x7fab08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 -- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab08003b40 msgr2=0x7fab08003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab08003b40 0x7fab08003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 -- 172.21.15.99:0/4271814612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab08003520 con 0x7fab080051c0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 -- 172.21.15.99:0/4271814612 reap_dead start 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab22cb8640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab080051c0 0x7fab0800b9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fab1400a7c0 tx=0x7fab08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14acb6605b407041 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.370+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1400a410 con 0x7fab080051c0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab14034650 con 0x7fab080051c0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab214b5640 1 -- 172.21.15.99:0/4271814612 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab14034a40 con 0x7fab080051c0 2024-03-04T10:04:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 -- 172.21.15.99:0/4271814612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab080051c0 msgr2=0x7fab0800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab080051c0 0x7fab0800b9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fab1400a7c0 tx=0x7fab08005830 comp rx=0 tx=0).stop 2024-03-04T10:04:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 -- 172.21.15.99:0/4271814612 shutdown_connections 2024-03-04T10:04:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 --2- 172.21.15.99:0/4271814612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab080051c0 0x7fab0800b9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 -- 172.21.15.99:0/4271814612 >> 172.21.15.99:0/4271814612 conn(0x7fab1c0d0470 msgr2=0x7fab1c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 -- 172.21.15.99:0/4271814612 shutdown_connections 2024-03-04T10:04:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.371+0000 7fab2471b640 1 -- 172.21.15.99:0/4271814612 wait complete. 2024-03-04T10:04:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.372+0000 7fab2471b640 1 Processor -- start 2024-03-04T10:04:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.372+0000 7fab2471b640 1 -- start start 2024-03-04T10:04:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab2471b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab080051c0 0x7fab1c0c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab2471b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 0x7fab1c0b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab2471b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1c0cc6b0 0x7fab1c0b7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab2471b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0d2820 con 0x7fab1c0cc250 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab2471b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0d24b0 con 0x7fab1c0cc6b0 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab2471b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c0b9380 con 0x7fab080051c0 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 0x7fab1c0b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 0x7fab1c0b77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52174/0 (socket says 172.21.15.99:52174) 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 -- 172.21.15.99:0/2652764151 learned_addr learned my addr 172.21.15.99:0/2652764151 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab224b7640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab080051c0 0x7fab1c0c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab22cb8640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1c0cc6b0 0x7fab1c0b7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 -- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab080051c0 msgr2=0x7fab1c0c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab080051c0 0x7fab1c0c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 -- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1c0cc6b0 msgr2=0x7fab1c0b7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1c0cc6b0 0x7fab1c0b7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.373+0000 7fab21cb6640 1 -- 172.21.15.99:0/2652764151 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c0ccc80 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.374+0000 7fab21cb6640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 0x7fab1c0b77d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fab180118e0 tx=0x7fab18011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.374+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1800fa30 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.374+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1800fbd0 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.374+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c0cac60 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.374+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab18014440 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.374+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1c0cb220 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.375+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1c074f60 con 0x7fab1c0cc250 2024-03-04T10:04:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.375+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fab18012040 con 0x7fab1c0cc250 2024-03-04T10:04:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.376+0000 7fab12ffd640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fab0006b4d0 0x7fab0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.376+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fab1808ab00 con 0x7fab1c0cc250 2024-03-04T10:04:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.376+0000 7fab224b7640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fab0006b4d0 0x7fab0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.376+0000 7fab224b7640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fab0006b4d0 0x7fab0006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab0c000c00 tx=0x7fab0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.380+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fab18019030 con 0x7fab1c0cc250 2024-03-04T10:04:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.589+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fab1c06cf30 con 0x7fab1c0cc250 2024-03-04T10:04:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.589+0000 7fab12ffd640 1 -- 172.21.15.99:0/2652764151 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fab180595b0 con 0x7fab1c0cc250 2024-03-04T10:04:16.591 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T10:04:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fab0006b4d0 msgr2=0x7fab0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fab0006b4d0 0x7fab0006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab0c000c00 tx=0x7fab0c009040 comp rx=0 tx=0).stop 2024-03-04T10:04:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 msgr2=0x7fab1c0b77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 0x7fab1c0b77d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fab180118e0 tx=0x7fab18011ce0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 shutdown_connections 2024-03-04T10:04:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fab0006b4d0 0x7fab0006d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1c0cc6b0 0x7fab1c0b7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab1c0cc250 0x7fab1c0b77d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 --2- 172.21.15.99:0/2652764151 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab080051c0 0x7fab1c0c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 >> 172.21.15.99:0/2652764151 conn(0x7fab1c0d0470 msgr2=0x7fab1c0b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 shutdown_connections 2024-03-04T10:04:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:16.591+0000 7fab2471b640 1 -- 172.21.15.99:0/2652764151 wait complete. 2024-03-04T10:04:16.603 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi162.front.sepia.ceph.com 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:04:16.604 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-04T10:04:16.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.604 DEBUG:teuthology.orchestra.run.smithi162:> ip addr 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:16.624 INFO:teuthology.orchestra.run.smithi162.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:16.625 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:d9:37:f8 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:16.625 INFO:teuthology.orchestra.run.smithi162.stdout: altname enp7s0f0 2024-03-04T10:04:16.625 INFO:teuthology.orchestra.run.smithi162.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:16.625 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:d9:37:f9 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:16.625 INFO:teuthology.orchestra.run.smithi162.stdout: altname enp7s0f1 2024-03-04T10:04:16.625 INFO:teuthology.orchestra.run.smithi162.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:88:7f:40 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:88:7f:41 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout: inet 172.21.15.162/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft 42731sec preferred_lft 42731sec 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 fe80::ec4:7aff:fe88:7f41/64 scope link 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:16.626 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T10:04:16.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.627 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:16.627 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:16.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link add name ceph-brx type bridge 2024-03-04T10:04:16.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip addr flush dev ceph-brx 2024-03-04T10:04:16.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link set ceph-brx up 2024-03-04T10:04:16.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T10:04:16.628 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:16.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.829 DEBUG:teuthology.orchestra.run.smithi162:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:04:16.857 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-03-04T10:04:16.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.858 DEBUG:teuthology.orchestra.run.smithi162:> route 2024-03-04T10:04:16.953 INFO:teuthology.orchestra.run.smithi162.stdout:Kernel IP routing table 2024-03-04T10:04:16.953 INFO:teuthology.orchestra.run.smithi162.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:04:16.953 INFO:teuthology.orchestra.run.smithi162.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:04:16.953 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:04:16.953 INFO:teuthology.orchestra.run.smithi162.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:04:16.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:16.954 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:16.954 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:16.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:04:16.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:04:16.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:04:16.954 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:17.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.164 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list 2024-03-04T10:04:17.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.179 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list-id 2024-03-04T10:04:17.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.234 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:17.234 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:17.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T10:04:17.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-04T10:04:17.234 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:17.317 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-04T10:04:17.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-04T10:04:17.318 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:17.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.457 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:17.457 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:17.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link set brx.0 up 2024-03-04T10:04:17.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T10:04:17.457 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:17.499 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:17.499 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-04T10:04:17.551 INFO:teuthology.orchestra.run.smithi162.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-04T10:04:17.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:17.552 DEBUG:teuthology.orchestra.run.smithi162:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-04T10:04:17.607 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-03-04T10:04:17.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:17.834 INFO:teuthology.orchestra.run.smithi162.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:17.834 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:17.951 INFO:teuthology.orchestra.run.smithi162.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:04:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: trying mount with new device syntax: 1@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:04:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:04:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-04T10:04:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:04:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:04:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:04:17.953 INFO:teuthology.orchestra.run.smithi162.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:04:17.954 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:04:17.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-04T10:04:17.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:17.981 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:18.037 INFO:teuthology.orchestra.run.smithi162.stdout:client4845 2024-03-04T10:04:18.038 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:18.038 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:18.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.095 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:18.151 INFO:teuthology.orchestra.run.smithi162.stdout:client4845 2024-03-04T10:04:18.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.152 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:18.207 INFO:teuthology.orchestra.run.smithi162.stdout:client4845 2024-03-04T10:04:18.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.208 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.client4845/status') 2024-03-04T10:04:18.274 INFO:teuthology.orchestra.run.smithi162.stdout:instance: client.4845 (3)172.21.15.162:0/3220944920 2024-03-04T10:04:18.274 INFO:teuthology.orchestra.run.smithi162.stdout:blocklisted: false 2024-03-04T10:04:18.274 INFO:teuthology.orchestra.run.smithi162.stderr:0+1 records in 2024-03-04T10:04:18.274 INFO:teuthology.orchestra.run.smithi162.stderr:0+1 records out 2024-03-04T10:04:18.274 INFO:teuthology.orchestra.run.smithi162.stderr:71 bytes copied, 2.5083e-05 s, 2.8 MB/s 2024-03-04T10:04:18.275 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-03-04T10:04:18.275 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T10:04:18.275 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-04T10:04:18.276 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:04:18.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.389+0000 7fc9c93b2640 1 Processor -- start 2024-03-04T10:04:18.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.389+0000 7fc9c93b2640 1 -- start start 2024-03-04T10:04:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9c93b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc9c40b2e60 0x7fc9c40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9c93b2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc9c40b5470 0x7fc9c40b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9c93b2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c415e1e0 con 0x7fc9c415de00 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9c93b2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40b98c0 con 0x7fc9c40b2e60 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9c93b2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40b32c0 con 0x7fc9c40b5470 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc9c40b2e60 0x7fc9c40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc9c40b2e60 0x7fc9c40b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52180/0 (socket says 172.21.15.99:52180) 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 -- 172.21.15.99:0/785242277 learned_addr learned my addr 172.21.15.99:0/785242277 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9c37fe640 1 --2- 172.21.15.99:0/785242277 >> v2:172.21.15.162:3300/0 conn(0x7fc9c40b5470 0x7fc9c40b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 -- 172.21.15.99:0/785242277 >> v2:172.21.15.162:3300/0 conn(0x7fc9c40b5470 msgr2=0x7fc9c40b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 --2- 172.21.15.99:0/785242277 >> v2:172.21.15.162:3300/0 conn(0x7fc9c40b5470 0x7fc9c40b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 -- 172.21.15.99:0/785242277 >> v1:172.21.15.162:6789/0 conn(0x7fc9c415de00 legacy=0x7fc9c40b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.390+0000 7fc9bbfff640 1 -- 172.21.15.99:0/785242277 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40d1c10 con 0x7fc9c40b2e60 2024-03-04T10:04:18.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.391+0000 7fc9bbfff640 1 --2- 172.21.15.99:0/785242277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c40b9380 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40b2d40 tx=0x7fc9ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26d49e5206ae31e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.391+0000 7fc9c27fc640 1 -- 172.21.15.99:0/785242277 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac035020 con 0x7fc9c40b2e60 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.391+0000 7fc9c27fc640 1 -- 172.21.15.99:0/785242277 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac031870 con 0x7fc9c40b2e60 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.391+0000 7fc9c27fc640 1 -- 172.21.15.99:0/785242277 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac031c40 con 0x7fc9c40b2e60 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 -- 172.21.15.99:0/785242277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 msgr2=0x7fc9c40b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/785242277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c40b9380 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc9c40b2d40 tx=0x7fc9ac02b910 comp rx=0 tx=0).stop 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 -- 172.21.15.99:0/785242277 shutdown_connections 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/785242277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c40b9380 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/785242277 >> v2:172.21.15.162:3300/0 conn(0x7fc9c40b5470 0x7fc9c40b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 -- 172.21.15.99:0/785242277 >> 172.21.15.99:0/785242277 conn(0x7fc9c40d0470 msgr2=0x7fc9c40d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 -- 172.21.15.99:0/785242277 shutdown_connections 2024-03-04T10:04:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.392+0000 7fc9c93b2640 1 -- 172.21.15.99:0/785242277 wait complete. 2024-03-04T10:04:18.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.393+0000 7fc9c93b2640 1 Processor -- start 2024-03-04T10:04:18.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.393+0000 7fc9c93b2640 1 -- start start 2024-03-04T10:04:18.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c93b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c415f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:18.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c93b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9c40b5470 0x7fc9c4159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:18.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c93b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9c415de00 0x7fc9c40caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:18.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c93b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40d21d0 con 0x7fc9c40b2e60 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c93b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40d2820 con 0x7fc9c415de00 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c93b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c40d24b0 con 0x7fc9c40b5470 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c415f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c415f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52190/0 (socket says 172.21.15.99:52190) 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 -- 172.21.15.99:0/3049272002 learned_addr learned my addr 172.21.15.99:0/3049272002 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c37fe640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9c415de00 0x7fc9c40caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 -- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9c40b5470 msgr2=0x7fc9c4159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9c40b5470 0x7fc9c4159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 -- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9c415de00 msgr2=0x7fc9c40caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9c415de00 0x7fc9c40caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 -- 172.21.15.99:0/3049272002 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c40c7ac0 con 0x7fc9c40b2e60 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.394+0000 7fc9c2ffd640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c415f080 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400b3f0 tx=0x7fc9b400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.395+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4015020 con 0x7fc9c40b2e60 2024-03-04T10:04:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.395+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4011830 con 0x7fc9c40b2e60 2024-03-04T10:04:18.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.395+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4011c20 con 0x7fc9c40b2e60 2024-03-04T10:04:18.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.395+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c40c8030 con 0x7fc9c40b2e60 2024-03-04T10:04:18.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.395+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c40c85f0 con 0x7fc9c40b2e60 2024-03-04T10:04:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.396+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c415f680 con 0x7fc9c40b2e60 2024-03-04T10:04:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.396+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc9b4018020 con 0x7fc9c40b2e60 2024-03-04T10:04:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.396+0000 7fc9bb7fe640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc9a406b4d0 0x7fc9a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.397+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc9b408a440 con 0x7fc9c40b2e60 2024-03-04T10:04:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.397+0000 7fc9bbfff640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc9a406b4d0 0x7fc9a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.397+0000 7fc9bbfff640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc9a406b4d0 0x7fc9a406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac036000 tx=0x7fc9ac030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:18.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.401+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc9b401f020 con 0x7fc9c40b2e60 2024-03-04T10:04:18.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.610+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc9c4074f60 con 0x7fc9c40b2e60 2024-03-04T10:04:18.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.610+0000 7fc9bb7fe640 1 -- 172.21.15.99:0/3049272002 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fc9b40172c0 con 0x7fc9c40b2e60 2024-03-04T10:04:18.612 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T10:04:18.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc9a406b4d0 msgr2=0x7fc9a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc9a406b4d0 0x7fc9a406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac036000 tx=0x7fc9ac030000 comp rx=0 tx=0).stop 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 msgr2=0x7fc9c415f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c415f080 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400b3f0 tx=0x7fc9b400b8b0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 shutdown_connections 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc9a406b4d0 0x7fc9a406d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9c415de00 0x7fc9c40caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9c40b5470 0x7fc9c4159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 --2- 172.21.15.99:0/3049272002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9c40b2e60 0x7fc9c415f080 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 >> 172.21.15.99:0/3049272002 conn(0x7fc9c40d0470 msgr2=0x7fc9c40b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 shutdown_connections 2024-03-04T10:04:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:18.612+0000 7fc9c93b2640 1 -- 172.21.15.99:0/3049272002 wait complete. 2024-03-04T10:04:18.624 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T10:04:18.624 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:04:18.624 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi162.front.sepia.ceph.com 2024-03-04T10:04:18.624 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-04T10:04:18.624 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-04T10:04:18.625 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T10:04:18.625 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T10:04:18.625 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:04:18.625 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-04T10:04:18.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.625 DEBUG:teuthology.orchestra.run.smithi162:> ip addr 2024-03-04T10:04:18.641 INFO:teuthology.orchestra.run.smithi162.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:18.641 INFO:teuthology.orchestra.run.smithi162.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:d9:37:f8 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: altname enp7s0f0 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:d9:37:f9 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: altname enp7s0f1 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:88:7f:40 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:18.642 INFO:teuthology.orchestra.run.smithi162.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 0c:c4:7a:88:7f:41 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: inet 172.21.15.162/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft 42729sec preferred_lft 42729sec 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 fe80::ec4:7aff:fe88:7f41/64 scope link 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 96:21:6d:ad:0a:22 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 fe80::d438:5dff:fe4b:d6b/64 scope link 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 96:21:6d:ad:0a:22 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T10:04:18.643 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 fe80::9421:6dff:fead:a22/64 scope link tentative 2024-03-04T10:04:18.644 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:18.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.644 DEBUG:teuthology.orchestra.run.smithi162:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:04:18.709 INFO:teuthology.orchestra.run.smithi162.stdout:1 2024-03-04T10:04:18.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.711 DEBUG:teuthology.orchestra.run.smithi162:> route 2024-03-04T10:04:18.767 INFO:teuthology.orchestra.run.smithi162.stdout:Kernel IP routing table 2024-03-04T10:04:18.767 INFO:teuthology.orchestra.run.smithi162.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:04:18.767 INFO:teuthology.orchestra.run.smithi162.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:04:18.767 INFO:teuthology.orchestra.run.smithi162.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:04:18.767 INFO:teuthology.orchestra.run.smithi162.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:04:18.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.768 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:18.768 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:18.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:04:18.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:04:18.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:04:18.768 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:18.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.863 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list 2024-03-04T10:04:18.877 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-04T10:04:18.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.878 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list-id 2024-03-04T10:04:18.932 INFO:teuthology.orchestra.run.smithi162.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-04T10:04:18.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:18.932 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:18.932 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:18.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-04T10:04:18.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-04T10:04:18.932 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:19.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-04T10:04:19.081 INFO:teuthology.orchestra.run.smithi162.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:19.081 INFO:teuthology.orchestra.run.smithi162.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 32:5c:d9:28:67:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 fe80::305c:d9ff:fe28:6713/64 scope link tentative 2024-03-04T10:04:19.082 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-04T10:04:19.150 INFO:teuthology.orchestra.run.smithi162.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:19.150 INFO:teuthology.orchestra.run.smithi162.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:19.150 INFO:teuthology.orchestra.run.smithi162.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:19.150 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: link/ether 32:5c:d9:28:67:13 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: inet6 fe80::305c:d9ff:fe28:6713/64 scope link tentative 2024-03-04T10:04:19.151 INFO:teuthology.orchestra.run.smithi162.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:19.152 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-04T10:04:19.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-04T10:04:19.153 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:19.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.278 DEBUG:teuthology.orchestra.run.smithi162:> (cd / && exec bash -c ' 2024-03-04T10:04:19.278 DEBUG:teuthology.orchestra.run.smithi162:> set -e 2024-03-04T10:04:19.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link set brx.1 up 2024-03-04T10:04:19.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link set dev brx.1 master ceph-brx 2024-03-04T10:04:19.278 DEBUG:teuthology.orchestra.run.smithi162:> ') 2024-03-04T10:04:19.321 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:19.321 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-04T10:04:19.375 INFO:teuthology.orchestra.run.smithi162.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-04T10:04:19.375 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:19.375 DEBUG:teuthology.orchestra.run.smithi162:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-04T10:04:19.429 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-03-04T10:04:19.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:19.515 INFO:teuthology.orchestra.run.smithi162.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:19.515 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:19.523 INFO:teuthology.orchestra.run.smithi162.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:19.523 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: trying mount with new device syntax: 3@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:04:19.524 INFO:teuthology.orchestra.run.smithi162.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:04:19.526 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:04:19.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-04T10:04:19.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.554 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:19.611 INFO:teuthology.orchestra.run.smithi162.stdout:client4854 2024-03-04T10:04:19.611 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:19.611 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:19.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.668 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:19.725 INFO:teuthology.orchestra.run.smithi162.stdout:client4854 2024-03-04T10:04:19.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.725 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:19.782 INFO:teuthology.orchestra.run.smithi162.stdout:client4854 2024-03-04T10:04:19.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:19.782 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.client4854/status') 2024-03-04T10:04:19.849 INFO:teuthology.orchestra.run.smithi162.stdout:instance: client.4854 (3)172.21.15.162:0/1533320844 2024-03-04T10:04:19.849 INFO:teuthology.orchestra.run.smithi162.stdout:blocklisted: false 2024-03-04T10:04:19.849 INFO:teuthology.orchestra.run.smithi162.stderr:0+1 records in 2024-03-04T10:04:19.849 INFO:teuthology.orchestra.run.smithi162.stderr:0+1 records out 2024-03-04T10:04:19.849 INFO:teuthology.orchestra.run.smithi162.stderr:71 bytes copied, 2.5808e-05 s, 2.8 MB/s 2024-03-04T10:04:19.850 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-04T10:04:19.866 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:19.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.974+0000 7fa2be929640 1 Processor -- start 2024-03-04T10:04:19.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.974+0000 7fa2be929640 1 -- start start 2024-03-04T10:04:19.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2be929640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2b8161e70 0x7fa2b8168af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2be929640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b806b500 con 0x7fa2b80b39c0 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2be929640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b806b7e0 con 0x7fa2b8158410 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2be929640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b806bac0 con 0x7fa2b8161e70 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2bcec6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2b8161e70 0x7fa2b8168af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2b77fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa2b80b39c0 0x7fa2b8161760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56286/0 (socket says 172.21.15.99:56286) 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.975+0000 7fa2b77fe640 1 -- 172.21.15.99:0/4086150285 learned_addr learned my addr 172.21.15.99:0/4086150285 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.976+0000 7fa2bcec6640 1 -- 172.21.15.99:0/4086150285 >> v1:172.21.15.162:6789/0 conn(0x7fa2b8158410 legacy=0x7fa2b806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.976+0000 7fa2bcec6640 1 -- 172.21.15.99:0/4086150285 >> v1:172.21.15.99:6789/0 conn(0x7fa2b80b39c0 legacy=0x7fa2b8161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:19.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.976+0000 7fa2bcec6640 1 -- 172.21.15.99:0/4086150285 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b806bda0 con 0x7fa2b8161e70 2024-03-04T10:04:19.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.976+0000 7fa2bcec6640 1 --2- 172.21.15.99:0/4086150285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b8161e70 0x7fa2b8168af0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009d90 tx=0x7fa2b002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aadbbab1d436c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:19.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.976+0000 7fa2b6ffd640 1 -- 172.21.15.99:0/4086150285 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0039070 con 0x7fa2b8161e70 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2b6ffd640 1 -- 172.21.15.99:0/4086150285 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b002cab0 con 0x7fa2b8161e70 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2b6ffd640 1 -- 172.21.15.99:0/4086150285 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0034440 con 0x7fa2b8161e70 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 -- 172.21.15.99:0/4086150285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b8161e70 msgr2=0x7fa2b8168af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 --2- 172.21.15.99:0/4086150285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b8161e70 0x7fa2b8168af0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009d90 tx=0x7fa2b002bd80 comp rx=0 tx=0).stop 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 -- 172.21.15.99:0/4086150285 shutdown_connections 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 --2- 172.21.15.99:0/4086150285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b8161e70 0x7fa2b8168af0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 -- 172.21.15.99:0/4086150285 >> 172.21.15.99:0/4086150285 conn(0x7fa2b80cfef0 msgr2=0x7fa2b80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 -- 172.21.15.99:0/4086150285 shutdown_connections 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.977+0000 7fa2be929640 1 -- 172.21.15.99:0/4086150285 wait complete. 2024-03-04T10:04:19.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.978+0000 7fa2be929640 1 Processor -- start 2024-03-04T10:04:19.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 -- start start 2024-03-04T10:04:19.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 0x7fa2b80cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:19.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b8158410 0x7fa2b815d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b8161e70 0x7fa2b80cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b806c5a0 con 0x7fa2b80b39c0 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80b3e50 con 0x7fa2b8161e70 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2be929640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b806ee60 con 0x7fa2b8158410 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2b7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 0x7fa2b80cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2b7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 0x7fa2b80cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52210/0 (socket says 172.21.15.99:52210) 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.979+0000 7fa2b7fff640 1 -- 172.21.15.99:0/169004019 learned_addr learned my addr 172.21.15.99:0/169004019 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2bcec6640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b8161e70 0x7fa2b80cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b77fe640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b8158410 0x7fa2b815d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b7fff640 1 -- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b8158410 msgr2=0x7fa2b815d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b7fff640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b8158410 0x7fa2b815d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b7fff640 1 -- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b8161e70 msgr2=0x7fa2b80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b7fff640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b8161e70 0x7fa2b80cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b7fff640 1 -- 172.21.15.99:0/169004019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b80ccb20 con 0x7fa2b80b39c0 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b77fe640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b8158410 0x7fa2b815d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b7fff640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 0x7fa2b80cdbc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c460 tx=0x7fa2a800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8016020 con 0x7fa2b80b39c0 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8012870 con 0x7fa2b80b39c0 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b80cce00 con 0x7fa2b80b39c0 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8012c60 con 0x7fa2b80b39c0 2024-03-04T10:04:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.980+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b80cd3c0 con 0x7fa2b80b39c0 2024-03-04T10:04:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.981+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b8074f60 con 0x7fa2b80b39c0 2024-03-04T10:04:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.981+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa2a8019020 con 0x7fa2b80b39c0 2024-03-04T10:04:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.982+0000 7fa2b4ff9640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a406b500 0x7fa2a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.982+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa2a808a2a0 con 0x7fa2b80b39c0 2024-03-04T10:04:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.982+0000 7fa2b77fe640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a406b500 0x7fa2a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.982+0000 7fa2b77fe640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a406b500 0x7fa2a406d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80cf3f0 tx=0x7fa2ac007d20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:19.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:19.986+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa2a8020030 con 0x7fa2b80b39c0 2024-03-04T10:04:20.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.159+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2b8066e10 con 0x7fa2b80b39c0 2024-03-04T10:04:20.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.160+0000 7fa2b4ff9640 1 -- 172.21.15.99:0/169004019 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18839 (secure 0 0 0) 0x7fa2a8058ca0 con 0x7fa2b80b39c0 2024-03-04T10:04:20.161 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:20.162 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":25,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:19.658255+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a406b500 msgr2=0x7fa2a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a406b500 0x7fa2a406d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa2b80cf3f0 tx=0x7fa2ac007d20 comp rx=0 tx=0).stop 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 msgr2=0x7fa2b80cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 0x7fa2b80cdbc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c460 tx=0x7fa2a800c920 comp rx=0 tx=0).stop 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 shutdown_connections 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a406b500 0x7fa2a406d9c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b8161e70 0x7fa2b80cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b8158410 0x7fa2b815d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 --2- 172.21.15.99:0/169004019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b80b39c0 0x7fa2b80cdbc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 >> 172.21.15.99:0/169004019 conn(0x7fa2b80cfef0 msgr2=0x7fa2b80aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:20.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 shutdown_connections 2024-03-04T10:04:20.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.162+0000 7fa2be929640 1 -- 172.21.15.99:0/169004019 wait complete. 2024-03-04T10:04:20.175 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:20.175 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:20.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.278+0000 7ff8f557e640 1 Processor -- start 2024-03-04T10:04:20.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.278+0000 7ff8f557e640 1 -- start start 2024-03-04T10:04:20.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.278+0000 7ff8f557e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff8f00b9410 0x7ff8f01622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.278+0000 7ff8f557e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00d2400 con 0x7ff8f00d34a0 2024-03-04T10:04:20.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.278+0000 7ff8f557e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f00d26e0 con 0x7ff8f00b5700 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.278+0000 7ff8f557e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f00d29c0 con 0x7ff8f00b9410 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff8f00d34a0 0x7ff8f0161be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56290/0 (socket says 172.21.15.99:56290) 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/222267219 learned_addr learned my addr 172.21.15.99:0/222267219 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ef7fe640 1 --2- 172.21.15.99:0/222267219 >> v2:172.21.15.162:3300/0 conn(0x7ff8f00b9410 0x7ff8f01622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518171233 0 0) 0x7ff8f00d2400 con 0x7ff8f00d34a0 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d8003520 con 0x7ff8f00d34a0 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1671678892 0 0) 0x7ff8d8003520 con 0x7ff8f00d34a0 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 >> v2:172.21.15.162:3300/0 conn(0x7ff8f00b9410 msgr2=0x7ff8f01622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 --2- 172.21.15.99:0/222267219 >> v2:172.21.15.162:3300/0 conn(0x7ff8f00b9410 0x7ff8f01622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 >> v1:172.21.15.162:6789/0 conn(0x7ff8f00b5700 legacy=0x7ff8f00d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00d2ca0 con 0x7ff8f00d34a0 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7ff8e4001670 con 0x7ff8f00d34a0 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 >> v1:172.21.15.99:6789/0 conn(0x7ff8f00d34a0 legacy=0x7ff8f0161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d80039b0 0x7ff8d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8d8004800 0x7ff8d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8d80051c0 0x7ff8d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d29c0 con 0x7ff8d8004800 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d800c160 con 0x7ff8d80039b0 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d800c360 con 0x7ff8d80051c0 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8d8004800 0x7ff8d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8eeffd640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d80039b0 0x7ff8d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ef7fe640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8d80051c0 0x7ff8d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8d80051c0 msgr2=0x7ff8d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8d80051c0 0x7ff8d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d80039b0 msgr2=0x7ff8d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d80039b0 0x7ff8d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ef7fe640 1 -- 172.21.15.99:0/222267219 reap_dead start 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.279+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/222267219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d8003520 con 0x7ff8d8004800 2024-03-04T10:04:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.280+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8d8004800 0x7ff8d8004c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400c210 tx=0x7ff8d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a8a8b9c7195134 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.280+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4007cf0 con 0x7ff8d8004800 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.280+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4030070 con 0x7ff8d8004800 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.280+0000 7ff8edffb640 1 -- 172.21.15.99:0/222267219 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4005d70 con 0x7ff8d8004800 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 -- 172.21.15.99:0/222267219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8d8004800 msgr2=0x7ff8d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8d8004800 0x7ff8d8004c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400c210 tx=0x7ff8d8005830 comp rx=0 tx=0).stop 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 -- 172.21.15.99:0/222267219 shutdown_connections 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 --2- 172.21.15.99:0/222267219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8d8004800 0x7ff8d8004c80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 -- 172.21.15.99:0/222267219 >> 172.21.15.99:0/222267219 conn(0x7ff8f00d0490 msgr2=0x7ff8f00d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 -- 172.21.15.99:0/222267219 shutdown_connections 2024-03-04T10:04:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.281+0000 7ff8f557e640 1 -- 172.21.15.99:0/222267219 wait complete. 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.282+0000 7ff8f557e640 1 Processor -- start 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.282+0000 7ff8f557e640 1 -- start start 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8f557e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d8004800 0x7ff8f015a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8f557e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 0x7ff8f00c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8f557e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8f00c88e0 0x7ff8f0147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8f557e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d1b00 con 0x7ff8f01e7570 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8f557e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00b5cf0 con 0x7ff8d8004800 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8f557e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f00d1c80 con 0x7ff8f00c88e0 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 0x7ff8f00c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 0x7ff8f00c83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52222/0 (socket says 172.21.15.99:52222) 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/198120568 learned_addr learned my addr 172.21.15.99:0/198120568 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ef7fe640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8f00c88e0 0x7ff8f0147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8eeffd640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d8004800 0x7ff8f015a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8f00c88e0 msgr2=0x7ff8f0147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8f00c88e0 0x7ff8f0147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d8004800 msgr2=0x7ff8f015a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d8004800 0x7ff8f015a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 -- 172.21.15.99:0/198120568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f01488f0 con 0x7ff8f01e7570 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8eeffd640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d8004800 0x7ff8f015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.283+0000 7ff8ee7fc640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 0x7ff8f00c83a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400c330 tx=0x7ff8f00d28b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.284+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4007cf0 con 0x7ff8f01e7570 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.284+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e40049f0 con 0x7ff8f01e7570 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.284+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f01e6a90 con 0x7ff8f01e7570 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.284+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4001c30 con 0x7ff8f01e7570 2024-03-04T10:04:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.284+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f015a5a0 con 0x7ff8f01e7570 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.285+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f0067f30 con 0x7ff8f01e7570 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.285+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff8e4007070 con 0x7ff8f01e7570 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.285+0000 7ff8d77fe640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.285+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6714+0+0 (secure 0 0 0) 0x7ff8e40ab6f0 con 0x7ff8f01e7570 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.285+0000 7ff8eeffd640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.286+0000 7ff8eeffd640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0000c00 tx=0x7ff8e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.290+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff8f0067f30 con 0x7ff8f01e7570 2024-03-04T10:04:20.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.461+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8f01e6880 con 0x7ff8f01e7570 2024-03-04T10:04:20.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.461+0000 7ff8d77fe640 1 -- 172.21.15.99:0/198120568 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8e407a180 con 0x7ff8f01e7570 2024-03-04T10:04:20.462 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:20.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff8dc06b500 msgr2=0x7ff8dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0000c00 tx=0x7ff8e0009040 comp rx=0 tx=0).stop 2024-03-04T10:04:20.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 msgr2=0x7ff8f00c83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 0x7ff8f00c83a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400c330 tx=0x7ff8f00d28b0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 shutdown_connections 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff8dc06b500 0x7ff8dc06d9c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8f00c88e0 0x7ff8f0147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8f01e7570 0x7ff8f00c83a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 --2- 172.21.15.99:0/198120568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8d8004800 0x7ff8f015a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 >> 172.21.15.99:0/198120568 conn(0x7ff8f00d0490 msgr2=0x7ff8f00b6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 shutdown_connections 2024-03-04T10:04:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.463+0000 7ff8f557e640 1 -- 172.21.15.99:0/198120568 wait complete. 2024-03-04T10:04:20.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:20.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.577+0000 7fd77d559640 1 Processor -- start 2024-03-04T10:04:20.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.577+0000 7fd77d559640 1 -- start start 2024-03-04T10:04:20.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd77d559640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd7780d34a0 0x7fd778161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd77d559640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780d2400 con 0x7fd7780b5700 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd77d559640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7780d26e0 con 0x7fd7780b9410 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd77d559640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7780d29c0 con 0x7fd7780d34a0 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd776ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd7780b5700 0x7fd7780d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56296/0 (socket says 172.21.15.99:56296) 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd776ffd640 1 -- 172.21.15.99:0/295326029 learned_addr learned my addr 172.21.15.99:0/295326029 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339682916 0 0) 0x7fd7780d2400 con 0x7fd7780b5700 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd76e7fc640 1 --2- 172.21.15.99:0/295326029 >> v2:172.21.15.162:3300/0 conn(0x7fd7780d34a0 0x7fd778161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd764003520 con 0x7fd7780b5700 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304796638 0 0) 0x7fd764003520 con 0x7fd7780b5700 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 >> v2:172.21.15.162:3300/0 conn(0x7fd7780d34a0 msgr2=0x7fd778161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 --2- 172.21.15.99:0/295326029 >> v2:172.21.15.162:3300/0 conn(0x7fd7780d34a0 0x7fd778161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 >> v1:172.21.15.162:6789/0 conn(0x7fd7780b9410 legacy=0x7fd778162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780d2ca0 con 0x7fd7780b5700 2024-03-04T10:04:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fd768001670 con 0x7fd7780b5700 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd76e7fc640 1 --2- 172.21.15.99:0/295326029 >> v2:172.21.15.162:3300/0 conn(0x7fd7780d34a0 0x7fd778161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 >> v1:172.21.15.99:6789/0 conn(0x7fd7780b5700 legacy=0x7fd7780d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7640039b0 0x7fd764003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd764004800 0x7fd764004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7640052e0 0x7fd76400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780d29c0 con 0x7fd7640039b0 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76400c1b0 con 0x7fd764004800 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.578+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76400c3b0 con 0x7fd7640052e0 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7640039b0 0x7fd764003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd7777fe640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7640052e0 0x7fd76400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd76e7fc640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd764004800 0x7fd764004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 -- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7640052e0 msgr2=0x7fd76400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7640052e0 0x7fd76400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 -- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd764004800 msgr2=0x7fd764004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd764004800 0x7fd764004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 -- 172.21.15.99:0/295326029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764003520 con 0x7fd7640039b0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd7777fe640 1 -- 172.21.15.99:0/295326029 reap_dead start 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.579+0000 7fd776ffd640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7640039b0 0x7fd764003dd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd76800c210 tx=0x7fd764005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0cc4c582625317 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768007cf0 con 0x7fd7640039b0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd768030070 con 0x7fd7640039b0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd7767fc640 1 -- 172.21.15.99:0/295326029 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768005d70 con 0x7fd7640039b0 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 -- 172.21.15.99:0/295326029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7640039b0 msgr2=0x7fd764003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7640039b0 0x7fd764003dd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd76800c210 tx=0x7fd764005830 comp rx=0 tx=0).stop 2024-03-04T10:04:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 -- 172.21.15.99:0/295326029 shutdown_connections 2024-03-04T10:04:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 --2- 172.21.15.99:0/295326029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7640039b0 0x7fd764003dd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 -- 172.21.15.99:0/295326029 >> 172.21.15.99:0/295326029 conn(0x7fd7780d0490 msgr2=0x7fd7780d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 -- 172.21.15.99:0/295326029 shutdown_connections 2024-03-04T10:04:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.580+0000 7fd77d559640 1 -- 172.21.15.99:0/295326029 wait complete. 2024-03-04T10:04:20.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.581+0000 7fd77d559640 1 Processor -- start 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 -- start start 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7640039b0 0x7fd77815f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 0x7fd7780ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7780caae0 0x7fd7780caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780d1b00 con 0x7fd7780aae00 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780b5cf0 con 0x7fd7640039b0 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd77d559640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7780d1c80 con 0x7fd7780caae0 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd76e7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 0x7fd7780ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd76e7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 0x7fd7780ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52228/0 (socket says 172.21.15.99:52228) 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd76e7fc640 1 -- 172.21.15.99:0/476756484 learned_addr learned my addr 172.21.15.99:0/476756484 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd7777fe640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7780caae0 0x7fd7780caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.582+0000 7fd776ffd640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7640039b0 0x7fd77815f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76e7fc640 1 -- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7780caae0 msgr2=0x7fd7780caf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76e7fc640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7780caae0 0x7fd7780caf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76e7fc640 1 -- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7640039b0 msgr2=0x7fd77815f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76e7fc640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7640039b0 0x7fd77815f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76e7fc640 1 -- 172.21.15.99:0/476756484 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7780ab550 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd7777fe640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7780caae0 0x7fd7780caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76e7fc640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 0x7fd7780ccbc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd7600116d0 tx=0x7fd760011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760019070 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760004220 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76000f040 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7780ab830 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.583+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7780abdf0 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.584+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd778074f60 con 0x7fd7780aae00 2024-03-04T10:04:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.584+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd76000f7c0 con 0x7fd7780aae00 2024-03-04T10:04:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.585+0000 7fd76ffff640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd75806b4d0 0x7fd75806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.585+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd76008a920 con 0x7fd7780aae00 2024-03-04T10:04:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.585+0000 7fd776ffd640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd75806b4d0 0x7fd75806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.585+0000 7fd776ffd640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd75806b4d0 0x7fd75806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd768034000 tx=0x7fd768007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.589+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd760059320 con 0x7fd7780aae00 2024-03-04T10:04:20.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.765+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd7780b4690 con 0x7fd7780aae00 2024-03-04T10:04:20.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.766+0000 7fd76ffff640 1 -- 172.21.15.99:0/476756484 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd760058c20 con 0x7fd7780aae00 2024-03-04T10:04:20.767 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd75806b4d0 msgr2=0x7fd75806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd75806b4d0 0x7fd75806d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd768034000 tx=0x7fd768007000 comp rx=0 tx=0).stop 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 msgr2=0x7fd7780ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 0x7fd7780ccbc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd7600116d0 tx=0x7fd760011b90 comp rx=0 tx=0).stop 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 shutdown_connections 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd75806b4d0 0x7fd75806d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd7780caae0 0x7fd7780caf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd7780aae00 0x7fd7780ccbc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 --2- 172.21.15.99:0/476756484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd7640039b0 0x7fd77815f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 >> 172.21.15.99:0/476756484 conn(0x7fd7780d0490 msgr2=0x7fd778160120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 shutdown_connections 2024-03-04T10:04:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.768+0000 7fd77d559640 1 -- 172.21.15.99:0/476756484 wait complete. 2024-03-04T10:04:20.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.883+0000 7efe00ffa640 1 Processor -- start 2024-03-04T10:04:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.883+0000 7efe00ffa640 1 -- start start 2024-03-04T10:04:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efe00ffa640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7efdfc15da20 0x7efdfc0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efe00ffa640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efdfc0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efe00ffa640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdfc162450 con 0x7efdfc1616e0 2024-03-04T10:04:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efe00ffa640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdfc0d0000 con 0x7efdfc15da20 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efe00ffa640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0cc920 con 0x7efdfc0cf660 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdfad9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7efdfc1616e0 0x7efdfc161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56312/0 (socket says 172.21.15.99:56312) 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdfad9d640 1 -- 172.21.15.99:0/211810852 learned_addr learned my addr 172.21.15.99:0/211810852 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdfa59c640 1 --2- 172.21.15.99:0/211810852 >> v2:172.21.15.162:3300/0 conn(0x7efdfc15da20 0x7efdfc0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efdfc0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782495913 0 0) 0x7efdfc162450 con 0x7efdfc1616e0 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efde4003520 con 0x7efdfc1616e0 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554195316 0 0) 0x7efde4003520 con 0x7efdfc1616e0 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 msgr2=0x7efdfc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf959a640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efdfc0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.884+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 >> v2:172.21.15.162:3300/0 conn(0x7efdfc15da20 msgr2=0x7efdfc0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 --2- 172.21.15.99:0/211810852 >> v2:172.21.15.162:3300/0 conn(0x7efdfc15da20 0x7efdfc0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc0cc2d0 con 0x7efdfc1616e0 2024-03-04T10:04:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7efdec001670 con 0x7efdfc1616e0 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdfa59c640 1 --2- 172.21.15.99:0/211810852 >> v2:172.21.15.162:3300/0 conn(0x7efdfc15da20 0x7efdfc0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efdfc0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 >> v1:172.21.15.99:6789/0 conn(0x7efdfc1616e0 legacy=0x7efdfc161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde40039b0 0x7efde4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde4004800 0x7efde4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efde40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0cc920 con 0x7efde4004800 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde400b9c0 con 0x7efde40039b0 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde400bb40 con 0x7efdfc0cf660 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde4004800 0x7efde4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdfa59c640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde40039b0 0x7efde4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdfad9d640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efde40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 -- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 msgr2=0x7efde40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efde40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 -- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde40039b0 msgr2=0x7efde4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde40039b0 0x7efde4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdf9d9b640 1 -- 172.21.15.99:0/211810852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4003520 con 0x7efde4004800 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdfad9d640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efdfc0cf660 0x7efde40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.885+0000 7efdfa59c640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efde40039b0 0x7efde4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde4004800 0x7efde4004c80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009c90 tx=0x7efde4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe9cd39dd9c7057 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0039070 con 0x7efde4004800 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf002c8f0 con 0x7efde4004800 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efdf959a640 1 -- 172.21.15.99:0/211810852 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0009860 con 0x7efde4004800 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efe00ffa640 1 -- 172.21.15.99:0/211810852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde4004800 msgr2=0x7efde4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efe00ffa640 1 --2- 172.21.15.99:0/211810852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efde4004800 0x7efde4004c80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009c90 tx=0x7efde4005830 comp rx=0 tx=0).stop 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efdfad9d640 1 -- 172.21.15.99:0/211810852 reap_dead start 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efe00ffa640 1 -- 172.21.15.99:0/211810852 shutdown_connections 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.886+0000 7efe00ffa640 1 -- 172.21.15.99:0/211810852 >> 172.21.15.99:0/211810852 conn(0x7efdfc0b33a0 msgr2=0x7efdfc0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.887+0000 7efe00ffa640 1 -- 172.21.15.99:0/211810852 shutdown_connections 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.887+0000 7efe00ffa640 1 -- 172.21.15.99:0/211810852 wait complete. 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.888+0000 7efe00ffa640 1 Processor -- start 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.888+0000 7efe00ffa640 1 -- start start 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde4004800 0x7efdfc160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 0x7efdfc0ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdfc14d840 0x7efdfc14dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0cbf80 con 0x7efdfc160aa0 2024-03-04T10:04:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc0cd2f0 con 0x7efdfc14d840 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc15de00 con 0x7efde4004800 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 0x7efdfc0ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 0x7efdfc0ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52238/0 (socket says 172.21.15.99:52238) 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 -- 172.21.15.99:0/3250884972 learned_addr learned my addr 172.21.15.99:0/3250884972 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdfa59c640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde4004800 0x7efdfc160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdfad9d640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdfc14d840 0x7efdfc14dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 -- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde4004800 msgr2=0x7efdfc160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde4004800 0x7efdfc160130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 -- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdfc14d840 msgr2=0x7efdfc14dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdfc14d840 0x7efdfc14dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 -- 172.21.15.99:0/3250884972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc157f70 con 0x7efdfc160aa0 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdf9d9b640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 0x7efdfc0ca2c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009c90 tx=0x7efdf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0039050 con 0x7efdfc160aa0 2024-03-04T10:04:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf0045070 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc15eb20 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0009860 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.889+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc15f0e0 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.890+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc074f60 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.891+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7efdf004a460 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.891+0000 7efdeaffd640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efddc06b500 0x7efddc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.891+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7efdf00ac8a0 con 0x7efdfc160aa0 2024-03-04T10:04:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.891+0000 7efdfa59c640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efddc06b500 0x7efddc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.892+0000 7efdfa59c640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efddc06b500 0x7efddc06d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0b42a0 tx=0x7efdf400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:20.895+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efdf007b2a0 con 0x7efdfc160aa0 2024-03-04T10:04:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.067+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efdfc0b2bd0 con 0x7efdfc160aa0 2024-03-04T10:04:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.068+0000 7efdeaffd640 1 -- 172.21.15.99:0/3250884972 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7efdf007aba0 con 0x7efdfc160aa0 2024-03-04T10:04:21.070 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efddc06b500 msgr2=0x7efddc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efddc06b500 0x7efddc06d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0b42a0 tx=0x7efdf400b040 comp rx=0 tx=0).stop 2024-03-04T10:04:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 msgr2=0x7efdfc0ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 0x7efdfc0ca2c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009c90 tx=0x7efdf0004730 comp rx=0 tx=0).stop 2024-03-04T10:04:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 shutdown_connections 2024-03-04T10:04:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efddc06b500 0x7efddc06d9c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdfc14d840 0x7efdfc14dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efdfc160aa0 0x7efdfc0ca2c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 --2- 172.21.15.99:0/3250884972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efde4004800 0x7efdfc160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 >> 172.21.15.99:0/3250884972 conn(0x7efdfc0b33a0 msgr2=0x7efdfc0b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 shutdown_connections 2024-03-04T10:04:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.070+0000 7efe00ffa640 1 -- 172.21.15.99:0/3250884972 wait complete. 2024-03-04T10:04:21.083 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:21.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.186+0000 7f227ef29640 1 Processor -- start 2024-03-04T10:04:21.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.186+0000 7f227ef29640 1 -- start start 2024-03-04T10:04:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f227ef29640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f227815bba0 0x7f22780728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f227ef29640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227815cec0 con 0x7f227806bce0 2024-03-04T10:04:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f227ef29640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227815c100 con 0x7f22780700a0 2024-03-04T10:04:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f227ef29640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227815c3e0 con 0x7f227815bba0 2024-03-04T10:04:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f227ccc5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f227806bce0 0x7f227806c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56320/0 (socket says 172.21.15.99:56320) 2024-03-04T10:04:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f227ccc5640 1 -- 172.21.15.99:0/928902273 learned_addr learned my addr 172.21.15.99:0/928902273 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f2277fff640 1 --2- 172.21.15.99:0/928902273 >> v2:172.21.15.162:3300/0 conn(0x7f227815bba0 0x7f22780728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666184231 0 0) 0x7f227815cec0 con 0x7f227806bce0 2024-03-04T10:04:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.187+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2260003520 con 0x7f227806bce0 2024-03-04T10:04:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407956164 0 0) 0x7f2260003520 con 0x7f227806bce0 2024-03-04T10:04:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 >> v2:172.21.15.162:3300/0 conn(0x7f227815bba0 msgr2=0x7f22780728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 --2- 172.21.15.99:0/928902273 >> v2:172.21.15.162:3300/0 conn(0x7f227815bba0 0x7f22780728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 >> v1:172.21.15.162:6789/0 conn(0x7f22780700a0 legacy=0x7f227815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227815c6c0 con 0x7f227806bce0 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f2268001670 con 0x7f227806bce0 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 >> v1:172.21.15.99:6789/0 conn(0x7f227806bce0 legacy=0x7f227806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f2260003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2260004800 0x7f2260004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22600051c0 0x7f226000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227815c3e0 con 0x7f22600039b0 2024-03-04T10:04:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226000c160 con 0x7f2260004800 2024-03-04T10:04:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226000c360 con 0x7f22600051c0 2024-03-04T10:04:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227ccc5640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f2260003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f2277fff640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2260004800 0x7f2260004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227d4c6640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22600051c0 0x7f226000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227ccc5640 1 -- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22600051c0 msgr2=0x7f226000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227ccc5640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22600051c0 0x7f226000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227ccc5640 1 -- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2260004800 msgr2=0x7f2260004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227ccc5640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2260004800 0x7f2260004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227ccc5640 1 -- 172.21.15.99:0/928902273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260003520 con 0x7f22600039b0 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227d4c6640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22600051c0 0x7f226000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.188+0000 7f227d4c6640 1 -- 172.21.15.99:0/928902273 reap_dead start 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.189+0000 7f227ccc5640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f2260003dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f226800c210 tx=0x7f2260005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc11da087fbe2e0f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.189+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268007d30 con 0x7f22600039b0 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.189+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2268030070 con 0x7f22600039b0 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.189+0000 7f22777fe640 1 -- 172.21.15.99:0/928902273 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226802cda0 con 0x7f22600039b0 2024-03-04T10:04:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 -- 172.21.15.99:0/928902273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 msgr2=0x7f2260003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f2260003dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f226800c210 tx=0x7f2260005830 comp rx=0 tx=0).stop 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 -- 172.21.15.99:0/928902273 shutdown_connections 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 --2- 172.21.15.99:0/928902273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f2260003dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 -- 172.21.15.99:0/928902273 >> 172.21.15.99:0/928902273 conn(0x7f2278071bc0 msgr2=0x7f2278071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 -- 172.21.15.99:0/928902273 shutdown_connections 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.190+0000 7f227ef29640 1 -- 172.21.15.99:0/928902273 wait complete. 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.191+0000 7f227ef29640 1 Processor -- start 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.191+0000 7f227ef29640 1 -- start start 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f22781e4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22781e46c0 0x7f227813b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227813b940 0x7f227813bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780728f0 con 0x7f22600039b0 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780cc3e0 con 0x7f227813b940 2024-03-04T10:04:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2278072a70 con 0x7f22781e46c0 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f22781e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f22781e4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52260/0 (socket says 172.21.15.99:52260) 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 -- 172.21.15.99:0/1646374940 learned_addr learned my addr 172.21.15.99:0/1646374940 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227d4c6640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227813b940 0x7f227813bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f2277fff640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22781e46c0 0x7f227813b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 -- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22781e46c0 msgr2=0x7f227813b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22781e46c0 0x7f227813b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 -- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227813b940 msgr2=0x7f227813bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227813b940 0x7f227813bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 -- 172.21.15.99:0/1646374940 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22781e4b40 con 0x7f22600039b0 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227d4c6640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227813b940 0x7f227813bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f2277fff640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22781e46c0 0x7f227813b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ccc5640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f22781e4180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f226800c210 tx=0x7f226802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268004bf0 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2268004d90 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227814e9e0 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2268007270 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.192+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22781397f0 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.193+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278074f60 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.194+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2268048020 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.194+0000 7f22757fa640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f226406b500 0x7f226406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.194+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f22680aba00 con 0x7f22600039b0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.194+0000 7f2277fff640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f226406b500 0x7f226406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.195+0000 7f2277fff640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f226406b500 0x7f226406d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f227814ef60 tx=0x7f226c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.198+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f226807a5b0 con 0x7f22600039b0 2024-03-04T10:04:21.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.368+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2278074f60 con 0x7f22600039b0 2024-03-04T10:04:21.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.368+0000 7f22757fa640 1 -- 172.21.15.99:0/1646374940 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2268079eb0 con 0x7f22600039b0 2024-03-04T10:04:21.370 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:21.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f226406b500 msgr2=0x7f226406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f226406b500 0x7f226406d9c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f227814ef60 tx=0x7f226c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 msgr2=0x7f22781e4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f22781e4180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f226800c210 tx=0x7f226802e070 comp rx=0 tx=0).stop 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 shutdown_connections 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f226406b500 0x7f226406d9c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f227813b940 0x7f227813bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22781e46c0 0x7f227813b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 --2- 172.21.15.99:0/1646374940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22600039b0 0x7f22781e4180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 >> 172.21.15.99:0/1646374940 conn(0x7f2278071bc0 msgr2=0x7f22780b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 shutdown_connections 2024-03-04T10:04:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.370+0000 7f227ef29640 1 -- 172.21.15.99:0/1646374940 wait complete. 2024-03-04T10:04:21.382 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:21.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.482+0000 7f4f278ea640 1 Processor -- start 2024-03-04T10:04:21.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.482+0000 7f4f278ea640 1 -- start start 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f278ea640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4f2015b960 0x7f4f200d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f278ea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f4f200b0e10 0x7f4f200b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f278ea640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f200ccad0 con 0x7f4f200d0ad0 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f278ea640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f200cf860 con 0x7f4f2015b960 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f278ea640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f200d0100 con 0x7f4f200b0e10 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f25686640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4f2015b960 0x7f4f200d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f25686640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4f2015b960 0x7f4f200d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52266/0 (socket says 172.21.15.99:52266) 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f25686640 1 -- 172.21.15.99:0/2797203317 learned_addr learned my addr 172.21.15.99:0/2797203317 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f25e87640 1 --2- 172.21.15.99:0/2797203317 >> v2:172.21.15.162:3300/0 conn(0x7f4f200b0e10 0x7f4f200b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f25686640 1 -- 172.21.15.99:0/2797203317 >> v2:172.21.15.162:3300/0 conn(0x7f4f200b0e10 msgr2=0x7f4f200b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.483+0000 7f4f25686640 1 --2- 172.21.15.99:0/2797203317 >> v2:172.21.15.162:3300/0 conn(0x7f4f200b0e10 0x7f4f200b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.484+0000 7f4f25686640 1 -- 172.21.15.99:0/2797203317 >> v1:172.21.15.162:6789/0 conn(0x7f4f200d0ad0 legacy=0x7f4f200b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.484+0000 7f4f25686640 1 -- 172.21.15.99:0/2797203317 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f200cfab0 con 0x7f4f2015b960 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.484+0000 7f4f25686640 1 --2- 172.21.15.99:0/2797203317 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f2015b960 0x7f4f200d0590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f200ce1b0 tx=0x7f4f1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f30ab262bd34df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.484+0000 7f4f17fff640 1 -- 172.21.15.99:0/2797203317 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c035020 con 0x7f4f2015b960 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.484+0000 7f4f17fff640 1 -- 172.21.15.99:0/2797203317 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c031870 con 0x7f4f2015b960 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.484+0000 7f4f17fff640 1 -- 172.21.15.99:0/2797203317 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c031c40 con 0x7f4f2015b960 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 -- 172.21.15.99:0/2797203317 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f2015b960 msgr2=0x7f4f200d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 --2- 172.21.15.99:0/2797203317 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f2015b960 0x7f4f200d0590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4f200ce1b0 tx=0x7f4f1c02b910 comp rx=0 tx=0).stop 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 -- 172.21.15.99:0/2797203317 shutdown_connections 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 --2- 172.21.15.99:0/2797203317 >> v2:172.21.15.162:3300/0 conn(0x7f4f200b0e10 0x7f4f200b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 --2- 172.21.15.99:0/2797203317 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f2015b960 0x7f4f200d0590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 -- 172.21.15.99:0/2797203317 >> 172.21.15.99:0/2797203317 conn(0x7f4f2006f5f0 msgr2=0x7f4f2006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 -- 172.21.15.99:0/2797203317 shutdown_connections 2024-03-04T10:04:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.485+0000 7f4f278ea640 1 -- 172.21.15.99:0/2797203317 wait complete. 2024-03-04T10:04:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.486+0000 7f4f278ea640 1 Processor -- start 2024-03-04T10:04:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.486+0000 7f4f278ea640 1 -- start start 2024-03-04T10:04:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f278ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f200b0e10 0x7f4f200cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f278ea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 0x7f4f2015d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f278ea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f2015b960 0x7f4f20155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f278ea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2015bec0 con 0x7f4f200d0ad0 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f278ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2015bd40 con 0x7f4f200b0e10 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f278ea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f200b4d00 con 0x7f4f2015b960 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 0x7f4f2015d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 0x7f4f2015d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52270/0 (socket says 172.21.15.99:52270) 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 -- 172.21.15.99:0/542713866 learned_addr learned my addr 172.21.15.99:0/542713866 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f25686640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f200b0e10 0x7f4f200cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f25e87640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f2015b960 0x7f4f20155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 -- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f2015b960 msgr2=0x7f4f20155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f2015b960 0x7f4f20155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 -- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f200b0e10 msgr2=0x7f4f200cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f200b0e10 0x7f4f200cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 -- 172.21.15.99:0/542713866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20156270 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f25686640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f200b0e10 0x7f4f200cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f25e87640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f2015b960 0x7f4f20155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.487+0000 7f4f24e85640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 0x7f4f2015d1b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000c400 tx=0x7f4f1000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.488+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f100052d0 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.488+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f10005490 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.488+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2015d870 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.488+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10007260 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.488+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2015de30 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.489+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f20074f60 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.489+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4f10007400 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.489+0000 7f4f15ffb640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f0406b4d0 0x7f4f0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.489+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f4f10089ae0 con 0x7f4f200d0ad0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.490+0000 7f4f25686640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f0406b4d0 0x7f4f0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.490+0000 7f4f25686640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f0406b4d0 0x7f4f0406d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c036000 tx=0x7f4f1c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.494+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4f10058600 con 0x7f4f200d0ad0 2024-03-04T10:04:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.665+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f2015b3b0 con 0x7f4f200d0ad0 2024-03-04T10:04:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.666+0000 7f4f15ffb640 1 -- 172.21.15.99:0/542713866 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7f4f10057f00 con 0x7f4f200d0ad0 2024-03-04T10:04:21.667 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:21.668 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f0406b4d0 msgr2=0x7f4f0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f0406b4d0 0x7f4f0406d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c036000 tx=0x7f4f1c03c040 comp rx=0 tx=0).stop 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 msgr2=0x7f4f2015d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 0x7f4f2015d1b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000c400 tx=0x7f4f1000c8c0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 shutdown_connections 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f0406b4d0 0x7f4f0406d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f2015b960 0x7f4f20155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f200d0ad0 0x7f4f2015d1b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 --2- 172.21.15.99:0/542713866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f200b0e10 0x7f4f200cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 >> 172.21.15.99:0/542713866 conn(0x7f4f2006f5f0 msgr2=0x7f4f200b5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 shutdown_connections 2024-03-04T10:04:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.668+0000 7f4f278ea640 1 -- 172.21.15.99:0/542713866 wait complete. 2024-03-04T10:04:21.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:21.681 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:21.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.783+0000 7f45494f1640 1 Processor -- start 2024-03-04T10:04:21.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.783+0000 7f45494f1640 1 -- start start 2024-03-04T10:04:21.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45494f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f45440b48a0 0x7f45440b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45494f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 0x7f45440b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45494f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454406d8a0 con 0x7f454415a810 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45494f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454406db80 con 0x7f45440b48a0 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45494f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454406de60 con 0x7f454406d3c0 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45437fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 0x7f45440b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f453bfff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f45440b48a0 0x7f45440b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45437fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 0x7f45440b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:41576/0 (socket says 172.21.15.99:41576) 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45437fe640 1 -- 172.21.15.99:0/3812162406 learned_addr learned my addr 172.21.15.99:0/3812162406 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f4542ffd640 1 --1- 172.21.15.99:0/3812162406 >> v1:172.21.15.162:6789/0 conn(0x7f454415a810 0x7f45440b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:44980/0 (socket says 172.21.15.99:44980) 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f453bfff640 1 -- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 msgr2=0x7f45440b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f453bfff640 1 --2- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 0x7f45440b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f453bfff640 1 -- 172.21.15.99:0/3812162406 >> v1:172.21.15.162:6789/0 conn(0x7f454415a810 legacy=0x7f45440b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f453bfff640 1 -- 172.21.15.99:0/3812162406 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454406e140 con 0x7f45440b48a0 2024-03-04T10:04:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.784+0000 7f45437fe640 1 --2- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 0x7f45440b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.785+0000 7f453bfff640 1 --2- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45440b48a0 0x7f45440b29f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f452c0097a0 tx=0x7f452c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=312f13204e65dea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.785+0000 7f45427fc640 1 -- 172.21.15.99:0/3812162406 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c035020 con 0x7f45440b48a0 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.785+0000 7f45427fc640 1 -- 172.21.15.99:0/3812162406 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f452c031830 con 0x7f45440b48a0 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.785+0000 7f45427fc640 1 -- 172.21.15.99:0/3812162406 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452c031c00 con 0x7f45440b48a0 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 -- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45440b48a0 msgr2=0x7f45440b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 --2- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45440b48a0 0x7f45440b29f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f452c0097a0 tx=0x7f452c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 -- 172.21.15.99:0/3812162406 shutdown_connections 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 --2- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f454406d3c0 0x7f45440b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 --2- 172.21.15.99:0/3812162406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f45440b48a0 0x7f45440b29f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 -- 172.21.15.99:0/3812162406 >> 172.21.15.99:0/3812162406 conn(0x7f454406acd0 msgr2=0x7f454406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 -- 172.21.15.99:0/3812162406 shutdown_connections 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.786+0000 7f45494f1640 1 -- 172.21.15.99:0/3812162406 wait complete. 2024-03-04T10:04:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.787+0000 7f45494f1640 1 Processor -- start 2024-03-04T10:04:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.787+0000 7f45494f1640 1 -- start start 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f454406d3c0 0x7f4544155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f45440b48a0 0x7f45440a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 0x7f45440cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440cc130 con 0x7f454415a810 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440cbfb0 con 0x7f454406d3c0 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4544158ba0 con 0x7f45440b48a0 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 0x7f45440cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 0x7f45440cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52284/0 (socket says 172.21.15.99:52284) 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 -- 172.21.15.99:0/1246605 learned_addr learned my addr 172.21.15.99:0/1246605 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f4542ffd640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f454406d3c0 0x7f4544155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f453bfff640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f45440b48a0 0x7f45440a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 -- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f45440b48a0 msgr2=0x7f45440a9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f45440b48a0 0x7f45440a9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 -- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f454406d3c0 msgr2=0x7f4544155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f454406d3c0 0x7f4544155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 -- 172.21.15.99:0/1246605 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440a8670 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f4542ffd640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f454406d3c0 0x7f4544155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45437fe640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 0x7f45440cec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f453c00ef90 tx=0x7f453c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c00cde0 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c003990 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45440a8950 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.788+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c003d80 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.789+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45441563e0 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.789+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4544156670 con 0x7f454415a810 2024-03-04T10:04:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.790+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f453c012050 con 0x7f454415a810 2024-03-04T10:04:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.790+0000 7f453b7fe640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f452406b4d0 0x7f452406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.790+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f453c08aa30 con 0x7f454415a810 2024-03-04T10:04:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.790+0000 7f4542ffd640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f452406b4d0 0x7f452406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.791+0000 7f4542ffd640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f452406b4d0 0x7f452406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4534007fd0 tx=0x7f4534007d80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:21.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.794+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f453c010040 con 0x7f454415a810 2024-03-04T10:04:21.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.965+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4544074f60 con 0x7f454415a810 2024-03-04T10:04:21.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.965+0000 7f453b7fe640 1 -- 172.21.15.99:0/1246605 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f453c059550 con 0x7f454415a810 2024-03-04T10:04:21.966 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:21.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f452406b4d0 msgr2=0x7f452406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f452406b4d0 0x7f452406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4534007fd0 tx=0x7f4534007d80 comp rx=0 tx=0).stop 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 msgr2=0x7f45440cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 0x7f45440cec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f453c00ef90 tx=0x7f453c00c490 comp rx=0 tx=0).stop 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 shutdown_connections 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f452406b4d0 0x7f452406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f454415a810 0x7f45440cec00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f45440b48a0 0x7f45440a9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 --2- 172.21.15.99:0/1246605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f454406d3c0 0x7f4544155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 >> 172.21.15.99:0/1246605 conn(0x7f454406acd0 msgr2=0x7f4544071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 shutdown_connections 2024-03-04T10:04:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:21.967+0000 7f45494f1640 1 -- 172.21.15.99:0/1246605 wait complete. 2024-03-04T10:04:21.979 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.081+0000 7f3d98f37640 1 Processor -- start 2024-03-04T10:04:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.081+0000 7f3d98f37640 1 -- start start 2024-03-04T10:04:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d98f37640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415bb70 0x7f3d940b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d98f37640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3d9415f660 0x7f3d940d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d98f37640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9415bf50 con 0x7f3d940d1cc0 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d98f37640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d940cf890 con 0x7f3d9415f660 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d98f37640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9415fb60 con 0x7f3d9415bb70 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3d9415f660 0x7f3d940d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3d9415f660 0x7f3d940d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52298/0 (socket says 172.21.15.99:52298) 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3131089698 learned_addr learned my addr 172.21.15.99:0/3131089698 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3131089698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415bb70 msgr2=0x7f3d940b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 --2- 172.21.15.99:0/3131089698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415bb70 0x7f3d940b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3131089698 >> v1:172.21.15.162:6789/0 conn(0x7f3d940d1cc0 legacy=0x7f3d940ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.082+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3131089698 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9415fe40 con 0x7f3d9415f660 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.083+0000 7f3d8bfff640 1 --2- 172.21.15.99:0/3131089698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415f660 0x7f3d940d1780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009870 tx=0x7f3d7c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=849baa537d2a4a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.083+0000 7f3d91d9b640 1 -- 172.21.15.99:0/3131089698 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0317c0 con 0x7f3d9415f660 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.083+0000 7f3d91d9b640 1 -- 172.21.15.99:0/3131089698 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c037070 con 0x7f3d9415f660 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.083+0000 7f3d91d9b640 1 -- 172.21.15.99:0/3131089698 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c031b80 con 0x7f3d9415f660 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 -- 172.21.15.99:0/3131089698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415f660 msgr2=0x7f3d940d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3131089698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415f660 0x7f3d940d1780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009870 tx=0x7f3d7c02b860 comp rx=0 tx=0).stop 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 -- 172.21.15.99:0/3131089698 shutdown_connections 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3131089698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415f660 0x7f3d940d1780 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3131089698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415bb70 0x7f3d940b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 -- 172.21.15.99:0/3131089698 >> 172.21.15.99:0/3131089698 conn(0x7f3d940b33a0 msgr2=0x7f3d940b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 -- 172.21.15.99:0/3131089698 shutdown_connections 2024-03-04T10:04:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.084+0000 7f3d98f37640 1 -- 172.21.15.99:0/3131089698 wait complete. 2024-03-04T10:04:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.085+0000 7f3d98f37640 1 Processor -- start 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.085+0000 7f3d98f37640 1 -- start start 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d940d1cc0 0x7f3d941f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 0x7f3d941f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415f660 0x7f3d9415e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940b0f80 con 0x7f3d9415bb70 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94160400 con 0x7f3d940d1cc0 2024-03-04T10:04:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940b1100 con 0x7f3d9415f660 2024-03-04T10:04:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 0x7f3d941f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 0x7f3d941f4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52300/0 (socket says 172.21.15.99:52300) 2024-03-04T10:04:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3270158169 learned_addr learned my addr 172.21.15.99:0/3270158169 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d92d9d640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415f660 0x7f3d9415e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415f660 msgr2=0x7f3d9415e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d9259c640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d940d1cc0 0x7f3d941f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415f660 0x7f3d9415e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d940d1cc0 msgr2=0x7f3d941f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d940d1cc0 0x7f3d941f1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 -- 172.21.15.99:0/3270158169 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9415c2c0 con 0x7f3d9415bb70 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d92d9d640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415f660 0x7f3d9415e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d9259c640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d940d1cc0 0x7f3d941f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8bfff640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 0x7f3d941f4740 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000c00 tx=0x7f3d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c032860 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c032a20 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c032e10 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.086+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9415c5a0 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.087+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9415cb60 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.087+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d94074f60 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.088+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3d7c04c060 con 0x7f3d9415bb70 2024-03-04T10:04:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.088+0000 7f3d8affd640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d7406b4d0 0x7f3d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.088+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f3d7c0abe40 con 0x7f3d9415bb70 2024-03-04T10:04:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.088+0000 7f3d9259c640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d7406b4d0 0x7f3d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.089+0000 7f3d9259c640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d7406b4d0 0x7f3d7406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006fd0 tx=0x7f3d84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.093+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3d7c040070 con 0x7f3d9415bb70 2024-03-04T10:04:22.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.264+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d940b1720 con 0x7f3d9415bb70 2024-03-04T10:04:22.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.264+0000 7f3d8affd640 1 -- 172.21.15.99:0/3270158169 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3d7c04c350 con 0x7f3d9415bb70 2024-03-04T10:04:22.265 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d7406b4d0 msgr2=0x7f3d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d7406b4d0 0x7f3d7406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3d84006fd0 tx=0x7f3d84008040 comp rx=0 tx=0).stop 2024-03-04T10:04:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 msgr2=0x7f3d941f4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 0x7f3d941f4740 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000c00 tx=0x7f3d7c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 shutdown_connections 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d7406b4d0 0x7f3d7406d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d9415f660 0x7f3d9415e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d9415bb70 0x7f3d941f4740 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 --2- 172.21.15.99:0/3270158169 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d940d1cc0 0x7f3d941f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 >> 172.21.15.99:0/3270158169 conn(0x7f3d940b33a0 msgr2=0x7f3d940b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 shutdown_connections 2024-03-04T10:04:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.266+0000 7f3d98f37640 1 -- 172.21.15.99:0/3270158169 wait complete. 2024-03-04T10:04:22.278 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.380+0000 7fc62d60b640 1 Processor -- start 2024-03-04T10:04:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.380+0000 7fc62d60b640 1 -- start start 2024-03-04T10:04:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc62d60b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc62815bc50 0x7fc6280cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc62d60b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 0x7fc6280cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc62d60b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6280cc7a0 con 0x7fc6280cdd80 2024-03-04T10:04:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc62d60b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6280cca80 con 0x7fc62815bc50 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc62d60b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280ccd60 con 0x7fc6280adc40 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc61e7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc6280cdd80 0x7fc6280ad530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56336/0 (socket says 172.21.15.99:56336) 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc61e7fc640 1 -- 172.21.15.99:0/918034472 learned_addr learned my addr 172.21.15.99:0/918034472 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc626ffd640 1 --2- 172.21.15.99:0/918034472 >> v2:172.21.15.162:3300/0 conn(0x7fc62815bc50 0x7fc6280cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6277fe640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 0x7fc6280cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110241429 0 0) 0x7fc6280cc7a0 con 0x7fc6280cdd80 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc614003520 con 0x7fc6280cdd80 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008375344 0 0) 0x7fc614003520 con 0x7fc6280cdd80 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 msgr2=0x7fc6280cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 0x7fc6280cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 >> v2:172.21.15.162:3300/0 conn(0x7fc62815bc50 msgr2=0x7fc6280cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 --2- 172.21.15.99:0/918034472 >> v2:172.21.15.162:3300/0 conn(0x7fc62815bc50 0x7fc6280cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6280cd040 con 0x7fc6280cdd80 2024-03-04T10:04:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fc610001670 con 0x7fc6280cdd80 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc626ffd640 1 --2- 172.21.15.99:0/918034472 >> v2:172.21.15.162:3300/0 conn(0x7fc62815bc50 0x7fc6280cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 >> v1:172.21.15.99:6789/0 conn(0x7fc6280cdd80 legacy=0x7fc6280ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc614003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.381+0000 7fc6267fc640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc614004800 0x7fc614004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc6267fc640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 0x7fc6140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280cca80 con 0x7fc6140039b0 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280ccd60 con 0x7fc614004800 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61400b9c0 con 0x7fc6280adc40 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc626ffd640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc614003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc61e7fc640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc614004800 0x7fc614004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc6277fe640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 0x7fc6140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc626ffd640 1 -- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 msgr2=0x7fc6140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc626ffd640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6280adc40 0x7fc6140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc626ffd640 1 -- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc614004800 msgr2=0x7fc614004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc626ffd640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc614004800 0x7fc614004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc626ffd640 1 -- 172.21.15.99:0/918034472 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614003520 con 0x7fc6140039b0 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.382+0000 7fc61e7fc640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc614004800 0x7fc614004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc626ffd640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc614003dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc618009c90 tx=0x7fc614005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabc15bc59811e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618039070 con 0x7fc6140039b0 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61802c980 con 0x7fc6140039b0 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc6267fc640 1 -- 172.21.15.99:0/918034472 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618009860 con 0x7fc6140039b0 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc62d60b640 1 -- 172.21.15.99:0/918034472 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 msgr2=0x7fc614003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc62d60b640 1 --2- 172.21.15.99:0/918034472 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc614003dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc618009c90 tx=0x7fc614005830 comp rx=0 tx=0).stop 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc6277fe640 1 -- 172.21.15.99:0/918034472 reap_dead start 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc62d60b640 1 -- 172.21.15.99:0/918034472 shutdown_connections 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc62d60b640 1 -- 172.21.15.99:0/918034472 >> 172.21.15.99:0/918034472 conn(0x7fc6280b2250 msgr2=0x7fc6280b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc62d60b640 1 -- 172.21.15.99:0/918034472 shutdown_connections 2024-03-04T10:04:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.383+0000 7fc62d60b640 1 -- 172.21.15.99:0/918034472 wait complete. 2024-03-04T10:04:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.384+0000 7fc62d60b640 1 Processor -- start 2024-03-04T10:04:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 -- start start 2024-03-04T10:04:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc62815c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc62815c570 0x7fc62815dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc62815f0c0 0x7fc62815e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280cc4b0 con 0x7fc6140039b0 2024-03-04T10:04:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280cc330 con 0x7fc62815f0c0 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc62d60b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62806acd0 con 0x7fc62815c570 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc626ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc62815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc626ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc62815c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52320/0 (socket says 172.21.15.99:52320) 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc626ffd640 1 -- 172.21.15.99:0/908885302 learned_addr learned my addr 172.21.15.99:0/908885302 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc6277fe640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc62815f0c0 0x7fc62815e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.385+0000 7fc61e7fc640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc62815c570 0x7fc62815dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc626ffd640 1 -- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc62815c570 msgr2=0x7fc62815dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc626ffd640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc62815c570 0x7fc62815dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc626ffd640 1 -- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc62815f0c0 msgr2=0x7fc62815e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc626ffd640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc62815f0c0 0x7fc62815e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc626ffd640 1 -- 172.21.15.99:0/908885302 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6280c92d0 con 0x7fc6140039b0 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc626ffd640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc62815c030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc61802c530 tx=0x7fc618004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618039050 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc618045070 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6280c95b0 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618009860 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.386+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6280c9b70 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.387+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc628074f60 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.387+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc618040450 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.388+0000 7fc61ffff640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc60806b4d0 0x7fc60806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.388+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc6180ac250 con 0x7fc6140039b0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.388+0000 7fc61e7fc640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc60806b4d0 0x7fc60806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.389+0000 7fc61e7fc640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc60806b4d0 0x7fc60806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc628158030 tx=0x7fc610009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.392+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc61804e020 con 0x7fc6140039b0 2024-03-04T10:04:22.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.561+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc62806d460 con 0x7fc6140039b0 2024-03-04T10:04:22.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.561+0000 7fc61ffff640 1 -- 172.21.15.99:0/908885302 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc61807ace0 con 0x7fc6140039b0 2024-03-04T10:04:22.562 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:22.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc60806b4d0 msgr2=0x7fc60806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc60806b4d0 0x7fc60806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc628158030 tx=0x7fc610009040 comp rx=0 tx=0).stop 2024-03-04T10:04:22.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 msgr2=0x7fc62815c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc62815c030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc61802c530 tx=0x7fc618004730 comp rx=0 tx=0).stop 2024-03-04T10:04:22.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 shutdown_connections 2024-03-04T10:04:22.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc60806b4d0 0x7fc60806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc62815f0c0 0x7fc62815e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc62815c570 0x7fc62815dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 --2- 172.21.15.99:0/908885302 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6140039b0 0x7fc62815c030 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 >> 172.21.15.99:0/908885302 conn(0x7fc6280b2250 msgr2=0x7fc6280cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 shutdown_connections 2024-03-04T10:04:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.563+0000 7fc62d60b640 1 -- 172.21.15.99:0/908885302 wait complete. 2024-03-04T10:04:22.576 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:22.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.678+0000 7f8cfed16640 1 Processor -- start 2024-03-04T10:04:22.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.678+0000 7f8cfed16640 1 -- start start 2024-03-04T10:04:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfed16640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b5700 0x7f8cf80d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfed16640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8cf80d34a0 0x7f8cf806c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfed16640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf80d2400 con 0x7f8cf80b9410 2024-03-04T10:04:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfed16640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf80d26e0 con 0x7f8cf80d34a0 2024-03-04T10:04:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfed16640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80d29c0 con 0x7f8cf80b5700 2024-03-04T10:04:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8cf80d34a0 0x7f8cf806c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8cf80d34a0 0x7f8cf806c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52330/0 (socket says 172.21.15.99:52330) 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 -- 172.21.15.99:0/1831738137 learned_addr learned my addr 172.21.15.99:0/1831738137 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfd2b3640 1 --1- 172.21.15.99:0/1831738137 >> v1:172.21.15.99:6789/0 conn(0x7f8cf80b9410 0x7f8cf806c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56338/0 (socket says 172.21.15.99:56338) 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cfcab2640 1 --2- 172.21.15.99:0/1831738137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b5700 0x7f8cf80d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 -- 172.21.15.99:0/1831738137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b5700 msgr2=0x7f8cf80d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 --2- 172.21.15.99:0/1831738137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b5700 0x7f8cf80d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 -- 172.21.15.99:0/1831738137 >> v1:172.21.15.99:6789/0 conn(0x7f8cf80b9410 legacy=0x7f8cf806c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf7fff640 1 -- 172.21.15.99:0/1831738137 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf80d2ca0 con 0x7f8cf80d34a0 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.679+0000 7f8cf77fe640 1 -- 172.21.15.99:0/1831738137 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748205411 0 0) 0x7f8cf80d2400 con 0x7f8cf80b9410 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.680+0000 7f8cf7fff640 1 --2- 172.21.15.99:0/1831738137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf806c0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009870 tx=0x7f8ce402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25dc66a13a3ab7ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.680+0000 7f8cf77fe640 1 -- 172.21.15.99:0/1831738137 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce4035020 con 0x7f8cf80d34a0 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.680+0000 7f8cf77fe640 1 -- 172.21.15.99:0/1831738137 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce4031640 con 0x7f8cf80d34a0 2024-03-04T10:04:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.680+0000 7f8cf77fe640 1 -- 172.21.15.99:0/1831738137 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ce402d040 con 0x7f8cf80d34a0 2024-03-04T10:04:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 -- 172.21.15.99:0/1831738137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 msgr2=0x7f8cf806c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1831738137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf806c0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4009870 tx=0x7f8ce402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 -- 172.21.15.99:0/1831738137 shutdown_connections 2024-03-04T10:04:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1831738137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf806c0e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1831738137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b5700 0x7f8cf80d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 -- 172.21.15.99:0/1831738137 >> 172.21.15.99:0/1831738137 conn(0x7f8cf80d0490 msgr2=0x7f8cf80d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 -- 172.21.15.99:0/1831738137 shutdown_connections 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.681+0000 7f8cfed16640 1 -- 172.21.15.99:0/1831738137 wait complete. 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.682+0000 7f8cfed16640 1 Processor -- start 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.682+0000 7f8cfed16640 1 -- start start 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf80b5700 0x7f8cf8159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b9410 0x7f8cf814dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf80c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf806ceb0 con 0x7f8cf80d34a0 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf806cd30 con 0x7f8cf80b5700 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf80b5ae0 con 0x7f8cf80b9410 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf80c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf80c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52342/0 (socket says 172.21.15.99:52342) 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 -- 172.21.15.99:0/1826939682 learned_addr learned my addr 172.21.15.99:0/1826939682 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfcab2640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf80b5700 0x7f8cf8159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cf7fff640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b9410 0x7f8cf814dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 -- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b9410 msgr2=0x7f8cf814dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b9410 0x7f8cf814dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 -- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf80b5700 msgr2=0x7f8cf8159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf80b5700 0x7f8cf8159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 -- 172.21.15.99:0/1826939682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf8157e00 con 0x7f8cf80d34a0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfd2b3640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf80c9520 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000c7d0 tx=0x7f8cf000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0018070 con 0x7f8cf80d34a0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf000a640 con 0x7f8cf80d34a0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf000aa70 con 0x7f8cf80d34a0 2024-03-04T10:04:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf81f73c0 con 0x7f8cf80d34a0 2024-03-04T10:04:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.683+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf81f7980 con 0x7f8cf80d34a0 2024-03-04T10:04:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.685+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf80cf540 con 0x7f8cf80d34a0 2024-03-04T10:04:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.685+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8cf001b070 con 0x7f8cf80d34a0 2024-03-04T10:04:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.685+0000 7f8cf57fa640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cdc06b4d0 0x7f8cdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.685+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f8cf0089a90 con 0x7f8cf80d34a0 2024-03-04T10:04:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.686+0000 7f8cfcab2640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cdc06b4d0 0x7f8cdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.686+0000 7f8cfcab2640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cdc06b4d0 0x7f8cdc06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8cec006fd0 tx=0x7f8cec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.690+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cf80cf540 con 0x7f8cf80d34a0 2024-03-04T10:04:22.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.860+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8cf8074f60 con 0x7f8cf80d34a0 2024-03-04T10:04:22.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.861+0000 7f8cf57fa640 1 -- 172.21.15.99:0/1826939682 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8cf0057f60 con 0x7f8cf80d34a0 2024-03-04T10:04:22.863 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:22.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cdc06b4d0 msgr2=0x7f8cdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cdc06b4d0 0x7f8cdc06d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8cec006fd0 tx=0x7f8cec008040 comp rx=0 tx=0).stop 2024-03-04T10:04:22.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 msgr2=0x7f8cf80c9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf80c9520 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000c7d0 tx=0x7f8cf000cc90 comp rx=0 tx=0).stop 2024-03-04T10:04:22.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 shutdown_connections 2024-03-04T10:04:22.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cdc06b4d0 0x7f8cdc06d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf80d34a0 0x7f8cf80c9520 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf80b9410 0x7f8cf814dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 --2- 172.21.15.99:0/1826939682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf80b5700 0x7f8cf8159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 >> 172.21.15.99:0/1826939682 conn(0x7f8cf80d0490 msgr2=0x7f8cf80b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 shutdown_connections 2024-03-04T10:04:22.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.863+0000 7f8cfed16640 1 -- 172.21.15.99:0/1826939682 wait complete. 2024-03-04T10:04:22.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:22.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.975+0000 7f8d63a61640 1 Processor -- start 2024-03-04T10:04:22.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.976+0000 7f8d63a61640 1 -- start start 2024-03-04T10:04:22.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.976+0000 7f8d63a61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d5c0d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.976+0000 7f8d63a61640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0cfb40 con 0x7f8d5c0d0e40 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.976+0000 7f8d63a61640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0cfe20 con 0x7f8d5c0cc690 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.976+0000 7f8d63a61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0d2880 con 0x7f8d5c15bbb0 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d60ffc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8d5c0d0e40 0x7f8d5c0cbf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56346/0 (socket says 172.21.15.99:56346) 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d60ffc640 1 -- 172.21.15.99:0/4145482363 learned_addr learned my addr 172.21.15.99:0/4145482363 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d617fd640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d5c0d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615488307 0 0) 0x7f8d5c0cfb40 con 0x7f8d5c0d0e40 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d48003520 con 0x7f8d5c0d0e40 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2843682763 0 0) 0x7f8d48003520 con 0x7f8d5c0d0e40 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 msgr2=0x7f8d5c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d5c0d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 >> v1:172.21.15.162:6789/0 conn(0x7f8d5c0cc690 legacy=0x7f8d5c0d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c0d0100 con 0x7f8d5c0d0e40 2024-03-04T10:04:22.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8d4c001670 con 0x7f8d5c0d0e40 2024-03-04T10:04:22.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 >> v1:172.21.15.99:6789/0 conn(0x7f8d5c0d0e40 legacy=0x7f8d5c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d480039e0 0x7f8d48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d48004800 0x7f8d48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0d2880 con 0x7f8d48004800 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4800ba70 con 0x7f8d480039e0 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4800bc70 con 0x7f8d5c15bbb0 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d60ffc640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d48004800 0x7f8d48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d617fd640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d480039e0 0x7f8d48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.977+0000 7f8d61ffe640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d60ffc640 1 -- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 msgr2=0x7f8d480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d60ffc640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d60ffc640 1 -- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d480039e0 msgr2=0x7f8d48003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d60ffc640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d480039e0 0x7f8d48003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d60ffc640 1 -- 172.21.15.99:0/4145482363 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48003520 con 0x7f8d48004800 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d617fd640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d480039e0 0x7f8d48003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d61ffe640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c15bbb0 0x7f8d480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d60ffc640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d48004800 0x7f8d48004c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0d1240 tx=0x7f8d48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca32ec1c81b364b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c007d10 con 0x7f8d48004800 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c030070 con 0x7f8d48004800 2024-03-04T10:04:22.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.978+0000 7f8d53fff640 1 -- 172.21.15.99:0/4145482363 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c005d50 con 0x7f8d48004800 2024-03-04T10:04:22.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d63a61640 1 -- 172.21.15.99:0/4145482363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d48004800 msgr2=0x7f8d48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d63a61640 1 --2- 172.21.15.99:0/4145482363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d48004800 0x7f8d48004c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0d1240 tx=0x7f8d48005830 comp rx=0 tx=0).stop 2024-03-04T10:04:22.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d61ffe640 1 -- 172.21.15.99:0/4145482363 reap_dead start 2024-03-04T10:04:22.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d63a61640 1 -- 172.21.15.99:0/4145482363 shutdown_connections 2024-03-04T10:04:22.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d63a61640 1 -- 172.21.15.99:0/4145482363 >> 172.21.15.99:0/4145482363 conn(0x7f8d5c0b3480 msgr2=0x7f8d5c0b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d63a61640 1 -- 172.21.15.99:0/4145482363 shutdown_connections 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.979+0000 7f8d63a61640 1 -- 172.21.15.99:0/4145482363 wait complete. 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.980+0000 7f8d63a61640 1 Processor -- start 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.980+0000 7f8d63a61640 1 -- start start 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d63a61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d48004800 0x7f8d5c154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d63a61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c155d10 0x7f8d5c0b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d63a61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 0x7f8d5c0b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d63a61640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c15bf90 con 0x7f8d5c156190 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d63a61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0d23b0 con 0x7f8d48004800 2024-03-04T10:04:22.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d63a61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1554c0 con 0x7f8d5c155d10 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 0x7f8d5c0b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 0x7f8d5c0b5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52368/0 (socket says 172.21.15.99:52368) 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 -- 172.21.15.99:0/508490094 learned_addr learned my addr 172.21.15.99:0/508490094 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d60ffc640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c155d10 0x7f8d5c0b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d617fd640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d48004800 0x7f8d5c154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 -- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c155d10 msgr2=0x7f8d5c0b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c155d10 0x7f8d5c0b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 -- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d48004800 msgr2=0x7f8d5c154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d48004800 0x7f8d5c154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 -- 172.21.15.99:0/508490094 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c1566f0 con 0x7f8d5c156190 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d617fd640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d48004800 0x7f8d5c154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.981+0000 7f8d61ffe640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 0x7f8d5c0b5530 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0b4290 tx=0x7f8d5400e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.982+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54019070 con 0x7f8d5c156190 2024-03-04T10:04:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.982+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d540052e0 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.982+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54003830 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.982+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c1f1a90 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.982+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c1f2050 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.983+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c074f60 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.983+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8d5401c070 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.983+0000 7f8d51ffb640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.984+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f8d54089910 con 0x7f8d5c156190 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.984+0000 7f8d617fd640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.984+0000 7f8d617fd640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8d4006b4d0 0x7f8d4006d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0b67d0 tx=0x7f8d5800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:22.988+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8d54057d50 con 0x7f8d5c156190 2024-03-04T10:04:23.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.161+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d5c1f2220 con 0x7f8d5c156190 2024-03-04T10:04:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.162+0000 7f8d51ffb640 1 -- 172.21.15.99:0/508490094 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7f8d5405bdf0 con 0x7f8d5c156190 2024-03-04T10:04:23.163 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:23.163 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8d4006b4d0 msgr2=0x7f8d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8d4006b4d0 0x7f8d4006d990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0b67d0 tx=0x7f8d5800b040 comp rx=0 tx=0).stop 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 msgr2=0x7f8d5c0b5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 0x7f8d5c0b5530 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c0b4290 tx=0x7f8d5400e820 comp rx=0 tx=0).stop 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 shutdown_connections 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8d4006b4d0 0x7f8d4006d990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d5c156190 0x7f8d5c0b5530 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d5c155d10 0x7f8d5c0b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 --2- 172.21.15.99:0/508490094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d48004800 0x7f8d5c154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 >> 172.21.15.99:0/508490094 conn(0x7f8d5c0b3480 msgr2=0x7f8d5c0b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 shutdown_connections 2024-03-04T10:04:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.164+0000 7f8d63a61640 1 -- 172.21.15.99:0/508490094 wait complete. 2024-03-04T10:04:23.177 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:23.177 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:23.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.279+0000 7f9be6485640 1 Processor -- start 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.279+0000 7f9be6485640 1 -- start start 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.280+0000 7f9be6485640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9be00b9410 0x7f9be01622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.280+0000 7f9be6485640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be00d2400 con 0x7f9be00b5700 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.280+0000 7f9be6485640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be00d26e0 con 0x7f9be00d34a0 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.280+0000 7f9be6485640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be00d29c0 con 0x7f9be00b9410 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.280+0000 7f9bdffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9be00b5700 0x7f9be00d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56362/0 (socket says 172.21.15.99:56362) 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.280+0000 7f9bdffff640 1 -- 172.21.15.99:0/52167198 learned_addr learned my addr 172.21.15.99:0/52167198 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 --2- 172.21.15.99:0/52167198 >> v2:172.21.15.162:3300/0 conn(0x7f9be00b9410 0x7f9be01622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180219787 0 0) 0x7f9be00d2400 con 0x7f9be00b5700 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc8003520 con 0x7f9be00b5700 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806407626 0 0) 0x7f9be00d26e0 con 0x7f9be00d34a0 2024-03-04T10:04:23.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be00d2400 con 0x7f9be00d34a0 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253053105 0 0) 0x7f9bc8003520 con 0x7f9be00b5700 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 >> v2:172.21.15.162:3300/0 conn(0x7f9be00b9410 msgr2=0x7f9be01622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 --2- 172.21.15.99:0/52167198 >> v2:172.21.15.162:3300/0 conn(0x7f9be00b9410 0x7f9be01622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 >> v1:172.21.15.162:6789/0 conn(0x7f9be00d34a0 legacy=0x7f9be0161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be00d2ca0 con 0x7f9be00b5700 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f9bd0001670 con 0x7f9be00b5700 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 --2- 172.21.15.99:0/52167198 >> v2:172.21.15.162:3300/0 conn(0x7f9be00b9410 0x7f9be01622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 >> v1:172.21.15.99:6789/0 conn(0x7f9be00b5700 legacy=0x7f9be00d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc8003b40 0x7f9bc8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc8004800 0x7f9bc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc80051c0 0x7f9bc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00d29c0 con 0x7f9bc80051c0 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc800c0f0 con 0x7f9bc8003b40 2024-03-04T10:04:23.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc800c2f0 con 0x7f9bc8004800 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc80051c0 0x7f9bc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdffff640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc8003b40 0x7f9bc8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9bdf7fe640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc8004800 0x7f9bc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 -- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc8004800 msgr2=0x7f9bc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc8004800 0x7f9bc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 -- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc8003b40 msgr2=0x7f9bc8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc8003b40 0x7f9bc8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.281+0000 7f9be4a22640 1 -- 172.21.15.99:0/52167198 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8003520 con 0x7f9bc80051c0 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.282+0000 7f9be4a22640 1 -- 172.21.15.99:0/52167198 reap_dead start 2024-03-04T10:04:23.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.282+0000 7f9be4a22640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc80051c0 0x7f9bc800b9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e350 tx=0x7f9bc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beee1ae10897fab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.282+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8009de0 con 0x7f9bc80051c0 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.282+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd802ce40 con 0x7f9bc80051c0 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.282+0000 7f9bdeffd640 1 -- 172.21.15.99:0/52167198 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8003bd0 con 0x7f9bc80051c0 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 -- 172.21.15.99:0/52167198 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc80051c0 msgr2=0x7f9bc800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc80051c0 0x7f9bc800b9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e350 tx=0x7f9bc8005830 comp rx=0 tx=0).stop 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 -- 172.21.15.99:0/52167198 shutdown_connections 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 --2- 172.21.15.99:0/52167198 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc80051c0 0x7f9bc800b9c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 -- 172.21.15.99:0/52167198 >> 172.21.15.99:0/52167198 conn(0x7f9be00d0490 msgr2=0x7f9be00d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 -- 172.21.15.99:0/52167198 shutdown_connections 2024-03-04T10:04:23.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.283+0000 7f9be6485640 1 -- 172.21.15.99:0/52167198 wait complete. 2024-03-04T10:04:23.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.284+0000 7f9be6485640 1 Processor -- start 2024-03-04T10:04:23.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.284+0000 7f9be6485640 1 -- start start 2024-03-04T10:04:23.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be6485640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc80051c0 0x7f9be0156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be6485640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 0x7f9be0158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be6485640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9be00caf60 0x7f9be0158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be6485640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00d1b00 con 0x7f9be00caae0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be6485640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00b5cf0 con 0x7f9bc80051c0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be6485640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be00d1c80 con 0x7f9be00caf60 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 0x7f9be0158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 0x7f9be0158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52378/0 (socket says 172.21.15.99:52378) 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 -- 172.21.15.99:0/3644702315 learned_addr learned my addr 172.21.15.99:0/3644702315 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9be4a22640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9be00caf60 0x7f9be0158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdffff640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc80051c0 0x7f9be0156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 -- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9be00caf60 msgr2=0x7f9be0158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9be00caf60 0x7f9be0158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 -- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc80051c0 msgr2=0x7f9be0156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc80051c0 0x7f9be0156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.285+0000 7f9bdf7fe640 1 -- 172.21.15.99:0/3644702315 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be00cb4c0 con 0x7f9be00caae0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.286+0000 7f9bdf7fe640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 0x7f9be0158990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.286+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd401a070 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.286+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd4005110 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.286+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be00cb7a0 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.286+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4007030 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.286+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be0158470 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.287+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be00b33f0 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.287+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9bd401d070 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.288+0000 7f9bdcff9640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.288+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f9bd4089c30 con 0x7f9be00caae0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.288+0000 7f9bdffff640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.289+0000 7f9bdffff640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0001eb0 tx=0x7f9bd0001e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.292+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9bd4058100 con 0x7f9be00caae0 2024-03-04T10:04:23.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.463+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9be0074f60 con 0x7f9be00caae0 2024-03-04T10:04:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.464+0000 7f9bdcff9640 1 -- 172.21.15.99:0/3644702315 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9bd405c1a0 con 0x7f9be00caae0 2024-03-04T10:04:23.465 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.465+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bcc06b500 msgr2=0x7f9bcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.465+0000 7f9be6485640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0001eb0 tx=0x7f9bd0001e40 comp rx=0 tx=0).stop 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.465+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 msgr2=0x7f9be0158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.465+0000 7f9be6485640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 0x7f9be0158990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400f770 tx=0x7f9bd400fc30 comp rx=0 tx=0).stop 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.465+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 shutdown_connections 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bcc06b500 0x7f9bcc06d9c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9be00caf60 0x7f9be0158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9be00caae0 0x7f9be0158990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 --2- 172.21.15.99:0/3644702315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc80051c0 0x7f9be0156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 >> 172.21.15.99:0/3644702315 conn(0x7f9be00d0490 msgr2=0x7f9be00b7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 shutdown_connections 2024-03-04T10:04:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.466+0000 7f9be6485640 1 -- 172.21.15.99:0/3644702315 wait complete. 2024-03-04T10:04:23.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:23.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.581+0000 7f6f2b91b640 1 Processor -- start 2024-03-04T10:04:23.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.581+0000 7f6f2b91b640 1 -- start start 2024-03-04T10:04:23.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f2b91b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 0x7f6f240701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f2b91b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f240aded0 con 0x7f6f2415e030 2024-03-04T10:04:23.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f2b91b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2406fbe0 con 0x7f6f240d1230 2024-03-04T10:04:23.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f2b91b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2406fec0 con 0x7f6f240cf0e0 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f296b7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6f2415e030 0x7f6f240ce9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57324/0 (socket says 172.21.15.99:57324) 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f296b7640 1 -- 172.21.15.99:0/4279556642 learned_addr learned my addr 172.21.15.99:0/4279556642 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 0x7f6f240701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695149414 0 0) 0x7f6f240aded0 con 0x7f6f2415e030 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.582+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f10003520 con 0x7f6f2415e030 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 269887807 0 0) 0x7f6f10003520 con 0x7f6f2415e030 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 msgr2=0x7f6f240701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 0x7f6f240701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 >> v1:172.21.15.162:6789/0 conn(0x7f6f240d1230 legacy=0x7f6f240ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f240ae190 con 0x7f6f2415e030 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6f20001670 con 0x7f6f2415e030 2024-03-04T10:04:23.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 >> v1:172.21.15.99:6789/0 conn(0x7f6f2415e030 legacy=0x7f6f240ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f100039e0 0x7f6f10003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f10004800 0x7f6f10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 0x7f6f100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2406fec0 con 0x7f6f10004800 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1000dd10 con 0x7f6f100039e0 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1000df10 con 0x7f6f240cf0e0 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f10004800 0x7f6f10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f296b7640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f100039e0 0x7f6f10003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f29eb8640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 0x7f6f100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f28eb6640 1 -- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 msgr2=0x7f6f100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f240cf0e0 0x7f6f100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f28eb6640 1 -- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f100039e0 msgr2=0x7f6f10003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f100039e0 0x7f6f10003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.583+0000 7f6f28eb6640 1 -- 172.21.15.99:0/4279556642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f10003520 con 0x7f6f10004800 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.584+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f10004800 0x7f6f10004c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009ac0 tx=0x7f6f10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3508d08cd77c8d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.584+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14039070 con 0x7f6f10004800 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.584+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1402c6b0 con 0x7f6f10004800 2024-03-04T10:04:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.584+0000 7f6f1bfff640 1 -- 172.21.15.99:0/4279556642 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14009690 con 0x7f6f10004800 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f2b91b640 1 -- 172.21.15.99:0/4279556642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f10004800 msgr2=0x7f6f10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/4279556642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f10004800 0x7f6f10004c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009ac0 tx=0x7f6f10005830 comp rx=0 tx=0).stop 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f29eb8640 1 -- 172.21.15.99:0/4279556642 reap_dead start 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f2b91b640 1 -- 172.21.15.99:0/4279556642 shutdown_connections 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f2b91b640 1 -- 172.21.15.99:0/4279556642 >> 172.21.15.99:0/4279556642 conn(0x7f6f240d0710 msgr2=0x7f6f240d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f2b91b640 1 -- 172.21.15.99:0/4279556642 shutdown_connections 2024-03-04T10:04:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.585+0000 7f6f2b91b640 1 -- 172.21.15.99:0/4279556642 wait complete. 2024-03-04T10:04:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.586+0000 7f6f2b91b640 1 Processor -- start 2024-03-04T10:04:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.586+0000 7f6f2b91b640 1 -- start start 2024-03-04T10:04:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f10004800 0x7f6f2414f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 0x7f6f240c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f240b04d0 0x7f6f24158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2415e410 con 0x7f6f240b0070 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2406f8b0 con 0x7f6f240b04d0 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2415e590 con 0x7f6f10004800 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 0x7f6f240c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 0x7f6f240c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51066/0 (socket says 172.21.15.99:51066) 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 -- 172.21.15.99:0/136340529 learned_addr learned my addr 172.21.15.99:0/136340529 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f29eb8640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f240b04d0 0x7f6f24158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f296b7640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f10004800 0x7f6f2414f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 -- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f10004800 msgr2=0x7f6f2414f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f10004800 0x7f6f2414f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 -- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f240b04d0 msgr2=0x7f6f24158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f240b04d0 0x7f6f24158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 -- 172.21.15.99:0/136340529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2414f7f0 con 0x7f6f240b0070 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f29eb8640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f240b04d0 0x7f6f24158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f296b7640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f10004800 0x7f6f2414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f28eb6640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 0x7f6f240c77e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009ac0 tx=0x7f6f14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14039050 con 0x7f6f240b0070 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14045070 con 0x7f6f240b0070 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f240c81d0 con 0x7f6f240b0070 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.587+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f140044e0 con 0x7f6f240b0070 2024-03-04T10:04:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.588+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f240b5110 con 0x7f6f240b0070 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.588+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f240b4f00 con 0x7f6f240b0070 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.589+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6f14003830 con 0x7f6f240b0070 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.589+0000 7f6f19ffb640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f0806b4d0 0x7f6f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.590+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f6f140ac700 con 0x7f6f240b0070 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.590+0000 7f6f296b7640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f0806b4d0 0x7f6f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.590+0000 7f6f296b7640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f0806b4d0 0x7f6f0806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6f20001eb0 tx=0x7f6f20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.594+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6f1407b220 con 0x7f6f240b0070 2024-03-04T10:04:23.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.764+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f24074f60 con 0x7f6f240b0070 2024-03-04T10:04:23.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.764+0000 7f6f19ffb640 1 -- 172.21.15.99:0/136340529 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f1407ab20 con 0x7f6f240b0070 2024-03-04T10:04:23.766 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f0806b4d0 msgr2=0x7f6f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f0806b4d0 0x7f6f0806d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6f20001eb0 tx=0x7f6f20009040 comp rx=0 tx=0).stop 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 msgr2=0x7f6f240c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 0x7f6f240c77e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6f14009ac0 tx=0x7f6f14004730 comp rx=0 tx=0).stop 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 shutdown_connections 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6f0806b4d0 0x7f6f0806d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6f240b04d0 0x7f6f24158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6f240b0070 0x7f6f240c77e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 --2- 172.21.15.99:0/136340529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6f10004800 0x7f6f2414f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 >> 172.21.15.99:0/136340529 conn(0x7f6f240d0710 msgr2=0x7f6f240ae990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 shutdown_connections 2024-03-04T10:04:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.766+0000 7f6f2b91b640 1 -- 172.21.15.99:0/136340529 wait complete. 2024-03-04T10:04:23.778 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.881+0000 7f379cbb3640 1 Processor -- start 2024-03-04T10:04:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.881+0000 7f379cbb3640 1 -- start start 2024-03-04T10:04:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f379cbb3640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f37980ccfe0 0x7f379815db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f379cbb3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379806ae00 con 0x7f37980cc900 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f379cbb3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379806b0e0 con 0x7f37980cdd70 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f379cbb3640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37980ccde0 con 0x7f37980ccfe0 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f3795d9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f37980cc900 0x7f379815d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57326/0 (socket says 172.21.15.99:57326) 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f3795d9b640 1 -- 172.21.15.99:0/218528647 learned_addr learned my addr 172.21.15.99:0/218528647 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f3796d9d640 1 --2- 172.21.15.99:0/218528647 >> v2:172.21.15.162:3300/0 conn(0x7f37980ccfe0 0x7f379815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f379559a640 1 -- 172.21.15.99:0/218528647 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909334699 0 0) 0x7f379806ae00 con 0x7f37980cc900 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.882+0000 7f379559a640 1 -- 172.21.15.99:0/218528647 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003520 con 0x7f37980cc900 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.883+0000 7f3796d9d640 1 -- 172.21.15.99:0/218528647 >> v1:172.21.15.162:6789/0 conn(0x7f37980cdd70 legacy=0x7f37980cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.883+0000 7f3796d9d640 1 -- 172.21.15.99:0/218528647 >> v1:172.21.15.99:6789/0 conn(0x7f37980cc900 legacy=0x7f379815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.883+0000 7f3796d9d640 1 -- 172.21.15.99:0/218528647 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379806b3c0 con 0x7f37980ccfe0 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.883+0000 7f3796d9d640 1 --2- 172.21.15.99:0/218528647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980ccfe0 0x7f379815db20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3790009d90 tx=0x7f379002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d23195e7a20e3383 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.883+0000 7f379559a640 1 -- 172.21.15.99:0/218528647 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790039070 con 0x7f37980ccfe0 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379559a640 1 -- 172.21.15.99:0/218528647 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379002cc60 con 0x7f37980ccfe0 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379559a640 1 -- 172.21.15.99:0/218528647 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790034640 con 0x7f37980ccfe0 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 -- 172.21.15.99:0/218528647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980ccfe0 msgr2=0x7f379815db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 --2- 172.21.15.99:0/218528647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980ccfe0 0x7f379815db20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3790009d90 tx=0x7f379002beb0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 -- 172.21.15.99:0/218528647 shutdown_connections 2024-03-04T10:04:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 --2- 172.21.15.99:0/218528647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980ccfe0 0x7f379815db20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 -- 172.21.15.99:0/218528647 >> 172.21.15.99:0/218528647 conn(0x7f37980b22e0 msgr2=0x7f37980b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 -- 172.21.15.99:0/218528647 shutdown_connections 2024-03-04T10:04:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.884+0000 7f379cbb3640 1 -- 172.21.15.99:0/218528647 wait complete. 2024-03-04T10:04:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.885+0000 7f379cbb3640 1 Processor -- start 2024-03-04T10:04:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.885+0000 7f379cbb3640 1 -- start start 2024-03-04T10:04:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379cbb3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980cc900 0x7f37981f3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379cbb3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 0x7f37981e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379cbb3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37980cdd70 0x7f37980c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379cbb3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379815e1e0 con 0x7f37980ccfe0 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379cbb3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379815e060 con 0x7f37980cc900 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379cbb3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37980b2e90 con 0x7f37980cdd70 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f3795d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 0x7f37981e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f3795d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 0x7f37981e8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51082/0 (socket says 172.21.15.99:51082) 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f3795d9b640 1 -- 172.21.15.99:0/3132026988 learned_addr learned my addr 172.21.15.99:0/3132026988 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f379659c640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980cc900 0x7f37981f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.886+0000 7f3796d9d640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37980cdd70 0x7f37980c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3795d9b640 1 -- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37980cdd70 msgr2=0x7f37980c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3795d9b640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37980cdd70 0x7f37980c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3795d9b640 1 -- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980cc900 msgr2=0x7f37981f3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3795d9b640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980cc900 0x7f37981f3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3795d9b640 1 -- 172.21.15.99:0/3132026988 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37981e8b20 con 0x7f37980ccfe0 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3795d9b640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 0x7f37981e8140 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c4d0 tx=0x7f378c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c017070 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c005ce0 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37981e8e00 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c007270 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.887+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379815f330 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.888+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3798074f60 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.888+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f378c01a070 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.889+0000 7f3786ffd640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f377806b4d0 0x7f377806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.889+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f378c089880 con 0x7f37980ccfe0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.889+0000 7f379659c640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f377806b4d0 0x7f377806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.889+0000 7f379659c640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f377806b4d0 0x7f377806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f37880016e0 tx=0x7f3788009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:23.893+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f378c057d50 con 0x7f37980ccfe0 2024-03-04T10:04:24.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.061+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f37980b6710 con 0x7f37980ccfe0 2024-03-04T10:04:24.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.062+0000 7f3786ffd640 1 -- 172.21.15.99:0/3132026988 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f378c05bdf0 con 0x7f37980ccfe0 2024-03-04T10:04:24.063 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:24.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.063+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f377806b4d0 msgr2=0x7f377806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.063+0000 7f379cbb3640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f377806b4d0 0x7f377806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f37880016e0 tx=0x7f3788009040 comp rx=0 tx=0).stop 2024-03-04T10:04:24.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.063+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 msgr2=0x7f37981e8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.063+0000 7f379cbb3640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 0x7f37981e8140 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c4d0 tx=0x7f378c00c990 comp rx=0 tx=0).stop 2024-03-04T10:04:24.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 shutdown_connections 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f377806b4d0 0x7f377806d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37980cdd70 0x7f37980c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37980ccfe0 0x7f37981e8140 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 --2- 172.21.15.99:0/3132026988 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37980cc900 0x7f37981f3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 >> 172.21.15.99:0/3132026988 conn(0x7f37980b22e0 msgr2=0x7f37980cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 shutdown_connections 2024-03-04T10:04:24.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.064+0000 7f379cbb3640 1 -- 172.21.15.99:0/3132026988 wait complete. 2024-03-04T10:04:24.076 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:24.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.178+0000 7fbd52745640 1 Processor -- start 2024-03-04T10:04:24.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.178+0000 7fbd52745640 1 -- start start 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd52745640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd52745640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4c162450 con 0x7fbd4c15da20 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd52745640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4c0cc2d0 con 0x7fbd4c0cf660 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd52745640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0d0000 con 0x7fbd4c161df0 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd4bfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbd4c15da20 0x7fbd4c0ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57342/0 (socket says 172.21.15.99:57342) 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd4bfff640 1 -- 172.21.15.99:0/1072992134 learned_addr learned my addr 172.21.15.99:0/1072992134 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd50ce2640 1 --2- 172.21.15.99:0/1072992134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd4affd640 1 -- 172.21.15.99:0/1072992134 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395887797 0 0) 0x7fbd4c162450 con 0x7fbd4c15da20 2024-03-04T10:04:24.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.179+0000 7fbd4affd640 1 -- 172.21.15.99:0/1072992134 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd34003520 con 0x7fbd4c15da20 2024-03-04T10:04:24.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.180+0000 7fbd50ce2640 1 -- 172.21.15.99:0/1072992134 >> v1:172.21.15.162:6789/0 conn(0x7fbd4c0cf660 legacy=0x7fbd4c1616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.180+0000 7fbd4affd640 1 -- 172.21.15.99:0/1072992134 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306293673 0 0) 0x7fbd4c0cc2d0 con 0x7fbd4c0cf660 2024-03-04T10:04:24.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.180+0000 7fbd50ce2640 1 -- 172.21.15.99:0/1072992134 >> v1:172.21.15.99:6789/0 conn(0x7fbd4c15da20 legacy=0x7fbd4c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.180+0000 7fbd50ce2640 1 -- 172.21.15.99:0/1072992134 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c0cc5b0 con 0x7fbd4c161df0 2024-03-04T10:04:24.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd50ce2640 1 --2- 172.21.15.99:0/1072992134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0cfac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009dc0 tx=0x7fbd4402bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e073fd443f5891 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd4affd640 1 -- 172.21.15.99:0/1072992134 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44039070 con 0x7fbd4c161df0 2024-03-04T10:04:24.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd4affd640 1 -- 172.21.15.99:0/1072992134 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4402c7b0 con 0x7fbd4c161df0 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd4affd640 1 -- 172.21.15.99:0/1072992134 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c001670 con 0x7fbd4c161df0 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 -- 172.21.15.99:0/1072992134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 msgr2=0x7fbd4c0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 --2- 172.21.15.99:0/1072992134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0cfac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009dc0 tx=0x7fbd4402bc80 comp rx=0 tx=0).stop 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 -- 172.21.15.99:0/1072992134 shutdown_connections 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 --2- 172.21.15.99:0/1072992134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0cfac0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 -- 172.21.15.99:0/1072992134 >> 172.21.15.99:0/1072992134 conn(0x7fbd4c0b33a0 msgr2=0x7fbd4c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 -- 172.21.15.99:0/1072992134 shutdown_connections 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.181+0000 7fbd52745640 1 -- 172.21.15.99:0/1072992134 wait complete. 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.182+0000 7fbd52745640 1 Processor -- start 2024-03-04T10:04:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 -- start start 2024-03-04T10:04:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c0cf660 0x7fbd4c15ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 0x7fbd4c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c15de00 con 0x7fbd4c15da20 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c0cc920 con 0x7fbd4c0cf660 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd52745640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c06bde0 con 0x7fbd4c161df0 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 0x7fbd4c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 0x7fbd4c0c7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51090/0 (socket says 172.21.15.99:51090) 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.183+0000 7fbd4b7fe640 1 -- 172.21.15.99:0/1435583627 learned_addr learned my addr 172.21.15.99:0/1435583627 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd50ce2640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4bfff640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c0cf660 0x7fbd4c15ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4b7fe640 1 -- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 msgr2=0x7fbd4c0c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4b7fe640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4b7fe640 1 -- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c0cf660 msgr2=0x7fbd4c15ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4b7fe640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c0cf660 0x7fbd4c15ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4b7fe640 1 -- 172.21.15.99:0/1435583627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c0c4f20 con 0x7fbd4c15da20 2024-03-04T10:04:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd4b7fe640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 0x7fbd4c0c7ae0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c460 tx=0x7fbd4000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40005480 con 0x7fbd4c15da20 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd400035f0 con 0x7fbd4c15da20 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c0c5200 con 0x7fbd4c15da20 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd40003a00 con 0x7fbd4c15da20 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.184+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c0c57c0 con 0x7fbd4c15da20 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.185+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c074f60 con 0x7fbd4c15da20 2024-03-04T10:04:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.185+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbd40003ba0 con 0x7fbd4c15da20 2024-03-04T10:04:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.186+0000 7fbd48ff9640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd3806b670 0x7fbd3806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.186+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fbd4008a0b0 con 0x7fbd4c15da20 2024-03-04T10:04:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.186+0000 7fbd4bfff640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd3806b670 0x7fbd3806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.187+0000 7fbd4bfff640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd3806b670 0x7fbd3806db30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c001eb0 tx=0x7fbd3c001650 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.190+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbd40058580 con 0x7fbd4c15da20 2024-03-04T10:04:24.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.359+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbd4c06fbf0 con 0x7fbd4c15da20 2024-03-04T10:04:24.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.359+0000 7fbd48ff9640 1 -- 172.21.15.99:0/1435583627 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbd4005c620 con 0x7fbd4c15da20 2024-03-04T10:04:24.361 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:24.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd3806b670 msgr2=0x7fbd3806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd3806b670 0x7fbd3806db30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c001eb0 tx=0x7fbd3c001650 comp rx=0 tx=0).stop 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 msgr2=0x7fbd4c0c7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 0x7fbd4c0c7ae0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000c460 tx=0x7fbd4000c920 comp rx=0 tx=0).stop 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 shutdown_connections 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd3806b670 0x7fbd3806db30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4c161df0 0x7fbd4c0c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4c15da20 0x7fbd4c0c7ae0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 --2- 172.21.15.99:0/1435583627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd4c0cf660 0x7fbd4c15ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 >> 172.21.15.99:0/1435583627 conn(0x7fbd4c0b33a0 msgr2=0x7fbd4c06c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 shutdown_connections 2024-03-04T10:04:24.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.361+0000 7fbd52745640 1 -- 172.21.15.99:0/1435583627 wait complete. 2024-03-04T10:04:24.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 Processor -- start 2024-03-04T10:04:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 -- start start 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb11c15dcd0 0x7fb11c0ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c0cfeb0 0x7fb11c1618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0cf200 con 0x7fb11c161de0 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c0cc0c0 con 0x7fb11c15dcd0 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.474+0000 7fb121cd7640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c162230 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb11c15dcd0 0x7fb11c0ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb11c15dcd0 0x7fb11c0ce0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51104/0 (socket says 172.21.15.99:51104) 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 -- 172.21.15.99:0/2616032591 learned_addr learned my addr 172.21.15.99:0/2616032591 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb112ffd640 1 --2- 172.21.15.99:0/2616032591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c0cfeb0 0x7fb11c1618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 -- 172.21.15.99:0/2616032591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c0cfeb0 msgr2=0x7fb11c1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 --2- 172.21.15.99:0/2616032591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c0cfeb0 0x7fb11c1618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 -- 172.21.15.99:0/2616032591 >> v1:172.21.15.162:6789/0 conn(0x7fb11c161de0 legacy=0x7fb11c0d18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.475+0000 7fb11b7fe640 1 -- 172.21.15.99:0/2616032591 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0ce680 con 0x7fb11c15dcd0 2024-03-04T10:04:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb11b7fe640 1 --2- 172.21.15.99:0/2616032591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c15dcd0 0x7fb11c0ce0a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009870 tx=0x7fb10c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59f749bf8c6402c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb11affd640 1 -- 172.21.15.99:0/2616032591 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c034070 con 0x7fb11c15dcd0 2024-03-04T10:04:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb11affd640 1 -- 172.21.15.99:0/2616032591 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c030840 con 0x7fb11c15dcd0 2024-03-04T10:04:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb11affd640 1 -- 172.21.15.99:0/2616032591 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c030c10 con 0x7fb11c15dcd0 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 -- 172.21.15.99:0/2616032591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c15dcd0 msgr2=0x7fb11c0ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 --2- 172.21.15.99:0/2616032591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c15dcd0 0x7fb11c0ce0a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009870 tx=0x7fb10c02b980 comp rx=0 tx=0).stop 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 -- 172.21.15.99:0/2616032591 shutdown_connections 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 --2- 172.21.15.99:0/2616032591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c0cfeb0 0x7fb11c1618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 --2- 172.21.15.99:0/2616032591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c15dcd0 0x7fb11c0ce0a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 -- 172.21.15.99:0/2616032591 >> 172.21.15.99:0/2616032591 conn(0x7fb11c0b3480 msgr2=0x7fb11c0b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 -- 172.21.15.99:0/2616032591 shutdown_connections 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.476+0000 7fb121cd7640 1 -- 172.21.15.99:0/2616032591 wait complete. 2024-03-04T10:04:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.477+0000 7fb121cd7640 1 Processor -- start 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 -- start start 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 0x7fb11c160d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb11c15dcd0 0x7fb11c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c161de0 0x7fb11c15bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0cef20 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0cec40 con 0x7fb11c15dcd0 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb121cd7640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c06bff0 con 0x7fb11c161de0 2024-03-04T10:04:24.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb11b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 0x7fb11c160d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb11b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 0x7fb11c160d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51112/0 (socket says 172.21.15.99:51112) 2024-03-04T10:04:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.478+0000 7fb11b7fe640 1 -- 172.21.15.99:0/1134830477 learned_addr learned my addr 172.21.15.99:0/1134830477 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb112ffd640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb11c15dcd0 0x7fb11c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11bfff640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c161de0 0x7fb11c15bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11b7fe640 1 -- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c161de0 msgr2=0x7fb11c15bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11b7fe640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c161de0 0x7fb11c15bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11b7fe640 1 -- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb11c15dcd0 msgr2=0x7fb11c15c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11b7fe640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb11c15dcd0 0x7fb11c15c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11b7fe640 1 -- 172.21.15.99:0/1134830477 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0b4e80 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb112ffd640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb11c15dcd0 0x7fb11c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11bfff640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c161de0 0x7fb11c15bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb11b7fe640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 0x7fb11c160d70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb10c000c00 tx=0x7fb10c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c048070 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c030640 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c034040 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c0b5160 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.479+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c15cbf0 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.480+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c074f60 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.480+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb10c04b070 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.481+0000 7fb118ff9640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb10006b4d0 0x7fb10006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.481+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb10c0ac440 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.481+0000 7fb112ffd640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb10006b4d0 0x7fb10006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.481+0000 7fb112ffd640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb10006b4d0 0x7fb10006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb104006fd0 tx=0x7fb104008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.485+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb10c07af60 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.657+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb11c0b78b0 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.658+0000 7fb118ff9640 1 -- 172.21.15.99:0/1134830477 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7fb10c07a860 con 0x7fb11c0cfeb0 2024-03-04T10:04:24.659 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:24.660 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb10006b4d0 msgr2=0x7fb10006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb10006b4d0 0x7fb10006d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb104006fd0 tx=0x7fb104008040 comp rx=0 tx=0).stop 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 msgr2=0x7fb11c160d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 0x7fb11c160d70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb10c000c00 tx=0x7fb10c004730 comp rx=0 tx=0).stop 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 shutdown_connections 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb10006b4d0 0x7fb10006d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb11c161de0 0x7fb11c15bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb11c15dcd0 0x7fb11c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 --2- 172.21.15.99:0/1134830477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb11c0cfeb0 0x7fb11c160d70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 >> 172.21.15.99:0/1134830477 conn(0x7fb11c0b3480 msgr2=0x7fb11c0cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 shutdown_connections 2024-03-04T10:04:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.660+0000 7fb121cd7640 1 -- 172.21.15.99:0/1134830477 wait complete. 2024-03-04T10:04:24.672 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:24.673 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:24.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.774+0000 7f3725ac0640 1 Processor -- start 2024-03-04T10:04:24.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f3725ac0640 1 -- start start 2024-03-04T10:04:24.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f3725ac0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 0x7f37200b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f3725ac0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f372006d3c0 0x7f37200b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f3725ac0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372006d8a0 con 0x7f37200cbfb0 2024-03-04T10:04:24.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f3725ac0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372006db80 con 0x7f372006d3c0 2024-03-04T10:04:24.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f3725ac0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372006de60 con 0x7f372015a810 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f371ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f37200cbfb0 0x7f372015abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57346/0 (socket says 172.21.15.99:57346) 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.775+0000 7f371ffff640 1 -- 172.21.15.99:0/308441490 learned_addr learned my addr 172.21.15.99:0/308441490 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 0x7f37200b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371effd640 1 --2- 172.21.15.99:0/308441490 >> v2:172.21.15.162:3300/0 conn(0x7f372006d3c0 0x7f37200b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094008684 0 0) 0x7f372006d8a0 con 0x7f37200cbfb0 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3708003520 con 0x7f37200cbfb0 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 170860782 0 0) 0x7f3708003520 con 0x7f37200cbfb0 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 msgr2=0x7f37200b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 0x7f37200b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 >> v2:172.21.15.162:3300/0 conn(0x7f372006d3c0 msgr2=0x7f37200b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 --2- 172.21.15.99:0/308441490 >> v2:172.21.15.162:3300/0 conn(0x7f372006d3c0 0x7f37200b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372006e140 con 0x7f37200cbfb0 2024-03-04T10:04:24.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f37180019f0 con 0x7f37200cbfb0 2024-03-04T10:04:24.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 >> v1:172.21.15.99:6789/0 conn(0x7f37200cbfb0 legacy=0x7f372015abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 0x7f3708004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3708004d40 0x7f37080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372006db80 con 0x7f37080039b0 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372006de60 con 0x7f3708004d40 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370800b9c0 con 0x7f372015a810 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371effd640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 0x7f3708004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371ffff640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3708004d40 0x7f37080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 -- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 msgr2=0x7f3708004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372015a810 0x7f3708004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 -- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3708004d40 msgr2=0x7f37080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3708004d40 0x7f37080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.776+0000 7f371f7fe640 1 -- 172.21.15.99:0/308441490 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3708003520 con 0x7f37080039b0 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.777+0000 7f371f7fe640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3710009ac0 tx=0x7f3708005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab5f835969e2122 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.777+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710039070 con 0x7f37080039b0 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.777+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371002c6b0 con 0x7f37080039b0 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.777+0000 7f371e7fc640 1 -- 172.21.15.99:0/308441490 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710009690 con 0x7f37080039b0 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f3725ac0640 1 -- 172.21.15.99:0/308441490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37080039b0 msgr2=0x7f3708003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f3725ac0640 1 --2- 172.21.15.99:0/308441490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3710009ac0 tx=0x7f3708005830 comp rx=0 tx=0).stop 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f371ffff640 1 -- 172.21.15.99:0/308441490 reap_dead start 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f3725ac0640 1 -- 172.21.15.99:0/308441490 shutdown_connections 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f3725ac0640 1 -- 172.21.15.99:0/308441490 >> 172.21.15.99:0/308441490 conn(0x7f372006acd0 msgr2=0x7f372006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f3725ac0640 1 -- 172.21.15.99:0/308441490 shutdown_connections 2024-03-04T10:04:24.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.778+0000 7f3725ac0640 1 -- 172.21.15.99:0/308441490 wait complete. 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.779+0000 7f3725ac0640 1 Processor -- start 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.779+0000 7f3725ac0640 1 -- start start 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37080039b0 0x7f3720141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372014b840 0x7f372014bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 0x7f3720142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372015e3c0 con 0x7f3720142450 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37200b3c10 con 0x7f37080039b0 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372015e540 con 0x7f372014b840 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 0x7f3720142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 0x7f3720142900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51134/0 (socket says 172.21.15.99:51134) 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 -- 172.21.15.99:0/1972582868 learned_addr learned my addr 172.21.15.99:0/1972582868 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371effd640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372014b840 0x7f372014bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371f7fe640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37080039b0 0x7f3720141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 -- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372014b840 msgr2=0x7f372014bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372014b840 0x7f372014bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 -- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37080039b0 msgr2=0x7f3720141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37080039b0 0x7f3720141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 -- 172.21.15.99:0/1972582868 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37201430a0 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f371ffff640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 0x7f3720142900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f372006e560 tx=0x7f371800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371800ec10 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3718018070 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3720143380 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.780+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718013420 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.781+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372015abf0 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.781+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37200cf570 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.782+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f37180135c0 con 0x7f3720142450 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.782+0000 7f3707fff640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f370c06b500 0x7f370c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:24.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.782+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f371808a8d0 con 0x7f3720142450 2024-03-04T10:04:24.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.782+0000 7f371f7fe640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f370c06b500 0x7f370c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:24.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.783+0000 7f371f7fe640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f370c06b500 0x7f370c06d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3710036000 tx=0x7f3710030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:24.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.786+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f37180593f0 con 0x7f3720142450 2024-03-04T10:04:24.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.957+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3720074f60 con 0x7f3720142450 2024-03-04T10:04:24.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.957+0000 7f3707fff640 1 -- 172.21.15.99:0/1972582868 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3718058cf0 con 0x7f3720142450 2024-03-04T10:04:24.959 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:24.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f370c06b500 msgr2=0x7f370c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f370c06b500 0x7f370c06d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3710036000 tx=0x7f3710030000 comp rx=0 tx=0).stop 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 msgr2=0x7f3720142900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 0x7f3720142900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f372006e560 tx=0x7f371800e530 comp rx=0 tx=0).stop 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 shutdown_connections 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f370c06b500 0x7f370c06d9c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3720142450 0x7f3720142900 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f372014b840 0x7f372014bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 --2- 172.21.15.99:0/1972582868 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37080039b0 0x7f3720141b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 >> 172.21.15.99:0/1972582868 conn(0x7f372006acd0 msgr2=0x7f37200b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 shutdown_connections 2024-03-04T10:04:24.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:24.959+0000 7f3725ac0640 1 -- 172.21.15.99:0/1972582868 wait complete. 2024-03-04T10:04:24.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.074+0000 7f78a7b9d640 1 Processor -- start 2024-03-04T10:04:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.074+0000 7f78a7b9d640 1 -- start start 2024-03-04T10:04:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a7b9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 0x7f78a00b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a7b9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f78a006d3c0 0x7f78a00b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a7b9d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a006d8a0 con 0x7f78a015a810 2024-03-04T10:04:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a7b9d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a006db80 con 0x7f78a006d3c0 2024-03-04T10:04:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a7b9d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a006de60 con 0x7f78a00b48a0 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a5138640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 0x7f78a00b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a613a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f78a006d3c0 0x7f78a00b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a5939640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f78a015a810 0x7f78a00b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35462/0 (socket says 172.21.15.99:35462) 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a5138640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 0x7f78a00b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:60246/0 (socket says 172.21.15.99:60246) 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.075+0000 7f78a5138640 1 -- 172.21.15.99:0/268818987 learned_addr learned my addr 172.21.15.99:0/268818987 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.076+0000 7f78a5138640 1 -- 172.21.15.99:0/268818987 >> v2:172.21.15.162:3300/0 conn(0x7f78a006d3c0 msgr2=0x7f78a00b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.076+0000 7f78a5138640 1 --2- 172.21.15.99:0/268818987 >> v2:172.21.15.162:3300/0 conn(0x7f78a006d3c0 0x7f78a00b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.076+0000 7f78a5138640 1 -- 172.21.15.99:0/268818987 >> v1:172.21.15.162:6789/0 conn(0x7f78a015a810 legacy=0x7f78a00b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.076+0000 7f78a5138640 1 -- 172.21.15.99:0/268818987 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a006e140 con 0x7f78a00b48a0 2024-03-04T10:04:25.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.076+0000 7f78a613a640 1 --2- 172.21.15.99:0/268818987 >> v2:172.21.15.162:3300/0 conn(0x7f78a006d3c0 0x7f78a00b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a5138640 1 --2- 172.21.15.99:0/268818987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 0x7f78a00b29f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7890009870 tx=0x7f789002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0851027bdbe37e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a4937640 1 -- 172.21.15.99:0/268818987 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890034070 con 0x7f78a00b48a0 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a4937640 1 -- 172.21.15.99:0/268818987 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890030840 con 0x7f78a00b48a0 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a4937640 1 -- 172.21.15.99:0/268818987 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890030c10 con 0x7f78a00b48a0 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 -- 172.21.15.99:0/268818987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 msgr2=0x7f78a00b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/268818987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 0x7f78a00b29f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7890009870 tx=0x7f789002b980 comp rx=0 tx=0).stop 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 -- 172.21.15.99:0/268818987 shutdown_connections 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/268818987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a00b48a0 0x7f78a00b29f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/268818987 >> v2:172.21.15.162:3300/0 conn(0x7f78a006d3c0 0x7f78a00b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 -- 172.21.15.99:0/268818987 >> 172.21.15.99:0/268818987 conn(0x7f78a006acd0 msgr2=0x7f78a006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 -- 172.21.15.99:0/268818987 shutdown_connections 2024-03-04T10:04:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.077+0000 7f78a7b9d640 1 -- 172.21.15.99:0/268818987 wait complete. 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 Processor -- start 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 -- start start 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a006d3c0 0x7f78a00ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 0x7f78a00ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a015a810 0x7f78a015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00cc130 con 0x7f78a00b48a0 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00cbfb0 con 0x7f78a006d3c0 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a7b9d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a014a9e0 con 0x7f78a015a810 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a5138640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 0x7f78a00ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a5138640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 0x7f78a00ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51144/0 (socket says 172.21.15.99:51144) 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.079+0000 7f78a5138640 1 -- 172.21.15.99:0/1660148326 learned_addr learned my addr 172.21.15.99:0/1660148326 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a613a640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a015a810 0x7f78a015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5939640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a006d3c0 0x7f78a00ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5138640 1 -- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a015a810 msgr2=0x7f78a015c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5138640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a015a810 0x7f78a015c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5138640 1 -- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a006d3c0 msgr2=0x7f78a00ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5138640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a006d3c0 0x7f78a00ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5138640 1 -- 172.21.15.99:0/1660148326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00cd7f0 con 0x7f78a00b48a0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5939640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a006d3c0 0x7f78a00ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a5138640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 0x7f78a00ad8e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f789002b9c0 tx=0x7f7890004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890048070 con 0x7f78a00b48a0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7890030640 con 0x7f78a00b48a0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890034040 con 0x7f78a00b48a0 2024-03-04T10:04:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a00cda60 con 0x7f78a00b48a0 2024-03-04T10:04:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.080+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a00ade20 con 0x7f78a00b48a0 2024-03-04T10:04:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.081+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a00cde50 con 0x7f78a00b48a0 2024-03-04T10:04:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.081+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7890037070 con 0x7f78a00b48a0 2024-03-04T10:04:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.082+0000 7f78967fc640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f788406b500 0x7f788406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.082+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f78900ac150 con 0x7f78a00b48a0 2024-03-04T10:04:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.082+0000 7f78a5939640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f788406b500 0x7f788406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.083+0000 7f78a5939640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f788406b500 0x7f788406d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f789c0012e0 tx=0x7f789c005b70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.086+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7890030810 con 0x7f78a00b48a0 2024-03-04T10:04:25.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.256+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78a0074f60 con 0x7f78a00b48a0 2024-03-04T10:04:25.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.257+0000 7f78967fc640 1 -- 172.21.15.99:0/1660148326 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f789007a510 con 0x7f78a00b48a0 2024-03-04T10:04:25.258 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:25.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f788406b500 msgr2=0x7f788406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f788406b500 0x7f788406d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f789c0012e0 tx=0x7f789c005b70 comp rx=0 tx=0).stop 2024-03-04T10:04:25.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 msgr2=0x7f78a00ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 0x7f78a00ad8e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f789002b9c0 tx=0x7f7890004730 comp rx=0 tx=0).stop 2024-03-04T10:04:25.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 shutdown_connections 2024-03-04T10:04:25.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f788406b500 0x7f788406d9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f78a015a810 0x7f78a015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f78a00b48a0 0x7f78a00ad8e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 --2- 172.21.15.99:0/1660148326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f78a006d3c0 0x7f78a00ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 >> 172.21.15.99:0/1660148326 conn(0x7f78a006acd0 msgr2=0x7f78a006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 shutdown_connections 2024-03-04T10:04:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.258+0000 7f78a7b9d640 1 -- 172.21.15.99:0/1660148326 wait complete. 2024-03-04T10:04:25.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:25.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.372+0000 7f02827f3640 1 Processor -- start 2024-03-04T10:04:25.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.372+0000 7f02827f3640 1 -- start start 2024-03-04T10:04:25.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f02827f3640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f027c15da20 0x7f027c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f02827f3640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f027c0cf660 0x7f027c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f02827f3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c162450 con 0x7f027c1616e0 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f02827f3640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c0d0000 con 0x7f027c0cf660 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f02827f3640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c0cc920 con 0x7f027c15da20 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f027b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f027c0cf660 0x7f027c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f027b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f027c0cf660 0x7f027c0cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51146/0 (socket says 172.21.15.99:51146) 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f027b7fe640 1 -- 172.21.15.99:0/3321974351 learned_addr learned my addr 172.21.15.99:0/3321974351 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f027bfff640 1 --2- 172.21.15.99:0/3321974351 >> v2:172.21.15.162:3300/0 conn(0x7f027c15da20 0x7f027c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f027b7fe640 1 -- 172.21.15.99:0/3321974351 >> v2:172.21.15.162:3300/0 conn(0x7f027c15da20 msgr2=0x7f027c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.373+0000 7f027b7fe640 1 --2- 172.21.15.99:0/3321974351 >> v2:172.21.15.162:3300/0 conn(0x7f027c15da20 0x7f027c0ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.374+0000 7f027b7fe640 1 -- 172.21.15.99:0/3321974351 >> v1:172.21.15.162:6789/0 conn(0x7f027c1616e0 legacy=0x7f027c161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.374+0000 7f027b7fe640 1 -- 172.21.15.99:0/3321974351 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0cc2d0 con 0x7f027c0cf660 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.374+0000 7f027b7fe640 1 --2- 172.21.15.99:0/3321974351 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c0cf660 0x7f027c0cfac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f027c0cf310 tx=0x7f027002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a539196ffcefd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.374+0000 7f027affd640 1 -- 172.21.15.99:0/3321974351 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270034070 con 0x7f027c0cf660 2024-03-04T10:04:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.374+0000 7f027affd640 1 -- 172.21.15.99:0/3321974351 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0270030860 con 0x7f027c0cf660 2024-03-04T10:04:25.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.374+0000 7f027affd640 1 -- 172.21.15.99:0/3321974351 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270030c30 con 0x7f027c0cf660 2024-03-04T10:04:25.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 -- 172.21.15.99:0/3321974351 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c0cf660 msgr2=0x7f027c0cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 --2- 172.21.15.99:0/3321974351 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c0cf660 0x7f027c0cfac0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f027c0cf310 tx=0x7f027002b9a0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 -- 172.21.15.99:0/3321974351 shutdown_connections 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 --2- 172.21.15.99:0/3321974351 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c0cf660 0x7f027c0cfac0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 --2- 172.21.15.99:0/3321974351 >> v2:172.21.15.162:3300/0 conn(0x7f027c15da20 0x7f027c0ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 -- 172.21.15.99:0/3321974351 >> 172.21.15.99:0/3321974351 conn(0x7f027c0b33a0 msgr2=0x7f027c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 -- 172.21.15.99:0/3321974351 shutdown_connections 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.375+0000 7f02827f3640 1 -- 172.21.15.99:0/3321974351 wait complete. 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.376+0000 7f02827f3640 1 Processor -- start 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.376+0000 7f02827f3640 1 -- start start 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f02827f3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f027c0cf660 0x7f027c1551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f02827f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f027c15da20 0x7f027c0cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f02827f3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 0x7f027c0cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f02827f3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0cbf80 con 0x7f027c1616e0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f02827f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0cd2f0 con 0x7f027c15da20 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f02827f3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c15de00 con 0x7f027c0cf660 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 0x7f027c0cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 0x7f027c0cb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51154/0 (socket says 172.21.15.99:51154) 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 -- 172.21.15.99:0/3696477029 learned_addr learned my addr 172.21.15.99:0/3696477029 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f027bfff640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f027c0cf660 0x7f027c1551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f027b7fe640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f027c15da20 0x7f027c0cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 -- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f027c0cf660 msgr2=0x7f027c1551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f027c0cf660 0x7f027c1551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 -- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f027c15da20 msgr2=0x7f027c0cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f027c15da20 0x7f027c0cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.377+0000 7f0280d90640 1 -- 172.21.15.99:0/3696477029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0c9ae0 con 0x7f027c1616e0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.378+0000 7f0280d90640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 0x7f027c0cb230 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f027400c6b0 tx=0x7f027400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.378+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274003b70 con 0x7f027c1616e0 2024-03-04T10:04:25.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.378+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0274003d30 con 0x7f027c1616e0 2024-03-04T10:04:25.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.378+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c0c9dc0 con 0x7f027c1616e0 2024-03-04T10:04:25.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.378+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0274005310 con 0x7f027c1616e0 2024-03-04T10:04:25.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.378+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c0ca380 con 0x7f027c1616e0 2024-03-04T10:04:25.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.379+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c074f60 con 0x7f027c1616e0 2024-03-04T10:04:25.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.379+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f02740054b0 con 0x7f027c1616e0 2024-03-04T10:04:25.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.379+0000 7f0278ff9640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f026806b4d0 0x7f026806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.380+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f027408a310 con 0x7f027c1616e0 2024-03-04T10:04:25.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.380+0000 7f027bfff640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f026806b4d0 0x7f026806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.380+0000 7f027bfff640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f026806b4d0 0x7f026806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f026c006fd0 tx=0x7f026c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.384+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0274058750 con 0x7f027c1616e0 2024-03-04T10:04:25.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.554+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f027c0c4e80 con 0x7f027c1616e0 2024-03-04T10:04:25.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.554+0000 7f0278ff9640 1 -- 172.21.15.99:0/3696477029 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f027405c7f0 con 0x7f027c1616e0 2024-03-04T10:04:25.555 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f026806b4d0 msgr2=0x7f026806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f026806b4d0 0x7f026806d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f026c006fd0 tx=0x7f026c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 msgr2=0x7f027c0cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 0x7f027c0cb230 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f027400c6b0 tx=0x7f027400cb70 comp rx=0 tx=0).stop 2024-03-04T10:04:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 shutdown_connections 2024-03-04T10:04:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f026806b4d0 0x7f026806d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f027c1616e0 0x7f027c0cb230 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f027c15da20 0x7f027c0cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 --2- 172.21.15.99:0/3696477029 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f027c0cf660 0x7f027c1551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 >> 172.21.15.99:0/3696477029 conn(0x7f027c0b33a0 msgr2=0x7f027c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 shutdown_connections 2024-03-04T10:04:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.556+0000 7f02827f3640 1 -- 172.21.15.99:0/3696477029 wait complete. 2024-03-04T10:04:25.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:25.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.669+0000 7f0c6bfd4640 1 Processor -- start 2024-03-04T10:04:25.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.670+0000 7f0c6bfd4640 1 -- start start 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.670+0000 7f0c6bfd4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0c640ccbd0 0x7f0c640b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.670+0000 7f0c6bfd4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0c6415c2e0 0x7f0c640ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.670+0000 7f0c6bfd4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6415ca50 con 0x7f0c640cecf0 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.670+0000 7f0c6bfd4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c640cf160 con 0x7f0c640ccbd0 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.670+0000 7f0c6bfd4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6415d2f0 con 0x7f0c6415c2e0 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0c640ccbd0 0x7f0c640b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0c640ccbd0 0x7f0c640b7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51158/0 (socket says 172.21.15.99:51158) 2024-03-04T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 -- 172.21.15.99:0/1609689502 learned_addr learned my addr 172.21.15.99:0/1609689502 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:25.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c6956f640 1 --2- 172.21.15.99:0/1609689502 >> v2:172.21.15.162:3300/0 conn(0x7f0c6415c2e0 0x7f0c640ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 -- 172.21.15.99:0/1609689502 >> v2:172.21.15.162:3300/0 conn(0x7f0c6415c2e0 msgr2=0x7f0c640ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:25.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 --2- 172.21.15.99:0/1609689502 >> v2:172.21.15.162:3300/0 conn(0x7f0c6415c2e0 0x7f0c640ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 -- 172.21.15.99:0/1609689502 >> v1:172.21.15.162:6789/0 conn(0x7f0c640cecf0 legacy=0x7f0c640b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.671+0000 7f0c69d70640 1 -- 172.21.15.99:0/1609689502 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6415cd30 con 0x7f0c640ccbd0 2024-03-04T10:04:25.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c69d70640 1 --2- 172.21.15.99:0/1609689502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c640b7320 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0c640b3af0 tx=0x7f0c5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd5b82af108a31b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c68d6e640 1 -- 172.21.15.99:0/1609689502 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54034070 con 0x7f0c640ccbd0 2024-03-04T10:04:25.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c68d6e640 1 -- 172.21.15.99:0/1609689502 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54030860 con 0x7f0c640ccbd0 2024-03-04T10:04:25.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c68d6e640 1 -- 172.21.15.99:0/1609689502 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54030c30 con 0x7f0c640ccbd0 2024-03-04T10:04:25.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/1609689502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 msgr2=0x7f0c640b7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/1609689502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c640b7320 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0c640b3af0 tx=0x7f0c5402b9a0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/1609689502 shutdown_connections 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/1609689502 >> v2:172.21.15.162:3300/0 conn(0x7f0c6415c2e0 0x7f0c640ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/1609689502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c640b7320 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/1609689502 >> 172.21.15.99:0/1609689502 conn(0x7f0c64066dc0 msgr2=0x7f0c640671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/1609689502 shutdown_connections 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.672+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/1609689502 wait complete. 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.673+0000 7f0c6bfd4640 1 Processor -- start 2024-03-04T10:04:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 -- start start 2024-03-04T10:04:25.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c64155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c640cecf0 0x7f0c6406d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c6415c2e0 0x7f0c640c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6415c760 con 0x7f0c640ccbd0 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6415d6d0 con 0x7f0c6415c2e0 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6bfd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c640ccfb0 con 0x7f0c640cecf0 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c69d70640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c64155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c69d70640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c64155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51168/0 (socket says 172.21.15.99:51168) 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c69d70640 1 -- 172.21.15.99:0/508136221 learned_addr learned my addr 172.21.15.99:0/508136221 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6956f640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c640cecf0 0x7f0c6406d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.674+0000 7f0c6a571640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c6415c2e0 0x7f0c640c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c69d70640 1 -- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c640cecf0 msgr2=0x7f0c6406d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c69d70640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c640cecf0 0x7f0c6406d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c69d70640 1 -- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c6415c2e0 msgr2=0x7f0c640c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c69d70640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c6415c2e0 0x7f0c640c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c69d70640 1 -- 172.21.15.99:0/508136221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c641577e0 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c6956f640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c640cecf0 0x7f0c6406d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c69d70640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c64155160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0c5402b9e0 tx=0x7f0c54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54048070 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54030660 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54034040 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6406ddd0 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.675+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c64157d60 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.676+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c64074f60 con 0x7f0c640ccbd0 2024-03-04T10:04:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.676+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0c5404b070 con 0x7f0c640ccbd0 2024-03-04T10:04:25.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.677+0000 7f0c5a7fc640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0c4806b4d0 0x7f0c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:25.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.677+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f0c540ac640 con 0x7f0c640ccbd0 2024-03-04T10:04:25.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.677+0000 7f0c6956f640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0c4806b4d0 0x7f0c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:25.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.677+0000 7f0c6956f640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0c4806b4d0 0x7f0c4806d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c6406d4f0 tx=0x7f0c60008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:25.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.681+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0c5407b280 con 0x7f0c640ccbd0 2024-03-04T10:04:25.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.852+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0c64156320 con 0x7f0c640ccbd0 2024-03-04T10:04:25.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.853+0000 7f0c5a7fc640 1 -- 172.21.15.99:0/508136221 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0c5407ab80 con 0x7f0c640ccbd0 2024-03-04T10:04:25.854 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0c4806b4d0 msgr2=0x7f0c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0c4806b4d0 0x7f0c4806d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0c6406d4f0 tx=0x7f0c60008040 comp rx=0 tx=0).stop 2024-03-04T10:04:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 msgr2=0x7f0c64155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c64155160 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0c5402b9e0 tx=0x7f0c54004730 comp rx=0 tx=0).stop 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 shutdown_connections 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0c4806b4d0 0x7f0c4806d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c6415c2e0 0x7f0c640c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0c640cecf0 0x7f0c6406d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 --2- 172.21.15.99:0/508136221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0c640ccbd0 0x7f0c64155160 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 >> 172.21.15.99:0/508136221 conn(0x7f0c64066dc0 msgr2=0x7f0c640671a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 shutdown_connections 2024-03-04T10:04:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:25.855+0000 7f0c6bfd4640 1 -- 172.21.15.99:0/508136221 wait complete. 2024-03-04T10:04:26.014 INFO:tasks.cephfs_test_runner:Starting test: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:04:26.015 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:26.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.114+0000 7ff385881640 1 Processor -- start 2024-03-04T10:04:26.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.114+0000 7ff385881640 1 -- start start 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff385881640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff3800b5b80 0x7ff3800b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff385881640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff38015e1e0 con 0x7ff38015de00 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff385881640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800d1c10 con 0x7ff3800b2e60 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff385881640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800b98c0 con 0x7ff3800b5b80 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff3800b5b80 0x7ff3800b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff38015de00 0x7ff3800b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57360/0 (socket says 172.21.15.99:57360) 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37effd640 1 -- 172.21.15.99:0/3379966165 learned_addr learned my addr 172.21.15.99:0/3379966165 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37f7fe640 1 -- 172.21.15.99:0/3379966165 >> v1:172.21.15.162:6789/0 conn(0x7ff3800b2e60 legacy=0x7ff3800b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37f7fe640 1 -- 172.21.15.99:0/3379966165 >> v1:172.21.15.99:6789/0 conn(0x7ff38015de00 legacy=0x7ff3800b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37dffb640 1 -- 172.21.15.99:0/3379966165 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678074841 0 0) 0x7ff38015e1e0 con 0x7ff38015de00 2024-03-04T10:04:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.115+0000 7ff37f7fe640 1 -- 172.21.15.99:0/3379966165 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800d1ef0 con 0x7ff3800b5b80 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.116+0000 7ff37f7fe640 1 --2- 172.21.15.99:0/3379966165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3800b5b80 0x7ff3800b9380 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff378009dc0 tx=0x7ff37802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da6c4ac08d98fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.116+0000 7ff37dffb640 1 -- 172.21.15.99:0/3379966165 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378039070 con 0x7ff3800b5b80 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.116+0000 7ff37dffb640 1 -- 172.21.15.99:0/3379966165 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37802ccd0 con 0x7ff3800b5b80 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.116+0000 7ff37dffb640 1 -- 172.21.15.99:0/3379966165 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378034660 con 0x7ff3800b5b80 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 -- 172.21.15.99:0/3379966165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3800b5b80 msgr2=0x7ff3800b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 --2- 172.21.15.99:0/3379966165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3800b5b80 0x7ff3800b9380 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff378009dc0 tx=0x7ff37802bf20 comp rx=0 tx=0).stop 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 -- 172.21.15.99:0/3379966165 shutdown_connections 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 --2- 172.21.15.99:0/3379966165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3800b5b80 0x7ff3800b9380 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 -- 172.21.15.99:0/3379966165 >> 172.21.15.99:0/3379966165 conn(0x7ff3800d0470 msgr2=0x7ff3800d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 -- 172.21.15.99:0/3379966165 shutdown_connections 2024-03-04T10:04:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.117+0000 7ff385881640 1 -- 172.21.15.99:0/3379966165 wait complete. 2024-03-04T10:04:26.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.118+0000 7ff385881640 1 Processor -- start 2024-03-04T10:04:26.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.118+0000 7ff385881640 1 -- start start 2024-03-04T10:04:26.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3800b2e60 0x7ff380160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3800b5b80 0x7ff3801f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 0x7ff3801f75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800d2820 con 0x7ff38015de00 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800d24b0 con 0x7ff3800b5b80 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800b72a0 con 0x7ff3800b2e60 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 0x7ff3801f75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 0x7ff3801f75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51180/0 (socket says 172.21.15.99:51180) 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 -- 172.21.15.99:0/1673010039 learned_addr learned my addr 172.21.15.99:0/1673010039 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37effd640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3800b2e60 0x7ff380160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37e7fc640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3800b5b80 0x7ff3801f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 -- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3800b2e60 msgr2=0x7ff380160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3800b2e60 0x7ff380160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 -- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3800b5b80 msgr2=0x7ff3801f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3800b5b80 0x7ff3801f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 -- 172.21.15.99:0/1673010039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800cc290 con 0x7ff38015de00 2024-03-04T10:04:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37effd640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3800b2e60 0x7ff380160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37e7fc640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3800b5b80 0x7ff3801f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff37f7fe640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 0x7ff3801f75e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff378037040 tx=0x7ff378004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378044020 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37803a040 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3800cb4d0 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff378039050 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.119+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3800cba50 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.120+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff380074f60 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.121+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff37802cb90 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.121+0000 7ff367fff640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff36c06b4d0 0x7ff36c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.121+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7ff3780acc80 con 0x7ff38015de00 2024-03-04T10:04:26.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.121+0000 7ff37effd640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff36c06b4d0 0x7ff36c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.122+0000 7ff37effd640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff36c06b4d0 0x7ff36c06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff370000f60 tx=0x7ff370009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.126+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff37804e060 con 0x7ff38015de00 2024-03-04T10:04:26.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.297+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3800ca3c0 con 0x7ff38015de00 2024-03-04T10:04:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.298+0000 7ff367fff640 1 -- 172.21.15.99:0/1673010039 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7ff37807b7a0 con 0x7ff38015de00 2024-03-04T10:04:26.300 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:26.300 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.300+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff36c06b4d0 msgr2=0x7ff36c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.300+0000 7ff385881640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff36c06b4d0 0x7ff36c06d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff370000f60 tx=0x7ff370009040 comp rx=0 tx=0).stop 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.300+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 msgr2=0x7ff3801f75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.300+0000 7ff385881640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 0x7ff3801f75e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff378037040 tx=0x7ff378004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 shutdown_connections 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff36c06b4d0 0x7ff36c06d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff38015de00 0x7ff3801f75e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3800b5b80 0x7ff3801f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 --2- 172.21.15.99:0/1673010039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3800b2e60 0x7ff380160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 >> 172.21.15.99:0/1673010039 conn(0x7ff3800d0470 msgr2=0x7ff3800b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 shutdown_connections 2024-03-04T10:04:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.301+0000 7ff385881640 1 -- 172.21.15.99:0/1673010039 wait complete. 2024-03-04T10:04:26.313 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:26.313 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.414+0000 7f83bab3b640 1 Processor -- start 2024-03-04T10:04:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.414+0000 7f83bab3b640 1 -- start start 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83bab3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f83b40b48a0 0x7f83b40b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83bab3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b406d3c0 0x7f83b40b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83bab3b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b406d8a0 con 0x7f83b415a810 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83bab3b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b406db80 con 0x7f83b40b48a0 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83bab3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b406de60 con 0x7f83b406d3c0 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f83b40b48a0 0x7f83b40b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b88d7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f83b415a810 0x7f83b40b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57368/0 (socket says 172.21.15.99:57368) 2024-03-04T10:04:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b88d7640 1 -- 172.21.15.99:0/681029800 learned_addr learned my addr 172.21.15.99:0/681029800 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b90d8640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b406d3c0 0x7f83b40b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b3fff640 1 -- 172.21.15.99:0/681029800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b406d3c0 msgr2=0x7f83b40b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b3fff640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b406d3c0 0x7f83b40b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b3fff640 1 -- 172.21.15.99:0/681029800 >> v1:172.21.15.99:6789/0 conn(0x7f83b415a810 legacy=0x7f83b40b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.415+0000 7f83b37fe640 1 -- 172.21.15.99:0/681029800 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152264001 0 0) 0x7f83b406d8a0 con 0x7f83b415a810 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.416+0000 7f83b3fff640 1 -- 172.21.15.99:0/681029800 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b406e140 con 0x7f83b40b48a0 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.416+0000 7f83b90d8640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b406d3c0 0x7f83b40b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.416+0000 7f83b3fff640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b40b48a0 0x7f83b40b29f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f83a80097a0 tx=0x7f83a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6039f7847a6bc31a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.416+0000 7f83b37fe640 1 -- 172.21.15.99:0/681029800 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8035020 con 0x7f83b40b48a0 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.416+0000 7f83b37fe640 1 -- 172.21.15.99:0/681029800 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8031830 con 0x7f83b40b48a0 2024-03-04T10:04:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.416+0000 7f83b37fe640 1 -- 172.21.15.99:0/681029800 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8031c00 con 0x7f83b40b48a0 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 -- 172.21.15.99:0/681029800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b40b48a0 msgr2=0x7f83b40b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b40b48a0 0x7f83b40b29f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f83a80097a0 tx=0x7f83a802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 -- 172.21.15.99:0/681029800 shutdown_connections 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b406d3c0 0x7f83b40b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 --2- 172.21.15.99:0/681029800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b40b48a0 0x7f83b40b29f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 -- 172.21.15.99:0/681029800 >> 172.21.15.99:0/681029800 conn(0x7f83b406acd0 msgr2=0x7f83b406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 -- 172.21.15.99:0/681029800 shutdown_connections 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.417+0000 7f83bab3b640 1 -- 172.21.15.99:0/681029800 wait complete. 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.418+0000 7f83bab3b640 1 Processor -- start 2024-03-04T10:04:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.418+0000 7f83bab3b640 1 -- start start 2024-03-04T10:04:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83bab3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b406d3c0 0x7f83b41f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83bab3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b40b48a0 0x7f83b40ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83bab3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 0x7f83b40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83bab3b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cc130 con 0x7f83b415a810 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83bab3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40cbfb0 con 0x7f83b406d3c0 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83bab3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b40ae170 con 0x7f83b40b48a0 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 0x7f83b40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 0x7f83b40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51192/0 (socket says 172.21.15.99:51192) 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 -- 172.21.15.99:0/3597877845 learned_addr learned my addr 172.21.15.99:0/3597877845 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b3fff640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b40b48a0 0x7f83b40ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b88d7640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b406d3c0 0x7f83b41f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 -- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b40b48a0 msgr2=0x7f83b40ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b40b48a0 0x7f83b40ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 -- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b406d3c0 msgr2=0x7f83b41f08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b406d3c0 0x7f83b41f08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 -- 172.21.15.99:0/3597877845 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b4157f30 con 0x7f83b415a810 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b88d7640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b406d3c0 0x7f83b41f08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.419+0000 7f83b90d8640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 0x7f83b40b1500 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00ce70 tx=0x7f83ac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.420+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac003b90 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.420+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac003d50 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.420+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac004260 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.420+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b4158210 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.420+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b40add40 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.421+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4074f60 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.421+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f83ac004400 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.421+0000 7f83b17fa640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.422+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f83ac08ae20 con 0x7f83b415a810 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.422+0000 7f83b88d7640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.422+0000 7f83b88d7640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f83a4000f60 tx=0x7f83a4007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.426+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f83ac0592f0 con 0x7f83b415a810 2024-03-04T10:04:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.598+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83b406ec30 con 0x7f83b415a810 2024-03-04T10:04:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.599+0000 7f83b17fa640 1 -- 172.21.15.99:0/3597877845 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83ac05d390 con 0x7f83b415a810 2024-03-04T10:04:26.600 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f83a006b4d0 msgr2=0x7f83a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f83a006b4d0 0x7f83a006d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f83a4000f60 tx=0x7f83a4007cc0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 msgr2=0x7f83b40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 0x7f83b40b1500 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f83ac00ce70 tx=0x7f83ac00a710 comp rx=0 tx=0).stop 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 shutdown_connections 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f83a006b4d0 0x7f83a006d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f83b415a810 0x7f83b40b1500 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f83b40b48a0 0x7f83b40ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 --2- 172.21.15.99:0/3597877845 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f83b406d3c0 0x7f83b41f08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 >> 172.21.15.99:0/3597877845 conn(0x7f83b406acd0 msgr2=0x7f83b40ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 shutdown_connections 2024-03-04T10:04:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.601+0000 7f83bab3b640 1 -- 172.21.15.99:0/3597877845 wait complete. 2024-03-04T10:04:26.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:26.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.716+0000 7f48157c4640 1 Processor -- start 2024-03-04T10:04:26.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.716+0000 7f48157c4640 1 -- start start 2024-03-04T10:04:26.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f48157c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f48100b5700 0x7f48100d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f48157c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100b9410 0x7f481006c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f48157c4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48100d2400 con 0x7f48100d34a0 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f48157c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48100d26e0 con 0x7f48100b5700 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f48157c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100d29c0 con 0x7f48100b9410 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480f7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100b9410 0x7f481006c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480e7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f48100d34a0 0x7f481006c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35496/0 (socket says 172.21.15.99:35496) 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480effd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f48100b5700 0x7f48100d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480e7fc640 1 -- 172.21.15.99:0/411689493 learned_addr learned my addr 172.21.15.99:0/411689493 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480f7fe640 1 -- 172.21.15.99:0/411689493 >> v2:172.21.15.162:3300/0 conn(0x7f48100b5700 msgr2=0x7f48100d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480f7fe640 1 --2- 172.21.15.99:0/411689493 >> v2:172.21.15.162:3300/0 conn(0x7f48100b5700 0x7f48100d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480f7fe640 1 -- 172.21.15.99:0/411689493 >> v1:172.21.15.162:6789/0 conn(0x7f48100d34a0 legacy=0x7f481006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.717+0000 7f480f7fe640 1 -- 172.21.15.99:0/411689493 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100d2ca0 con 0x7f48100b9410 2024-03-04T10:04:26.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.718+0000 7f480f7fe640 1 --2- 172.21.15.99:0/411689493 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100b9410 0x7f481006c7f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f480802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49cc1fcac97b310 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.718+0000 7f480dffb640 1 -- 172.21.15.99:0/411689493 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808039070 con 0x7f48100b9410 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.718+0000 7f480dffb640 1 -- 172.21.15.99:0/411689493 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480802cbc0 con 0x7f48100b9410 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.718+0000 7f480dffb640 1 -- 172.21.15.99:0/411689493 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808034440 con 0x7f48100b9410 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 -- 172.21.15.99:0/411689493 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100b9410 msgr2=0x7f481006c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 --2- 172.21.15.99:0/411689493 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100b9410 0x7f481006c7f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4808009d90 tx=0x7f480802be10 comp rx=0 tx=0).stop 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 -- 172.21.15.99:0/411689493 shutdown_connections 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 --2- 172.21.15.99:0/411689493 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100b9410 0x7f481006c7f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 --2- 172.21.15.99:0/411689493 >> v2:172.21.15.162:3300/0 conn(0x7f48100b5700 0x7f48100d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 -- 172.21.15.99:0/411689493 >> 172.21.15.99:0/411689493 conn(0x7f48100d0490 msgr2=0x7f48100d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 -- 172.21.15.99:0/411689493 shutdown_connections 2024-03-04T10:04:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.719+0000 7f48157c4640 1 -- 172.21.15.99:0/411689493 wait complete. 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.720+0000 7f48157c4640 1 Processor -- start 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.720+0000 7f48157c4640 1 -- start start 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f48157c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 0x7f48100ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f48157c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48100b9410 0x7f4810160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f48157c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100d34a0 0x7f48100cc960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f48157c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481006ceb0 con 0x7f48100b5700 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f48157c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481006cd30 con 0x7f48100b9410 2024-03-04T10:04:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f48157c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100b5ae0 con 0x7f48100d34a0 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 0x7f48100ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 0x7f48100ab5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51204/0 (socket says 172.21.15.99:51204) 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 -- 172.21.15.99:0/910748602 learned_addr learned my addr 172.21.15.99:0/910748602 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480f7fe640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100d34a0 0x7f48100cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480e7fc640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48100b9410 0x7f4810160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 -- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100d34a0 msgr2=0x7f48100cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100d34a0 0x7f48100cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 -- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48100b9410 msgr2=0x7f4810160000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48100b9410 0x7f4810160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480effd640 1 -- 172.21.15.99:0/910748602 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100cdda0 con 0x7f48100b5700 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.721+0000 7f480f7fe640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100d34a0 0x7f48100cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.722+0000 7f480effd640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 0x7f48100ab5d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f480000e9e0 tx=0x7f480000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.722+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800017070 con 0x7f48100b5700 2024-03-04T10:04:26.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.722+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48100cd220 con 0x7f48100b5700 2024-03-04T10:04:26.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.722+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480000c970 con 0x7f48100b5700 2024-03-04T10:04:26.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.722+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480000cd60 con 0x7f48100b5700 2024-03-04T10:04:26.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.722+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48100cd7e0 con 0x7f48100b5700 2024-03-04T10:04:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.723+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4810074f60 con 0x7f48100b5700 2024-03-04T10:04:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.723+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f480001a070 con 0x7f48100b5700 2024-03-04T10:04:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.724+0000 7f47f77fe640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47fc06b4d0 0x7f47fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.724+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f480008a660 con 0x7f48100b5700 2024-03-04T10:04:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.724+0000 7f480e7fc640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47fc06b4d0 0x7f47fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.724+0000 7f480e7fc640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47fc06b4d0 0x7f47fc06d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48100ac9b0 tx=0x7f4804009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.728+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4800059070 con 0x7f48100b5700 2024-03-04T10:04:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.898+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f48100cf980 con 0x7f48100b5700 2024-03-04T10:04:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.899+0000 7f47f77fe640 1 -- 172.21.15.99:0/910748602 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4800058970 con 0x7f48100b5700 2024-03-04T10:04:26.900 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47fc06b4d0 msgr2=0x7f47fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47fc06b4d0 0x7f47fc06d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f48100ac9b0 tx=0x7f4804009040 comp rx=0 tx=0).stop 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 msgr2=0x7f48100ab5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 0x7f48100ab5d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f480000e9e0 tx=0x7f480000eea0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 shutdown_connections 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47fc06b4d0 0x7f47fc06d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48100d34a0 0x7f48100cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f48100b9410 0x7f4810160000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 --2- 172.21.15.99:0/910748602 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48100b5700 0x7f48100ab5d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.900+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 >> 172.21.15.99:0/910748602 conn(0x7f48100d0490 msgr2=0x7f48100ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.901+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 shutdown_connections 2024-03-04T10:04:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:26.901+0000 7f48157c4640 1 -- 172.21.15.99:0/910748602 wait complete. 2024-03-04T10:04:26.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:27.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.014+0000 7f7db5ddf640 1 Processor -- start 2024-03-04T10:04:27.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7db5ddf640 1 -- start start 2024-03-04T10:04:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7db5ddf640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7db00cedf0 0x7f7db0168e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7db5ddf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00cf740 con 0x7f7db006c060 2024-03-04T10:04:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7db5ddf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db006c7d0 con 0x7f7db0161390 2024-03-04T10:04:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7db5ddf640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db006d3e0 con 0x7f7db00cedf0 2024-03-04T10:04:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7daeffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7db006c060 0x7f7db00ce6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57370/0 (socket says 172.21.15.99:57370) 2024-03-04T10:04:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.015+0000 7f7daeffd640 1 -- 172.21.15.99:0/1346913035 learned_addr learned my addr 172.21.15.99:0/1346913035 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795349826 0 0) 0x7f7db00cf740 con 0x7f7db006c060 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98003520 con 0x7f7db006c060 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7daffff640 1 --2- 172.21.15.99:0/1346913035 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cedf0 0x7f7db0168e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344872149 0 0) 0x7f7d98003520 con 0x7f7db006c060 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cedf0 msgr2=0x7f7db0168e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 --2- 172.21.15.99:0/1346913035 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cedf0 0x7f7db0168e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 >> v1:172.21.15.162:6789/0 conn(0x7f7db0161390 legacy=0x7f7db00d20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db006cab0 con 0x7f7db006c060 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7da4001670 con 0x7f7db006c060 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 >> v1:172.21.15.99:6789/0 conn(0x7f7db006c060 legacy=0x7f7db00ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d980039b0 0x7f7d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7d98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d980051c0 0x7f7d9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db006d3e0 con 0x7f7d98004800 2024-03-04T10:04:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800c060 con 0x7f7d980051c0 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9800c260 con 0x7f7d980039b0 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7daeffd640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7d98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d980039b0 0x7f7d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.016+0000 7f7daffff640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d980051c0 0x7f7d9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daeffd640 1 -- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d980039b0 msgr2=0x7f7d98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daeffd640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d980039b0 0x7f7d98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daeffd640 1 -- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d980051c0 msgr2=0x7f7d9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daeffd640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d980051c0 0x7f7d9800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daffff640 1 -- 172.21.15.99:0/1346913035 reap_dead start 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daeffd640 1 -- 172.21.15.99:0/1346913035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98003520 con 0x7f7d98004800 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7daeffd640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7d98004c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c1e0 tx=0x7f7d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07a30b60a42ff6a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4007d00 con 0x7f7d98004800 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da4030070 con 0x7f7d98004800 2024-03-04T10:04:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.017+0000 7f7dae7fc640 1 -- 172.21.15.99:0/1346913035 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da4005d50 con 0x7f7d98004800 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 -- 172.21.15.99:0/1346913035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 msgr2=0x7f7d98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7d98004c80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c1e0 tx=0x7f7d98005830 comp rx=0 tx=0).stop 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 -- 172.21.15.99:0/1346913035 shutdown_connections 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/1346913035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7d98004c80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 -- 172.21.15.99:0/1346913035 >> 172.21.15.99:0/1346913035 conn(0x7f7db0066dc0 msgr2=0x7f7db00671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 -- 172.21.15.99:0/1346913035 shutdown_connections 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.018+0000 7f7db5ddf640 1 -- 172.21.15.99:0/1346913035 wait complete. 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.019+0000 7f7db5ddf640 1 Processor -- start 2024-03-04T10:04:27.023 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.019+0000 7f7db5ddf640 1 -- start start 2024-03-04T10:04:27.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7db5ddf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7db015c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7db5ddf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00b58d0 0x7f7db00b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7db5ddf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00b67a0 0x7f7db01f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7db5ddf640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db006c4c0 con 0x7f7d98004800 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7db5ddf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db006d070 con 0x7f7db00b67a0 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7db5ddf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db0161770 con 0x7f7db00b58d0 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7db015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7db015c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51228/0 (socket says 172.21.15.99:51228) 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4250765999 learned_addr learned my addr 172.21.15.99:0/4250765999 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daffff640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00b67a0 0x7f7db01f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daeffd640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00b58d0 0x7f7db00b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00b58d0 msgr2=0x7f7db00b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00b58d0 0x7f7db00b6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00b67a0 msgr2=0x7f7db01f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00b67a0 0x7f7db01f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4250765999 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db01f75b0 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daeffd640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00b58d0 0x7f7db00b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.020+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7db015c710 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7da0007fd0 tx=0x7f7da000e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.021+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0019070 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.021+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da00052e0 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.021+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db00ad600 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.021+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0003810 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.021+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db00adbc0 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.022+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db0074f60 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.022+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7da001c070 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.022+0000 7f7d97fff640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9c06b4d0 0x7f7d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.023+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7da0089bf0 con 0x7f7d98004800 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.023+0000 7f7daeffd640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9c06b4d0 0x7f7d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.023+0000 7f7daeffd640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9c06b4d0 0x7f7d9c06d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7da4034000 tx=0x7f7da4007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.027+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7da0058030 con 0x7f7d98004800 2024-03-04T10:04:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.195+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7db00d2270 con 0x7f7d98004800 2024-03-04T10:04:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.195+0000 7f7d97fff640 1 -- 172.21.15.99:0/4250765999 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7da005c0d0 con 0x7f7d98004800 2024-03-04T10:04:27.196 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9c06b4d0 msgr2=0x7f7d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9c06b4d0 0x7f7d9c06d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7da4034000 tx=0x7f7da4007000 comp rx=0 tx=0).stop 2024-03-04T10:04:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 msgr2=0x7f7db015c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7db015c710 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7da0007fd0 tx=0x7f7da000e8a0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 shutdown_connections 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9c06b4d0 0x7f7d9c06d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00b67a0 0x7f7db01f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00b58d0 0x7f7db00b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 --2- 172.21.15.99:0/4250765999 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d98004800 0x7f7db015c710 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 >> 172.21.15.99:0/4250765999 conn(0x7f7db0066dc0 msgr2=0x7f7db00cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 shutdown_connections 2024-03-04T10:04:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.197+0000 7f7db5ddf640 1 -- 172.21.15.99:0/4250765999 wait complete. 2024-03-04T10:04:27.209 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:27.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.311+0000 7f9070630640 1 Processor -- start 2024-03-04T10:04:27.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.311+0000 7f9070630640 1 -- start start 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f9070630640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90680b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f9070630640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f906806d3c0 0x7f90680b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f9070630640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906806d8a0 con 0x7f906815a810 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f9070630640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906806db80 con 0x7f906806d3c0 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f9070630640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906806de60 con 0x7f90680b48a0 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f906e3cc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f906815a810 0x7f90680b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57384/0 (socket says 172.21.15.99:57384) 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.312+0000 7f906e3cc640 1 -- 172.21.15.99:0/3914731116 learned_addr learned my addr 172.21.15.99:0/3914731116 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:27.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906dbcb640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90680b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782436183 0 0) 0x7f906806d8a0 con 0x7f906815a810 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9054003520 con 0x7f906815a810 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 --2- 172.21.15.99:0/3914731116 >> v2:172.21.15.162:3300/0 conn(0x7f906806d3c0 0x7f90680b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440625626 0 0) 0x7f9054003520 con 0x7f906815a810 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 msgr2=0x7f90680b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90680b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 >> v2:172.21.15.162:3300/0 conn(0x7f906806d3c0 msgr2=0x7f90680b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 --2- 172.21.15.99:0/3914731116 >> v2:172.21.15.162:3300/0 conn(0x7f906806d3c0 0x7f90680b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906806e140 con 0x7f906815a810 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f9064001670 con 0x7f906815a810 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 >> v1:172.21.15.99:6789/0 conn(0x7f906815a810 legacy=0x7f90680b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90540039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9054003f30 0x7f9054004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f90540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906806db80 con 0x7f9054004d40 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906806de60 con 0x7f9054003f30 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905400b9c0 con 0x7f90680b48a0 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f90540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906dbcb640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9054003f30 0x7f9054004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906e3cc640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90540039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 -- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 msgr2=0x7f90540039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90540039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 -- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9054003f30 msgr2=0x7f9054004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9054003f30 0x7f9054004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906ebcd640 1 -- 172.21.15.99:0/3914731116 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9054003520 con 0x7f9054004d40 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906dbcb640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9054003f30 0x7f9054004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.313+0000 7f906e3cc640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90680b48a0 0x7f90540039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.314+0000 7f906ebcd640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f90540051c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f90680d0760 tx=0x7f9054005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbcdb86c8edf88a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.314+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060039070 con 0x7f9054004d40 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.314+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906002cbc0 con 0x7f9054004d40 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.314+0000 7f906d3ca640 1 -- 172.21.15.99:0/3914731116 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060009c10 con 0x7f9054004d40 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f9070630640 1 -- 172.21.15.99:0/3914731116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 msgr2=0x7f90540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f9070630640 1 --2- 172.21.15.99:0/3914731116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f90540051c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f90680d0760 tx=0x7f9054005830 comp rx=0 tx=0).stop 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f906ebcd640 1 -- 172.21.15.99:0/3914731116 reap_dead start 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f9070630640 1 -- 172.21.15.99:0/3914731116 shutdown_connections 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f9070630640 1 -- 172.21.15.99:0/3914731116 >> 172.21.15.99:0/3914731116 conn(0x7f906806acd0 msgr2=0x7f906806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f9070630640 1 -- 172.21.15.99:0/3914731116 shutdown_connections 2024-03-04T10:04:27.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.315+0000 7f9070630640 1 -- 172.21.15.99:0/3914731116 wait complete. 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.316+0000 7f9070630640 1 Processor -- start 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.316+0000 7f9070630640 1 -- start start 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f9068153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90681591b0 0x7f90680ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90680acdb0 0x7f90680afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90680cc130 con 0x7f9054004d40 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90680cbfb0 con 0x7f90680acdb0 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9068151b10 con 0x7f90681591b0 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f9068153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f9068153d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51242/0 (socket says 172.21.15.99:51242) 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 -- 172.21.15.99:0/2554819864 learned_addr learned my addr 172.21.15.99:0/2554819864 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:27.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 -- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90681591b0 msgr2=0x7f90680ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90681591b0 0x7f90680ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 -- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90680acdb0 msgr2=0x7f90680afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90680acdb0 0x7f90680afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 -- 172.21.15.99:0/2554819864 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9068157e70 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f906e3cc640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f9068153d30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f906400e910 tx=0x7f906400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064019070 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90640049f0 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9068158110 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.317+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064004dc0 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.318+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90681586d0 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.318+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9068074f60 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.319+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f906401c070 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.319+0000 7f905effd640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f904c06b500 0x7f904c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.319+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f9064089db0 con 0x7f9054004d40 2024-03-04T10:04:27.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.319+0000 7f906dbcb640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f904c06b500 0x7f904c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.320+0000 7f906dbcb640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f904c06b500 0x7f904c06d9c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9068159a90 tx=0x7f9058008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.323+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f90640581f0 con 0x7f9054004d40 2024-03-04T10:04:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.494+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f906806ec30 con 0x7f9054004d40 2024-03-04T10:04:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.494+0000 7f905effd640 1 -- 172.21.15.99:0/2554819864 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f906405e020 con 0x7f9054004d40 2024-03-04T10:04:27.495 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f904c06b500 msgr2=0x7f904c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f904c06b500 0x7f904c06d9c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9068159a90 tx=0x7f9058008040 comp rx=0 tx=0).stop 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 msgr2=0x7f9068153d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f9068153d30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f906400e910 tx=0x7f906400edd0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 shutdown_connections 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f904c06b500 0x7f904c06d9c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f90680acdb0 0x7f90680afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f90681591b0 0x7f90680ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 --2- 172.21.15.99:0/2554819864 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9054004d40 0x7f9068153d30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 >> 172.21.15.99:0/2554819864 conn(0x7f906806acd0 msgr2=0x7f90680b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 shutdown_connections 2024-03-04T10:04:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.496+0000 7f9070630640 1 -- 172.21.15.99:0/2554819864 wait complete. 2024-03-04T10:04:27.509 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-03-04T10:04:27.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.610+0000 7fb18b9df640 1 Processor -- start 2024-03-04T10:04:27.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.610+0000 7fb18b9df640 1 -- start start 2024-03-04T10:04:27.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.611+0000 7fb18b9df640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb1840cf650 0x7fb1840cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.611+0000 7fb18b9df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184070de0 0x7fb184071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.611+0000 7fb18b9df640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb18415f020 con 0x7fb184154a80 2024-03-04T10:04:27.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.611+0000 7fb18b9df640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1840cd010 con 0x7fb1840cf650 2024-03-04T10:04:27.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.611+0000 7fb18b9df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1840cdaa0 con 0x7fb184070de0 2024-03-04T10:04:27.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb189f7c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184070de0 0x7fb184071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb188f7a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb1840cf650 0x7fb1840cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb18977b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb184154a80 0x7fb18415e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35514/0 (socket says 172.21.15.99:35514) 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb18977b640 1 -- 172.21.15.99:0/177662967 learned_addr learned my addr 172.21.15.99:0/177662967 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb17bfff640 1 -- 172.21.15.99:0/177662967 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816286513 0 0) 0x7fb180001490 con 0x7fb184154a80 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb189f7c640 1 -- 172.21.15.99:0/177662967 >> v2:172.21.15.162:3300/0 conn(0x7fb1840cf650 msgr2=0x7fb1840cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb189f7c640 1 --2- 172.21.15.99:0/177662967 >> v2:172.21.15.162:3300/0 conn(0x7fb1840cf650 0x7fb1840cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb189f7c640 1 -- 172.21.15.99:0/177662967 >> v1:172.21.15.162:6789/0 conn(0x7fb184154a80 legacy=0x7fb18415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.612+0000 7fb189f7c640 1 -- 172.21.15.99:0/177662967 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1840cdd80 con 0x7fb184070de0 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb189f7c640 1 --2- 172.21.15.99:0/177662967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184070de0 0x7fb184071270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0097a0 tx=0x7fb17c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27067ef69ad98215 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb17bfff640 1 -- 172.21.15.99:0/177662967 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c035020 con 0x7fb184070de0 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb17bfff640 1 -- 172.21.15.99:0/177662967 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c031830 con 0x7fb184070de0 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb17bfff640 1 -- 172.21.15.99:0/177662967 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c031c00 con 0x7fb184070de0 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb18b9df640 1 -- 172.21.15.99:0/177662967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184070de0 msgr2=0x7fb184071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb18b9df640 1 --2- 172.21.15.99:0/177662967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184070de0 0x7fb184071270 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0097a0 tx=0x7fb17c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb18b9df640 1 -- 172.21.15.99:0/177662967 shutdown_connections 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb18b9df640 1 --2- 172.21.15.99:0/177662967 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184070de0 0x7fb184071270 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb18b9df640 1 --2- 172.21.15.99:0/177662967 >> v2:172.21.15.162:3300/0 conn(0x7fb1840cf650 0x7fb1840cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.613+0000 7fb18b9df640 1 -- 172.21.15.99:0/177662967 >> 172.21.15.99:0/177662967 conn(0x7fb18406f5f0 msgr2=0x7fb18406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.614+0000 7fb18b9df640 1 -- 172.21.15.99:0/177662967 shutdown_connections 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.614+0000 7fb18b9df640 1 -- 172.21.15.99:0/177662967 wait complete. 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 Processor -- start 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 -- start start 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 0x7fb1841536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1840cf650 0x7fb184156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184154a80 0x7fb1840b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1840ce620 con 0x7fb184070de0 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1840ce340 con 0x7fb1840cf650 2024-03-04T10:04:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.615+0000 7fb18b9df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb184154e60 con 0x7fb184154a80 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 0x7fb1841536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 0x7fb1841536b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51244/0 (socket says 172.21.15.99:51244) 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 -- 172.21.15.99:0/3434452060 learned_addr learned my addr 172.21.15.99:0/3434452060 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb189f7c640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184154a80 0x7fb1840b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb188f7a640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1840cf650 0x7fb184156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 -- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184154a80 msgr2=0x7fb1840b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184154a80 0x7fb1840b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 -- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1840cf650 msgr2=0x7fb184156be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1840cf650 0x7fb184156be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 -- 172.21.15.99:0/3434452060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb184153bf0 con 0x7fb184070de0 2024-03-04T10:04:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18977b640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 0x7fb1841536b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb18000c7c0 tx=0x7fb18000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180018070 con 0x7fb184070de0 2024-03-04T10:04:27.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18000a640 con 0x7fb184070de0 2024-03-04T10:04:27.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1841554e0 con 0x7fb184070de0 2024-03-04T10:04:27.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18000aa30 con 0x7fb184070de0 2024-03-04T10:04:27.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.616+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb184155aa0 con 0x7fb184070de0 2024-03-04T10:04:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.618+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb184074f60 con 0x7fb184070de0 2024-03-04T10:04:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.618+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb18001b070 con 0x7fb184070de0 2024-03-04T10:04:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.618+0000 7fb179ffb640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb16806b4d0 0x7fb16806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.618+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb180089a90 con 0x7fb184070de0 2024-03-04T10:04:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.618+0000 7fb188f7a640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb16806b4d0 0x7fb16806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.619+0000 7fb188f7a640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb16806b4d0 0x7fb16806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb1841f8340 tx=0x7fb174009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:27.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.623+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb18415f020 con 0x7fb184070de0 2024-03-04T10:04:27.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:27.864+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) v1 -- 0x7fb1840cd7c0 con 0x7fb184070de0 2024-03-04T10:04:28.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.692+0000 7fb179ffb640 1 -- 172.21.15.99:0/3434452060 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v69) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb180057f60 con 0x7fb184070de0 2024-03-04T10:04:28.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb16806b4d0 msgr2=0x7fb16806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:28.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb16806b4d0 0x7fb16806d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb1841f8340 tx=0x7fb174009190 comp rx=0 tx=0).stop 2024-03-04T10:04:28.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 msgr2=0x7fb1841536b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:28.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 0x7fb1841536b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb18000c7c0 tx=0x7fb18000cc80 comp rx=0 tx=0).stop 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 shutdown_connections 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb16806b4d0 0x7fb16806d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb184154a80 0x7fb1840b2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1840cf650 0x7fb184156be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 --2- 172.21.15.99:0/3434452060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb184070de0 0x7fb1841536b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.696+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 >> 172.21.15.99:0/3434452060 conn(0x7fb18406f5f0 msgr2=0x7fb18406c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.697+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 shutdown_connections 2024-03-04T10:04:28.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.697+0000 7fb18b9df640 1 -- 172.21.15.99:0/3434452060 wait complete. 2024-03-04T10:04:28.711 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T10:04:28.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.822+0000 7fb4c16c0640 1 Processor -- start 2024-03-04T10:04:28.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.822+0000 7fb4c16c0640 1 -- start start 2024-03-04T10:04:28.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4c16c0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb4bc0cec40 0x7fb4bc0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4c16c0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0cf020 0x7fb4bc15d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4c16c0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb4bc0d3300 0x7fb4bc15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4c16c0640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4bc0ca970 con 0x7fb4bc0cec40 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4c16c0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4bc161d90 con 0x7fb4bc0d3300 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4c16c0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc162070 con 0x7fb4bc0cf020 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb4bc0cec40 0x7fb4bc0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb4bc0cec40 0x7fb4bc0b6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51260/0 (socket says 172.21.15.99:51260) 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 -- 172.21.15.99:0/3025021333 learned_addr learned my addr 172.21.15.99:0/3025021333 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4bb7fe640 1 --2- 172.21.15.99:0/3025021333 >> v2:172.21.15.162:3300/0 conn(0x7fb4bc0d3300 0x7fb4bc15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4b3fff640 1 --2- 172.21.15.99:0/3025021333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0cf020 0x7fb4bc15d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 -- 172.21.15.99:0/3025021333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0cf020 msgr2=0x7fb4bc15d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 --2- 172.21.15.99:0/3025021333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0cf020 0x7fb4bc15d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 -- 172.21.15.99:0/3025021333 >> v2:172.21.15.162:3300/0 conn(0x7fb4bc0d3300 msgr2=0x7fb4bc15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 --2- 172.21.15.99:0/3025021333 >> v2:172.21.15.162:3300/0 conn(0x7fb4bc0d3300 0x7fb4bc15dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.823+0000 7fb4baffd640 1 -- 172.21.15.99:0/3025021333 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc162350 con 0x7fb4bc0cec40 2024-03-04T10:04:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.824+0000 7fb4baffd640 1 --2- 172.21.15.99:0/3025021333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0b6ea0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0d18d0 tx=0x7fb4ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417396541157ba04 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.824+0000 7fb4ba7fc640 1 -- 172.21.15.99:0/3025021333 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac035020 con 0x7fb4bc0cec40 2024-03-04T10:04:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.824+0000 7fb4ba7fc640 1 -- 172.21.15.99:0/3025021333 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac031870 con 0x7fb4bc0cec40 2024-03-04T10:04:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.824+0000 7fb4ba7fc640 1 -- 172.21.15.99:0/3025021333 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac031c40 con 0x7fb4bc0cec40 2024-03-04T10:04:28.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 -- 172.21.15.99:0/3025021333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 msgr2=0x7fb4bc0b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/3025021333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0b6ea0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0d18d0 tx=0x7fb4ac02b910 comp rx=0 tx=0).stop 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 -- 172.21.15.99:0/3025021333 shutdown_connections 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/3025021333 >> v2:172.21.15.162:3300/0 conn(0x7fb4bc0d3300 0x7fb4bc15dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/3025021333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0cf020 0x7fb4bc15d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/3025021333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0b6ea0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 -- 172.21.15.99:0/3025021333 >> 172.21.15.99:0/3025021333 conn(0x7fb4bc0d0580 msgr2=0x7fb4bc0d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 -- 172.21.15.99:0/3025021333 shutdown_connections 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.825+0000 7fb4c16c0640 1 -- 172.21.15.99:0/3025021333 wait complete. 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.826+0000 7fb4c16c0640 1 Processor -- start 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.826+0000 7fb4c16c0640 1 -- start start 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4c16c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4c16c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4bc0cf020 0x7fb4bc15f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4c16c0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0d3300 0x7fb4bc15f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4c16c0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc1617d0 con 0x7fb4bc0cec40 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4c16c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc0d1bf0 con 0x7fb4bc0cf020 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4c16c0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc161950 con 0x7fb4bc0d3300 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51272/0 (socket says 172.21.15.99:51272) 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 -- 172.21.15.99:0/4221774500 learned_addr learned my addr 172.21.15.99:0/4221774500 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4b3fff640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4bc0cf020 0x7fb4bc15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4bb7fe640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0d3300 0x7fb4bc15f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 -- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0d3300 msgr2=0x7fb4bc15f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0d3300 0x7fb4bc15f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 -- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4bc0cf020 msgr2=0x7fb4bc15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4bc0cf020 0x7fb4bc15f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 -- 172.21.15.99:0/4221774500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4bc15fff0 con 0x7fb4bc0cec40 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.827+0000 7fb4baffd640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0cce20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac02b950 tx=0x7fb4ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.828+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac040070 con 0x7fb4bc0cec40 2024-03-04T10:04:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.828+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ac030d00 con 0x7fb4bc0cec40 2024-03-04T10:04:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.828+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac035040 con 0x7fb4bc0cec40 2024-03-04T10:04:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.828+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc1602d0 con 0x7fb4bc0cec40 2024-03-04T10:04:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.828+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc0aa730 con 0x7fb4bc0cec40 2024-03-04T10:04:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.829+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4bc074f60 con 0x7fb4bc0cec40 2024-03-04T10:04:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.829+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb4ac04b060 con 0x7fb4bc0cec40 2024-03-04T10:04:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.829+0000 7fb4b37fe640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb49c06b4d0 0x7fb49c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.829+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fb4ac0ac310 con 0x7fb4bc0cec40 2024-03-04T10:04:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.830+0000 7fb4b3fff640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb49c06b4d0 0x7fb49c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.830+0000 7fb4b3fff640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb49c06b4d0 0x7fb49c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc15f300 tx=0x7fb4a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:28.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:28.834+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb4ac07ae30 con 0x7fb4bc0cec40 2024-03-04T10:04:29.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.005+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb4bc0d1930 con 0x7fb4bc0cec40 2024-03-04T10:04:29.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.009+0000 7fb4b37fe640 1 -- 172.21.15.99:0/4221774500 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb4ac07a730 con 0x7fb4bc0cec40 2024-03-04T10:04:29.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb49c06b4d0 msgr2=0x7fb49c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb49c06b4d0 0x7fb49c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc15f300 tx=0x7fb4a4008040 comp rx=0 tx=0).stop 2024-03-04T10:04:29.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 msgr2=0x7fb4bc0cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0cce20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac02b950 tx=0x7fb4ac0046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 shutdown_connections 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb49c06b4d0 0x7fb49c06d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb4bc0d3300 0x7fb4bc15f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb4bc0cf020 0x7fb4bc15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 --2- 172.21.15.99:0/4221774500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb4bc0cec40 0x7fb4bc0cce20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 >> 172.21.15.99:0/4221774500 conn(0x7fb4bc0d0580 msgr2=0x7fb4bc15cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 shutdown_connections 2024-03-04T10:04:29.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.011+0000 7fb4c16c0640 1 -- 172.21.15.99:0/4221774500 wait complete. 2024-03-04T10:04:29.023 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:04:29.023 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:04:29.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:04:29.135 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:04:29.135 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:29.136 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:29.151 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:29.151 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:29.206 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:29.206 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:29.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:29.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 Processor -- start 2024-03-04T10:04:29.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 -- start start 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa35415de00 0x7fa3540b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3540b9380 0x7fa3540b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa3540b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3540d2820 con 0x7fa3540b9380 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35415e1e0 con 0x7fa35415de00 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa35b229640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540d1c10 con 0x7fa3540b70d0 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa353fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3540b9380 0x7fa3540b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa353fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3540b9380 0x7fa3540b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51284/0 (socket says 172.21.15.99:51284) 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.403+0000 7fa353fff640 1 -- 172.21.15.99:0/4105416901 learned_addr learned my addr 172.21.15.99:0/4105416901 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.404+0000 7fa353fff640 1 -- 172.21.15.99:0/4105416901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 msgr2=0x7fa3540b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.404+0000 7fa353fff640 1 --2- 172.21.15.99:0/4105416901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa3540b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.404+0000 7fa353fff640 1 -- 172.21.15.99:0/4105416901 >> v2:172.21.15.162:3300/0 conn(0x7fa35415de00 msgr2=0x7fa3540b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.404+0000 7fa353fff640 1 --2- 172.21.15.99:0/4105416901 >> v2:172.21.15.162:3300/0 conn(0x7fa35415de00 0x7fa3540b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.404+0000 7fa353fff640 1 -- 172.21.15.99:0/4105416901 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540d1ef0 con 0x7fa3540b9380 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.404+0000 7fa353fff640 1 --2- 172.21.15.99:0/4105416901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa3540b6b90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa3540ce240 tx=0x7fa34002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a4a05f3daa56c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa3537fe640 1 -- 172.21.15.99:0/4105416901 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340035020 con 0x7fa3540b9380 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa3537fe640 1 -- 172.21.15.99:0/4105416901 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa340031870 con 0x7fa3540b9380 2024-03-04T10:04:29.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa3537fe640 1 -- 172.21.15.99:0/4105416901 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340031c40 con 0x7fa3540b9380 2024-03-04T10:04:29.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 -- 172.21.15.99:0/4105416901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 msgr2=0x7fa3540b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 --2- 172.21.15.99:0/4105416901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa3540b6b90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa3540ce240 tx=0x7fa34002b910 comp rx=0 tx=0).stop 2024-03-04T10:04:29.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 -- 172.21.15.99:0/4105416901 shutdown_connections 2024-03-04T10:04:29.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 --2- 172.21.15.99:0/4105416901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa3540b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 --2- 172.21.15.99:0/4105416901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa3540b6b90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 --2- 172.21.15.99:0/4105416901 >> v2:172.21.15.162:3300/0 conn(0x7fa35415de00 0x7fa3540b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 -- 172.21.15.99:0/4105416901 >> 172.21.15.99:0/4105416901 conn(0x7fa3540d0470 msgr2=0x7fa3540d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 -- 172.21.15.99:0/4105416901 shutdown_connections 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.405+0000 7fa35b229640 1 -- 172.21.15.99:0/4105416901 wait complete. 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.406+0000 7fa35b229640 1 Processor -- start 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.406+0000 7fa35b229640 1 -- start start 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa35b229640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa35415b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa35b229640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa354160c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa35b229640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa35415de00 0x7fa3541e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa35b229640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b97e0 con 0x7fa3540b9380 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa35b229640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540d24b0 con 0x7fa35415de00 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa35b229640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3540b59b0 con 0x7fa3540b70d0 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa353fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa354160c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa353fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa354160c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51294/0 (socket says 172.21.15.99:51294) 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.407+0000 7fa353fff640 1 -- 172.21.15.99:0/338821313 learned_addr learned my addr 172.21.15.99:0/338821313 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa353fff640 1 -- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 msgr2=0x7fa35415b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa358fc5640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa35415b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa3597c6640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa35415de00 0x7fa3541e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa353fff640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa35415b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa353fff640 1 -- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa35415de00 msgr2=0x7fa3541e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa353fff640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa35415de00 0x7fa3541e75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa353fff640 1 -- 172.21.15.99:0/338821313 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540b4e10 con 0x7fa3540b9380 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa353fff640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa354160c90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa340000c00 tx=0x7fa3400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34003ad60 con 0x7fa3540b9380 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3400443f0 con 0x7fa3540b9380 2024-03-04T10:04:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3400447e0 con 0x7fa3540b9380 2024-03-04T10:04:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3540b50f0 con 0x7fa3540b9380 2024-03-04T10:04:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.408+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3540b2b40 con 0x7fa3540b9380 2024-03-04T10:04:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.409+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa354074f60 con 0x7fa3540b9380 2024-03-04T10:04:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.410+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa34004b060 con 0x7fa3540b9380 2024-03-04T10:04:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.410+0000 7fa3517fa640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa33806b500 0x7fa33806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.410+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fa3400ac870 con 0x7fa3540b9380 2024-03-04T10:04:29.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.410+0000 7fa358fc5640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa33806b500 0x7fa33806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.411+0000 7fa358fc5640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa33806b500 0x7fa33806d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3541f6d20 tx=0x7fa348008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:29.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.414+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa34003f070 con 0x7fa3540b9380 2024-03-04T10:04:29.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.596+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3540b9160 con 0x7fa3540b9380 2024-03-04T10:04:29.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.597+0000 7fa3517fa640 1 -- 172.21.15.99:0/338821313 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7fa34007b300 con 0x7fa3540b9380 2024-03-04T10:04:29.599 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:29.599 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:29.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa33806b500 msgr2=0x7fa33806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa33806b500 0x7fa33806d9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa3541f6d20 tx=0x7fa348008040 comp rx=0 tx=0).stop 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 msgr2=0x7fa354160c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa354160c90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa340000c00 tx=0x7fa3400046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 shutdown_connections 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa33806b500 0x7fa33806d9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa35415de00 0x7fa3541e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3540b9380 0x7fa354160c90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 --2- 172.21.15.99:0/338821313 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3540b70d0 0x7fa35415b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 >> 172.21.15.99:0/338821313 conn(0x7fa3540d0470 msgr2=0x7fa3540d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 shutdown_connections 2024-03-04T10:04:29.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.599+0000 7fa35b229640 1 -- 172.21.15.99:0/338821313 wait complete. 2024-03-04T10:04:29.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:29.613 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:29.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.719+0000 7fc4a6881640 1 Processor -- start 2024-03-04T10:04:29.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.719+0000 7fc4a6881640 1 -- start start 2024-03-04T10:04:29.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a6881640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a00cfc70 0x7fc4a00cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a6881640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc4a00cc340 0x7fc4a00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a6881640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a00cebd0 con 0x7fc4a01450e0 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a6881640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a00cf7e0 con 0x7fc4a00cc340 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a6881640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00cc780 con 0x7fc4a00cfc70 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a4e1e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc4a00cc340 0x7fc4a00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc49ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc4a01450e0 0x7fc4a015c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57388/0 (socket says 172.21.15.99:57388) 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc49ffff640 1 -- 172.21.15.99:0/2201354436 learned_addr learned my addr 172.21.15.99:0/2201354436 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.720+0000 7fc4a4e1e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc4a00cc340 0x7fc4a00b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51298/0 (socket says 172.21.15.99:51298) 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc49f7fe640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a00cfc70 0x7fc4a00cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc4a4e1e640 1 -- 172.21.15.99:0/2201354436 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a00cfc70 msgr2=0x7fc4a00cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc4a4e1e640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a00cfc70 0x7fc4a00cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc4a4e1e640 1 -- 172.21.15.99:0/2201354436 >> v1:172.21.15.99:6789/0 conn(0x7fc4a01450e0 legacy=0x7fc4a015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc4a4e1e640 1 -- 172.21.15.99:0/2201354436 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a00ceeb0 con 0x7fc4a00cc340 2024-03-04T10:04:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc49effd640 1 -- 172.21.15.99:0/2201354436 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848922268 0 0) 0x7fc4a00cebd0 con 0x7fc4a01450e0 2024-03-04T10:04:29.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.721+0000 7fc49f7fe640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a00cfc70 0x7fc4a00cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.722+0000 7fc4a4e1e640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cc340 0x7fc4a00b3250 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d90 tx=0x7fc49802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7e12781de87612 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.722+0000 7fc49effd640 1 -- 172.21.15.99:0/2201354436 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498039070 con 0x7fc4a00cc340 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.722+0000 7fc49effd640 1 -- 172.21.15.99:0/2201354436 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49802cc60 con 0x7fc4a00cc340 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.722+0000 7fc49effd640 1 -- 172.21.15.99:0/2201354436 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498034660 con 0x7fc4a00cc340 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.722+0000 7fc4a6881640 1 -- 172.21.15.99:0/2201354436 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cc340 msgr2=0x7fc4a00b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.722+0000 7fc4a6881640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cc340 0x7fc4a00b3250 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d90 tx=0x7fc49802beb0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.723+0000 7fc4a6881640 1 -- 172.21.15.99:0/2201354436 shutdown_connections 2024-03-04T10:04:29.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.723+0000 7fc4a6881640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cc340 0x7fc4a00b3250 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.723+0000 7fc4a6881640 1 --2- 172.21.15.99:0/2201354436 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a00cfc70 0x7fc4a00cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.723+0000 7fc4a6881640 1 -- 172.21.15.99:0/2201354436 >> 172.21.15.99:0/2201354436 conn(0x7fc4a006acd0 msgr2=0x7fc4a006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.723+0000 7fc4a6881640 1 -- 172.21.15.99:0/2201354436 shutdown_connections 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.723+0000 7fc4a6881640 1 -- 172.21.15.99:0/2201354436 wait complete. 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.724+0000 7fc4a6881640 1 Processor -- start 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.724+0000 7fc4a6881640 1 -- start start 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4a00cc340 0x7fc4a013d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 0x7fc4a013b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a01450e0 0x7fc4a013f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a01454c0 con 0x7fc4a00cfc70 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00cf470 con 0x7fc4a00cc340 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00ccd10 con 0x7fc4a01450e0 2024-03-04T10:04:29.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 0x7fc4a013b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 0x7fc4a013b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51310/0 (socket says 172.21.15.99:51310) 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 -- 172.21.15.99:0/790349161 learned_addr learned my addr 172.21.15.99:0/790349161 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a4e1e640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a01450e0 0x7fc4a013f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49ffff640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4a00cc340 0x7fc4a013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 -- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a01450e0 msgr2=0x7fc4a013f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a01450e0 0x7fc4a013f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 -- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4a00cc340 msgr2=0x7fc4a013d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4a00cc340 0x7fc4a013d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 -- 172.21.15.99:0/790349161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a013bf70 con 0x7fc4a00cfc70 2024-03-04T10:04:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a4e1e640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a01450e0 0x7fc4a013f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:29.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49f7fe640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 0x7fc4a013b6f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e7e0 tx=0x7fc49400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:29.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49400f040 con 0x7fc4a00cfc70 2024-03-04T10:04:29.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc494004030 con 0x7fc4a00cfc70 2024-03-04T10:04:29.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a013c250 con 0x7fc4a00cfc70 2024-03-04T10:04:29.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc494004400 con 0x7fc4a00cfc70 2024-03-04T10:04:29.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.725+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a013c810 con 0x7fc4a00cfc70 2024-03-04T10:04:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.727+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0074f60 con 0x7fc4a00cfc70 2024-03-04T10:04:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.727+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc49400c6d0 con 0x7fc4a00cfc70 2024-03-04T10:04:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.727+0000 7fc49cff9640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc48c06b4d0 0x7fc48c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.728+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc49408a340 con 0x7fc4a00cfc70 2024-03-04T10:04:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.728+0000 7fc49ffff640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc48c06b4d0 0x7fc48c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:29.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.728+0000 7fc49ffff640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc48c06b4d0 0x7fc48c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc490000f60 tx=0x7fc490007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:29.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.732+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc494017020 con 0x7fc4a00cfc70 2024-03-04T10:04:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.910+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4a00d0590 con 0x7fc4a00cfc70 2024-03-04T10:04:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.910+0000 7fc49cff9640 1 -- 172.21.15.99:0/790349161 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc494058e60 con 0x7fc4a00cfc70 2024-03-04T10:04:29.912 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.912+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc48c06b4d0 msgr2=0x7fc48c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.912+0000 7fc4a6881640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc48c06b4d0 0x7fc48c06d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc490000f60 tx=0x7fc490007c00 comp rx=0 tx=0).stop 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 msgr2=0x7fc4a013b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 0x7fc4a013b6f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e7e0 tx=0x7fc49400eca0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 shutdown_connections 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc48c06b4d0 0x7fc48c06d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc4a01450e0 0x7fc4a013f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc4a00cfc70 0x7fc4a013b6f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 --2- 172.21.15.99:0/790349161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc4a00cc340 0x7fc4a013d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 >> 172.21.15.99:0/790349161 conn(0x7fc4a006acd0 msgr2=0x7fc4a00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 shutdown_connections 2024-03-04T10:04:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:29.913+0000 7fc4a6881640 1 -- 172.21.15.99:0/790349161 wait complete. 2024-03-04T10:04:29.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:30.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.029+0000 7f1d3a11b640 1 Processor -- start 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.029+0000 7f1d3a11b640 1 -- start start 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d3a11b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1d340ae5a0 0x7f1d340ae980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d3a11b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d340cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d3a11b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d340aeec0 con 0x7f1d3415eca0 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d3a11b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d341617b0 con 0x7f1d340ae5a0 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d3a11b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34161a90 con 0x7f1d3415e840 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d337fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1d340ae5a0 0x7f1d340ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d33fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1d3415eca0 0x7f1d340cf860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57398/0 (socket says 172.21.15.99:57398) 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d337fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1d340ae5a0 0x7f1d340ae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51318/0 (socket says 172.21.15.99:51318) 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d33fff640 1 -- 172.21.15.99:0/2267600214 learned_addr learned my addr 172.21.15.99:0/2267600214 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:30.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d32ffd640 1 --2- 172.21.15.99:0/2267600214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d340cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d337fe640 1 -- 172.21.15.99:0/2267600214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 msgr2=0x7f1d340cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d337fe640 1 --2- 172.21.15.99:0/2267600214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d340cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d337fe640 1 -- 172.21.15.99:0/2267600214 >> v1:172.21.15.99:6789/0 conn(0x7f1d3415eca0 legacy=0x7f1d340cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.030+0000 7f1d337fe640 1 -- 172.21.15.99:0/2267600214 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d34161d70 con 0x7f1d340ae5a0 2024-03-04T10:04:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.031+0000 7f1d337fe640 1 --2- 172.21.15.99:0/2267600214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d340ae5a0 0x7f1d340ae980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009d90 tx=0x7f1d2c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b616bbbc44a681f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.031+0000 7f1d327fc640 1 -- 172.21.15.99:0/2267600214 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c039070 con 0x7f1d340ae5a0 2024-03-04T10:04:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.031+0000 7f1d327fc640 1 -- 172.21.15.99:0/2267600214 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c02cb30 con 0x7f1d340ae5a0 2024-03-04T10:04:30.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.031+0000 7f1d327fc640 1 -- 172.21.15.99:0/2267600214 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c034440 con 0x7f1d340ae5a0 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 -- 172.21.15.99:0/2267600214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d340ae5a0 msgr2=0x7f1d340ae980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/2267600214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d340ae5a0 0x7f1d340ae980 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009d90 tx=0x7f1d2c02bd80 comp rx=0 tx=0).stop 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 -- 172.21.15.99:0/2267600214 shutdown_connections 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/2267600214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d340cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/2267600214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d340ae5a0 0x7f1d340ae980 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 -- 172.21.15.99:0/2267600214 >> 172.21.15.99:0/2267600214 conn(0x7f1d3406acd0 msgr2=0x7f1d3406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 -- 172.21.15.99:0/2267600214 shutdown_connections 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.032+0000 7f1d3a11b640 1 -- 172.21.15.99:0/2267600214 wait complete. 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.033+0000 7f1d3a11b640 1 Processor -- start 2024-03-04T10:04:30.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.033+0000 7f1d3a11b640 1 -- start start 2024-03-04T10:04:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d3a11b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d340ae5a0 0x7f1d3413f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d3a11b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d34140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d3a11b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 0x7f1d341510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d3a11b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3415e2a0 con 0x7f1d3415eca0 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d3a11b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34162330 con 0x7f1d340ae5a0 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d3a11b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3415e420 con 0x7f1d3415e840 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 0x7f1d341510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 0x7f1d341510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51326/0 (socket says 172.21.15.99:51326) 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 -- 172.21.15.99:0/3764416601 learned_addr learned my addr 172.21.15.99:0/3764416601 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d32ffd640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d34140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d337fe640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d340ae5a0 0x7f1d3413f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 -- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 msgr2=0x7f1d34140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d34140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 -- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d340ae5a0 msgr2=0x7f1d3413f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d340ae5a0 0x7f1d3413f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.034+0000 7f1d33fff640 1 -- 172.21.15.99:0/3764416601 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3413fda0 con 0x7f1d3415eca0 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d337fe640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d340ae5a0 0x7f1d3413f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d33fff640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 0x7f1d341510c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c370 tx=0x7f1d2400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d24017070 con 0x7f1d3415eca0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d24005ce0 con 0x7f1d3415eca0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d34140b50 con 0x7f1d3415eca0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d240072d0 con 0x7f1d3415eca0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.035+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d34141110 con 0x7f1d3415eca0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.036+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d34074f60 con 0x7f1d3415eca0 2024-03-04T10:04:30.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.036+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1d2401a070 con 0x7f1d3415eca0 2024-03-04T10:04:30.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.037+0000 7f1d2bfff640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d1406b4d0 0x7f1d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.037+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f1d24089830 con 0x7f1d3415eca0 2024-03-04T10:04:30.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.037+0000 7f1d337fe640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d1406b4d0 0x7f1d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.038+0000 7f1d337fe640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d1406b4d0 0x7f1d1406d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c036000 tx=0x7f1d2c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.041+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1d24057c60 con 0x7f1d3415eca0 2024-03-04T10:04:30.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.218+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1d340adcb0 con 0x7f1d3415eca0 2024-03-04T10:04:30.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.218+0000 7f1d2bfff640 1 -- 172.21.15.99:0/3764416601 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1d2405bd00 con 0x7f1d3415eca0 2024-03-04T10:04:30.221 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d1406b4d0 msgr2=0x7f1d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d1406b4d0 0x7f1d1406d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c036000 tx=0x7f1d2c030000 comp rx=0 tx=0).stop 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 msgr2=0x7f1d341510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 0x7f1d341510c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400c370 tx=0x7f1d2400c830 comp rx=0 tx=0).stop 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 shutdown_connections 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1d1406b4d0 0x7f1d1406d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1d3415eca0 0x7f1d341510c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1d3415e840 0x7f1d34140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 --2- 172.21.15.99:0/3764416601 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d340ae5a0 0x7f1d3413f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.220+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 >> 172.21.15.99:0/3764416601 conn(0x7f1d3406acd0 msgr2=0x7f1d340acea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.221+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 shutdown_connections 2024-03-04T10:04:30.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.221+0000 7f1d3a11b640 1 -- 172.21.15.99:0/3764416601 wait complete. 2024-03-04T10:04:30.233 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:30.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.336+0000 7fd28e9b2640 1 Processor -- start 2024-03-04T10:04:30.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.336+0000 7fd28e9b2640 1 -- start start 2024-03-04T10:04:30.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28e9b2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd2880b5700 0x7fd2880d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28e9b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880d34a0 0x7fd28806c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28e9b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2880b9410 0x7fd28806c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28e9b2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2880d2400 con 0x7fd2880b9410 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28e9b2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2880d26e0 con 0x7fd2880b5700 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28e9b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880d29c0 con 0x7fd2880d34a0 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2880b9410 0x7fd28806c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2880b9410 0x7fd28806c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51334/0 (socket says 172.21.15.99:51334) 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 -- 172.21.15.99:0/3162342581 learned_addr learned my addr 172.21.15.99:0/3162342581 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 -- 172.21.15.99:0/3162342581 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880d34a0 msgr2=0x7fd28806c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 --2- 172.21.15.99:0/3162342581 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880d34a0 0x7fd28806c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd287fff640 1 --2- 172.21.15.99:0/3162342581 >> v2:172.21.15.162:3300/0 conn(0x7fd2880b5700 0x7fd2880d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 -- 172.21.15.99:0/3162342581 >> v2:172.21.15.162:3300/0 conn(0x7fd2880b5700 msgr2=0x7fd2880d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 --2- 172.21.15.99:0/3162342581 >> v2:172.21.15.162:3300/0 conn(0x7fd2880b5700 0x7fd2880d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.337+0000 7fd28cf4f640 1 -- 172.21.15.99:0/3162342581 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880d2ca0 con 0x7fd2880b9410 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.338+0000 7fd28cf4f640 1 --2- 172.21.15.99:0/3162342581 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880b9410 0x7fd28806c620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd28806dad0 tx=0x7fd28002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b897ac52f92a69e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.338+0000 7fd286ffd640 1 -- 172.21.15.99:0/3162342581 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28002c9e0 con 0x7fd2880b9410 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.338+0000 7fd286ffd640 1 -- 172.21.15.99:0/3162342581 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28002cb80 con 0x7fd2880b9410 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.338+0000 7fd286ffd640 1 -- 172.21.15.99:0/3162342581 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2800343f0 con 0x7fd2880b9410 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.338+0000 7fd28e9b2640 1 -- 172.21.15.99:0/3162342581 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880b9410 msgr2=0x7fd28806c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.338+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/3162342581 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880b9410 0x7fd28806c620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd28806dad0 tx=0x7fd28002be40 comp rx=0 tx=0).stop 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 -- 172.21.15.99:0/3162342581 shutdown_connections 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/3162342581 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880b9410 0x7fd28806c620 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/3162342581 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880d34a0 0x7fd28806c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/3162342581 >> v2:172.21.15.162:3300/0 conn(0x7fd2880b5700 0x7fd2880d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 -- 172.21.15.99:0/3162342581 >> 172.21.15.99:0/3162342581 conn(0x7fd2880d0490 msgr2=0x7fd2880d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 -- 172.21.15.99:0/3162342581 shutdown_connections 2024-03-04T10:04:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.339+0000 7fd28e9b2640 1 -- 172.21.15.99:0/3162342581 wait complete. 2024-03-04T10:04:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.340+0000 7fd28e9b2640 1 Processor -- start 2024-03-04T10:04:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.340+0000 7fd28e9b2640 1 -- start start 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880b5700 0x7fd2880b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2880b9410 0x7fd2880cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 0x7fd2881f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880b5c60 con 0x7fd2880d34a0 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880b5ae0 con 0x7fd2880b9410 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880b5de0 con 0x7fd2880b5700 2024-03-04T10:04:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 0x7fd2881f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 0x7fd2881f3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51350/0 (socket says 172.21.15.99:51350) 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 -- 172.21.15.99:0/540145273 learned_addr learned my addr 172.21.15.99:0/540145273 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 -- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880b5700 msgr2=0x7fd2880b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880b5700 0x7fd2880b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 -- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2880b9410 msgr2=0x7fd2880cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2880b9410 0x7fd2880cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 -- 172.21.15.99:0/540145273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2880c88a0 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28cf4f640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 0x7fd2881f3490 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd280009f40 tx=0x7fd280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280043020 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28003e450 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280009510 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2880c8b80 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.341+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2880c9140 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.342+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd288074f60 con 0x7fd2880d34a0 2024-03-04T10:04:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.343+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd280005d10 con 0x7fd2880d34a0 2024-03-04T10:04:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.343+0000 7fd284ff9640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27406b500 0x7fd27406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.343+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fd2800acaf0 con 0x7fd2880d34a0 2024-03-04T10:04:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.343+0000 7fd287fff640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27406b500 0x7fd27406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.344+0000 7fd287fff640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27406b500 0x7fd27406d9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00a810 tx=0x7fd27c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.348+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd28004d020 con 0x7fd2880d34a0 2024-03-04T10:04:30.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.523+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd28806b1b0 con 0x7fd2880d34a0 2024-03-04T10:04:30.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.523+0000 7fd284ff9640 1 -- 172.21.15.99:0/540145273 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd28007b610 con 0x7fd2880d34a0 2024-03-04T10:04:30.525 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:30.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27406b500 msgr2=0x7fd27406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27406b500 0x7fd27406d9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00a810 tx=0x7fd27c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 msgr2=0x7fd2881f3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 0x7fd2881f3490 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd280009f40 tx=0x7fd280004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 shutdown_connections 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27406b500 0x7fd27406d9c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2880d34a0 0x7fd2881f3490 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2880b9410 0x7fd2880cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 --2- 172.21.15.99:0/540145273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2880b5700 0x7fd2880b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 >> 172.21.15.99:0/540145273 conn(0x7fd2880d0490 msgr2=0x7fd2880ceae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 shutdown_connections 2024-03-04T10:04:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.525+0000 7fd28e9b2640 1 -- 172.21.15.99:0/540145273 wait complete. 2024-03-04T10:04:30.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.638+0000 7f3f5b061640 1 Processor -- start 2024-03-04T10:04:30.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.638+0000 7f3f5b061640 1 -- start start 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f5b061640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415bb70 0x7f3f540b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f5b061640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3f5415f660 0x7f3f540d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f5b061640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5415bf50 con 0x7f3f540d1cc0 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f5b061640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f540cf890 con 0x7f3f5415f660 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f5b061640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5415fb60 con 0x7f3f5415bb70 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f53fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3f5415f660 0x7f3f540d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f595fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3f540d1cc0 0x7f3f540ce1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57406/0 (socket says 172.21.15.99:57406) 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.639+0000 7f3f595fe640 1 -- 172.21.15.99:0/3498790959 learned_addr learned my addr 172.21.15.99:0/3498790959 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f58dfd640 1 --2- 172.21.15.99:0/3498790959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415bb70 0x7f3f540b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f53fff640 1 -- 172.21.15.99:0/3498790959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415bb70 msgr2=0x7f3f540b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f53fff640 1 --2- 172.21.15.99:0/3498790959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415bb70 0x7f3f540b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f53fff640 1 -- 172.21.15.99:0/3498790959 >> v1:172.21.15.99:6789/0 conn(0x7f3f540d1cc0 legacy=0x7f3f540ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f53fff640 1 -- 172.21.15.99:0/3498790959 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5415fe40 con 0x7f3f5415f660 2024-03-04T10:04:30.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f537fe640 1 -- 172.21.15.99:0/3498790959 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925960213 0 0) 0x7f3f5415bf50 con 0x7f3f540d1cc0 2024-03-04T10:04:30.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f53fff640 1 --2- 172.21.15.99:0/3498790959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415f660 0x7f3f540d1780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3f480097a0 tx=0x7f3f4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49faccbeb945fd0c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.640+0000 7f3f537fe640 1 -- 172.21.15.99:0/3498790959 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f480315a0 con 0x7f3f5415f660 2024-03-04T10:04:30.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f537fe640 1 -- 172.21.15.99:0/3498790959 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48037070 con 0x7f3f5415f660 2024-03-04T10:04:30.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f537fe640 1 -- 172.21.15.99:0/3498790959 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48031960 con 0x7f3f5415f660 2024-03-04T10:04:30.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 -- 172.21.15.99:0/3498790959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415f660 msgr2=0x7f3f540d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 --2- 172.21.15.99:0/3498790959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415f660 0x7f3f540d1780 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3f480097a0 tx=0x7f3f4802b840 comp rx=0 tx=0).stop 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 -- 172.21.15.99:0/3498790959 shutdown_connections 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 --2- 172.21.15.99:0/3498790959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415f660 0x7f3f540d1780 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 --2- 172.21.15.99:0/3498790959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415bb70 0x7f3f540b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 -- 172.21.15.99:0/3498790959 >> 172.21.15.99:0/3498790959 conn(0x7f3f540b33a0 msgr2=0x7f3f540b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 -- 172.21.15.99:0/3498790959 shutdown_connections 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.641+0000 7f3f5b061640 1 -- 172.21.15.99:0/3498790959 wait complete. 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.642+0000 7f3f5b061640 1 Processor -- start 2024-03-04T10:04:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.642+0000 7f3f5b061640 1 -- start start 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f5b061640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f540d1cc0 0x7f3f540c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f5b061640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 0x7f3f54159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f5b061640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415f660 0x7f3f5415e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f5b061640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540b0f80 con 0x7f3f5415bb70 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f5b061640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f54160400 con 0x7f3f540d1cc0 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f5b061640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540b1100 con 0x7f3f5415f660 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f53fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 0x7f3f54159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f53fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 0x7f3f54159d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51370/0 (socket says 172.21.15.99:51370) 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f53fff640 1 -- 172.21.15.99:0/1291600452 learned_addr learned my addr 172.21.15.99:0/1291600452 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f595fe640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415f660 0x7f3f5415e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.643+0000 7f3f58dfd640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f540d1cc0 0x7f3f540c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f53fff640 1 -- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415f660 msgr2=0x7f3f5415e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f53fff640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415f660 0x7f3f5415e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f53fff640 1 -- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f540d1cc0 msgr2=0x7f3f540c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f53fff640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f540d1cc0 0x7f3f540c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f53fff640 1 -- 172.21.15.99:0/1291600452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5415a2b0 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f58dfd640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f540d1cc0 0x7f3f540c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f53fff640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 0x7f3f54159d70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3f48034040 tx=0x7f3f480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48032630 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f480327d0 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f54146990 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48032ba0 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.644+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f54146f50 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.645+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f54074f60 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.645+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3f4804e430 con 0x7f3f5415bb70 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.646+0000 7f3f517fa640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f4006b4d0 0x7f3f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.646+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f3f4802f030 con 0x7f3f5415bb70 2024-03-04T10:04:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.646+0000 7f3f58dfd640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f4006b4d0 0x7f3f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.646+0000 7f3f58dfd640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f4006b4d0 0x7f3f4006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006fd0 tx=0x7f3f44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.650+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3f48040070 con 0x7f3f5415bb70 2024-03-04T10:04:30.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.821+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3f540c8580 con 0x7f3f5415bb70 2024-03-04T10:04:30.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.821+0000 7f3f517fa640 1 -- 172.21.15.99:0/1291600452 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3f4807aee0 con 0x7f3f5415bb70 2024-03-04T10:04:30.824 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:30.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f4006b4d0 msgr2=0x7f3f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f4006b4d0 0x7f3f4006d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3f44006fd0 tx=0x7f3f44008040 comp rx=0 tx=0).stop 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 msgr2=0x7f3f54159d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 0x7f3f54159d70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3f48034040 tx=0x7f3f480046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 shutdown_connections 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f4006b4d0 0x7f3f4006d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f5415f660 0x7f3f5415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f5415bb70 0x7f3f54159d70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 --2- 172.21.15.99:0/1291600452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f540d1cc0 0x7f3f540c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 >> 172.21.15.99:0/1291600452 conn(0x7f3f540b33a0 msgr2=0x7f3f540ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 shutdown_connections 2024-03-04T10:04:30.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.823+0000 7f3f5b061640 1 -- 172.21.15.99:0/1291600452 wait complete. 2024-03-04T10:04:30.836 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:04:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 Processor -- start 2024-03-04T10:04:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 -- start start 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbe740aeb20 0x7fbe74071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbe74071890 0x7fbe74150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7415da40 con 0x7fbe7415d660 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe74071ce0 con 0x7fbe74071890 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.936+0000 7fbe7ab7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7415cbc0 con 0x7fbe740aeb20 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7911b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbe74071890 0x7fbe74150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7891a640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbe7415d660 0x7fbe7415e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57418/0 (socket says 172.21.15.99:57418) 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7891a640 1 -- 172.21.15.99:0/2974629294 learned_addr learned my addr 172.21.15.99:0/2974629294 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe73fff640 1 --2- 172.21.15.99:0/2974629294 >> v2:172.21.15.162:3300/0 conn(0x7fbe740aeb20 0x7fbe74071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7911b640 1 -- 172.21.15.99:0/2974629294 >> v2:172.21.15.162:3300/0 conn(0x7fbe740aeb20 msgr2=0x7fbe74071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7911b640 1 --2- 172.21.15.99:0/2974629294 >> v2:172.21.15.162:3300/0 conn(0x7fbe740aeb20 0x7fbe74071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7911b640 1 -- 172.21.15.99:0/2974629294 >> v1:172.21.15.99:6789/0 conn(0x7fbe7415d660 legacy=0x7fbe7415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe7911b640 1 -- 172.21.15.99:0/2974629294 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe740af090 con 0x7fbe74071890 2024-03-04T10:04:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.937+0000 7fbe737fe640 1 -- 172.21.15.99:0/2974629294 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628446594 0 0) 0x7fbe7415da40 con 0x7fbe7415d660 2024-03-04T10:04:30.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7911b640 1 --2- 172.21.15.99:0/2974629294 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe74071890 0x7fbe74150960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbe7406d200 tx=0x7fbe6402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b690197282b5a10 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe737fe640 1 -- 172.21.15.99:0/2974629294 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64038070 con 0x7fbe74071890 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe737fe640 1 -- 172.21.15.99:0/2974629294 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe64031710 con 0x7fbe74071890 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe737fe640 1 -- 172.21.15.99:0/2974629294 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64031ae0 con 0x7fbe74071890 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2974629294 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe74071890 msgr2=0x7fbe74150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2974629294 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe74071890 0x7fbe74150960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbe7406d200 tx=0x7fbe6402b7b0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2974629294 shutdown_connections 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2974629294 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe74071890 0x7fbe74150960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2974629294 >> v2:172.21.15.162:3300/0 conn(0x7fbe740aeb20 0x7fbe74071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2974629294 >> 172.21.15.99:0/2974629294 conn(0x7fbe7406acd0 msgr2=0x7fbe7406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2974629294 shutdown_connections 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.938+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2974629294 wait complete. 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.939+0000 7fbe7ab7e640 1 Processor -- start 2024-03-04T10:04:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 -- start start 2024-03-04T10:04:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe74071890 0x7fbe74148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe740aeb20 0x7fbe74157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 0x7fbe741582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7415afb0 con 0x7fbe7415d660 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7415acd0 con 0x7fbe740aeb20 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7ab7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7415a660 con 0x7fbe74071890 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.940+0000 7fbe7911b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 0x7fbe741582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 0x7fbe741582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51394/0 (socket says 172.21.15.99:51394) 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 -- 172.21.15.99:0/2065919948 learned_addr learned my addr 172.21.15.99:0/2065919948 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7891a640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe74071890 0x7fbe74148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 -- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe74071890 msgr2=0x7fbe74148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe73fff640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe740aeb20 0x7fbe74157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe74071890 0x7fbe74148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 -- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe740aeb20 msgr2=0x7fbe74157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe740aeb20 0x7fbe74157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 -- 172.21.15.99:0/2065919948 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe741e4ee0 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7911b640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 0x7fbe741582a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009890 tx=0x7fbe640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe64038040 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe64004230 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe740ac870 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6403d670 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.941+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe740ace30 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.942+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe74074f60 con 0x7fbe7415d660 2024-03-04T10:04:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.943+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbe6403b070 con 0x7fbe7415d660 2024-03-04T10:04:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.943+0000 7fbe717fa640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbe5406b4d0 0x7fbe5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.943+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fbe640abc70 con 0x7fbe7415d660 2024-03-04T10:04:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.943+0000 7fbe7891a640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbe5406b4d0 0x7fbe5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.944+0000 7fbe7891a640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbe5406b4d0 0x7fbe5406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0012e0 tx=0x7fbe6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:30.947+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbe64042020 con 0x7fbe7415d660 2024-03-04T10:04:31.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.132+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbe74072460 con 0x7fbe7415d660 2024-03-04T10:04:31.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.132+0000 7fbe717fa640 1 -- 172.21.15.99:0/2065919948 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbe6407a700 con 0x7fbe7415d660 2024-03-04T10:04:31.134 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:04:31.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbe5406b4d0 msgr2=0x7fbe5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbe5406b4d0 0x7fbe5406d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0012e0 tx=0x7fbe6c009040 comp rx=0 tx=0).stop 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 msgr2=0x7fbe741582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 0x7fbe741582a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009890 tx=0x7fbe640046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 shutdown_connections 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbe5406b4d0 0x7fbe5406d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbe7415d660 0x7fbe741582a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbe740aeb20 0x7fbe74157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 --2- 172.21.15.99:0/2065919948 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbe74071890 0x7fbe74148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 >> 172.21.15.99:0/2065919948 conn(0x7fbe7406acd0 msgr2=0x7fbe7406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 shutdown_connections 2024-03-04T10:04:31.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.134+0000 7fbe7ab7e640 1 -- 172.21.15.99:0/2065919948 wait complete. 2024-03-04T10:04:31.148 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:04:31.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:31.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-04T10:04:31.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:04:31.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:31.178 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:04:31.233 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:31.233 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:31.252 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:31.252 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:31.310 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:31.310 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:31.368 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.467+0000 7fc34131b640 1 Processor -- start 2024-03-04T10:04:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.467+0000 7fc34131b640 1 -- start start 2024-03-04T10:04:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.468+0000 7fc34131b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc33c0cfd60 0x7fc33c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.468+0000 7fc34131b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.468+0000 7fc34131b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c0d0140 con 0x7fc33c06d3b0 2024-03-04T10:04:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.468+0000 7fc34131b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c15cbb0 con 0x7fc33c0cfd60 2024-03-04T10:04:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.468+0000 7fc34131b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c15c280 con 0x7fc33c0cf0d0 2024-03-04T10:04:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33affd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc33c0cfd60 0x7fc33c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33b7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33a7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc33c06d3b0 0x7fc33c0ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35524/0 (socket says 172.21.15.99:35524) 2024-03-04T10:04:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33a7fc640 1 -- 172.21.15.99:0/231203091 learned_addr learned my addr 172.21.15.99:0/231203091 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33affd640 1 -- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 msgr2=0x7fc33c0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33affd640 1 --2- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33affd640 1 -- 172.21.15.99:0/231203091 >> v1:172.21.15.162:6789/0 conn(0x7fc33c06d3b0 legacy=0x7fc33c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33affd640 1 -- 172.21.15.99:0/231203091 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c15c560 con 0x7fc33c0cfd60 2024-03-04T10:04:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.469+0000 7fc33b7fe640 1 --2- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.470+0000 7fc33affd640 1 --2- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c0cfd60 0x7fc33c0cbdf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009d90 tx=0x7fc32c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c7f5a4dc81804d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.470+0000 7fc339ffb640 1 -- 172.21.15.99:0/231203091 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c039070 con 0x7fc33c0cfd60 2024-03-04T10:04:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.470+0000 7fc339ffb640 1 -- 172.21.15.99:0/231203091 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c02cbc0 con 0x7fc33c0cfd60 2024-03-04T10:04:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.470+0000 7fc339ffb640 1 -- 172.21.15.99:0/231203091 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c034440 con 0x7fc33c0cfd60 2024-03-04T10:04:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.470+0000 7fc34131b640 1 -- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c0cfd60 msgr2=0x7fc33c0cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.470+0000 7fc34131b640 1 --2- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c0cfd60 0x7fc33c0cbdf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009d90 tx=0x7fc32c02be10 comp rx=0 tx=0).stop 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.471+0000 7fc34131b640 1 -- 172.21.15.99:0/231203091 shutdown_connections 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.471+0000 7fc34131b640 1 --2- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.471+0000 7fc34131b640 1 --2- 172.21.15.99:0/231203091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c0cfd60 0x7fc33c0cbdf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.471+0000 7fc34131b640 1 -- 172.21.15.99:0/231203091 >> 172.21.15.99:0/231203091 conn(0x7fc33c06acd0 msgr2=0x7fc33c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.471+0000 7fc34131b640 1 -- 172.21.15.99:0/231203091 shutdown_connections 2024-03-04T10:04:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.471+0000 7fc34131b640 1 -- 172.21.15.99:0/231203091 wait complete. 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.472+0000 7fc34131b640 1 Processor -- start 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.472+0000 7fc34131b640 1 -- start start 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.472+0000 7fc34131b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c06d3b0 0x7fc33c0c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.472+0000 7fc34131b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc34131b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 0x7fc33c1f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc34131b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c15f520 con 0x7fc33c0cfd60 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc34131b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c15bf30 con 0x7fc33c06d3b0 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc34131b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c15f6a0 con 0x7fc33c0cf0d0 2024-03-04T10:04:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 0x7fc33c1f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 0x7fc33c1f4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51406/0 (socket says 172.21.15.99:51406) 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 -- 172.21.15.99:0/1098415048 learned_addr learned my addr 172.21.15.99:0/1098415048 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33affd640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c06d3b0 0x7fc33c0c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33a7fc640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 -- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 msgr2=0x7fc33c0a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0a9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 -- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c06d3b0 msgr2=0x7fc33c0c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c06d3b0 0x7fc33c0c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 -- 172.21.15.99:0/1098415048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c1f26a0 con 0x7fc33c0cfd60 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc33b7fe640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 0x7fc33c1f4df0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc33400e930 tx=0x7fc33400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334017070 con 0x7fc33c0cfd60 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33400ca00 con 0x7fc33c0cfd60 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33400cdd0 con 0x7fc33c0cfd60 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c0aa810 con 0x7fc33c0cfd60 2024-03-04T10:04:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.473+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c0aadd0 con 0x7fc33c0cfd60 2024-03-04T10:04:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.474+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c1f2b10 con 0x7fc33c0cfd60 2024-03-04T10:04:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.475+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc334013470 con 0x7fc33c0cfd60 2024-03-04T10:04:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.475+0000 7fc3237fe640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc32806b4d0 0x7fc32806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.475+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fc33408a140 con 0x7fc33c0cfd60 2024-03-04T10:04:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.475+0000 7fc33affd640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc32806b4d0 0x7fc32806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.476+0000 7fc33affd640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc32806b4d0 0x7fc32806d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc32c036000 tx=0x7fc32c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:31.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.479+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc33c1f2b10 con 0x7fc33c0cfd60 2024-03-04T10:04:31.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.656+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc33c074f60 con 0x7fc33c0cfd60 2024-03-04T10:04:31.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.657+0000 7fc3237fe640 1 -- 172.21.15.99:0/1098415048 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7fc334058c90 con 0x7fc33c0cfd60 2024-03-04T10:04:31.659 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:31.659 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc32806b4d0 msgr2=0x7fc32806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc32806b4d0 0x7fc32806d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc32c036000 tx=0x7fc32c030000 comp rx=0 tx=0).stop 2024-03-04T10:04:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 msgr2=0x7fc33c1f4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 0x7fc33c1f4df0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc33400e930 tx=0x7fc33400edf0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 shutdown_connections 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc32806b4d0 0x7fc32806d990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc33c0cfd60 0x7fc33c1f4df0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc33c0cf0d0 0x7fc33c0a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 --2- 172.21.15.99:0/1098415048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc33c06d3b0 0x7fc33c0c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 >> 172.21.15.99:0/1098415048 conn(0x7fc33c06acd0 msgr2=0x7fc33c0ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 shutdown_connections 2024-03-04T10:04:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.659+0000 7fc34131b640 1 -- 172.21.15.99:0/1098415048 wait complete. 2024-03-04T10:04:31.672 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:31.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:31.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.773+0000 7fcad88a4640 1 Processor -- start 2024-03-04T10:04:31.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.773+0000 7fcad88a4640 1 -- start start 2024-03-04T10:04:31.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad88a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcad40b2d70 0x7fcad40cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad88a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad40cd650 0x7fcad415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad88a4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad4070340 con 0x7fcad406ff60 2024-03-04T10:04:31.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad88a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad40cc050 con 0x7fcad40b2d70 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad88a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad406f530 con 0x7fcad40cd650 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad2d9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad40cd650 0x7fcad415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad1d9b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcad40b2d70 0x7fcad40cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fcad406ff60 0x7fcad40cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35530/0 (socket says 172.21.15.99:35530) 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad259c640 1 -- 172.21.15.99:0/1498445487 learned_addr learned my addr 172.21.15.99:0/1498445487 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad2d9d640 1 -- 172.21.15.99:0/1498445487 >> v2:172.21.15.162:3300/0 conn(0x7fcad40b2d70 msgr2=0x7fcad40cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad2d9d640 1 --2- 172.21.15.99:0/1498445487 >> v2:172.21.15.162:3300/0 conn(0x7fcad40b2d70 0x7fcad40cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad2d9d640 1 -- 172.21.15.99:0/1498445487 >> v1:172.21.15.162:6789/0 conn(0x7fcad406ff60 legacy=0x7fcad40cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad2d9d640 1 -- 172.21.15.99:0/1498445487 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad40cb440 con 0x7fcad40cd650 2024-03-04T10:04:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.774+0000 7fcad1d9b640 1 --2- 172.21.15.99:0/1498445487 >> v2:172.21.15.162:3300/0 conn(0x7fcad40b2d70 0x7fcad40cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.775+0000 7fcad2d9d640 1 --2- 172.21.15.99:0/1498445487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad40cd650 0x7fcad415a350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcac40097a0 tx=0x7fcac402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51c11e6bf46569ad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.775+0000 7fcad159a640 1 -- 172.21.15.99:0/1498445487 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4035020 con 0x7fcad40cd650 2024-03-04T10:04:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.775+0000 7fcad159a640 1 -- 172.21.15.99:0/1498445487 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac4031830 con 0x7fcad40cd650 2024-03-04T10:04:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.775+0000 7fcad159a640 1 -- 172.21.15.99:0/1498445487 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4031c00 con 0x7fcad40cd650 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 -- 172.21.15.99:0/1498445487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad40cd650 msgr2=0x7fcad415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 --2- 172.21.15.99:0/1498445487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad40cd650 0x7fcad415a350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcac40097a0 tx=0x7fcac402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 -- 172.21.15.99:0/1498445487 shutdown_connections 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 --2- 172.21.15.99:0/1498445487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad40cd650 0x7fcad415a350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 --2- 172.21.15.99:0/1498445487 >> v2:172.21.15.162:3300/0 conn(0x7fcad40b2d70 0x7fcad40cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 -- 172.21.15.99:0/1498445487 >> 172.21.15.99:0/1498445487 conn(0x7fcad40b2250 msgr2=0x7fcad40b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 -- 172.21.15.99:0/1498445487 shutdown_connections 2024-03-04T10:04:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.776+0000 7fcad88a4640 1 -- 172.21.15.99:0/1498445487 wait complete. 2024-03-04T10:04:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.777+0000 7fcad88a4640 1 Processor -- start 2024-03-04T10:04:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.777+0000 7fcad88a4640 1 -- start start 2024-03-04T10:04:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad88a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad406ff60 0x7fcad40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad88a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad40b2d70 0x7fcad4148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad88a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 0x7fcad40b3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad88a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad40cbce0 con 0x7fcad40cd650 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad88a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad40cba00 con 0x7fcad40b2d70 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad88a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad40cc4e0 con 0x7fcad406ff60 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 0x7fcad40b3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 0x7fcad40b3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51414/0 (socket says 172.21.15.99:51414) 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 -- 172.21.15.99:0/2519984627 learned_addr learned my addr 172.21.15.99:0/2519984627 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad259c640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad406ff60 0x7fcad40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad1d9b640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad40b2d70 0x7fcad4148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 -- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad406ff60 msgr2=0x7fcad40b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad406ff60 0x7fcad40b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 -- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad40b2d70 msgr2=0x7fcad4148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad40b2d70 0x7fcad4148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.778+0000 7fcad2d9d640 1 -- 172.21.15.99:0/2519984627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad4149060 con 0x7fcad40cd650 2024-03-04T10:04:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.779+0000 7fcad2d9d640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 0x7fcad40b3340 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcac40098c0 tx=0x7fcac40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.779+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4035040 con 0x7fcad40cd650 2024-03-04T10:04:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.779+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac403f070 con 0x7fcad40cd650 2024-03-04T10:04:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.779+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad40b3d20 con 0x7fcad40cd650 2024-03-04T10:04:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.779+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac403baa0 con 0x7fcad40cd650 2024-03-04T10:04:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.779+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad40b42e0 con 0x7fcad40cd650 2024-03-04T10:04:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.780+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad4074f60 con 0x7fcad40cd650 2024-03-04T10:04:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.780+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcac4038020 con 0x7fcad40cd650 2024-03-04T10:04:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.781+0000 7fcac2ffd640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.781+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fcac40ac070 con 0x7fcad40cd650 2024-03-04T10:04:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.781+0000 7fcad259c640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.781+0000 7fcad259c640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcab406b4d0 0x7fcab406d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007fd0 tx=0x7fcacc007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:31.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.785+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcad4149060 con 0x7fcad40cd650 2024-03-04T10:04:31.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.956+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcad40ccef0 con 0x7fcad40cd650 2024-03-04T10:04:31.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.956+0000 7fcac2ffd640 1 -- 172.21.15.99:0/2519984627 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcac4031800 con 0x7fcad40cd650 2024-03-04T10:04:31.958 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:31.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcab406b4d0 msgr2=0x7fcab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcab406b4d0 0x7fcab406d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcacc007fd0 tx=0x7fcacc007e80 comp rx=0 tx=0).stop 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 msgr2=0x7fcad40b3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 0x7fcad40b3340 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcac40098c0 tx=0x7fcac40046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 shutdown_connections 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcab406b4d0 0x7fcab406d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcad40cd650 0x7fcad40b3340 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcad40b2d70 0x7fcad4148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 --2- 172.21.15.99:0/2519984627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcad406ff60 0x7fcad40b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 >> 172.21.15.99:0/2519984627 conn(0x7fcad40b2250 msgr2=0x7fcad415e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 shutdown_connections 2024-03-04T10:04:31.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:31.958+0000 7fcad88a4640 1 -- 172.21.15.99:0/2519984627 wait complete. 2024-03-04T10:04:31.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.071+0000 7f75d8e73640 1 Processor -- start 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.071+0000 7f75d8e73640 1 -- start start 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d8e73640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f75d41450e0 0x7f75d40cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d8e73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75d415c9b0 0x7f75d40cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d8e73640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40cebd0 con 0x7f75d40cc340 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d8e73640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d40cf7e0 con 0x7f75d41450e0 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d8e73640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d40cc780 con 0x7f75d415c9b0 2024-03-04T10:04:32.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d2d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f75d40cc340 0x7f75d40b3250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57434/0 (socket says 172.21.15.99:57434) 2024-03-04T10:04:32.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d2d9d640 1 -- 172.21.15.99:0/1585748532 learned_addr learned my addr 172.21.15.99:0/1585748532 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:32.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d259c640 1 --2- 172.21.15.99:0/1585748532 >> v2:172.21.15.162:3300/0 conn(0x7f75d41450e0 0x7f75d40cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75d415c9b0 0x7f75d40cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121608474 0 0) 0x7f75d40cebd0 con 0x7f75d40cc340 2024-03-04T10:04:32.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c0003520 con 0x7f75d40cc340 2024-03-04T10:04:32.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565668543 0 0) 0x7f75c0003520 con 0x7f75d40cc340 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75d415c9b0 msgr2=0x7f75d40cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75d415c9b0 0x7f75d40cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 >> v2:172.21.15.162:3300/0 conn(0x7f75d41450e0 msgr2=0x7f75d40cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 --2- 172.21.15.99:0/1585748532 >> v2:172.21.15.162:3300/0 conn(0x7f75d41450e0 0x7f75d40cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d40ceeb0 con 0x7f75d40cc340 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f75cc0019f0 con 0x7f75d40cc340 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 >> v1:172.21.15.99:6789/0 conn(0x7f75d40cc340 legacy=0x7f75d40b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c00039b0 0x7f75c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75c0004800 0x7f75c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c00051c0 0x7f75c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d40cf7e0 con 0x7f75c0004800 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d40cc780 con 0x7f75c00039b0 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c000bf00 con 0x7f75c00051c0 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75d415c9b0 0x7f75d40cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.072+0000 7f75d259c640 1 --2- 172.21.15.99:0/1585748532 >> v2:172.21.15.162:3300/0 conn(0x7f75d41450e0 0x7f75d40cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75c0004800 0x7f75c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75d2d9d640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c00051c0 0x7f75c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75d259c640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c00039b0 0x7f75c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75cbfff640 1 -- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c00051c0 msgr2=0x7f75c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c00051c0 0x7f75c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75cbfff640 1 -- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c00039b0 msgr2=0x7f75c0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c00039b0 0x7f75c0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75cbfff640 1 -- 172.21.15.99:0/1585748532 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c0003520 con 0x7f75c0004800 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.073+0000 7f75d259c640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75c00039b0 0x7f75c0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75c0004800 0x7f75c0004c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009b30 tx=0x7f75c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e2855afdbae510 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0097d0 con 0x7f75c0004800 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc004030 con 0x7f75c0004800 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d1d9b640 1 -- 172.21.15.99:0/1585748532 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc004400 con 0x7f75c0004800 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d8e73640 1 -- 172.21.15.99:0/1585748532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75c0004800 msgr2=0x7f75c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1585748532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75c0004800 0x7f75c0004c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009b30 tx=0x7f75c0005830 comp rx=0 tx=0).stop 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d2d9d640 1 -- 172.21.15.99:0/1585748532 reap_dead start 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d8e73640 1 -- 172.21.15.99:0/1585748532 shutdown_connections 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d8e73640 1 -- 172.21.15.99:0/1585748532 >> 172.21.15.99:0/1585748532 conn(0x7f75d406acd0 msgr2=0x7f75d406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d8e73640 1 -- 172.21.15.99:0/1585748532 shutdown_connections 2024-03-04T10:04:32.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.074+0000 7f75d8e73640 1 -- 172.21.15.99:0/1585748532 wait complete. 2024-03-04T10:04:32.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 Processor -- start 2024-03-04T10:04:32.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 -- start start 2024-03-04T10:04:32.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c0004800 0x7f75d413ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 0x7f75d41f4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75d40af590 0x7f75d40afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d41454c0 con 0x7f75d40a8460 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d40cf470 con 0x7f75d40af590 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75d8e73640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d415ce30 con 0x7f75c0004800 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75cbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 0x7f75d41f4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75cbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 0x7f75d41f4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51432/0 (socket says 172.21.15.99:51432) 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.076+0000 7f75cbfff640 1 -- 172.21.15.99:0/1951051226 learned_addr learned my addr 172.21.15.99:0/1951051226 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75d259c640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c0004800 0x7f75d413ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75d2d9d640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75d40af590 0x7f75d40afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75cbfff640 1 -- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c0004800 msgr2=0x7f75d413ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c0004800 0x7f75d413ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75cbfff640 1 -- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75d40af590 msgr2=0x7f75d40afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75d40af590 0x7f75d40afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75cbfff640 1 -- 172.21.15.99:0/1951051226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d413cfa0 con 0x7f75d40a8460 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75d259c640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c0004800 0x7f75d413ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:32.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75cbfff640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 0x7f75d41f4580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009b30 tx=0x7f75bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc042070 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc02c910 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d41f4ac0 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc02ccc0 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.077+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d413d520 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.078+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d40b08b0 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.079+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f75bc045070 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.079+0000 7f75caffd640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.079+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f75bc0ac850 con 0x7f75d40a8460 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.079+0000 7f75d259c640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.080+0000 7f75d259c640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75b406b4d0 0x7f75b406d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f75d413bfe0 tx=0x7f75c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.083+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f75d40b08b0 con 0x7f75d40a8460 2024-03-04T10:04:32.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.254+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f75d4074f60 con 0x7f75d40a8460 2024-03-04T10:04:32.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.254+0000 7f75caffd640 1 -- 172.21.15.99:0/1951051226 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f75bc07ac90 con 0x7f75d40a8460 2024-03-04T10:04:32.257 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75b406b4d0 msgr2=0x7f75b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75b406b4d0 0x7f75b406d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f75d413bfe0 tx=0x7f75c4008040 comp rx=0 tx=0).stop 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 msgr2=0x7f75d41f4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 0x7f75d41f4580 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009b30 tx=0x7f75bc004730 comp rx=0 tx=0).stop 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 shutdown_connections 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75d40af590 0x7f75d40afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75d40a8460 0x7f75d41f4580 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 --2- 172.21.15.99:0/1951051226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75c0004800 0x7f75d413ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 >> 172.21.15.99:0/1951051226 conn(0x7f75d406acd0 msgr2=0x7f75d40b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 shutdown_connections 2024-03-04T10:04:32.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.256+0000 7f75d8e73640 1 -- 172.21.15.99:0/1951051226 wait complete. 2024-03-04T10:04:32.269 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:32.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.370+0000 7f47f8996640 1 Processor -- start 2024-03-04T10:04:32.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.370+0000 7f47f8996640 1 -- start start 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f8996640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f415bb70 0x7f47f40b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f8996640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47f40d1e90 0x7f47f40cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f8996640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f415bf50 con 0x7f47f415f660 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f8996640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f40cf890 con 0x7f47f40d1e90 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f8996640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f415fb60 con 0x7f47f415bb70 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47f40d1e90 0x7f47f40cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47f40d1e90 0x7f47f40cfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51440/0 (socket says 172.21.15.99:51440) 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 -- 172.21.15.99:0/1624841398 learned_addr learned my addr 172.21.15.99:0/1624841398 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f259c640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f415bb70 0x7f47f40b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 -- 172.21.15.99:0/1624841398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f415bb70 msgr2=0x7f47f40b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f415bb70 0x7f47f40b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 -- 172.21.15.99:0/1624841398 >> v1:172.21.15.162:6789/0 conn(0x7f47f415f660 legacy=0x7f47f40d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f2d9d640 1 -- 172.21.15.99:0/1624841398 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f415fe40 con 0x7f47f40d1e90 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.371+0000 7f47f259c640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f415bb70 0x7f47f40b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f2d9d640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f40d1e90 0x7f47f40cfd20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009e50 tx=0x7f47ec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae021b27e084113e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f1d9b640 1 -- 172.21.15.99:0/1624841398 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec02c9e0 con 0x7f47f40d1e90 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f1d9b640 1 -- 172.21.15.99:0/1624841398 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ec02cb80 con 0x7f47f40d1e90 2024-03-04T10:04:32.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f1d9b640 1 -- 172.21.15.99:0/1624841398 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ec0343f0 con 0x7f47f40d1e90 2024-03-04T10:04:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f8996640 1 -- 172.21.15.99:0/1624841398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f40d1e90 msgr2=0x7f47f40cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f8996640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f40d1e90 0x7f47f40cfd20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009e50 tx=0x7f47ec02be40 comp rx=0 tx=0).stop 2024-03-04T10:04:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f8996640 1 -- 172.21.15.99:0/1624841398 shutdown_connections 2024-03-04T10:04:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f8996640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f40d1e90 0x7f47f40cfd20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.372+0000 7f47f8996640 1 --2- 172.21.15.99:0/1624841398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f415bb70 0x7f47f40b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.373+0000 7f47f8996640 1 -- 172.21.15.99:0/1624841398 >> 172.21.15.99:0/1624841398 conn(0x7f47f40b33a0 msgr2=0x7f47f40b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.373+0000 7f47f8996640 1 -- 172.21.15.99:0/1624841398 shutdown_connections 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.373+0000 7f47f8996640 1 -- 172.21.15.99:0/1624841398 wait complete. 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.374+0000 7f47f8996640 1 Processor -- start 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.374+0000 7f47f8996640 1 -- start start 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f40d1e90 0x7f47f4158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 0x7f47f415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f415f660 0x7f47f415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40b0f80 con 0x7f47f415bb70 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f4160400 con 0x7f47f415f660 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f40b1100 con 0x7f47f40d1e90 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 0x7f47f415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 0x7f47f415a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51452/0 (socket says 172.21.15.99:51452) 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 -- 172.21.15.99:0/830315149 learned_addr learned my addr 172.21.15.99:0/830315149 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 -- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f40d1e90 msgr2=0x7f47f4158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f259c640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f40d1e90 0x7f47f4158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f2d9d640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f415f660 0x7f47f415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f40d1e90 0x7f47f4158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 -- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f415f660 msgr2=0x7f47f415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f415f660 0x7f47f415a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 -- 172.21.15.99:0/830315149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f41f2a50 con 0x7f47f415bb70 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47ebfff640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 0x7f47f415a030 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b420 tx=0x7f47dc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc011830 con 0x7f47f415bb70 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc016070 con 0x7f47f415bb70 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f41f2cf0 con 0x7f47f415bb70 2024-03-04T10:04:32.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.375+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc011c10 con 0x7f47f415bb70 2024-03-04T10:04:32.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.376+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f41f32b0 con 0x7f47f415bb70 2024-03-04T10:04:32.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.376+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f4074f60 con 0x7f47f415bb70 2024-03-04T10:04:32.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.377+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f47dc01b460 con 0x7f47f415bb70 2024-03-04T10:04:32.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.377+0000 7f47eaffd640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47d406b500 0x7f47d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.377+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f47dc08a0a0 con 0x7f47f415bb70 2024-03-04T10:04:32.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.377+0000 7f47f259c640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47d406b500 0x7f47d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.378+0000 7f47f259c640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47d406b500 0x7f47d406d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f47e4008850 tx=0x7f47e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.381+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f47dc020020 con 0x7f47f415bb70 2024-03-04T10:04:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.551+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f47f40d07b0 con 0x7f47f415bb70 2024-03-04T10:04:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.551+0000 7f47eaffd640 1 -- 172.21.15.99:0/830315149 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47dc058b30 con 0x7f47f415bb70 2024-03-04T10:04:32.553 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:32.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47d406b500 msgr2=0x7f47d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47d406b500 0x7f47d406d9c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f47e4008850 tx=0x7f47e4008040 comp rx=0 tx=0).stop 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 msgr2=0x7f47f415a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 0x7f47f415a030 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00b420 tx=0x7f47dc00b8e0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 shutdown_connections 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f47d406b500 0x7f47d406d9c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f415f660 0x7f47f415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47f415bb70 0x7f47f415a030 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 --2- 172.21.15.99:0/830315149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47f40d1e90 0x7f47f4158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 >> 172.21.15.99:0/830315149 conn(0x7f47f40b33a0 msgr2=0x7f47f40b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 shutdown_connections 2024-03-04T10:04:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.553+0000 7f47f8996640 1 -- 172.21.15.99:0/830315149 wait complete. 2024-03-04T10:04:32.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:32.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 Processor -- start 2024-03-04T10:04:32.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 -- start start 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f31640b5700 0x7f31640d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b9410 0x7f316406c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31640d2400 con 0x7f31640d34a0 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31640d26e0 con 0x7f31640b5700 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f3169b3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640d29c0 con 0x7f31640b9410 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.667+0000 7f31637fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f31640b5700 0x7f31640d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f31637fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f31640b5700 0x7f31640d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51454/0 (socket says 172.21.15.99:51454) 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f31637fe640 1 -- 172.21.15.99:0/714234898 learned_addr learned my addr 172.21.15.99:0/714234898 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f3163fff640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b9410 0x7f316406c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f31637fe640 1 -- 172.21.15.99:0/714234898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b9410 msgr2=0x7f316406c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f31637fe640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b9410 0x7f316406c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f31637fe640 1 -- 172.21.15.99:0/714234898 >> v1:172.21.15.162:6789/0 conn(0x7f31640d34a0 legacy=0x7f316406c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f31637fe640 1 -- 172.21.15.99:0/714234898 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640d2ca0 con 0x7f31640b5700 2024-03-04T10:04:32.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.668+0000 7f3163fff640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b9410 0x7f316406c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:32.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f31637fe640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640b5700 0x7f31640d1b00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52eb2dbc50a153a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f31627fc640 1 -- 172.21.15.99:0/714234898 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154035020 con 0x7f31640b5700 2024-03-04T10:04:32.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f31627fc640 1 -- 172.21.15.99:0/714234898 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3154031640 con 0x7f31640b5700 2024-03-04T10:04:32.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f31627fc640 1 -- 172.21.15.99:0/714234898 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315402d040 con 0x7f31640b5700 2024-03-04T10:04:32.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 -- 172.21.15.99:0/714234898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640b5700 msgr2=0x7f31640d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640b5700 0x7f31640d1b00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3154009870 tx=0x7f315402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 -- 172.21.15.99:0/714234898 shutdown_connections 2024-03-04T10:04:32.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b9410 0x7f316406c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 --2- 172.21.15.99:0/714234898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640b5700 0x7f31640d1b00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 -- 172.21.15.99:0/714234898 >> 172.21.15.99:0/714234898 conn(0x7f31640d0490 msgr2=0x7f31640d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 -- 172.21.15.99:0/714234898 shutdown_connections 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.669+0000 7f3169b3b640 1 -- 172.21.15.99:0/714234898 wait complete. 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.670+0000 7f3169b3b640 1 Processor -- start 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 -- start start 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b5700 0x7f31641f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31640b9410 0x7f316415f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 0x7f316415fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316406ceb0 con 0x7f31640d34a0 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316406cd30 con 0x7f31640b9410 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3169b3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31640b5ae0 con 0x7f31640b5700 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3163fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 0x7f316415fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3163fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 0x7f316415fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51458/0 (socket says 172.21.15.99:51458) 2024-03-04T10:04:32.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.671+0000 7f3163fff640 1 -- 172.21.15.99:0/3538520870 learned_addr learned my addr 172.21.15.99:0/3538520870 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f31637fe640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b5700 0x7f31641f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3162ffd640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31640b9410 0x7f316415f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3163fff640 1 -- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b5700 msgr2=0x7f31641f0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3163fff640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b5700 0x7f31641f0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3163fff640 1 -- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31640b9410 msgr2=0x7f316415f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3163fff640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31640b9410 0x7f316415f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3163fff640 1 -- 172.21.15.99:0/3538520870 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31641f3b60 con 0x7f31640d34a0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3162ffd640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31640b9410 0x7f316415f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3163fff640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 0x7f316415fc70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f315c007ef0 tx=0x7f315c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c01a070 con 0x7f31640d34a0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f315c003b90 con 0x7f31640d34a0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31641f3e40 con 0x7f31640d34a0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315c004030 con 0x7f31640d34a0 2024-03-04T10:04:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.672+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31641f4400 con 0x7f31640d34a0 2024-03-04T10:04:32.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.673+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3164074f60 con 0x7f31640d34a0 2024-03-04T10:04:32.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.673+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f315c01d070 con 0x7f31640d34a0 2024-03-04T10:04:32.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.674+0000 7f314bfff640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.674+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f315c08a830 con 0x7f31640d34a0 2024-03-04T10:04:32.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.674+0000 7f31637fe640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.675+0000 7f31637fe640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f315006b4d0 0x7f315006d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3154036000 tx=0x7f315403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.678+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f315c010070 con 0x7f31640d34a0 2024-03-04T10:04:32.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.850+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f31640cf620 con 0x7f31640d34a0 2024-03-04T10:04:32.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.850+0000 7f314bfff640 1 -- 172.21.15.99:0/3538520870 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f315c058c70 con 0x7f31640d34a0 2024-03-04T10:04:32.852 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f315006b4d0 msgr2=0x7f315006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f315006b4d0 0x7f315006d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3154036000 tx=0x7f315403c040 comp rx=0 tx=0).stop 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 msgr2=0x7f316415fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 0x7f316415fc70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f315c007ef0 tx=0x7f315c00d4a0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 shutdown_connections 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f315006b4d0 0x7f315006d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f31640d34a0 0x7f316415fc70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f31640b9410 0x7f316415f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 --2- 172.21.15.99:0/3538520870 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f31640b5700 0x7f31641f0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 >> 172.21.15.99:0/3538520870 conn(0x7f31640d0490 msgr2=0x7f316406c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 shutdown_connections 2024-03-04T10:04:32.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.852+0000 7f3169b3b640 1 -- 172.21.15.99:0/3538520870 wait complete. 2024-03-04T10:04:32.865 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:04:32.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.964+0000 7f20b3f13640 1 Processor -- start 2024-03-04T10:04:32.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.965+0000 7f20b3f13640 1 -- start start 2024-03-04T10:04:32.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.965+0000 7f20b3f13640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac15be90 0x7f20ac0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.965+0000 7f20b3f13640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f20ac0cfcd0 0x7f20ac0cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.965+0000 7f20b3f13640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac15f8c0 con 0x7f20ac15bab0 2024-03-04T10:04:32.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.965+0000 7f20b3f13640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac0b7970 con 0x7f20ac0cfcd0 2024-03-04T10:04:32.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.965+0000 7f20b3f13640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15c7f0 con 0x7f20ac15be90 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b24b0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f20ac0cfcd0 0x7f20ac0cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b14ae640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac15be90 0x7f20ac0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b1caf640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f20ac15bab0 0x7f20ac0b7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35560/0 (socket says 172.21.15.99:35560) 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b1caf640 1 -- 172.21.15.99:0/4170829074 learned_addr learned my addr 172.21.15.99:0/4170829074 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b24b0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f20ac0cfcd0 0x7f20ac0cc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:52712/0 (socket says 172.21.15.99:52712) 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b0cad640 1 -- 172.21.15.99:0/4170829074 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186983550 0 0) 0x7f20ac15f8c0 con 0x7f20ac15bab0 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b14ae640 1 -- 172.21.15.99:0/4170829074 >> v2:172.21.15.162:3300/0 conn(0x7f20ac0cfcd0 msgr2=0x7f20ac0cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b14ae640 1 --2- 172.21.15.99:0/4170829074 >> v2:172.21.15.162:3300/0 conn(0x7f20ac0cfcd0 0x7f20ac0cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b14ae640 1 -- 172.21.15.99:0/4170829074 >> v1:172.21.15.162:6789/0 conn(0x7f20ac15bab0 legacy=0x7f20ac0b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.966+0000 7f20b14ae640 1 -- 172.21.15.99:0/4170829074 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac15cad0 con 0x7f20ac15be90 2024-03-04T10:04:32.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.967+0000 7f20b14ae640 1 --2- 172.21.15.99:0/4170829074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac15be90 0x7f20ac0cf790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20a00097a0 tx=0x7f20a002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd2cec22ac01993 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.967+0000 7f20b0cad640 1 -- 172.21.15.99:0/4170829074 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0035020 con 0x7f20ac15be90 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.967+0000 7f20b0cad640 1 -- 172.21.15.99:0/4170829074 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0031830 con 0x7f20ac15be90 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.967+0000 7f20b0cad640 1 -- 172.21.15.99:0/4170829074 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0031c00 con 0x7f20ac15be90 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 -- 172.21.15.99:0/4170829074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac15be90 msgr2=0x7f20ac0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4170829074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac15be90 0x7f20ac0cf790 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20a00097a0 tx=0x7f20a002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 -- 172.21.15.99:0/4170829074 shutdown_connections 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4170829074 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac15be90 0x7f20ac0cf790 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4170829074 >> v2:172.21.15.162:3300/0 conn(0x7f20ac0cfcd0 0x7f20ac0cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 -- 172.21.15.99:0/4170829074 >> 172.21.15.99:0/4170829074 conn(0x7f20ac06f6e0 msgr2=0x7f20ac06faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 -- 172.21.15.99:0/4170829074 shutdown_connections 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.968+0000 7f20b3f13640 1 -- 172.21.15.99:0/4170829074 wait complete. 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.969+0000 7f20b3f13640 1 Processor -- start 2024-03-04T10:04:32.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.969+0000 7f20b3f13640 1 -- start start 2024-03-04T10:04:32.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac0cfcd0 0x7f20ac158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 0x7f20ac0b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20ac15be90 0x7f20ac0b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15c320 con 0x7f20ac15bab0 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15d090 con 0x7f20ac15be90 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15c4a0 con 0x7f20ac0cfcd0 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 0x7f20ac0b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 0x7f20ac0b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51460/0 (socket says 172.21.15.99:51460) 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 -- 172.21.15.99:0/4179222767 learned_addr learned my addr 172.21.15.99:0/4179222767 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b1caf640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac0cfcd0 0x7f20ac158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b24b0640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20ac15be90 0x7f20ac0b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 -- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac0cfcd0 msgr2=0x7f20ac158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac0cfcd0 0x7f20ac158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 -- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20ac15be90 msgr2=0x7f20ac0b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20ac15be90 0x7f20ac0b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 -- 172.21.15.99:0/4179222767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac1f1940 con 0x7f20ac15bab0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b14ae640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 0x7f20ac0b2340 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20a0000c00 tx=0x7f20a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0035040 con 0x7f20ac15bab0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0004250 con 0x7f20ac15bab0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac1f1c20 con 0x7f20ac15bab0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.970+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a003a640 con 0x7f20ac15bab0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.971+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac1f21e0 con 0x7f20ac15bab0 2024-03-04T10:04:32.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.971+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac074f60 con 0x7f20ac15bab0 2024-03-04T10:04:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.972+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f20a0038020 con 0x7f20ac15bab0 2024-03-04T10:04:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.972+0000 7f20aa7fc640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.972+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f20a00ac1b0 con 0x7f20ac15bab0 2024-03-04T10:04:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.973+0000 7f20b1caf640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.973+0000 7f20b1caf640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f209806b4d0 0x7f209806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20a40012e0 tx=0x7f20a4001700 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:32.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:32.977+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f20a007a5f0 con 0x7f20ac15bab0 2024-03-04T10:04:33.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.159+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f20ac15e170 con 0x7f20ac15bab0 2024-03-04T10:04:33.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.160+0000 7f20aa7fc640 1 -- 172.21.15.99:0/4179222767 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f20a007e690 con 0x7f20ac15bab0 2024-03-04T10:04:33.162 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.161+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f209806b4d0 msgr2=0x7f209806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.161+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f209806b4d0 0x7f209806d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20a40012e0 tx=0x7f20a4001700 comp rx=0 tx=0).stop 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 msgr2=0x7f20ac0b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 0x7f20ac0b2340 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20a0000c00 tx=0x7f20a00046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 shutdown_connections 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f209806b4d0 0x7f209806d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20ac15be90 0x7f20ac0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f20ac15bab0 0x7f20ac0b2340 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 --2- 172.21.15.99:0/4179222767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f20ac0cfcd0 0x7f20ac158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 >> 172.21.15.99:0/4179222767 conn(0x7f20ac06f6e0 msgr2=0x7f20ac15af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 shutdown_connections 2024-03-04T10:04:33.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.162+0000 7f20b3f13640 1 -- 172.21.15.99:0/4179222767 wait complete. 2024-03-04T10:04:33.174 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-04T10:04:33.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:33.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-04T10:04:33.209 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:04:33.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:33.209 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-04T10:04:33.280 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:33.281 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:33.298 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:33.298 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:33.353 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:33.353 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:33.408 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:33.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.546+0000 7f070c406640 1 Processor -- start 2024-03-04T10:04:33.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.546+0000 7f070c406640 1 -- start start 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f070c406640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 0x7f07040cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f070c406640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070415f020 con 0x7f0704154a80 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f070c406640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07040cd010 con 0x7f0704070de0 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f070c406640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040cfff0 con 0x7f07040cf650 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f070a1a2640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0704154a80 0x7f070415e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34040/0 (socket says 172.21.15.99:34040) 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f070a1a2640 1 -- 172.21.15.99:0/904767070 learned_addr learned my addr 172.21.15.99:0/904767070 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965883342 0 0) 0x7f0700001110 con 0x7f0704154a80 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f0003520 con 0x7f0704154a80 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07099a1640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 0x7f07040cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 847907328 0 0) 0x7f06f0003520 con 0x7f0704154a80 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 msgr2=0x7f07040cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 0x7f07040cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 >> v1:172.21.15.162:6789/0 conn(0x7f0704070de0 legacy=0x7f0704071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040cdaa0 con 0x7f0704154a80 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f0700001840 con 0x7f0704154a80 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 >> v1:172.21.15.99:6789/0 conn(0x7f0704154a80 legacy=0x7f070415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 0x7f06f00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06f0004010 0x7f06f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004d40 0x7f06f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f000bb50 con 0x7f06f0004010 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f000bd00 con 0x7f06f0004d40 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f000bf00 con 0x7f07040cf650 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.547+0000 7f07099a1640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06f0004010 0x7f06f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f070a1a2640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 0x7f06f00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f070a9a3640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004d40 0x7f06f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f07099a1640 1 -- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 msgr2=0x7f06f00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f07099a1640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040cf650 0x7f06f00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f07099a1640 1 -- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004d40 msgr2=0x7f06f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f07099a1640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004d40 0x7f06f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f07099a1640 1 -- 172.21.15.99:0/904767070 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0003520 con 0x7f06f0004010 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.548+0000 7f07099a1640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06f0004010 0x7f06f0004800 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f06f400ba30 tx=0x7f06f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b6b0511b09c4d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4036020 con 0x7f06f0004010 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f4032640 con 0x7f06f0004010 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f07091a0640 1 -- 172.21.15.99:0/904767070 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f400b600 con 0x7f06f0004010 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070c406640 1 -- 172.21.15.99:0/904767070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06f0004010 msgr2=0x7f06f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070c406640 1 --2- 172.21.15.99:0/904767070 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06f0004010 0x7f06f0004800 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f06f400ba30 tx=0x7f06f0005830 comp rx=0 tx=0).stop 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070a9a3640 1 -- 172.21.15.99:0/904767070 reap_dead start 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070c406640 1 -- 172.21.15.99:0/904767070 shutdown_connections 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070c406640 1 -- 172.21.15.99:0/904767070 >> 172.21.15.99:0/904767070 conn(0x7f070406f5f0 msgr2=0x7f070406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070c406640 1 -- 172.21.15.99:0/904767070 shutdown_connections 2024-03-04T10:04:33.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.549+0000 7f070c406640 1 -- 172.21.15.99:0/904767070 wait complete. 2024-03-04T10:04:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.550+0000 7f070c406640 1 Processor -- start 2024-03-04T10:04:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.550+0000 7f070c406640 1 -- start start 2024-03-04T10:04:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070c406640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004010 0x7f0704153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070c406640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040b4490 0x7f070415caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070c406640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 0x7f07040c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070c406640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040ce340 con 0x7f070415cfe0 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070c406640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040ce060 con 0x7f06f0004010 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070c406640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040ce620 con 0x7f07040b4490 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a9a3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 0x7f07040c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a9a3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 0x7f07040c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35052/0 (socket says 172.21.15.99:35052) 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a9a3640 1 -- 172.21.15.99:0/1991996261 learned_addr learned my addr 172.21.15.99:0/1991996261 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a1a2640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004010 0x7f0704153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f07099a1640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040b4490 0x7f070415caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a9a3640 1 -- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040b4490 msgr2=0x7f070415caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a9a3640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040b4490 0x7f070415caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.551+0000 7f070a9a3640 1 -- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004010 msgr2=0x7f0704153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f070a9a3640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004010 0x7f0704153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f070a9a3640 1 -- 172.21.15.99:0/1991996261 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040c6140 con 0x7f070415cfe0 2024-03-04T10:04:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f07099a1640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040b4490 0x7f070415caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f070a9a3640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 0x7f07040c5410 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f070406ea30 tx=0x7f06fc00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc01a070 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc006d60 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07040c6420 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc003b90 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.552+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07040b48f0 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.553+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0704074f60 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.553+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f06fc005110 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.554+0000 7f06faffd640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e806b500 0x7f06e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.554+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f06fc089b10 con 0x7f070415cfe0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.554+0000 7f070a1a2640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e806b500 0x7f06e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.554+0000 7f070a1a2640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e806b500 0x7f06e806d9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0700000ff0 tx=0x7f0700009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:33.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.558+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f06fc058630 con 0x7f070415cfe0 2024-03-04T10:04:33.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.730+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07040c65f0 con 0x7f070415cfe0 2024-03-04T10:04:33.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.731+0000 7f06faffd640 1 -- 172.21.15.99:0/1991996261 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7f06fc057f30 con 0x7f070415cfe0 2024-03-04T10:04:33.733 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:33.733 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:33.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e806b500 msgr2=0x7f06e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e806b500 0x7f06e806d9c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0700000ff0 tx=0x7f0700009040 comp rx=0 tx=0).stop 2024-03-04T10:04:33.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 msgr2=0x7f07040c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 0x7f07040c5410 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f070406ea30 tx=0x7f06fc00fc90 comp rx=0 tx=0).stop 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 shutdown_connections 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e806b500 0x7f06e806d9c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f070415cfe0 0x7f07040c5410 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07040b4490 0x7f070415caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 --2- 172.21.15.99:0/1991996261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06f0004010 0x7f0704153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 >> 172.21.15.99:0/1991996261 conn(0x7f070406f5f0 msgr2=0x7f070406d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:33.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 shutdown_connections 2024-03-04T10:04:33.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.733+0000 7f070c406640 1 -- 172.21.15.99:0/1991996261 wait complete. 2024-03-04T10:04:33.746 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:33.747 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:33.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.847+0000 7f43b5dc4640 1 Processor -- start 2024-03-04T10:04:33.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.847+0000 7f43b5dc4640 1 -- start start 2024-03-04T10:04:33.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43b5dc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b00b48a0 0x7f43b00b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43b5dc4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f43b006d3c0 0x7f43b00b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43b5dc4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b006d8a0 con 0x7f43b015a810 2024-03-04T10:04:33.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43b5dc4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b006db80 con 0x7f43b006d3c0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43b5dc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b006de60 con 0x7f43b00b48a0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43affff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f43b006d3c0 0x7f43b00b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43affff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f43b006d3c0 0x7f43b00b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35060/0 (socket says 172.21.15.99:35060) 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43affff640 1 -- 172.21.15.99:0/1250870300 learned_addr learned my addr 172.21.15.99:0/1250870300 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43aeffd640 1 --2- 172.21.15.99:0/1250870300 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b00b48a0 0x7f43b00b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.848+0000 7f43affff640 1 -- 172.21.15.99:0/1250870300 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b00b48a0 msgr2=0x7f43b00b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43affff640 1 --2- 172.21.15.99:0/1250870300 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b00b48a0 0x7f43b00b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43affff640 1 -- 172.21.15.99:0/1250870300 >> v1:172.21.15.162:6789/0 conn(0x7f43b015a810 legacy=0x7f43b00b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43affff640 1 -- 172.21.15.99:0/1250870300 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b006e140 con 0x7f43b006d3c0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43affff640 1 --2- 172.21.15.99:0/1250870300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b006d3c0 0x7f43b00b2f30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43a800d150 tx=0x7f43a802e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fcc3586f06d595b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43ae7fc640 1 -- 172.21.15.99:0/1250870300 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a800b0d0 con 0x7f43b006d3c0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43ae7fc640 1 -- 172.21.15.99:0/1250870300 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a800b270 con 0x7f43b006d3c0 2024-03-04T10:04:33.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.849+0000 7f43ae7fc640 1 -- 172.21.15.99:0/1250870300 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8009600 con 0x7f43b006d3c0 2024-03-04T10:04:33.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 -- 172.21.15.99:0/1250870300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b006d3c0 msgr2=0x7f43b00b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/1250870300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b006d3c0 0x7f43b00b2f30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43a800d150 tx=0x7f43a802e160 comp rx=0 tx=0).stop 2024-03-04T10:04:33.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 -- 172.21.15.99:0/1250870300 shutdown_connections 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/1250870300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b006d3c0 0x7f43b00b2f30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/1250870300 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b00b48a0 0x7f43b00b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 -- 172.21.15.99:0/1250870300 >> 172.21.15.99:0/1250870300 conn(0x7f43b006acd0 msgr2=0x7f43b006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 -- 172.21.15.99:0/1250870300 shutdown_connections 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.850+0000 7f43b5dc4640 1 -- 172.21.15.99:0/1250870300 wait complete. 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.851+0000 7f43b5dc4640 1 Processor -- start 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.851+0000 7f43b5dc4640 1 -- start start 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43b5dc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b006d3c0 0x7f43b00ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43b5dc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43b00b48a0 0x7f43b0143fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43b5dc4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 0x7f43b015c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43b5dc4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00cc130 con 0x7f43b015a810 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43b5dc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b00cbfb0 con 0x7f43b00b48a0 2024-03-04T10:04:33.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43b5dc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0151be0 con 0x7f43b006d3c0 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 0x7f43b015c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 0x7f43b015c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35076/0 (socket says 172.21.15.99:35076) 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 -- 172.21.15.99:0/3316761071 learned_addr learned my addr 172.21.15.99:0/3316761071 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43af7fe640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b006d3c0 0x7f43b00ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43aeffd640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43b00b48a0 0x7f43b0143fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 -- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b006d3c0 msgr2=0x7f43b00ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b006d3c0 0x7f43b00ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 -- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43b00b48a0 msgr2=0x7f43b0143fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43b00b48a0 0x7f43b0143fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43affff640 1 -- 172.21.15.99:0/3316761071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b01444f0 con 0x7f43b015a810 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.852+0000 7f43aeffd640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43b00b48a0 0x7f43b0143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.853+0000 7f43affff640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 0x7f43b015c8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f43a8038040 tx=0x7f43a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:33.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.853+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8045020 con 0x7f43b015a810 2024-03-04T10:04:33.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.853+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a800b420 con 0x7f43b015a810 2024-03-04T10:04:33.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.853+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b0158370 con 0x7f43b015a810 2024-03-04T10:04:33.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.853+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a800b0d0 con 0x7f43b015a810 2024-03-04T10:04:33.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.853+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b0158930 con 0x7f43b015a810 2024-03-04T10:04:33.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.854+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b00ccfc0 con 0x7f43b015a810 2024-03-04T10:04:33.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.854+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f43a8048020 con 0x7f43b015a810 2024-03-04T10:04:33.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.854+0000 7f4397fff640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c06b4d0 0x7f439c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:33.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.855+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f43a80af010 con 0x7f43b015a810 2024-03-04T10:04:33.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.855+0000 7f43af7fe640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c06b4d0 0x7f439c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:33.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.855+0000 7f43af7fe640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c06b4d0 0x7f439c06d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f43a00016e0 tx=0x7f43a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:33.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:33.859+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f43b00ccfc0 con 0x7f43b015a810 2024-03-04T10:04:34.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.029+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43b0074f60 con 0x7f43b015a810 2024-03-04T10:04:34.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.029+0000 7f4397fff640 1 -- 172.21.15.99:0/3316761071 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f43a804f030 con 0x7f43b015a810 2024-03-04T10:04:34.032 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c06b4d0 msgr2=0x7f439c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c06b4d0 0x7f439c06d990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f43a00016e0 tx=0x7f43a0009040 comp rx=0 tx=0).stop 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 msgr2=0x7f43b015c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 0x7f43b015c8d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f43a8038040 tx=0x7f43a8004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 shutdown_connections 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c06b4d0 0x7f439c06d990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43b015a810 0x7f43b015c8d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43b00b48a0 0x7f43b0143fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 --2- 172.21.15.99:0/3316761071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43b006d3c0 0x7f43b00ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.031+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 >> 172.21.15.99:0/3316761071 conn(0x7f43b006acd0 msgr2=0x7f43b00b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.032+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 shutdown_connections 2024-03-04T10:04:34.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.032+0000 7f43b5dc4640 1 -- 172.21.15.99:0/3316761071 wait complete. 2024-03-04T10:04:34.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:34.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.145+0000 7ff9b09d1640 1 Processor -- start 2024-03-04T10:04:34.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.145+0000 7ff9b09d1640 1 -- start start 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9b09d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff9ac0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9b09d1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac162450 con 0x7ff9ac1616e0 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9b09d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac0cc2d0 con 0x7ff9ac15da20 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9b09d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0d0000 con 0x7ff9ac0cf660 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9aad9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff9ac1616e0 0x7ff9ac161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34046/0 (socket says 172.21.15.99:34046) 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9aad9d640 1 -- 172.21.15.99:0/3611740653 learned_addr learned my addr 172.21.15.99:0/3611740653 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430337571 0 0) 0x7ff9ac162450 con 0x7ff9ac1616e0 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff994003520 con 0x7ff9ac1616e0 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.146+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff9ac0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54608598 0 0) 0x7ff994003520 con 0x7ff9ac1616e0 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 msgr2=0x7ff9ac0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff9ac0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 >> v1:172.21.15.162:6789/0 conn(0x7ff9ac15da20 legacy=0x7ff9ac0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0cc5b0 con 0x7ff9ac1616e0 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7ff9a40019f0 con 0x7ff9ac1616e0 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff9ac0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 >> v1:172.21.15.99:6789/0 conn(0x7ff9ac1616e0 legacy=0x7ff9ac161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9940039e0 0x7ff994003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff994004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99400bb50 con 0x7ff994004d40 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99400bcd0 con 0x7ff9940039e0 2024-03-04T10:04:34.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99400bed0 con 0x7ff9ac0cf660 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aad9d640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff994004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aa59c640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9940039e0 0x7ff994003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aad9d640 1 -- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 msgr2=0x7ff994004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aad9d640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0cf660 0x7ff994004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aad9d640 1 -- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9940039e0 msgr2=0x7ff994003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aad9d640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9940039e0 0x7ff994003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.147+0000 7ff9aad9d640 1 -- 172.21.15.99:0/3611740653 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff994003520 con 0x7ff994004d40 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.148+0000 7ff9aad9d640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9940051c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff9a400a850 tx=0x7ff994005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c97cd13b21195d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.148+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400a4a0 con 0x7ff994004d40 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.148+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a4034650 con 0x7ff994004d40 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.148+0000 7ff9a959a640 1 -- 172.21.15.99:0/3611740653 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4034a40 con 0x7ff994004d40 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.148+0000 7ff9b09d1640 1 -- 172.21.15.99:0/3611740653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 msgr2=0x7ff9940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.148+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/3611740653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9940051c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff9a400a850 tx=0x7ff994005830 comp rx=0 tx=0).stop 2024-03-04T10:04:34.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.149+0000 7ff9aad9d640 1 -- 172.21.15.99:0/3611740653 reap_dead start 2024-03-04T10:04:34.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.149+0000 7ff9b09d1640 1 -- 172.21.15.99:0/3611740653 shutdown_connections 2024-03-04T10:04:34.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.149+0000 7ff9b09d1640 1 -- 172.21.15.99:0/3611740653 >> 172.21.15.99:0/3611740653 conn(0x7ff9ac0b33a0 msgr2=0x7ff9ac0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.149+0000 7ff9b09d1640 1 -- 172.21.15.99:0/3611740653 shutdown_connections 2024-03-04T10:04:34.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.149+0000 7ff9b09d1640 1 -- 172.21.15.99:0/3611740653 wait complete. 2024-03-04T10:04:34.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.150+0000 7ff9b09d1640 1 Processor -- start 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.150+0000 7ff9b09d1640 1 -- start start 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9ac0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0c69a0 0x7ff9ac0b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9ac1584f0 0x7ff9ac0b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac15de00 con 0x7ff994004d40 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0cc920 con 0x7ff9ac1584f0 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac0cfac0 con 0x7ff9ac0c69a0 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9ac0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9ac0c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35086/0 (socket says 172.21.15.99:35086) 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 -- 172.21.15.99:0/2708691500 learned_addr learned my addr 172.21.15.99:0/2708691500 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0c69a0 0x7ff9ac0b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aad9d640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9ac1584f0 0x7ff9ac0b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 -- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0c69a0 msgr2=0x7ff9ac0b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0c69a0 0x7ff9ac0b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 -- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9ac1584f0 msgr2=0x7ff9ac0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9ac1584f0 0x7ff9ac0b2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 -- 172.21.15.99:0/2708691500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0c6e60 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0c69a0 0x7ff9ac0b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9aa59c640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9ac0c6460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00f820 tx=0x7ff99c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0050d0 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c018070 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c005490 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.151+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac0c7140 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.152+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac0c7700 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.152+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac074f60 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.153+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff99c00d730 con 0x7ff994004d40 2024-03-04T10:04:34.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.153+0000 7ff99affd640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff98c06b500 0x7ff98c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.153+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7ff99c08a430 con 0x7ff994004d40 2024-03-04T10:04:34.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.153+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff98c06b500 0x7ff98c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.154+0000 7ff9a9d9b640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff98c06b500 0x7ff98c06d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac160430 tx=0x7ff9a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.157+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff99c058f50 con 0x7ff994004d40 2024-03-04T10:04:34.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.327+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9ac0d1ed0 con 0x7ff994004d40 2024-03-04T10:04:34.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.327+0000 7ff99affd640 1 -- 172.21.15.99:0/2708691500 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff99c058850 con 0x7ff994004d40 2024-03-04T10:04:34.329 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:34.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff98c06b500 msgr2=0x7ff98c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff98c06b500 0x7ff98c06d9c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac160430 tx=0x7ff9a0008040 comp rx=0 tx=0).stop 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 msgr2=0x7ff9ac0c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9ac0c6460 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff99c00f820 tx=0x7ff99c00fce0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 shutdown_connections 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff98c06b500 0x7ff98c06d9c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9ac1584f0 0x7ff9ac0b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9ac0c69a0 0x7ff9ac0b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 --2- 172.21.15.99:0/2708691500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff994004d40 0x7ff9ac0c6460 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 >> 172.21.15.99:0/2708691500 conn(0x7ff9ac0b33a0 msgr2=0x7ff9ac0ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 shutdown_connections 2024-03-04T10:04:34.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.329+0000 7ff9b09d1640 1 -- 172.21.15.99:0/2708691500 wait complete. 2024-03-04T10:04:34.342 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:34.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.442+0000 7f99b4ef8640 1 Processor -- start 2024-03-04T10:04:34.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.442+0000 7f99b4ef8640 1 -- start start 2024-03-04T10:04:34.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99b4ef8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99b4ef8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f99b00cf660 0x7f99b00cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99b4ef8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b0162450 con 0x7f99b01616e0 2024-03-04T10:04:34.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99b4ef8640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00d0000 con 0x7f99b00cf660 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99b4ef8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00cc920 con 0x7f99b015da20 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99add9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f99b00cf660 0x7f99b00cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99aed9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f99b01616e0 0x7f99b0161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34058/0 (socket says 172.21.15.99:34058) 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99aed9d640 1 -- 172.21.15.99:0/1742977122 learned_addr learned my addr 172.21.15.99:0/1742977122 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99ae59c640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99add9b640 1 -- 172.21.15.99:0/1742977122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 msgr2=0x7f99b00ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99add9b640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.443+0000 7f99add9b640 1 -- 172.21.15.99:0/1742977122 >> v1:172.21.15.99:6789/0 conn(0x7f99b01616e0 legacy=0x7f99b0161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.444+0000 7f99add9b640 1 -- 172.21.15.99:0/1742977122 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00cc2d0 con 0x7f99b00cf660 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.444+0000 7f99ae59c640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.444+0000 7f99add9b640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b00cf660 0x7f99b00cfac0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f99a40097a0 tx=0x7f99a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964235cad0acd35a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.444+0000 7f99ad59a640 1 -- 172.21.15.99:0/1742977122 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4035020 con 0x7f99b00cf660 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.444+0000 7f99ad59a640 1 -- 172.21.15.99:0/1742977122 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a4031830 con 0x7f99b00cf660 2024-03-04T10:04:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.444+0000 7f99ad59a640 1 -- 172.21.15.99:0/1742977122 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4031c00 con 0x7f99b00cf660 2024-03-04T10:04:34.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 -- 172.21.15.99:0/1742977122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b00cf660 msgr2=0x7f99b00cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b00cf660 0x7f99b00cfac0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f99a40097a0 tx=0x7f99a402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 -- 172.21.15.99:0/1742977122 shutdown_connections 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b00cf660 0x7f99b00cfac0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/1742977122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 -- 172.21.15.99:0/1742977122 >> 172.21.15.99:0/1742977122 conn(0x7f99b00b33a0 msgr2=0x7f99b00b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 -- 172.21.15.99:0/1742977122 shutdown_connections 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.445+0000 7f99b4ef8640 1 -- 172.21.15.99:0/1742977122 wait complete. 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.446+0000 7f99b4ef8640 1 Processor -- start 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.446+0000 7f99b4ef8640 1 -- start start 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99b4ef8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b00cf660 0x7f99b00c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99b4ef8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99b4ef8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 0x7f99b00ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99b4ef8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00cbf80 con 0x7f99b01616e0 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99b4ef8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00cd2f0 con 0x7f99b00cf660 2024-03-04T10:04:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99b4ef8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015de00 con 0x7f99b015da20 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 0x7f99b00ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 0x7f99b00ca580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35100/0 (socket says 172.21.15.99:35100) 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 -- 172.21.15.99:0/2510504753 learned_addr learned my addr 172.21.15.99:0/2510504753 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99add9b640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99ae59c640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b00cf660 0x7f99b00c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 -- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 msgr2=0x7f99b00b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 -- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b00cf660 msgr2=0x7f99b00c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b00cf660 0x7f99b00c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99aed9d640 1 -- 172.21.15.99:0/2510504753 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00c5d30 con 0x7f99b01616e0 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.447+0000 7f99add9b640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.448+0000 7f99aed9d640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 0x7f99b00ca580 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ca20 tx=0x7f99a800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.448+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8017070 con 0x7f99b01616e0 2024-03-04T10:04:34.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.448+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a800abd0 con 0x7f99b01616e0 2024-03-04T10:04:34.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.448+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8012400 con 0x7f99b01616e0 2024-03-04T10:04:34.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.448+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b00c6010 con 0x7f99b01616e0 2024-03-04T10:04:34.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.448+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b00c65d0 con 0x7f99b01616e0 2024-03-04T10:04:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.449+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0074f60 con 0x7f99b01616e0 2024-03-04T10:04:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.449+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f99a801a070 con 0x7f99b01616e0 2024-03-04T10:04:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.450+0000 7f999effd640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f999006b4d0 0x7f999006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.450+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f99a808a230 con 0x7f99b01616e0 2024-03-04T10:04:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.450+0000 7f99ae59c640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f999006b4d0 0x7f999006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.450+0000 7f99ae59c640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f999006b4d0 0x7f999006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99b00cf210 tx=0x7f99a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.454+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f99a8058d50 con 0x7f99b01616e0 2024-03-04T10:04:34.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.626+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99b00cce40 con 0x7f99b01616e0 2024-03-04T10:04:34.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.627+0000 7f999effd640 1 -- 172.21.15.99:0/2510504753 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99a8058650 con 0x7f99b01616e0 2024-03-04T10:04:34.629 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.628+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f999006b4d0 msgr2=0x7f999006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.628+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f999006b4d0 0x7f999006d990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99b00cf210 tx=0x7f99a0008040 comp rx=0 tx=0).stop 2024-03-04T10:04:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 msgr2=0x7f99b00ca580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 0x7f99b00ca580 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ca20 tx=0x7f99a800cee0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 shutdown_connections 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f999006b4d0 0x7f999006d990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99b01616e0 0x7f99b00ca580 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f99b015da20 0x7f99b00b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 --2- 172.21.15.99:0/2510504753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99b00cf660 0x7f99b00c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 >> 172.21.15.99:0/2510504753 conn(0x7f99b00b33a0 msgr2=0x7f99b00b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 shutdown_connections 2024-03-04T10:04:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.629+0000 7f99b4ef8640 1 -- 172.21.15.99:0/2510504753 wait complete. 2024-03-04T10:04:34.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 Processor -- start 2024-03-04T10:04:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 -- start start 2024-03-04T10:04:34.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 0x7f5d680ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5d68161df0 0x7f5d680cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d68162450 con 0x7f5d680cf660 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d680d0000 con 0x7f5d68161df0 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.744+0000 7f5d6df69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680cc920 con 0x7f5d6815da20 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d66ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5d680cf660 0x7f5d681616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34060/0 (socket says 172.21.15.99:34060) 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d66ffd640 1 -- 172.21.15.99:0/1139530218 learned_addr learned my addr 172.21.15.99:0/1139530218 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818599548 0 0) 0x7f5d68162450 con 0x7f5d680cf660 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d50003520 con 0x7f5d680cf660 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d67fff640 1 --2- 172.21.15.99:0/1139530218 >> v2:172.21.15.162:3300/0 conn(0x7f5d68161df0 0x7f5d680cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d677fe640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 0x7f5d680ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2433431078 0 0) 0x7f5d50003520 con 0x7f5d680cf660 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 msgr2=0x7f5d680ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 0x7f5d680ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 >> v2:172.21.15.162:3300/0 conn(0x7f5d68161df0 msgr2=0x7f5d680cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 --2- 172.21.15.99:0/1139530218 >> v2:172.21.15.162:3300/0 conn(0x7f5d68161df0 0x7f5d680cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d680cc2d0 con 0x7f5d680cf660 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5d58001670 con 0x7f5d680cf660 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 >> v1:172.21.15.99:6789/0 conn(0x7f5d680cf660 legacy=0x7f5d681616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 0x7f5d500039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50004d40 0x7f5d500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680cc920 con 0x7f5d50003fe0 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5000ba70 con 0x7f5d50004d40 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5000bc40 con 0x7f5d6815da20 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d677fe640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 0x7f5d500039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d67fff640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50004d40 0x7f5d500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d66ffd640 1 -- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 msgr2=0x7f5d500039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.745+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d6815da20 0x7f5d500039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d66ffd640 1 -- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50004d40 msgr2=0x7f5d500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d50004d40 0x7f5d500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d66ffd640 1 -- 172.21.15.99:0/1139530218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d50003520 con 0x7f5d50003fe0 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d50004800 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800c210 tx=0x7f5d50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57aeac53baae2cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d58007d50 con 0x7f5d50003fe0 2024-03-04T10:04:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d58030070 con 0x7f5d50003fe0 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.746+0000 7f5d667fc640 1 -- 172.21.15.99:0/1139530218 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5802cde0 con 0x7f5d50003fe0 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d6df69640 1 -- 172.21.15.99:0/1139530218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 msgr2=0x7f5d50004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d6df69640 1 --2- 172.21.15.99:0/1139530218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d50004800 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800c210 tx=0x7f5d50005830 comp rx=0 tx=0).stop 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d67fff640 1 -- 172.21.15.99:0/1139530218 reap_dead start 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d6df69640 1 -- 172.21.15.99:0/1139530218 shutdown_connections 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d6df69640 1 -- 172.21.15.99:0/1139530218 >> 172.21.15.99:0/1139530218 conn(0x7f5d680b33a0 msgr2=0x7f5d680b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d6df69640 1 -- 172.21.15.99:0/1139530218 shutdown_connections 2024-03-04T10:04:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.747+0000 7f5d6df69640 1 -- 172.21.15.99:0/1139530218 wait complete. 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.748+0000 7f5d6df69640 1 Processor -- start 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.748+0000 7f5d6df69640 1 -- start start 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d6df69640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d680c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d6df69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d680cb2a0 0x7f5d680b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d6df69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d68151760 0x7f5d68151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d6df69640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6815de00 con 0x7f5d50003fe0 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d6df69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680cbf80 con 0x7f5d68151760 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d6df69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680ccdb0 con 0x7f5d680cb2a0 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d680c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d680c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35110/0 (socket says 172.21.15.99:35110) 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 -- 172.21.15.99:0/3747015039 learned_addr learned my addr 172.21.15.99:0/3747015039 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d680cb2a0 0x7f5d680b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d67fff640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d68151760 0x7f5d68151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 -- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d680cb2a0 msgr2=0x7f5d680b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d680cb2a0 0x7f5d680b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 -- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d68151760 msgr2=0x7f5d68151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d68151760 0x7f5d68151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 -- 172.21.15.99:0/3747015039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d680cb720 con 0x7f5d50003fe0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.749+0000 7f5d677fe640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d680c5320 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00b7f0 tx=0x7f5d5c00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.750+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c018070 con 0x7f5d50003fe0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.750+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c0107b0 con 0x7f5d50003fe0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.750+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c010b80 con 0x7f5d50003fe0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.750+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d68152120 con 0x7f5d50003fe0 2024-03-04T10:04:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.750+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d680c5c30 con 0x7f5d50003fe0 2024-03-04T10:04:34.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.751+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d680c6630 con 0x7f5d50003fe0 2024-03-04T10:04:34.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.751+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5d5c01b070 con 0x7f5d50003fe0 2024-03-04T10:04:34.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.751+0000 7f5d4ffff640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d5406b500 0x7f5d5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:34.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.752+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f5d5c08aa70 con 0x7f5d50003fe0 2024-03-04T10:04:34.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.752+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d5406b500 0x7f5d5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:34.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.752+0000 7f5d66ffd640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d5406b500 0x7f5d5406d9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5d58034000 tx=0x7f5d58007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.756+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5d5c059590 con 0x7f5d50003fe0 2024-03-04T10:04:34.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.925+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5d68074f60 con 0x7f5d50003fe0 2024-03-04T10:04:34.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.925+0000 7f5d4ffff640 1 -- 172.21.15.99:0/3747015039 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5d5c058e90 con 0x7f5d50003fe0 2024-03-04T10:04:34.928 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d5406b500 msgr2=0x7f5d5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d5406b500 0x7f5d5406d9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5d58034000 tx=0x7f5d58007000 comp rx=0 tx=0).stop 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 msgr2=0x7f5d680c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d680c5320 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00b7f0 tx=0x7f5d5c00bcb0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 shutdown_connections 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d5406b500 0x7f5d5406d9c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d68151760 0x7f5d68151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d680cb2a0 0x7f5d680b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.927+0000 7f5d6df69640 1 --2- 172.21.15.99:0/3747015039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d50003fe0 0x7f5d680c5320 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:34.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.928+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 >> 172.21.15.99:0/3747015039 conn(0x7f5d680b33a0 msgr2=0x7f5d681614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:34.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.928+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 shutdown_connections 2024-03-04T10:04:34.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:34.928+0000 7f5d6df69640 1 -- 172.21.15.99:0/3747015039 wait complete. 2024-03-04T10:04:34.941 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:04:35.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.040+0000 7fbbc110a640 1 Processor -- start 2024-03-04T10:04:35.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 -- start start 2024-03-04T10:04:35.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbbbc15b030 0x7fbbbc15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc06d3b0 0x7fbbbc15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbbbc15b410 0x7fbbbc15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc0b2720 con 0x7fbbbc15b410 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc15be00 con 0x7fbbbc15b030 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbc110a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc15e2d0 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbbb59e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbbbc15b410 0x7fbbbc15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbbb59e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbbbc15b410 0x7fbbbc15b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35120/0 (socket says 172.21.15.99:35120) 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.041+0000 7fbbbb59e640 1 -- 172.21.15.99:0/1683892463 learned_addr learned my addr 172.21.15.99:0/1683892463 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbba59c640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc06d3b0 0x7fbbbc15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbad9d640 1 --2- 172.21.15.99:0/1683892463 >> v2:172.21.15.162:3300/0 conn(0x7fbbbc15b030 0x7fbbbc15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbb59e640 1 -- 172.21.15.99:0/1683892463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc06d3b0 msgr2=0x7fbbbc15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:35.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbb59e640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc06d3b0 0x7fbbbc15dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbb59e640 1 -- 172.21.15.99:0/1683892463 >> v2:172.21.15.162:3300/0 conn(0x7fbbbc15b030 msgr2=0x7fbbbc15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbb59e640 1 --2- 172.21.15.99:0/1683892463 >> v2:172.21.15.162:3300/0 conn(0x7fbbbc15b030 0x7fbbbc15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbb59e640 1 -- 172.21.15.99:0/1683892463 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc06d8b0 con 0x7fbbbc15b410 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbba59c640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc06d3b0 0x7fbbbc15dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbad9d640 1 --2- 172.21.15.99:0/1683892463 >> v2:172.21.15.162:3300/0 conn(0x7fbbbc15b030 0x7fbbbc15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.042+0000 7fbbbb59e640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc15b410 0x7fbbbc15b8c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0725c0 tx=0x7fbbb402e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913c0e7c36c8045c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbb9d9b640 1 -- 172.21.15.99:0/1683892463 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb402ea00 con 0x7fbbbc15b410 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbb9d9b640 1 -- 172.21.15.99:0/1683892463 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb402eba0 con 0x7fbbbc15b410 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbb9d9b640 1 -- 172.21.15.99:0/1683892463 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4009600 con 0x7fbbbc15b410 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 -- 172.21.15.99:0/1683892463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc15b410 msgr2=0x7fbbbc15b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc15b410 0x7fbbbc15b8c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0725c0 tx=0x7fbbb402e060 comp rx=0 tx=0).stop 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 -- 172.21.15.99:0/1683892463 shutdown_connections 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc15b410 0x7fbbbc15b8c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 --2- 172.21.15.99:0/1683892463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc06d3b0 0x7fbbbc15dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 --2- 172.21.15.99:0/1683892463 >> v2:172.21.15.162:3300/0 conn(0x7fbbbc15b030 0x7fbbbc15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 -- 172.21.15.99:0/1683892463 >> 172.21.15.99:0/1683892463 conn(0x7fbbbc06acd0 msgr2=0x7fbbbc06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:35.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 -- 172.21.15.99:0/1683892463 shutdown_connections 2024-03-04T10:04:35.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.043+0000 7fbbc110a640 1 -- 172.21.15.99:0/1683892463 wait complete. 2024-03-04T10:04:35.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 Processor -- start 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 -- start start 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 0x7fbbbc0c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbbc15b030 0x7fbbbc0ad060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc15b410 0x7fbbbc15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06e150 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06de70 con 0x7fbbbc15b030 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbc110a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc06e690 con 0x7fbbbc15b410 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 0x7fbbbc0c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.045+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 0x7fbbbc0c4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35134/0 (socket says 172.21.15.99:35134) 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 -- 172.21.15.99:0/3774558591 learned_addr learned my addr 172.21.15.99:0/3774558591 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbba59c640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbbc15b030 0x7fbbbc0ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbb59e640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc15b410 0x7fbbbc15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 -- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc15b410 msgr2=0x7fbbbc15c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc15b410 0x7fbbbc15c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 -- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbbc15b030 msgr2=0x7fbbbc0ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbbc15b030 0x7fbbbc0ad060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 -- 172.21.15.99:0/3774558591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc1490f0 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbba59c640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbbc15b030 0x7fbbbc0ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbb59e640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc15b410 0x7fbbbc15c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbbad9d640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 0x7fbbbc0c4690 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000c820 tx=0x7fbbb000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0015070 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb0007a30 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc0c5110 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0007de0 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.046+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc149690 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.047+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc074f60 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.048+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbbb0018070 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.048+0000 7fbba37fe640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbba806b4d0 0x7fbba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.048+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fbbb008a520 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.048+0000 7fbbba59c640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbba806b4d0 0x7fbba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.049+0000 7fbbba59c640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbba806b4d0 0x7fbba806d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbbac006fd0 tx=0x7fbbac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.052+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbbb0058fb0 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.234+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbbbc0cc080 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.235+0000 7fbba37fe640 1 -- 172.21.15.99:0/3774558591 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbbb00588b0 con 0x7fbbbc06d3b0 2024-03-04T10:04:35.237 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbba806b4d0 msgr2=0x7fbba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbba806b4d0 0x7fbba806d990 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbbac006fd0 tx=0x7fbbac008040 comp rx=0 tx=0).stop 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 msgr2=0x7fbbbc0c4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 0x7fbbbc0c4690 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000c820 tx=0x7fbbb000cce0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 shutdown_connections 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbba806b4d0 0x7fbba806d990 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbbbc15b410 0x7fbbbc15c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbbbc15b030 0x7fbbbc0ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 --2- 172.21.15.99:0/3774558591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbbbc06d3b0 0x7fbbbc0c4690 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 >> 172.21.15.99:0/3774558591 conn(0x7fbbbc06acd0 msgr2=0x7fbbbc0b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:35.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 shutdown_connections 2024-03-04T10:04:35.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.237+0000 7fbbc110a640 1 -- 172.21.15.99:0/3774558591 wait complete. 2024-03-04T10:04:35.249 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-04T10:04:35.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:35.250 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-04T10:04:35.277 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:04:35.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:35.277 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-04T10:04:35.333 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:35.333 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:35.352 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:35.352 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:35.409 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:04:35.409 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:35.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:35.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.568+0000 7f7db5b2b640 1 Processor -- start 2024-03-04T10:04:35.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7db5b2b640 1 -- start start 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7db5b2b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7db00cfd60 0x7f7db00cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7db5b2b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7db00cf0d0 0x7f7db00cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7db5b2b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00d0140 con 0x7f7db006d3b0 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7db5b2b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db015cbb0 con 0x7f7db00cfd60 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7db5b2b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db015c280 con 0x7f7db00cf0d0 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7daf7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7db00cfd60 0x7f7db00cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7daf7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7db00cfd60 0x7f7db00cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35142/0 (socket says 172.21.15.99:35142) 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.569+0000 7f7daf7fe640 1 -- 172.21.15.99:0/2364659337 learned_addr learned my addr 172.21.15.99:0/2364659337 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.570+0000 7f7daffff640 1 --2- 172.21.15.99:0/2364659337 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cf0d0 0x7f7db00cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.570+0000 7f7daf7fe640 1 -- 172.21.15.99:0/2364659337 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cf0d0 msgr2=0x7f7db00cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:35.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.570+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/2364659337 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cf0d0 0x7f7db00cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.570+0000 7f7daf7fe640 1 -- 172.21.15.99:0/2364659337 >> v1:172.21.15.162:6789/0 conn(0x7f7db006d3b0 legacy=0x7f7db00ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.570+0000 7f7daf7fe640 1 -- 172.21.15.99:0/2364659337 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db015c560 con 0x7f7db00cfd60 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/2364659337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db00cfd60 0x7f7db00cbdf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7db015cd30 tx=0x7f7da002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedeb58aa34a72c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7daeffd640 1 -- 172.21.15.99:0/2364659337 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0034070 con 0x7f7db00cfd60 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7daeffd640 1 -- 172.21.15.99:0/2364659337 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da0030860 con 0x7f7db00cfd60 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7daeffd640 1 -- 172.21.15.99:0/2364659337 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0030c30 con 0x7f7db00cfd60 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 -- 172.21.15.99:0/2364659337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db00cfd60 msgr2=0x7f7db00cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/2364659337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db00cfd60 0x7f7db00cbdf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7db015cd30 tx=0x7f7da002b9a0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 -- 172.21.15.99:0/2364659337 shutdown_connections 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/2364659337 >> v2:172.21.15.162:3300/0 conn(0x7f7db00cf0d0 0x7f7db00cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/2364659337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db00cfd60 0x7f7db00cbdf0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 -- 172.21.15.99:0/2364659337 >> 172.21.15.99:0/2364659337 conn(0x7f7db006acd0 msgr2=0x7f7db006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:35.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 -- 172.21.15.99:0/2364659337 shutdown_connections 2024-03-04T10:04:35.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.571+0000 7f7db5b2b640 1 -- 172.21.15.99:0/2364659337 wait complete. 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.572+0000 7f7db5b2b640 1 Processor -- start 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 -- start start 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 0x7f7db014cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00cf0d0 0x7f7db00af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00cfd60 0x7f7db006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015f520 con 0x7f7db006d3b0 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015bf30 con 0x7f7db00cf0d0 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7db5b2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015f6a0 con 0x7f7db00cfd60 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 0x7f7db014cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7daf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 0x7f7db014cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35158/0 (socket says 172.21.15.99:35158) 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.573+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4184506514 learned_addr learned my addr 172.21.15.99:0/4184506514 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7da6ffd640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00cf0d0 0x7f7db00af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daffff640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00cfd60 0x7f7db006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00cfd60 msgr2=0x7f7db006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00cfd60 0x7f7db006dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00cf0d0 msgr2=0x7f7db00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00cf0d0 0x7f7db00af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daf7fe640 1 -- 172.21.15.99:0/4184506514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00afc80 con 0x7f7db006d3b0 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7da6ffd640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00cf0d0 0x7f7db00af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daffff640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00cfd60 0x7f7db006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7daf7fe640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 0x7f7db014cae0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7da002b9e0 tx=0x7f7da00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0034040 con 0x7f7db006d3b0 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da0039d90 con 0x7f7db006d3b0 2024-03-04T10:04:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db00aff60 con 0x7f7db006d3b0 2024-03-04T10:04:35.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0043610 con 0x7f7db006d3b0 2024-03-04T10:04:35.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.574+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db00b0520 con 0x7f7db006d3b0 2024-03-04T10:04:35.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.575+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db014f7e0 con 0x7f7db006d3b0 2024-03-04T10:04:35.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.575+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7da0037070 con 0x7f7db006d3b0 2024-03-04T10:04:35.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.576+0000 7f7dacff9640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.576+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7da00ac780 con 0x7f7db006d3b0 2024-03-04T10:04:35.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.576+0000 7f7da6ffd640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.576+0000 7f7da6ffd640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9006b4d0 0x7f7d9006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7db006f410 tx=0x7f7d98009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.580+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7db014f7e0 con 0x7f7db006d3b0 2024-03-04T10:04:35.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.753+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7db015b560 con 0x7f7db006d3b0 2024-03-04T10:04:35.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.755+0000 7f7dacff9640 1 -- 172.21.15.99:0/4184506514 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7f7da003e070 con 0x7f7db006d3b0 2024-03-04T10:04:35.757 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:35.757 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9006b4d0 msgr2=0x7f7d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9006b4d0 0x7f7d9006d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7db006f410 tx=0x7f7d98009040 comp rx=0 tx=0).stop 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 msgr2=0x7f7db014cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 0x7f7db014cae0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7da002b9e0 tx=0x7f7da00046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 shutdown_connections 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d9006b4d0 0x7f7d9006d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7db00cfd60 0x7f7db006dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db00cf0d0 0x7f7db00af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 --2- 172.21.15.99:0/4184506514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7db006d3b0 0x7f7db014cae0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 >> 172.21.15.99:0/4184506514 conn(0x7f7db006acd0 msgr2=0x7f7db00ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:35.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 shutdown_connections 2024-03-04T10:04:35.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.757+0000 7f7db5b2b640 1 -- 172.21.15.99:0/4184506514 wait complete. 2024-03-04T10:04:35.770 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:35.770 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.871+0000 7f1572bc4640 1 Processor -- start 2024-03-04T10:04:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.871+0000 7f1572bc4640 1 -- start start 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1572bc4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f156c15ee80 0x7f156c0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1572bc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06eef0 0x7f156c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1572bc4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f156c06dd40 0x7f156c0cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1572bc4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c0cce60 con 0x7f156c06dd40 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1572bc4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f156c06f490 con 0x7f156c15ee80 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1572bc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0cd0b0 con 0x7f156c06eef0 2024-03-04T10:04:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f156c06dd40 0x7f156c0cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f156c06dd40 0x7f156c0cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35164/0 (socket says 172.21.15.99:35164) 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 -- 172.21.15.99:0/1059786045 learned_addr learned my addr 172.21.15.99:0/1059786045 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f156bfff640 1 --2- 172.21.15.99:0/1059786045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06eef0 0x7f156c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 -- 172.21.15.99:0/1059786045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06eef0 msgr2=0x7f156c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1570960640 1 --2- 172.21.15.99:0/1059786045 >> v2:172.21.15.162:3300/0 conn(0x7f156c15ee80 0x7f156c0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 --2- 172.21.15.99:0/1059786045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06eef0 0x7f156c06d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 -- 172.21.15.99:0/1059786045 >> v2:172.21.15.162:3300/0 conn(0x7f156c15ee80 msgr2=0x7f156c0724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 --2- 172.21.15.99:0/1059786045 >> v2:172.21.15.162:3300/0 conn(0x7f156c15ee80 0x7f156c0724e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.872+0000 7f1571161640 1 -- 172.21.15.99:0/1059786045 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c0cd390 con 0x7f156c06dd40 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.873+0000 7f1571161640 1 --2- 172.21.15.99:0/1059786045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06dd40 0x7f156c0cefb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f156c073130 tx=0x7f156402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870b54e855bb9920 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.873+0000 7f156b7fe640 1 -- 172.21.15.99:0/1059786045 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156402ca70 con 0x7f156c06dd40 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.873+0000 7f156b7fe640 1 -- 172.21.15.99:0/1059786045 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156402cc10 con 0x7f156c06dd40 2024-03-04T10:04:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.873+0000 7f156b7fe640 1 -- 172.21.15.99:0/1059786045 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15640343f0 con 0x7f156c06dd40 2024-03-04T10:04:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 -- 172.21.15.99:0/1059786045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06dd40 msgr2=0x7f156c0cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1059786045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06dd40 0x7f156c0cefb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f156c073130 tx=0x7f156402bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 -- 172.21.15.99:0/1059786045 shutdown_connections 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1059786045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06dd40 0x7f156c0cefb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1059786045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06eef0 0x7f156c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1059786045 >> v2:172.21.15.162:3300/0 conn(0x7f156c15ee80 0x7f156c0724e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 -- 172.21.15.99:0/1059786045 >> 172.21.15.99:0/1059786045 conn(0x7f156c0cfad0 msgr2=0x7f156c0cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 -- 172.21.15.99:0/1059786045 shutdown_connections 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.874+0000 7f1572bc4640 1 -- 172.21.15.99:0/1059786045 wait complete. 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.875+0000 7f1572bc4640 1 Processor -- start 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.875+0000 7f1572bc4640 1 -- start start 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1572bc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06dd40 0x7f156c140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1572bc4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 0x7f156c13dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1572bc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f156c15ee80 0x7f156c1f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1572bc4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c15f260 con 0x7f156c06eef0 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1572bc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156c072a20 con 0x7f156c15ee80 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1572bc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156c0cf4f0 con 0x7f156c06dd40 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 0x7f156c13dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 0x7f156c13dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35176/0 (socket says 172.21.15.99:35176) 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 -- 172.21.15.99:0/1504930191 learned_addr learned my addr 172.21.15.99:0/1504930191 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1570960640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06dd40 0x7f156c140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1571161640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f156c15ee80 0x7f156c1f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 -- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06dd40 msgr2=0x7f156c140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06dd40 0x7f156c140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 -- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f156c15ee80 msgr2=0x7f156c1f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f156c15ee80 0x7f156c1f6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 -- 172.21.15.99:0/1504930191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c13e130 con 0x7f156c06eef0 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1570960640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06dd40 0x7f156c140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f1571161640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f156c15ee80 0x7f156c1f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.876+0000 7f156bfff640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 0x7f156c13dbf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f156000c820 tx=0x7f156000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.877+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560015070 con 0x7f156c06eef0 2024-03-04T10:04:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.877+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1560004220 con 0x7f156c06eef0 2024-03-04T10:04:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.877+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156c1f7640 con 0x7f156c06eef0 2024-03-04T10:04:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.877+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156000ce10 con 0x7f156c06eef0 2024-03-04T10:04:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.877+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f156c140770 con 0x7f156c06eef0 2024-03-04T10:04:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.878+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156c074f60 con 0x7f156c06eef0 2024-03-04T10:04:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.878+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f15600077d0 con 0x7f156c06eef0 2024-03-04T10:04:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.879+0000 7f15697fa640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155806b4d0 0x7f155806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.879+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f156008a400 con 0x7f156c06eef0 2024-03-04T10:04:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.879+0000 7f1570960640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155806b4d0 0x7f155806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.879+0000 7f1570960640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155806b4d0 0x7f155806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f156c0b4950 tx=0x7f155c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:35.883+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1560058e00 con 0x7f156c06eef0 2024-03-04T10:04:36.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.052+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f156c06c780 con 0x7f156c06eef0 2024-03-04T10:04:36.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.053+0000 7f15697fa640 1 -- 172.21.15.99:0/1504930191 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1560058700 con 0x7f156c06eef0 2024-03-04T10:04:36.055 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155806b4d0 msgr2=0x7f155806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155806b4d0 0x7f155806d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f156c0b4950 tx=0x7f155c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 msgr2=0x7f156c13dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 0x7f156c13dbf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f156000c820 tx=0x7f156000cce0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 shutdown_connections 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155806b4d0 0x7f155806d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f156c15ee80 0x7f156c1f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f156c06eef0 0x7f156c13dbf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 --2- 172.21.15.99:0/1504930191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f156c06dd40 0x7f156c140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 >> 172.21.15.99:0/1504930191 conn(0x7f156c0cfad0 msgr2=0x7f156c0cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 shutdown_connections 2024-03-04T10:04:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.055+0000 7f1572bc4640 1 -- 172.21.15.99:0/1504930191 wait complete. 2024-03-04T10:04:36.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:36.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.169+0000 7f08c36d1640 1 Processor -- start 2024-03-04T10:04:36.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.169+0000 7f08c36d1640 1 -- start start 2024-03-04T10:04:36.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c36d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc15a810 0x7f08bc0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c36d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f08bc06d3c0 0x7f08bc0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c36d1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f08bc0cbfb0 0x7f08bc15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c36d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc06d8a0 con 0x7f08bc06d3c0 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c36d1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc06db80 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c36d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc06de60 con 0x7f08bc15a810 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f08bc06d3c0 0x7f08bc0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f08bc06d3c0 0x7f08bc0b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35190/0 (socket says 172.21.15.99:35190) 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 -- 172.21.15.99:0/2899657046 learned_addr learned my addr 172.21.15.99:0/2899657046 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c146d640 1 --2- 172.21.15.99:0/2899657046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc15a810 0x7f08bc0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c1c6e640 1 --2- 172.21.15.99:0/2899657046 >> v2:172.21.15.162:3300/0 conn(0x7f08bc0cbfb0 0x7f08bc15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 -- 172.21.15.99:0/2899657046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc15a810 msgr2=0x7f08bc0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 --2- 172.21.15.99:0/2899657046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc15a810 0x7f08bc0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 -- 172.21.15.99:0/2899657046 >> v2:172.21.15.162:3300/0 conn(0x7f08bc0cbfb0 msgr2=0x7f08bc15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:36.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 --2- 172.21.15.99:0/2899657046 >> v2:172.21.15.162:3300/0 conn(0x7f08bc0cbfb0 0x7f08bc15abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.170+0000 7f08c0c6c640 1 -- 172.21.15.99:0/2899657046 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc06e140 con 0x7f08bc06d3c0 2024-03-04T10:04:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.171+0000 7f08c0c6c640 1 --2- 172.21.15.99:0/2899657046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc06d3c0 0x7f08bc0b46d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f08bc06bc80 tx=0x7f08ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136dbb76c4a66afb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.171+0000 7f08b3fff640 1 -- 172.21.15.99:0/2899657046 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac035020 con 0x7f08bc06d3c0 2024-03-04T10:04:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.171+0000 7f08b3fff640 1 -- 172.21.15.99:0/2899657046 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac031870 con 0x7f08bc06d3c0 2024-03-04T10:04:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.171+0000 7f08b3fff640 1 -- 172.21.15.99:0/2899657046 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac031c40 con 0x7f08bc06d3c0 2024-03-04T10:04:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 -- 172.21.15.99:0/2899657046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc06d3c0 msgr2=0x7f08bc0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 --2- 172.21.15.99:0/2899657046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc06d3c0 0x7f08bc0b46d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f08bc06bc80 tx=0x7f08ac02b910 comp rx=0 tx=0).stop 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 -- 172.21.15.99:0/2899657046 shutdown_connections 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 --2- 172.21.15.99:0/2899657046 >> v2:172.21.15.162:3300/0 conn(0x7f08bc0cbfb0 0x7f08bc15abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 --2- 172.21.15.99:0/2899657046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc06d3c0 0x7f08bc0b46d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 --2- 172.21.15.99:0/2899657046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc15a810 0x7f08bc0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 -- 172.21.15.99:0/2899657046 >> 172.21.15.99:0/2899657046 conn(0x7f08bc06acd0 msgr2=0x7f08bc06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 -- 172.21.15.99:0/2899657046 shutdown_connections 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.172+0000 7f08c36d1640 1 -- 172.21.15.99:0/2899657046 wait complete. 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.173+0000 7f08c36d1640 1 Processor -- start 2024-03-04T10:04:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.173+0000 7f08c36d1640 1 -- start start 2024-03-04T10:04:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc06d3c0 0x7f08bc147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 0x7f08bc0a87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc15a810 0x7f08bc0ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc15b130 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc15e5d0 con 0x7f08bc15a810 2024-03-04T10:04:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc15b2b0 con 0x7f08bc06d3c0 2024-03-04T10:04:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 0x7f08bc0a87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 0x7f08bc0a87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35198/0 (socket says 172.21.15.99:35198) 2024-03-04T10:04:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 -- 172.21.15.99:0/1100540086 learned_addr learned my addr 172.21.15.99:0/1100540086 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c146d640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc06d3c0 0x7f08bc147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c1c6e640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc15a810 0x7f08bc0ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 -- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc06d3c0 msgr2=0x7f08bc147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc06d3c0 0x7f08bc147fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 -- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc15a810 msgr2=0x7f08bc0ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc15a810 0x7f08bc0ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 -- 172.21.15.99:0/1100540086 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0ad490 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c0c6c640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 0x7f08bc0a87a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f08ac000c00 tx=0x7f08ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac048070 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac0305d0 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc0ad770 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.174+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac035040 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.175+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc0ad140 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.176+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc074f60 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.176+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f08ac04b070 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.176+0000 7f08b1ffb640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08a006b500 0x7f08a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.177+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f08ac0ae590 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.177+0000 7f08c146d640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08a006b500 0x7f08a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.177+0000 7f08c146d640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08a006b500 0x7f08a006d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08b80073f0 tx=0x7f08b800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.181+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f08ac040070 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.349+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08bc06ec30 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.350+0000 7f08b1ffb640 1 -- 172.21.15.99:0/1100540086 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f08ac07d0b0 con 0x7f08bc0cbfb0 2024-03-04T10:04:36.352 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08a006b500 msgr2=0x7f08a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08a006b500 0x7f08a006d9c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f08b80073f0 tx=0x7f08b800b040 comp rx=0 tx=0).stop 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 msgr2=0x7f08bc0a87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 0x7f08bc0a87a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f08ac000c00 tx=0x7f08ac0046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 shutdown_connections 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08a006b500 0x7f08a006d9c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08bc15a810 0x7f08bc0ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08bc0cbfb0 0x7f08bc0a87a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 --2- 172.21.15.99:0/1100540086 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08bc06d3c0 0x7f08bc147fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 >> 172.21.15.99:0/1100540086 conn(0x7f08bc06acd0 msgr2=0x7f08bc15d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 shutdown_connections 2024-03-04T10:04:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.352+0000 7f08c36d1640 1 -- 172.21.15.99:0/1100540086 wait complete. 2024-03-04T10:04:36.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.465+0000 7fe1267b2640 1 Processor -- start 2024-03-04T10:04:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.465+0000 7fe1267b2640 1 -- start start 2024-03-04T10:04:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe1267b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015da20 0x7fe1200ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe1267b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe1200cf660 0x7fe1200cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe1267b2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe120162450 con 0x7fe1201616e0 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe1267b2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1200d0000 con 0x7fe1200cf660 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe1267b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200cc920 con 0x7fe12015da20 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe11f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe1200cf660 0x7fe1200cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe11f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe1200cf660 0x7fe1200cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35212/0 (socket says 172.21.15.99:35212) 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1741591014 learned_addr learned my addr 172.21.15.99:0/1741591014 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.466+0000 7fe11ffff640 1 --2- 172.21.15.99:0/1741591014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015da20 0x7fe1200ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1741591014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015da20 msgr2=0x7fe1200ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11f7fe640 1 --2- 172.21.15.99:0/1741591014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015da20 0x7fe1200ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1741591014 >> v1:172.21.15.162:6789/0 conn(0x7fe1201616e0 legacy=0x7fe120161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1741591014 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200cc2d0 con 0x7fe1200cf660 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11f7fe640 1 --2- 172.21.15.99:0/1741591014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe1200cf660 0x7fe1200cfac0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe114009870 tx=0x7fe11402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e936ee3a880afb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11effd640 1 -- 172.21.15.99:0/1741591014 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114035020 con 0x7fe1200cf660 2024-03-04T10:04:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.467+0000 7fe11effd640 1 -- 172.21.15.99:0/1741591014 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe114031640 con 0x7fe1200cf660 2024-03-04T10:04:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe11effd640 1 -- 172.21.15.99:0/1741591014 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11402d040 con 0x7fe1200cf660 2024-03-04T10:04:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 -- 172.21.15.99:0/1741591014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe1200cf660 msgr2=0x7fe1200cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1741591014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe1200cf660 0x7fe1200cfac0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe114009870 tx=0x7fe11402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 -- 172.21.15.99:0/1741591014 shutdown_connections 2024-03-04T10:04:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1741591014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe1200cf660 0x7fe1200cfac0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1741591014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015da20 0x7fe1200ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 -- 172.21.15.99:0/1741591014 >> 172.21.15.99:0/1741591014 conn(0x7fe1200b33a0 msgr2=0x7fe1200b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 -- 172.21.15.99:0/1741591014 shutdown_connections 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.468+0000 7fe1267b2640 1 -- 172.21.15.99:0/1741591014 wait complete. 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.469+0000 7fe1267b2640 1 Processor -- start 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.469+0000 7fe1267b2640 1 -- start start 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe1267b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cf660 0x7fe120151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe1267b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 0x7fe1201607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe1267b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1201616e0 0x7fe1201e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe1267b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200cbf80 con 0x7fe12015da20 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe1267b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200cd2f0 con 0x7fe1200cf660 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe1267b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12015de00 con 0x7fe1201616e0 2024-03-04T10:04:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 0x7fe1201607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 0x7fe1201607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35226/0 (socket says 172.21.15.99:35226) 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1619705147 learned_addr learned my addr 172.21.15.99:0/1619705147 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe124d4f640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1201616e0 0x7fe1201e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11ffff640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cf660 0x7fe120151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1201616e0 msgr2=0x7fe1201e94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1201616e0 0x7fe1201e94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cf660 msgr2=0x7fe120151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cf660 0x7fe120151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.470+0000 7fe11f7fe640 1 -- 172.21.15.99:0/1619705147 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12015eb50 con 0x7fe12015da20 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.471+0000 7fe11f7fe640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 0x7fe1201607c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe114000c00 tx=0x7fe1140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.471+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114040070 con 0x7fe12015da20 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.471+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1140399f0 con 0x7fe12015da20 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.471+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12015ee30 con 0x7fe12015da20 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.471+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe114035040 con 0x7fe12015da20 2024-03-04T10:04:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.471+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12015f3f0 con 0x7fe12015da20 2024-03-04T10:04:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.472+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe120074f60 con 0x7fe12015da20 2024-03-04T10:04:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.472+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe114038020 con 0x7fe12015da20 2024-03-04T10:04:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.472+0000 7fe11cff9640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10c06b500 0x7fe10c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.473+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fe1140abf60 con 0x7fe12015da20 2024-03-04T10:04:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.473+0000 7fe11ffff640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10c06b500 0x7fe10c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.473+0000 7fe11ffff640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10c06b500 0x7fe10c06d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe110006fd0 tx=0x7fe110008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.477+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe11407a3a0 con 0x7fe12015da20 2024-03-04T10:04:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.648+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe12006d750 con 0x7fe12015da20 2024-03-04T10:04:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.648+0000 7fe11cff9640 1 -- 172.21.15.99:0/1619705147 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe11407e440 con 0x7fe12015da20 2024-03-04T10:04:36.651 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10c06b500 msgr2=0x7fe10c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10c06b500 0x7fe10c06d9c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe110006fd0 tx=0x7fe110008040 comp rx=0 tx=0).stop 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 msgr2=0x7fe1201607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 0x7fe1201607c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe114000c00 tx=0x7fe1140046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 shutdown_connections 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10c06b500 0x7fe10c06d9c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1201616e0 0x7fe1201e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12015da20 0x7fe1201607c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 --2- 172.21.15.99:0/1619705147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cf660 0x7fe120151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 >> 172.21.15.99:0/1619705147 conn(0x7fe1200b33a0 msgr2=0x7fe1200b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 shutdown_connections 2024-03-04T10:04:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.650+0000 7fe1267b2640 1 -- 172.21.15.99:0/1619705147 wait complete. 2024-03-04T10:04:36.663 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:36.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.766+0000 7f937a705640 1 Processor -- start 2024-03-04T10:04:36.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.766+0000 7f937a705640 1 -- start start 2024-03-04T10:04:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f937a705640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f93740cdd70 0x7f93740cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f937a705640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f93740cc900 0x7f93740cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f937a705640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937406ae00 con 0x7f93740ccfe0 2024-03-04T10:04:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f937a705640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93740ccde0 con 0x7f93740cdd70 2024-03-04T10:04:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f937a705640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93740d0480 con 0x7f93740cc900 2024-03-04T10:04:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9373fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f93740cdd70 0x7f93740cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9378ca2640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f93740ccfe0 0x7f937415d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34070/0 (socket says 172.21.15.99:34070) 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9378ca2640 1 -- 172.21.15.99:0/77996557 learned_addr learned my addr 172.21.15.99:0/77996557 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f93737fe640 1 --2- 172.21.15.99:0/77996557 >> v2:172.21.15.162:3300/0 conn(0x7f93740cc900 0x7f93740cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9373fff640 1 -- 172.21.15.99:0/77996557 >> v2:172.21.15.162:3300/0 conn(0x7f93740cc900 msgr2=0x7f93740cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9373fff640 1 --2- 172.21.15.99:0/77996557 >> v2:172.21.15.162:3300/0 conn(0x7f93740cc900 0x7f93740cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9373fff640 1 -- 172.21.15.99:0/77996557 >> v1:172.21.15.99:6789/0 conn(0x7f93740ccfe0 legacy=0x7f937415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9372ffd640 1 -- 172.21.15.99:0/77996557 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688248914 0 0) 0x7f937406ae00 con 0x7f93740ccfe0 2024-03-04T10:04:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.767+0000 7f9373fff640 1 -- 172.21.15.99:0/77996557 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937406b0e0 con 0x7f93740cdd70 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f9373fff640 1 --2- 172.21.15.99:0/77996557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cdd70 0x7f93740cfa00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f93740b66c0 tx=0x7f936402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db19b76b7e597fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f9372ffd640 1 -- 172.21.15.99:0/77996557 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640315a0 con 0x7f93740cdd70 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f9372ffd640 1 -- 172.21.15.99:0/77996557 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364037070 con 0x7f93740cdd70 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f9372ffd640 1 -- 172.21.15.99:0/77996557 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364031960 con 0x7f93740cdd70 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f937a705640 1 -- 172.21.15.99:0/77996557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cdd70 msgr2=0x7f93740cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f937a705640 1 --2- 172.21.15.99:0/77996557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cdd70 0x7f93740cfa00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f93740b66c0 tx=0x7f936402b840 comp rx=0 tx=0).stop 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f937a705640 1 -- 172.21.15.99:0/77996557 shutdown_connections 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f937a705640 1 --2- 172.21.15.99:0/77996557 >> v2:172.21.15.162:3300/0 conn(0x7f93740cc900 0x7f93740cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f937a705640 1 --2- 172.21.15.99:0/77996557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cdd70 0x7f93740cfa00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.768+0000 7f937a705640 1 -- 172.21.15.99:0/77996557 >> 172.21.15.99:0/77996557 conn(0x7f93740b22e0 msgr2=0x7f93740b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.769+0000 7f937a705640 1 -- 172.21.15.99:0/77996557 shutdown_connections 2024-03-04T10:04:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.769+0000 7f937a705640 1 -- 172.21.15.99:0/77996557 wait complete. 2024-03-04T10:04:36.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.770+0000 7f937a705640 1 Processor -- start 2024-03-04T10:04:36.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.770+0000 7f937a705640 1 -- start start 2024-03-04T10:04:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.770+0000 7f937a705640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 0x7f937414fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93740ccfe0 0x7f93740c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93740cdd70 0x7f937415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b2e90 con 0x7f93740cc900 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937406b6a0 con 0x7f93740ccfe0 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93740b3010 con 0x7f93740cdd70 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 0x7f937414fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 0x7f937414fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35254/0 (socket says 172.21.15.99:35254) 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 -- 172.21.15.99:0/2155244148 learned_addr learned my addr 172.21.15.99:0/2155244148 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9378ca2640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93740cdd70 0x7f937415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f93737fe640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93740ccfe0 0x7f93740c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 -- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93740cdd70 msgr2=0x7f937415fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93740cdd70 0x7f937415fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 -- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93740ccfe0 msgr2=0x7f93740c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93740ccfe0 0x7f93740c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 -- 172.21.15.99:0/2155244148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93740b4190 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9378ca2640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93740cdd70 0x7f937415fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f93737fe640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93740ccfe0 0x7f93740c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9373fff640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 0x7f937414fcb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f936402be50 tx=0x7f93640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364032840 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93640329e0 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93740c5dd0 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364032dd0 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.771+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93740c6390 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.772+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9374074f60 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.773+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f936404c020 con 0x7f93740cc900 2024-03-04T10:04:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.773+0000 7f9370ff9640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f936006b4d0 0x7f936006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:36.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.773+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f936402f020 con 0x7f93740cc900 2024-03-04T10:04:36.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.773+0000 7f93737fe640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f936006b4d0 0x7f936006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:36.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.774+0000 7f93737fe640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f936006b4d0 0x7f936006d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9374153890 tx=0x7f9368008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:36.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.777+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9364040070 con 0x7f93740cc900 2024-03-04T10:04:36.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.948+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f93740af570 con 0x7f93740cc900 2024-03-04T10:04:36.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.949+0000 7f9370ff9640 1 -- 172.21.15.99:0/2155244148 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f936407afa0 con 0x7f93740cc900 2024-03-04T10:04:36.951 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f936006b4d0 msgr2=0x7f936006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f936006b4d0 0x7f936006d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9374153890 tx=0x7f9368008040 comp rx=0 tx=0).stop 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 msgr2=0x7f937414fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 0x7f937414fcb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f936402be50 tx=0x7f93640046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 shutdown_connections 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f936006b4d0 0x7f936006d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f93740cdd70 0x7f937415fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93740ccfe0 0x7f93740c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 --2- 172.21.15.99:0/2155244148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f93740cc900 0x7f937414fcb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 >> 172.21.15.99:0/2155244148 conn(0x7f93740b22e0 msgr2=0x7f937406e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 shutdown_connections 2024-03-04T10:04:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:36.951+0000 7f937a705640 1 -- 172.21.15.99:0/2155244148 wait complete. 2024-03-04T10:04:36.964 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:04:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.063+0000 7fbccdca4640 1 Processor -- start 2024-03-04T10:04:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.063+0000 7fbccdca4640 1 -- start start 2024-03-04T10:04:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbccdca4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc815dea0 0x7fbcc80cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbccdca4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbcc806b3e0 0x7fbcc80cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbccdca4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc806fd20 con 0x7fbcc80cc8d0 2024-03-04T10:04:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbccdca4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc806fad0 con 0x7fbcc806b3e0 2024-03-04T10:04:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbccdca4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8070000 con 0x7fbcc815dea0 2024-03-04T10:04:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbcc7fff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbcc806b3e0 0x7fbcc80cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbcc77fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc815dea0 0x7fbcc80cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbcc6ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fbcc80cc8d0 0x7fbcc806acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:36070/0 (socket says 172.21.15.99:36070) 2024-03-04T10:04:37.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.064+0000 7fbcc6ffd640 1 -- 172.21.15.99:0/2681962021 learned_addr learned my addr 172.21.15.99:0/2681962021 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.065+0000 7fbcc77fe640 1 -- 172.21.15.99:0/2681962021 >> v2:172.21.15.162:3300/0 conn(0x7fbcc806b3e0 msgr2=0x7fbcc80cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.065+0000 7fbcc77fe640 1 --2- 172.21.15.99:0/2681962021 >> v2:172.21.15.162:3300/0 conn(0x7fbcc806b3e0 0x7fbcc80cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.065+0000 7fbcc67fc640 1 -- 172.21.15.99:0/2681962021 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851454407 0 0) 0x7fbcc806fd20 con 0x7fbcc80cc8d0 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.065+0000 7fbcc77fe640 1 -- 172.21.15.99:0/2681962021 >> v1:172.21.15.162:6789/0 conn(0x7fbcc80cc8d0 legacy=0x7fbcc806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.065+0000 7fbcc77fe640 1 -- 172.21.15.99:0/2681962021 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc80702e0 con 0x7fbcc815dea0 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.065+0000 7fbcc77fe640 1 --2- 172.21.15.99:0/2681962021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc815dea0 0x7fbcc80cad60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097a0 tx=0x7fbcb802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7de9c8ae87471757 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbcc67fc640 1 -- 172.21.15.99:0/2681962021 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8035020 con 0x7fbcc815dea0 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbcc67fc640 1 -- 172.21.15.99:0/2681962021 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb8031830 con 0x7fbcc815dea0 2024-03-04T10:04:37.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbcc67fc640 1 -- 172.21.15.99:0/2681962021 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8031c00 con 0x7fbcc815dea0 2024-03-04T10:04:37.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 -- 172.21.15.99:0/2681962021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc815dea0 msgr2=0x7fbcc80cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2681962021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc815dea0 0x7fbcc80cad60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097a0 tx=0x7fbcb802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 -- 172.21.15.99:0/2681962021 shutdown_connections 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2681962021 >> v2:172.21.15.162:3300/0 conn(0x7fbcc806b3e0 0x7fbcc80cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2681962021 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc815dea0 0x7fbcc80cad60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 -- 172.21.15.99:0/2681962021 >> 172.21.15.99:0/2681962021 conn(0x7fbcc80d0710 msgr2=0x7fbcc80d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 -- 172.21.15.99:0/2681962021 shutdown_connections 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.066+0000 7fbccdca4640 1 -- 172.21.15.99:0/2681962021 wait complete. 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.067+0000 7fbccdca4640 1 Processor -- start 2024-03-04T10:04:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 -- start start 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc806b3e0 0x7fbcc80b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcc80cc8d0 0x7fbcc8150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 0x7fbcc80b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80d1380 con 0x7fbcc815dea0 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80d1200 con 0x7fbcc80cc8d0 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbccdca4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc80d1500 con 0x7fbcc806b3e0 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbcc7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 0x7fbcc80b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbcc7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 0x7fbcc80b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35270/0 (socket says 172.21.15.99:35270) 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbcc7fff640 1 -- 172.21.15.99:0/2821335789 learned_addr learned my addr 172.21.15.99:0/2821335789 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbcc77fe640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc806b3e0 0x7fbcc80b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.068+0000 7fbcc6ffd640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcc80cc8d0 0x7fbcc8150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcc7fff640 1 -- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc806b3e0 msgr2=0x7fbcc80b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcc7fff640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc806b3e0 0x7fbcc80b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcc7fff640 1 -- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcc80cc8d0 msgr2=0x7fbcc8150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcc7fff640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcc80cc8d0 0x7fbcc8150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcc7fff640 1 -- 172.21.15.99:0/2821335789 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc80b09a0 con 0x7fbcc815dea0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcc7fff640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 0x7fbcc80b0170 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000ef90 tx=0x7fbcc000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0018020 con 0x7fbcc815dea0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc0014480 con 0x7fbcc815dea0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc80b0c80 con 0x7fbcc815dea0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0014870 con 0x7fbcc815dea0 2024-03-04T10:04:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.069+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc80b1240 con 0x7fbcc815dea0 2024-03-04T10:04:37.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.070+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc815d700 con 0x7fbcc815dea0 2024-03-04T10:04:37.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.070+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbcc001b020 con 0x7fbcc815dea0 2024-03-04T10:04:37.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.071+0000 7fbcaffff640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbcb406b4d0 0x7fbcb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.071+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fbcc008b300 con 0x7fbcc815dea0 2024-03-04T10:04:37.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.071+0000 7fbcc77fe640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbcb406b4d0 0x7fbcb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.072+0000 7fbcc77fe640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbcb406b4d0 0x7fbcb406d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8036000 tx=0x7fbcb8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.075+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbcc815d700 con 0x7fbcc815dea0 2024-03-04T10:04:37.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.257+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbcc8074f60 con 0x7fbcc815dea0 2024-03-04T10:04:37.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.257+0000 7fbcaffff640 1 -- 172.21.15.99:0/2821335789 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbcc0022020 con 0x7fbcc815dea0 2024-03-04T10:04:37.259 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:04:37.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbcb406b4d0 msgr2=0x7fbcb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbcb406b4d0 0x7fbcb406d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8036000 tx=0x7fbcb8008030 comp rx=0 tx=0).stop 2024-03-04T10:04:37.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 msgr2=0x7fbcc80b0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 0x7fbcc80b0170 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000ef90 tx=0x7fbcc000c490 comp rx=0 tx=0).stop 2024-03-04T10:04:37.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 shutdown_connections 2024-03-04T10:04:37.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbcb406b4d0 0x7fbcb406d990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbcc815dea0 0x7fbcc80b0170 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbcc80cc8d0 0x7fbcc8150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 --2- 172.21.15.99:0/2821335789 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbcc806b3e0 0x7fbcc80b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 >> 172.21.15.99:0/2821335789 conn(0x7fbcc80d0710 msgr2=0x7fbcc80adc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:37.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 shutdown_connections 2024-03-04T10:04:37.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.259+0000 7fbccdca4640 1 -- 172.21.15.99:0/2821335789 wait complete. 2024-03-04T10:04:37.272 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-04T10:04:37.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:37.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-04T10:04:37.306 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:04:37.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:37.306 DEBUG:teuthology.orchestra.run.smithi162:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-04T10:04:37.365 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T10:04:37.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:37.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.463+0000 7f1de2254640 1 Processor -- start 2024-03-04T10:04:37.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.463+0000 7f1de2254640 1 -- start start 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1de2254640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1ddc06c470 0x7f1ddc06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1de2254640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1ddc066de0 0x7f1ddc15f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1de2254640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ddc0ad6c0 con 0x7f1ddc06c090 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1de2254640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ddc0ad960 con 0x7f1ddc06c470 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1de2254640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ddc06ebc0 con 0x7f1ddc066de0 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1ddc06c470 0x7f1ddc06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1ddc06c470 0x7f1ddc06f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35272/0 (socket says 172.21.15.99:35272) 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2293791781 learned_addr learned my addr 172.21.15.99:0/2293791781 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddbfff640 1 --2- 172.21.15.99:0/2293791781 >> v2:172.21.15.162:3300/0 conn(0x7f1ddc066de0 0x7f1ddc15f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2293791781 >> v2:172.21.15.162:3300/0 conn(0x7f1ddc066de0 msgr2=0x7f1ddc15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 --2- 172.21.15.99:0/2293791781 >> v2:172.21.15.162:3300/0 conn(0x7f1ddc066de0 0x7f1ddc15f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2293791781 >> v1:172.21.15.162:6789/0 conn(0x7f1ddc06c090 legacy=0x7f1ddc0af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.464+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2293791781 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc06eea0 con 0x7f1ddc06c470 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.465+0000 7f1ddaffd640 1 --2- 172.21.15.99:0/2293791781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c470 0x7f1ddc06f680 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc1565c0 tx=0x7f1dd002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1254ad06a1052f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.465+0000 7f1dda7fc640 1 -- 172.21.15.99:0/2293791781 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00317c0 con 0x7f1ddc06c470 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.465+0000 7f1dda7fc640 1 -- 172.21.15.99:0/2293791781 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0037070 con 0x7f1ddc06c470 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.465+0000 7f1dda7fc640 1 -- 172.21.15.99:0/2293791781 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0031b80 con 0x7f1ddc06c470 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 -- 172.21.15.99:0/2293791781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c470 msgr2=0x7f1ddc06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 --2- 172.21.15.99:0/2293791781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c470 0x7f1ddc06f680 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc1565c0 tx=0x7f1dd002b860 comp rx=0 tx=0).stop 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 -- 172.21.15.99:0/2293791781 shutdown_connections 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 --2- 172.21.15.99:0/2293791781 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c470 0x7f1ddc06f680 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 --2- 172.21.15.99:0/2293791781 >> v2:172.21.15.162:3300/0 conn(0x7f1ddc066de0 0x7f1ddc15f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 -- 172.21.15.99:0/2293791781 >> 172.21.15.99:0/2293791781 conn(0x7f1ddc0cfef0 msgr2=0x7f1ddc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 -- 172.21.15.99:0/2293791781 shutdown_connections 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.466+0000 7f1de2254640 1 -- 172.21.15.99:0/2293791781 wait complete. 2024-03-04T10:04:37.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.467+0000 7f1de2254640 1 Processor -- start 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.467+0000 7f1de2254640 1 -- start start 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1de2254640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ddc066de0 0x7f1ddc1506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1de2254640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 0x7f1ddc155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1de2254640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ddc1543d0 0x7f1ddc154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1de2254640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc0addc0 con 0x7f1ddc06c090 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1de2254640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc0adc40 con 0x7f1ddc066de0 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1de2254640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc15faa0 con 0x7f1ddc1543d0 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 0x7f1ddc155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 0x7f1ddc155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35282/0 (socket says 172.21.15.99:35282) 2024-03-04T10:04:37.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2522801247 learned_addr learned my addr 172.21.15.99:0/2522801247 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddb7fe640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ddc066de0 0x7f1ddc1506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddbfff640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ddc1543d0 0x7f1ddc154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ddc1543d0 msgr2=0x7f1ddc154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ddc1543d0 0x7f1ddc154880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ddc066de0 msgr2=0x7f1ddc1506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ddc066de0 0x7f1ddc1506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 -- 172.21.15.99:0/2522801247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc150c00 con 0x7f1ddc06c090 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddbfff640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ddc1543d0 0x7f1ddc154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.468+0000 7f1ddaffd640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 0x7f1ddc155fe0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1dd002b8a0 tx=0x7f1dd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.469+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0032860 con 0x7f1ddc06c090 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.469+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0032a00 con 0x7f1ddc06c090 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.469+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc154f40 con 0x7f1ddc06c090 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.469+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0032df0 con 0x7f1ddc06c090 2024-03-04T10:04:37.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.469+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc155500 con 0x7f1ddc06c090 2024-03-04T10:04:37.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.470+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ddc0748c0 con 0x7f1ddc06c090 2024-03-04T10:04:37.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.470+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1dd00453f0 con 0x7f1ddc06c090 2024-03-04T10:04:37.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.470+0000 7f1dc3fff640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1dc806b500 0x7f1dc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.470+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f1dd00ac600 con 0x7f1ddc06c090 2024-03-04T10:04:37.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.471+0000 7f1ddb7fe640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1dc806b500 0x7f1dc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.471+0000 7f1ddb7fe640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1dc806b500 0x7f1dc806d9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc004ea0 tx=0x7f1dcc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.475+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1ddc0748c0 con 0x7f1ddc06c090 2024-03-04T10:04:37.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.681+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ddc06def0 con 0x7f1ddc06c090 2024-03-04T10:04:37.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.682+0000 7f1dc3fff640 1 -- 172.21.15.99:0/2522801247 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f1dd0040070 con 0x7f1ddc06c090 2024-03-04T10:04:37.684 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:37.684 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:37.684 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.683+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1dc806b500 msgr2=0x7f1dc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.683+0000 7f1de2254640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1dc806b500 0x7f1dc806d9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc004ea0 tx=0x7f1dcc009040 comp rx=0 tx=0).stop 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.683+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 msgr2=0x7f1ddc155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.683+0000 7f1de2254640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 0x7f1ddc155fe0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1dd002b8a0 tx=0x7f1dd00046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 shutdown_connections 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1dc806b500 0x7f1dc806d9c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ddc1543d0 0x7f1ddc154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ddc06c090 0x7f1ddc155fe0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 --2- 172.21.15.99:0/2522801247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ddc066de0 0x7f1ddc1506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 >> 172.21.15.99:0/2522801247 conn(0x7f1ddc0cfef0 msgr2=0x7f1ddc15f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:37.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 shutdown_connections 2024-03-04T10:04:37.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.684+0000 7f1de2254640 1 -- 172.21.15.99:0/2522801247 wait complete. 2024-03-04T10:04:37.697 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.795+0000 7f9a79545640 1 Processor -- start 2024-03-04T10:04:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.795+0000 7f9a79545640 1 -- start start 2024-03-04T10:04:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a79545640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a740b2e60 0x7f9a740b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a79545640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9a740b5470 0x7f9a740b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a79545640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7415e1e0 con 0x7f9a7415de00 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a79545640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740b98c0 con 0x7f9a740b2e60 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a79545640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740b32c0 con 0x7f9a740b5470 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a727fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a740b2e60 0x7f9a740b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a72ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9a7415de00 0x7f9a740b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34086/0 (socket says 172.21.15.99:34086) 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.796+0000 7f9a72ffd640 1 -- 172.21.15.99:0/2065210296 learned_addr learned my addr 172.21.15.99:0/2065210296 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a737fe640 1 --2- 172.21.15.99:0/2065210296 >> v2:172.21.15.162:3300/0 conn(0x7f9a740b5470 0x7f9a740b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a727fc640 1 -- 172.21.15.99:0/2065210296 >> v2:172.21.15.162:3300/0 conn(0x7f9a740b5470 msgr2=0x7f9a740b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:37.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a727fc640 1 --2- 172.21.15.99:0/2065210296 >> v2:172.21.15.162:3300/0 conn(0x7f9a740b5470 0x7f9a740b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a727fc640 1 -- 172.21.15.99:0/2065210296 >> v1:172.21.15.99:6789/0 conn(0x7f9a7415de00 legacy=0x7f9a740b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a727fc640 1 -- 172.21.15.99:0/2065210296 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740d1c10 con 0x7f9a740b2e60 2024-03-04T10:04:37.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a737fe640 1 --2- 172.21.15.99:0/2065210296 >> v2:172.21.15.162:3300/0 conn(0x7f9a740b5470 0x7f9a740b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:37.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a727fc640 1 --2- 172.21.15.99:0/2065210296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a740b9380 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9a740b2d40 tx=0x7f9a6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1ffac1be92903d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a71ffb640 1 -- 172.21.15.99:0/2065210296 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a68035020 con 0x7f9a740b2e60 2024-03-04T10:04:37.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a71ffb640 1 -- 172.21.15.99:0/2065210296 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a68031830 con 0x7f9a740b2e60 2024-03-04T10:04:37.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.797+0000 7f9a71ffb640 1 -- 172.21.15.99:0/2065210296 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a68031c00 con 0x7f9a740b2e60 2024-03-04T10:04:37.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 -- 172.21.15.99:0/2065210296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 msgr2=0x7f9a740b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 --2- 172.21.15.99:0/2065210296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a740b9380 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9a740b2d40 tx=0x7f9a6802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 -- 172.21.15.99:0/2065210296 shutdown_connections 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 --2- 172.21.15.99:0/2065210296 >> v2:172.21.15.162:3300/0 conn(0x7f9a740b5470 0x7f9a740b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 --2- 172.21.15.99:0/2065210296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a740b9380 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 -- 172.21.15.99:0/2065210296 >> 172.21.15.99:0/2065210296 conn(0x7f9a740d0470 msgr2=0x7f9a740d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 -- 172.21.15.99:0/2065210296 shutdown_connections 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.798+0000 7f9a79545640 1 -- 172.21.15.99:0/2065210296 wait complete. 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.799+0000 7f9a79545640 1 Processor -- start 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.799+0000 7f9a79545640 1 -- start start 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a79545640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a741606a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a79545640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a740b5470 0x7f9a7414cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a79545640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a7415de00 0x7f9a740b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a79545640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d21d0 con 0x7f9a740b2e60 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a79545640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d2820 con 0x7f9a740b5470 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a79545640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740d24b0 con 0x7f9a7415de00 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a741606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a741606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35296/0 (socket says 172.21.15.99:35296) 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 -- 172.21.15.99:0/1188689591 learned_addr learned my addr 172.21.15.99:0/1188689591 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a727fc640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a740b5470 0x7f9a7414cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a737fe640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a7415de00 0x7f9a740b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 -- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a7415de00 msgr2=0x7f9a740b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a7415de00 0x7f9a740b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 -- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a740b5470 msgr2=0x7f9a7414cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a740b5470 0x7f9a7414cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a72ffd640 1 -- 172.21.15.99:0/1188689591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a74161240 con 0x7f9a740b2e60 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a727fc640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a740b5470 0x7f9a7414cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.800+0000 7f9a737fe640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a7415de00 0x7f9a740b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.801+0000 7f9a72ffd640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a741606a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400ca70 tx=0x7f9a6400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.801+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64017070 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.801+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6400abd0 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.801+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a74160c80 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.801+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64012400 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.801+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740c78a0 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.802+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74161070 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.802+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9a6401a070 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.803+0000 7f9a5b7fe640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a6006b4d0 0x7f9a6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.803+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f9a6408a330 con 0x7f9a740b2e60 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.803+0000 7f9a727fc640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a6006b4d0 0x7f9a6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:37.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.803+0000 7f9a727fc640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a6006b4d0 0x7f9a6006d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9a68036000 tx=0x7f9a6803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:37.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.807+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9a64058dc0 con 0x7f9a740b2e60 2024-03-04T10:04:37.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.981+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a74074f60 con 0x7f9a740b2e60 2024-03-04T10:04:37.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.982+0000 7f9a5b7fe640 1 -- 172.21.15.99:0/1188689591 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7f9a640586c0 con 0x7f9a740b2e60 2024-03-04T10:04:37.985 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:37.985 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.984+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a6006b4d0 msgr2=0x7f9a6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.984+0000 7f9a79545640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a6006b4d0 0x7f9a6006d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9a68036000 tx=0x7f9a6803c040 comp rx=0 tx=0).stop 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.984+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 msgr2=0x7f9a741606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.984+0000 7f9a79545640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a741606a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400ca70 tx=0x7f9a6400cf30 comp rx=0 tx=0).stop 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.984+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 shutdown_connections 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a6006b4d0 0x7f9a6006d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a7415de00 0x7f9a740b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a740b5470 0x7f9a7414cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 --2- 172.21.15.99:0/1188689591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a740b2e60 0x7f9a741606a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 >> 172.21.15.99:0/1188689591 conn(0x7f9a740d0470 msgr2=0x7f9a740b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 shutdown_connections 2024-03-04T10:04:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:37.985+0000 7f9a79545640 1 -- 172.21.15.99:0/1188689591 wait complete. 2024-03-04T10:04:37.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:03:37.439702+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:03:37.840772+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T10:03:41.564149+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T10:04:37.998 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.099+0000 7f2114f90640 1 Processor -- start 2024-03-04T10:04:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.099+0000 7f2114f90640 1 -- start start 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2114f90640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f211015da20 0x7f21100ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2114f90640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21100cf660 0x7f21100cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2114f90640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f21101616e0 0x7f2110161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2114f90640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21100d0000 con 0x7f211015da20 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2114f90640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21100cc920 con 0x7f21101616e0 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2114f90640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2110162450 con 0x7f21100cf660 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f211015da20 0x7f21100ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f211015da20 0x7f21100ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35302/0 (socket says 172.21.15.99:35302) 2024-03-04T10:04:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 -- 172.21.15.99:0/1304810290 learned_addr learned my addr 172.21.15.99:0/1304810290 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210ed9d640 1 --2- 172.21.15.99:0/1304810290 >> v2:172.21.15.162:3300/0 conn(0x7f21101616e0 0x7f2110161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 -- 172.21.15.99:0/1304810290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21100cf660 msgr2=0x7f21100cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2107fff640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21100cf660 0x7f21100cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21100cf660 0x7f21100cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 -- 172.21.15.99:0/1304810290 >> v2:172.21.15.162:3300/0 conn(0x7f21101616e0 msgr2=0x7f2110161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 --2- 172.21.15.99:0/1304810290 >> v2:172.21.15.162:3300/0 conn(0x7f21101616e0 0x7f2110161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210e59c640 1 -- 172.21.15.99:0/1304810290 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21100cc2d0 con 0x7f211015da20 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f210ed9d640 1 --2- 172.21.15.99:0/1304810290 >> v2:172.21.15.162:3300/0 conn(0x7f21101616e0 0x7f2110161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.100+0000 7f2107fff640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21100cf660 0x7f21100cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.101+0000 7f210e59c640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f21100ccdb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f21100d1340 tx=0x7f210002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56df95a88637dbea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.101+0000 7f210dd9b640 1 -- 172.21.15.99:0/1304810290 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100035020 con 0x7f211015da20 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.101+0000 7f210dd9b640 1 -- 172.21.15.99:0/1304810290 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210002d040 con 0x7f211015da20 2024-03-04T10:04:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.101+0000 7f210dd9b640 1 -- 172.21.15.99:0/1304810290 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100031a80 con 0x7f211015da20 2024-03-04T10:04:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.101+0000 7f2114f90640 1 -- 172.21.15.99:0/1304810290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 msgr2=0x7f21100ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.101+0000 7f2114f90640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f21100ccdb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f21100d1340 tx=0x7f210002b900 comp rx=0 tx=0).stop 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 -- 172.21.15.99:0/1304810290 shutdown_connections 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 --2- 172.21.15.99:0/1304810290 >> v2:172.21.15.162:3300/0 conn(0x7f21101616e0 0x7f2110161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21100cf660 0x7f21100cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 --2- 172.21.15.99:0/1304810290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f21100ccdb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 -- 172.21.15.99:0/1304810290 >> 172.21.15.99:0/1304810290 conn(0x7f21100b33a0 msgr2=0x7f21100b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 -- 172.21.15.99:0/1304810290 shutdown_connections 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.102+0000 7f2114f90640 1 -- 172.21.15.99:0/1304810290 wait complete. 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.103+0000 7f2114f90640 1 Processor -- start 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.103+0000 7f2114f90640 1 -- start start 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21100cf660 0x7f2110151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f2110155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21101616e0 0x7f21100c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21100cbf80 con 0x7f211015da20 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21100cd2f0 con 0x7f21100cf660 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211015de00 con 0x7f21101616e0 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f2110155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f2110155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35310/0 (socket says 172.21.15.99:35310) 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 -- 172.21.15.99:0/2244577425 learned_addr learned my addr 172.21.15.99:0/2244577425 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f210ed9d640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21101616e0 0x7f21100c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f210e59c640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21100cf660 0x7f2110151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 -- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21101616e0 msgr2=0x7f21100c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21101616e0 0x7f21100c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 -- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21100cf660 msgr2=0x7f2110151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21100cf660 0x7f2110151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 -- 172.21.15.99:0/2244577425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21100cb780 con 0x7f211015da20 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f210e59c640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21100cf660 0x7f2110151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2107fff640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f2110155700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f20f800c7f0 tx=0x7f20f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8015070 con 0x7f211015da20 2024-03-04T10:04:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f8007a30 con 0x7f211015da20 2024-03-04T10:04:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2110153c00 con 0x7f211015da20 2024-03-04T10:04:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8007e00 con 0x7f211015da20 2024-03-04T10:04:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.104+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21100c8960 con 0x7f211015da20 2024-03-04T10:04:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.105+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2110074f60 con 0x7f211015da20 2024-03-04T10:04:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.106+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f20f8018070 con 0x7f211015da20 2024-03-04T10:04:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.106+0000 7f2106ffd640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.106+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f20f808a5a0 con 0x7f211015da20 2024-03-04T10:04:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.106+0000 7f210e59c640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.107+0000 7f210e59c640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2100036000 tx=0x7f2100008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.110+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f20f8059030 con 0x7f211015da20 2024-03-04T10:04:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.281+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f211015cf70 con 0x7f211015da20 2024-03-04T10:04:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.281+0000 7f2106ffd640 1 -- 172.21.15.99:0/2244577425 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20f8058930 con 0x7f211015da20 2024-03-04T10:04:38.283 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 msgr2=0x7f20f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2100036000 tx=0x7f2100008030 comp rx=0 tx=0).stop 2024-03-04T10:04:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 msgr2=0x7f2110155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f2110155700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f20f800c7f0 tx=0x7f20f800cbf0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 shutdown_connections 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f21101616e0 0x7f21100c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f211015da20 0x7f2110155700 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 --2- 172.21.15.99:0/2244577425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21100cf660 0x7f2110151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 >> 172.21.15.99:0/2244577425 conn(0x7f21100b33a0 msgr2=0x7f21100cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 shutdown_connections 2024-03-04T10:04:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.283+0000 7f2114f90640 1 -- 172.21.15.99:0/2244577425 wait complete. 2024-03-04T10:04:38.296 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.397+0000 7fecb2da2640 1 Processor -- start 2024-03-04T10:04:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.397+0000 7fecb2da2640 1 -- start start 2024-03-04T10:04:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb2da2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fecac0d33a0 0x7fecac0d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb2da2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac0b79e0 con 0x7fecac0d02b0 2024-03-04T10:04:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb2da2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecac0d2300 con 0x7fecac15dc90 2024-03-04T10:04:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb2da2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecac0d25e0 con 0x7fecac0d33a0 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb1da0640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fecac0d02b0 0x7fecac0d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34098/0 (socket says 172.21.15.99:34098) 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb1da0640 1 -- 172.21.15.99:0/1951501865 learned_addr learned my addr 172.21.15.99:0/1951501865 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022622470 0 0) 0x7fecac0b79e0 con 0x7fecac0d02b0 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb25a1640 1 --2- 172.21.15.99:0/1951501865 >> v2:172.21.15.162:3300/0 conn(0x7fecac0d33a0 0x7fecac0d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.398+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec98003520 con 0x7fecac0d02b0 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280750652 0 0) 0x7fec98003520 con 0x7fecac0d02b0 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 >> v2:172.21.15.162:3300/0 conn(0x7fecac0d33a0 msgr2=0x7fecac0d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 --2- 172.21.15.99:0/1951501865 >> v2:172.21.15.162:3300/0 conn(0x7fecac0d33a0 0x7fecac0d1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 >> v1:172.21.15.162:6789/0 conn(0x7fecac15dc90 legacy=0x7fecac0b92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0d28c0 con 0x7fecac0d02b0 2024-03-04T10:04:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7feca8001670 con 0x7fecac0d02b0 2024-03-04T10:04:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 >> v1:172.21.15.99:6789/0 conn(0x7fecac0d02b0 legacy=0x7fecac0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec980039e0 0x7fec98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec98004800 0x7fec98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec980051c0 0x7fec9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0d25e0 con 0x7fec98004800 2024-03-04T10:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9800c090 con 0x7fec980039e0 2024-03-04T10:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9800c290 con 0x7fec980051c0 2024-03-04T10:04:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb159f640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec98004800 0x7fec98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb1da0640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec980039e0 0x7fec98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb25a1640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec980051c0 0x7fec9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb159f640 1 -- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec980051c0 msgr2=0x7fec9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb159f640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec980051c0 0x7fec9800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb159f640 1 -- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec980039e0 msgr2=0x7fec98003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb159f640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec980039e0 0x7fec98003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb159f640 1 -- 172.21.15.99:0/1951501865 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec98003520 con 0x7fec98004800 2024-03-04T10:04:38.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb25a1640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec980051c0 0x7fec9800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:38.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.399+0000 7fecb25a1640 1 -- 172.21.15.99:0/1951501865 reap_dead start 2024-03-04T10:04:38.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.400+0000 7fecb159f640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec98004800 0x7fec98004c80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fec9c001750 tx=0x7fec98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45114eca525371d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.400+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c001370 con 0x7fec98004800 2024-03-04T10:04:38.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.400+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c0052d0 con 0x7fec98004800 2024-03-04T10:04:38.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.400+0000 7fecb0d9e640 1 -- 172.21.15.99:0/1951501865 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c003830 con 0x7fec98004800 2024-03-04T10:04:38.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.400+0000 7fecb2da2640 1 -- 172.21.15.99:0/1951501865 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec98004800 msgr2=0x7fec98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.400+0000 7fecb2da2640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec98004800 0x7fec98004c80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fec9c001750 tx=0x7fec98005830 comp rx=0 tx=0).stop 2024-03-04T10:04:38.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.401+0000 7fecb2da2640 1 -- 172.21.15.99:0/1951501865 shutdown_connections 2024-03-04T10:04:38.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.401+0000 7fecb2da2640 1 --2- 172.21.15.99:0/1951501865 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec98004800 0x7fec98004c80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.401+0000 7fecb2da2640 1 -- 172.21.15.99:0/1951501865 >> 172.21.15.99:0/1951501865 conn(0x7fecac072400 msgr2=0x7fecac0727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:38.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.401+0000 7fecb2da2640 1 -- 172.21.15.99:0/1951501865 shutdown_connections 2024-03-04T10:04:38.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.401+0000 7fecb2da2640 1 -- 172.21.15.99:0/1951501865 wait complete. 2024-03-04T10:04:38.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 Processor -- start 2024-03-04T10:04:38.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 -- start start 2024-03-04T10:04:38.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec98004800 0x7fecac0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 0x7fecac1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fecac0b5f50 0x7fecac154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0d1df0 con 0x7fecac0b5ad0 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0d1c70 con 0x7fec98004800 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.402+0000 7fecb2da2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecac0d1f70 con 0x7fecac0b5f50 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 0x7fecac1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 0x7fecac1541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35334/0 (socket says 172.21.15.99:35334) 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 -- 172.21.15.99:0/2530371366 learned_addr learned my addr 172.21.15.99:0/2530371366 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb25a1640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fecac0b5f50 0x7fecac154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb1da0640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec98004800 0x7fecac0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 -- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fecac0b5f50 msgr2=0x7fecac154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fecac0b5f50 0x7fecac154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 -- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec98004800 msgr2=0x7fecac0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec98004800 0x7fecac0c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 -- 172.21.15.99:0/2530371366 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac0c9cf0 con 0x7fecac0b5ad0 2024-03-04T10:04:38.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb1da0640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec98004800 0x7fecac0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:38.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb159f640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 0x7fecac1541d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000c00 tx=0x7fec9c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c041020 con 0x7fecac0b5ad0 2024-03-04T10:04:38.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c005480 con 0x7fecac0b5ad0 2024-03-04T10:04:38.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac0b4a30 con 0x7fecac0b5ad0 2024-03-04T10:04:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c007a30 con 0x7fecac0b5ad0 2024-03-04T10:04:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.403+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac0b4ff0 con 0x7fecac0b5ad0 2024-03-04T10:04:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.404+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecac04f090 con 0x7fecac0b5ad0 2024-03-04T10:04:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.405+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fec9c044020 con 0x7fecac0b5ad0 2024-03-04T10:04:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.405+0000 7feca27fc640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec9006b500 0x7fec9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.405+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fec9c0abf30 con 0x7fecac0b5ad0 2024-03-04T10:04:38.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.405+0000 7fecb1da0640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec9006b500 0x7fec9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.406+0000 7fecb1da0640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec9006b500 0x7fec9006d9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fecac150660 tx=0x7feca8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.409+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fec9c04b030 con 0x7fecac0b5ad0 2024-03-04T10:04:38.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.580+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fecac15bc20 con 0x7fecac0b5ad0 2024-03-04T10:04:38.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.581+0000 7feca27fc640 1 -- 172.21.15.99:0/2530371366 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec9c07a930 con 0x7fecac0b5ad0 2024-03-04T10:04:38.583 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:38.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.582+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec9006b500 msgr2=0x7fec9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.582+0000 7fecb2da2640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec9006b500 0x7fec9006d9c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fecac150660 tx=0x7feca8009040 comp rx=0 tx=0).stop 2024-03-04T10:04:38.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.582+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 msgr2=0x7fecac1541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.582+0000 7fecb2da2640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 0x7fecac1541d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fec9c000c00 tx=0x7fec9c001120 comp rx=0 tx=0).stop 2024-03-04T10:04:38.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 shutdown_connections 2024-03-04T10:04:38.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec9006b500 0x7fec9006d9c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fecac0b5f50 0x7fecac154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fecac0b5ad0 0x7fecac1541d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 --2- 172.21.15.99:0/2530371366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec98004800 0x7fecac0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 >> 172.21.15.99:0/2530371366 conn(0x7fecac072400 msgr2=0x7fecac0cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:38.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 shutdown_connections 2024-03-04T10:04:38.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.583+0000 7fecb2da2640 1 -- 172.21.15.99:0/2530371366 wait complete. 2024-03-04T10:04:38.596 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:38.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.696+0000 7f2fd4020640 1 Processor -- start 2024-03-04T10:04:38.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.697+0000 7f2fd4020640 1 -- start start 2024-03-04T10:04:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.697+0000 7f2fd4020640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc15a810 0x7f2fcc0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.697+0000 7f2fd4020640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2fcc06d3c0 0x7f2fcc0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.697+0000 7f2fd4020640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc06d8a0 con 0x7f2fcc0cbfb0 2024-03-04T10:04:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.697+0000 7f2fd4020640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc06db80 con 0x7f2fcc06d3c0 2024-03-04T10:04:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.697+0000 7f2fd4020640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc06de60 con 0x7f2fcc15a810 2024-03-04T10:04:38.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd15bb640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2fcc06d3c0 0x7f2fcc0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd25bd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2fcc0cbfb0 0x7f2fcc15abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34102/0 (socket says 172.21.15.99:34102) 2024-03-04T10:04:38.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd25bd640 1 -- 172.21.15.99:0/10100648 learned_addr learned my addr 172.21.15.99:0/10100648 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:38.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd1dbc640 1 --2- 172.21.15.99:0/10100648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc15a810 0x7f2fcc0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd15bb640 1 -- 172.21.15.99:0/10100648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc15a810 msgr2=0x7f2fcc0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd15bb640 1 --2- 172.21.15.99:0/10100648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc15a810 0x7f2fcc0b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd15bb640 1 -- 172.21.15.99:0/10100648 >> v1:172.21.15.99:6789/0 conn(0x7f2fcc0cbfb0 legacy=0x7f2fcc15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd15bb640 1 -- 172.21.15.99:0/10100648 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc06e140 con 0x7f2fcc06d3c0 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.698+0000 7f2fd0dba640 1 -- 172.21.15.99:0/10100648 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473453232 0 0) 0x7f2fcc06d8a0 con 0x7f2fcc0cbfb0 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd15bb640 1 --2- 172.21.15.99:0/10100648 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc0b46d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0097a0 tx=0x7f2fbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd1c5e9d6ed5740 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd0dba640 1 -- 172.21.15.99:0/10100648 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc035020 con 0x7f2fcc06d3c0 2024-03-04T10:04:38.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd0dba640 1 -- 172.21.15.99:0/10100648 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc031830 con 0x7f2fcc06d3c0 2024-03-04T10:04:38.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd0dba640 1 -- 172.21.15.99:0/10100648 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc031c00 con 0x7f2fcc06d3c0 2024-03-04T10:04:38.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 -- 172.21.15.99:0/10100648 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc06d3c0 msgr2=0x7f2fcc0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 --2- 172.21.15.99:0/10100648 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc0b46d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc0097a0 tx=0x7f2fbc02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 -- 172.21.15.99:0/10100648 shutdown_connections 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 --2- 172.21.15.99:0/10100648 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc0b46d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 --2- 172.21.15.99:0/10100648 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc15a810 0x7f2fcc0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 -- 172.21.15.99:0/10100648 >> 172.21.15.99:0/10100648 conn(0x7f2fcc06acd0 msgr2=0x7f2fcc06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 -- 172.21.15.99:0/10100648 shutdown_connections 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.699+0000 7f2fd4020640 1 -- 172.21.15.99:0/10100648 wait complete. 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 Processor -- start 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 -- start start 2024-03-04T10:04:38.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc13c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0cbfb0 0x7f2fcc15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 0x7f2fcc14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15e3c0 con 0x7f2fcc15a810 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b3c10 con 0x7f2fcc06d3c0 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd4020640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15e540 con 0x7f2fcc0cbfb0 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd25bd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 0x7f2fcc14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd25bd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 0x7f2fcc14eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35348/0 (socket says 172.21.15.99:35348) 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.701+0000 7f2fd25bd640 1 -- 172.21.15.99:0/1831345462 learned_addr learned my addr 172.21.15.99:0/1831345462 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd15bb640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0cbfb0 0x7f2fcc15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd25bd640 1 -- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0cbfb0 msgr2=0x7f2fcc15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd1dbc640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc13c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd25bd640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0cbfb0 0x7f2fcc15abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd25bd640 1 -- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc06d3c0 msgr2=0x7f2fcc13c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd25bd640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc13c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd25bd640 1 -- 172.21.15.99:0/1831345462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc14d850 con 0x7f2fcc15a810 2024-03-04T10:04:38.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd25bd640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 0x7f2fcc14eec0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c7d0 tx=0x7f2fc400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4018070 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc400a640 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc14e3e0 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400aa70 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.702+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc14db30 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.703+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc074f60 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.703+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2fc401b070 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.704+0000 7f2fc27fc640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.704+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f2fc4089ca0 con 0x7f2fcc15a810 2024-03-04T10:04:38.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.704+0000 7f2fd1dbc640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:38.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.704+0000 7f2fd1dbc640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800b7c0 tx=0x7f2fc8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:38.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.708+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2fc40586a0 con 0x7f2fcc15a810 2024-03-04T10:04:38.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.878+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2fcc06ec30 con 0x7f2fcc15a810 2024-03-04T10:04:38.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.878+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1831345462 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2fc4057fa0 con 0x7f2fcc15a810 2024-03-04T10:04:38.881 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 msgr2=0x7f2fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800b7c0 tx=0x7f2fc8008040 comp rx=0 tx=0).stop 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 msgr2=0x7f2fcc14eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 0x7f2fcc14eec0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2fc400c7d0 tx=0x7f2fc400cc90 comp rx=0 tx=0).stop 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 shutdown_connections 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fcc15a810 0x7f2fcc14eec0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fcc0cbfb0 0x7f2fcc15abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 --2- 172.21.15.99:0/1831345462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fcc06d3c0 0x7f2fcc13c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 >> 172.21.15.99:0/1831345462 conn(0x7f2fcc06acd0 msgr2=0x7f2fcc0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 shutdown_connections 2024-03-04T10:04:38.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.880+0000 7f2fd4020640 1 -- 172.21.15.99:0/1831345462 wait complete. 2024-03-04T10:04:38.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.995+0000 7f88762f8640 1 Processor -- start 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.995+0000 7f88762f8640 1 -- start start 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f88762f8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 0x7f88700b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f88762f8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f887015f660 0x7f88700d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f88762f8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887015bf50 con 0x7f88700d1cc0 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f88762f8640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88700cf890 con 0x7f887015f660 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f88762f8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887015fb60 con 0x7f887015bb70 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f8874895640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f88700d1cc0 0x7f88700ce1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34116/0 (socket says 172.21.15.99:34116) 2024-03-04T10:04:39.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f8874895640 1 -- 172.21.15.99:0/2670218243 learned_addr learned my addr 172.21.15.99:0/2670218243 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886f7fe640 1 --2- 172.21.15.99:0/2670218243 >> v2:172.21.15.162:3300/0 conn(0x7f887015f660 0x7f88700d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761517848 0 0) 0x7f887015bf50 con 0x7f88700d1cc0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886ffff640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 0x7f88700b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8858003520 con 0x7f88700d1cc0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1823032046 0 0) 0x7f8858003520 con 0x7f88700d1cc0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 msgr2=0x7f88700b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 0x7f88700b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 >> v2:172.21.15.162:3300/0 conn(0x7f887015f660 msgr2=0x7f88700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.996+0000 7f886effd640 1 --2- 172.21.15.99:0/2670218243 >> v2:172.21.15.162:3300/0 conn(0x7f887015f660 0x7f88700d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887015fe40 con 0x7f88700d1cc0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f88680019f0 con 0x7f88700d1cc0 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 >> v1:172.21.15.99:6789/0 conn(0x7f88700d1cc0 legacy=0x7f88700ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88580039b0 0x7f8858003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 0x7f8858004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8858004d40 0x7f88580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700cf890 con 0x7f88580039b0 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887015fb60 con 0x7f8858004d40 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885800b9c0 con 0x7f887015bb70 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886ffff640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88580039b0 0x7f8858003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886f7fe640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 0x7f8858004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f8874895640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8858004d40 0x7f88580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886ffff640 1 -- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 msgr2=0x7f8858004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886ffff640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f887015bb70 0x7f8858004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886ffff640 1 -- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8858004d40 msgr2=0x7f88580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886ffff640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8858004d40 0x7f88580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f886ffff640 1 -- 172.21.15.99:0/2670218243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8858003520 con 0x7f88580039b0 2024-03-04T10:04:39.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.997+0000 7f8874895640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8858004d40 0x7f88580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f886ffff640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88580039b0 0x7f8858003dd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8860009ac0 tx=0x7f8858005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a014b1f75d0f87ca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860039070 con 0x7f88580039b0 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886002c6b0 con 0x7f88580039b0 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f886effd640 1 -- 172.21.15.99:0/2670218243 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860009690 con 0x7f88580039b0 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f88762f8640 1 -- 172.21.15.99:0/2670218243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88580039b0 msgr2=0x7f8858003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f88762f8640 1 --2- 172.21.15.99:0/2670218243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88580039b0 0x7f8858003dd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8860009ac0 tx=0x7f8858005830 comp rx=0 tx=0).stop 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f8874895640 1 -- 172.21.15.99:0/2670218243 reap_dead start 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f88762f8640 1 -- 172.21.15.99:0/2670218243 shutdown_connections 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f88762f8640 1 -- 172.21.15.99:0/2670218243 >> 172.21.15.99:0/2670218243 conn(0x7f88700b33a0 msgr2=0x7f88700b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.998+0000 7f88762f8640 1 -- 172.21.15.99:0/2670218243 shutdown_connections 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:38.999+0000 7f88762f8640 1 -- 172.21.15.99:0/2670218243 wait complete. 2024-03-04T10:04:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 Processor -- start 2024-03-04T10:04:39.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 -- start start 2024-03-04T10:04:39.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88580039b0 0x7f88700c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88700b6f20 0x7f88700c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 0x7f887015df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700d0d30 con 0x7f88700b5120 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8870160400 con 0x7f88700b6f20 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.000+0000 7f88762f8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88700d0eb0 con 0x7f88580039b0 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 0x7f887015df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 0x7f887015df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35368/0 (socket says 172.21.15.99:35368) 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 -- 172.21.15.99:0/691188841 learned_addr learned my addr 172.21.15.99:0/691188841 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 -- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88580039b0 msgr2=0x7f88700c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88580039b0 0x7f88700c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 -- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88700b6f20 msgr2=0x7f88700c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88700b6f20 0x7f88700c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 -- 172.21.15.99:0/691188841 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88700b4960 con 0x7f88700b5120 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f8874895640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 0x7f887015df50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88700b74a0 tx=0x7f886800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886800eb90 con 0x7f88700b5120 2024-03-04T10:04:39.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886800ed30 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700c71f0 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868014650 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.001+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700b4c00 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.002+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870074f60 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.003+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f88680147f0 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.003+0000 7f886cff9640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.003+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f886808a3a0 con 0x7f88700b5120 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.003+0000 7f886ffff640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.004+0000 7f886ffff640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f885c06b4d0 0x7f885c06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8860036000 tx=0x7f886002e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.007+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8868058ec0 con 0x7f88700b5120 2024-03-04T10:04:39.049 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:39.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:04:39.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.182+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f88701572e0 con 0x7f88700b5120 2024-03-04T10:04:39.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.182+0000 7f886cff9640 1 -- 172.21.15.99:0/691188841 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f88680587c0 con 0x7f88700b5120 2024-03-04T10:04:39.184 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:39.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f885c06b4d0 msgr2=0x7f885c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f885c06b4d0 0x7f885c06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8860036000 tx=0x7f886002e000 comp rx=0 tx=0).stop 2024-03-04T10:04:39.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 msgr2=0x7f887015df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 0x7f887015df50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f88700b74a0 tx=0x7f886800e530 comp rx=0 tx=0).stop 2024-03-04T10:04:39.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 shutdown_connections 2024-03-04T10:04:39.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f885c06b4d0 0x7f885c06d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f88700b5120 0x7f887015df50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f88700b6f20 0x7f88700c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 --2- 172.21.15.99:0/691188841 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f88580039b0 0x7f88700c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 >> 172.21.15.99:0/691188841 conn(0x7f88700b33a0 msgr2=0x7f887015d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 shutdown_connections 2024-03-04T10:04:39.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.184+0000 7f88762f8640 1 -- 172.21.15.99:0/691188841 wait complete. 2024-03-04T10:04:39.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:39.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.295+0000 7fad33485640 1 Processor -- start 2024-03-04T10:04:39.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.295+0000 7fad33485640 1 -- start start 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad33485640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fad2c15de00 0x7fad2c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad33485640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c15e1e0 con 0x7fad2c0b5470 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad33485640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2c0d1c10 con 0x7fad2c0b9380 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad33485640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad2c0d2820 con 0x7fad2c15de00 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad31221640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fad2c15de00 0x7fad2c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad31a22640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fad2c0b5470 0x7fad2c0d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34128/0 (socket says 172.21.15.99:34128) 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.296+0000 7fad31a22640 1 -- 172.21.15.99:0/102215473 learned_addr learned my addr 172.21.15.99:0/102215473 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.297+0000 7fad31221640 1 -- 172.21.15.99:0/102215473 >> v1:172.21.15.162:6789/0 conn(0x7fad2c0b9380 legacy=0x7fad2c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.297+0000 7fad31221640 1 -- 172.21.15.99:0/102215473 >> v1:172.21.15.99:6789/0 conn(0x7fad2c0b5470 legacy=0x7fad2c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.297+0000 7fad31221640 1 -- 172.21.15.99:0/102215473 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c0d1ef0 con 0x7fad2c15de00 2024-03-04T10:04:39.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.297+0000 7fad23fff640 1 -- 172.21.15.99:0/102215473 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071796998 0 0) 0x7fad2c15e1e0 con 0x7fad2c0b5470 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.297+0000 7fad31221640 1 --2- 172.21.15.99:0/102215473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c15de00 0x7fad2c0b2e60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fad28009d90 tx=0x7fad2802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb86318800264a91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad23fff640 1 -- 172.21.15.99:0/102215473 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28039070 con 0x7fad2c15de00 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad23fff640 1 -- 172.21.15.99:0/102215473 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2802cc60 con 0x7fad2c15de00 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad23fff640 1 -- 172.21.15.99:0/102215473 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28034660 con 0x7fad2c15de00 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 -- 172.21.15.99:0/102215473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c15de00 msgr2=0x7fad2c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 --2- 172.21.15.99:0/102215473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c15de00 0x7fad2c0b2e60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fad28009d90 tx=0x7fad2802beb0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 -- 172.21.15.99:0/102215473 shutdown_connections 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 --2- 172.21.15.99:0/102215473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c15de00 0x7fad2c0b2e60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 -- 172.21.15.99:0/102215473 >> 172.21.15.99:0/102215473 conn(0x7fad2c0d0470 msgr2=0x7fad2c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 -- 172.21.15.99:0/102215473 shutdown_connections 2024-03-04T10:04:39.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.298+0000 7fad33485640 1 -- 172.21.15.99:0/102215473 wait complete. 2024-03-04T10:04:39.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.299+0000 7fad33485640 1 Processor -- start 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.299+0000 7fad33485640 1 -- start start 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad33485640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad2c0b5470 0x7fad2c1f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad33485640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 0x7fad2c15a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad33485640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad2c15de00 0x7fad2c14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad33485640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0b97e0 con 0x7fad2c0b9380 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad33485640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0d24b0 con 0x7fad2c0b5470 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad33485640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0b72a0 con 0x7fad2c15de00 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 0x7fad2c15a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 0x7fad2c15a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35378/0 (socket says 172.21.15.99:35378) 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 -- 172.21.15.99:0/3932060197 learned_addr learned my addr 172.21.15.99:0/3932060197 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad31221640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad2c0b5470 0x7fad2c1f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad31a22640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad2c15de00 0x7fad2c14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 -- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad2c15de00 msgr2=0x7fad2c14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad2c15de00 0x7fad2c14dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 -- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad2c0b5470 msgr2=0x7fad2c1f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.300+0000 7fad30a20640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad2c0b5470 0x7fad2c1f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad30a20640 1 -- 172.21.15.99:0/3932060197 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c15ab00 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad30a20640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 0x7fad2c15a5c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c430 tx=0x7fad1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c011790 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c017070 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2c1f4210 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c011bb0 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.301+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2c1f47d0 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.302+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2c0cbdc0 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.302+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fad1c01c460 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.303+0000 7fad21ffb640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad1006b4d0 0x7fad1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.303+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fad1c08a4b0 con 0x7fad2c0b9380 2024-03-04T10:04:39.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.303+0000 7fad31221640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad1006b4d0 0x7fad1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.303+0000 7fad31221640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad1006b4d0 0x7fad1006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad28036000 tx=0x7fad28030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.307+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fad1c058f40 con 0x7fad2c0b9380 2024-03-04T10:04:39.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.512+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad2c0b8f20 con 0x7fad2c0b9380 2024-03-04T10:04:39.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.513+0000 7fad21ffb640 1 -- 172.21.15.99:0/3932060197 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7fad2c0b8f20 con 0x7fad2c0b9380 2024-03-04T10:04:39.515 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:39.515 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:39.515 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad1006b4d0 msgr2=0x7fad1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad1006b4d0 0x7fad1006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fad28036000 tx=0x7fad28030000 comp rx=0 tx=0).stop 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 msgr2=0x7fad2c15a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 0x7fad2c15a5c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00c430 tx=0x7fad1c00c8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 shutdown_connections 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad1006b4d0 0x7fad1006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad2c15de00 0x7fad2c14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad2c0b9380 0x7fad2c15a5c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 --2- 172.21.15.99:0/3932060197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad2c0b5470 0x7fad2c1f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 >> 172.21.15.99:0/3932060197 conn(0x7fad2c0d0470 msgr2=0x7fad2c0737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 shutdown_connections 2024-03-04T10:04:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.515+0000 7fad33485640 1 -- 172.21.15.99:0/3932060197 wait complete. 2024-03-04T10:04:39.528 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:39.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.628+0000 7f448f99e640 1 Processor -- start 2024-03-04T10:04:39.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 -- start start 2024-03-04T10:04:39.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f448815de00 0x7f44880b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b9380 0x7f44880b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f44880b70d0 0x7f44880b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44880d2820 con 0x7f44880b70d0 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448815e1e0 con 0x7f448815de00 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.629+0000 7f448f99e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44880d1c10 con 0x7f44880b9380 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f44880b70d0 0x7f44880b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f44880b70d0 0x7f44880b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35388/0 (socket says 172.21.15.99:35388) 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 -- 172.21.15.99:0/603385634 learned_addr learned my addr 172.21.15.99:0/603385634 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448d73a640 1 --2- 172.21.15.99:0/603385634 >> v2:172.21.15.162:3300/0 conn(0x7f448815de00 0x7f44880b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448cf39640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b9380 0x7f44880b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 -- 172.21.15.99:0/603385634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b9380 msgr2=0x7f44880b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b9380 0x7f44880b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 -- 172.21.15.99:0/603385634 >> v2:172.21.15.162:3300/0 conn(0x7f448815de00 msgr2=0x7f44880b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 --2- 172.21.15.99:0/603385634 >> v2:172.21.15.162:3300/0 conn(0x7f448815de00 0x7f44880b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448df3b640 1 -- 172.21.15.99:0/603385634 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880d1ef0 con 0x7f44880b70d0 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448d73a640 1 --2- 172.21.15.99:0/603385634 >> v2:172.21.15.162:3300/0 conn(0x7f448815de00 0x7f44880b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.630+0000 7f448cf39640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b9380 0x7f44880b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448df3b640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44880b70d0 0x7f44880b5470 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f44880ce240 tx=0x7f448002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4cbd8c852deae39 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f447ffff640 1 -- 172.21.15.99:0/603385634 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448002ca70 con 0x7f44880b70d0 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f447ffff640 1 -- 172.21.15.99:0/603385634 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448002cc10 con 0x7f44880b70d0 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f447ffff640 1 -- 172.21.15.99:0/603385634 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800343f0 con 0x7f44880b70d0 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 -- 172.21.15.99:0/603385634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44880b70d0 msgr2=0x7f44880b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44880b70d0 0x7f44880b5470 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f44880ce240 tx=0x7f448002bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 -- 172.21.15.99:0/603385634 shutdown_connections 2024-03-04T10:04:39.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44880b70d0 0x7f44880b5470 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 --2- 172.21.15.99:0/603385634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b9380 0x7f44880b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 --2- 172.21.15.99:0/603385634 >> v2:172.21.15.162:3300/0 conn(0x7f448815de00 0x7f44880b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 -- 172.21.15.99:0/603385634 >> 172.21.15.99:0/603385634 conn(0x7f44880d0470 msgr2=0x7f44880d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 -- 172.21.15.99:0/603385634 shutdown_connections 2024-03-04T10:04:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.631+0000 7f448f99e640 1 -- 172.21.15.99:0/603385634 wait complete. 2024-03-04T10:04:39.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.632+0000 7f448f99e640 1 Processor -- start 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.632+0000 7f448f99e640 1 -- start start 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448f99e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b70d0 0x7f448815a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448f99e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44880b9380 0x7f448814dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448f99e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 0x7f44881f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448f99e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44880b97e0 con 0x7f448815de00 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448f99e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44880d24b0 con 0x7f44880b9380 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448f99e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44880b59b0 con 0x7f44880b70d0 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448df3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 0x7f44881f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448df3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 0x7f44881f3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35400/0 (socket says 172.21.15.99:35400) 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448df3b640 1 -- 172.21.15.99:0/2308042100 learned_addr learned my addr 172.21.15.99:0/2308042100 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448d73a640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b70d0 0x7f448815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.633+0000 7f448cf39640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44880b9380 0x7f448814dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448df3b640 1 -- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b70d0 msgr2=0x7f448815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448df3b640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b70d0 0x7f448815a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448df3b640 1 -- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44880b9380 msgr2=0x7f448814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448df3b640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44880b9380 0x7f448814dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448df3b640 1 -- 172.21.15.99:0/2308042100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880b3d90 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448df3b640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 0x7f44881f3840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4480009f40 tx=0x7f4480004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480043020 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448003e450 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44880b4070 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480009510 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.634+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44880b4630 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.635+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4488074f60 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.636+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4480046020 con 0x7f448815de00 2024-03-04T10:04:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.636+0000 7f447dffb640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f446c06b4d0 0x7f446c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.636+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f44800aced0 con 0x7f448815de00 2024-03-04T10:04:39.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.636+0000 7f448d73a640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f446c06b4d0 0x7f446c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.637+0000 7f448d73a640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f446c06b4d0 0x7f446c06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f447800a950 tx=0x7f4478008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.640+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f448004d030 con 0x7f448815de00 2024-03-04T10:04:39.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.816+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44880b6b50 con 0x7f448815de00 2024-03-04T10:04:39.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.817+0000 7f447dffb640 1 -- 172.21.15.99:0/2308042100 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7f448007b9f0 con 0x7f448815de00 2024-03-04T10:04:39.819 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:39.819 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:39.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f446c06b4d0 msgr2=0x7f446c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f446c06b4d0 0x7f446c06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f447800a950 tx=0x7f4478008040 comp rx=0 tx=0).stop 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 msgr2=0x7f44881f3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 0x7f44881f3840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4480009f40 tx=0x7f4480004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 shutdown_connections 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f446c06b4d0 0x7f446c06d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f448815de00 0x7f44881f3840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44880b9380 0x7f448814dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 --2- 172.21.15.99:0/2308042100 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44880b70d0 0x7f448815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 >> 172.21.15.99:0/2308042100 conn(0x7f44880d0470 msgr2=0x7f4488160d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 shutdown_connections 2024-03-04T10:04:39.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.819+0000 7f448f99e640 1 -- 172.21.15.99:0/2308042100 wait complete. 2024-03-04T10:04:39.832 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:04:39.833 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:39.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.932+0000 7fe58a7c4640 1 Processor -- start 2024-03-04T10:04:39.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.932+0000 7fe58a7c4640 1 -- start start 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe58a7c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe58415dea0 0x7fe5840cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe58a7c4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5840cc8d0 0x7fe58406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe58a7c4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58406fd20 con 0x7fe5840d1200 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe58a7c4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58406fad0 con 0x7fe5840cc8d0 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe58a7c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe584070000 con 0x7fe58415dea0 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe5837fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5840cc8d0 0x7fe58406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe588d61640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe5840d1200 0x7fe5840cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34138/0 (socket says 172.21.15.99:34138) 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe588d61640 1 -- 172.21.15.99:0/1937093660 learned_addr learned my addr 172.21.15.99:0/1937093660 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe583fff640 1 --2- 172.21.15.99:0/1937093660 >> v2:172.21.15.162:3300/0 conn(0x7fe58415dea0 0x7fe5840cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe5837fe640 1 -- 172.21.15.99:0/1937093660 >> v2:172.21.15.162:3300/0 conn(0x7fe58415dea0 msgr2=0x7fe5840cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe5837fe640 1 --2- 172.21.15.99:0/1937093660 >> v2:172.21.15.162:3300/0 conn(0x7fe58415dea0 0x7fe5840cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe5837fe640 1 -- 172.21.15.99:0/1937093660 >> v1:172.21.15.99:6789/0 conn(0x7fe5840d1200 legacy=0x7fe5840cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe582ffd640 1 -- 172.21.15.99:0/1937093660 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692252810 0 0) 0x7fe58406fd20 con 0x7fe5840d1200 2024-03-04T10:04:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.933+0000 7fe5837fe640 1 -- 172.21.15.99:0/1937093660 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5840702e0 con 0x7fe5840cc8d0 2024-03-04T10:04:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe5837fe640 1 --2- 172.21.15.99:0/1937093660 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe58406acd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ce200 tx=0x7fe57802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf229a1bf4dfb157 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe582ffd640 1 -- 172.21.15.99:0/1937093660 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578035020 con 0x7fe5840cc8d0 2024-03-04T10:04:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe582ffd640 1 -- 172.21.15.99:0/1937093660 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe578031830 con 0x7fe5840cc8d0 2024-03-04T10:04:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe582ffd640 1 -- 172.21.15.99:0/1937093660 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe578031c00 con 0x7fe5840cc8d0 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 -- 172.21.15.99:0/1937093660 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 msgr2=0x7fe58406acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/1937093660 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe58406acd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ce200 tx=0x7fe57802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 -- 172.21.15.99:0/1937093660 shutdown_connections 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/1937093660 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe58406acd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/1937093660 >> v2:172.21.15.162:3300/0 conn(0x7fe58415dea0 0x7fe5840cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 -- 172.21.15.99:0/1937093660 >> 172.21.15.99:0/1937093660 conn(0x7fe5840d0710 msgr2=0x7fe5840d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 -- 172.21.15.99:0/1937093660 shutdown_connections 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.934+0000 7fe58a7c4640 1 -- 172.21.15.99:0/1937093660 wait complete. 2024-03-04T10:04:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.936+0000 7fe58a7c4640 1 Processor -- start 2024-03-04T10:04:39.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.936+0000 7fe58a7c4640 1 -- start start 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.936+0000 7fe58a7c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe5840b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.936+0000 7fe58a7c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5840d1200 0x7fe584160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.936+0000 7fe58a7c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe58415dea0 0x7fe58414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe58a7c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58406b390 con 0x7fe5840cc8d0 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe58a7c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58406b210 con 0x7fe5840d1200 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe58a7c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58406b510 con 0x7fe58415dea0 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe5840b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe5840b1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35424/0 (socket says 172.21.15.99:35424) 2024-03-04T10:04:39.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 -- 172.21.15.99:0/2681189372 learned_addr learned my addr 172.21.15.99:0/2681189372 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 -- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe58415dea0 msgr2=0x7fe58414fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe58415dea0 0x7fe58414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 -- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5840d1200 msgr2=0x7fe584160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5840d1200 0x7fe584160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 -- 172.21.15.99:0/2681189372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5840ae340 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe583fff640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe5840b1400 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d830 tx=0x7fe57400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400b6c0 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe574019070 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57400ba80 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58415f980 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.937+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5840b1940 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.938+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58415f780 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.939+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe57401e460 con 0x7fe5840cc8d0 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.939+0000 7fe580ff9640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe57006b4d0 0x7fe57006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.939+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fe57408ac10 con 0x7fe5840cc8d0 2024-03-04T10:04:39.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.939+0000 7fe5837fe640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe57006b4d0 0x7fe57006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:39.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.940+0000 7fe5837fe640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe57006b4d0 0x7fe57006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe578036000 tx=0x7fe578030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:39.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:39.943+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe574059730 con 0x7fe5840cc8d0 2024-03-04T10:04:40.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.151+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe584074f60 con 0x7fe5840cc8d0 2024-03-04T10:04:40.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.152+0000 7fe580ff9640 1 -- 172.21.15.99:0/2681189372 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7fe574059030 con 0x7fe5840cc8d0 2024-03-04T10:04:40.154 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:40.154 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:40.155 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:40.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe57006b4d0 msgr2=0x7fe57006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe57006b4d0 0x7fe57006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe578036000 tx=0x7fe578030000 comp rx=0 tx=0).stop 2024-03-04T10:04:40.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 msgr2=0x7fe5840b1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe5840b1400 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe57400d830 tx=0x7fe57400dcf0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 shutdown_connections 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe57006b4d0 0x7fe57006d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe58415dea0 0x7fe58414fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5840d1200 0x7fe584160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 --2- 172.21.15.99:0/2681189372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5840cc8d0 0x7fe5840b1400 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 >> 172.21.15.99:0/2681189372 conn(0x7fe5840d0710 msgr2=0x7fe5840afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 shutdown_connections 2024-03-04T10:04:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.154+0000 7fe58a7c4640 1 -- 172.21.15.99:0/2681189372 wait complete. 2024-03-04T10:04:40.167 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:40.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 Processor -- start 2024-03-04T10:04:40.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 -- start start 2024-03-04T10:04:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0cb415bbb0 0x7f0cb40d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0cb40d0e40 0x7f0cb40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb40cfb40 con 0x7f0cb40cc4c0 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb40cc930 con 0x7f0cb415bbb0 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.265+0000 7f0cb9aea640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb40d2880 con 0x7f0cb40d0e40 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0cb415bbb0 0x7f0cb40d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0cb415bbb0 0x7f0cb40d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35434/0 (socket says 172.21.15.99:35434) 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 -- 172.21.15.99:0/586947274 learned_addr learned my addr 172.21.15.99:0/586947274 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb2ffd640 1 --2- 172.21.15.99:0/586947274 >> v2:172.21.15.162:3300/0 conn(0x7f0cb40d0e40 0x7f0cb40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 -- 172.21.15.99:0/586947274 >> v2:172.21.15.162:3300/0 conn(0x7f0cb40d0e40 msgr2=0x7f0cb40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 --2- 172.21.15.99:0/586947274 >> v2:172.21.15.162:3300/0 conn(0x7f0cb40d0e40 0x7f0cb40cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 -- 172.21.15.99:0/586947274 >> v1:172.21.15.162:6789/0 conn(0x7f0cb40cc4c0 legacy=0x7f0cb40d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb37fe640 1 -- 172.21.15.99:0/586947274 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb40cfe20 con 0x7f0cb415bbb0 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.266+0000 7f0cb2ffd640 1 --2- 172.21.15.99:0/586947274 >> v2:172.21.15.162:3300/0 conn(0x7f0cb40d0e40 0x7f0cb40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb37fe640 1 --2- 172.21.15.99:0/586947274 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40d0900 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4070790 tx=0x7f0ca402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321d2179e48dd1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb27fc640 1 -- 172.21.15.99:0/586947274 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4035020 con 0x7f0cb415bbb0 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb27fc640 1 -- 172.21.15.99:0/586947274 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca4031870 con 0x7f0cb415bbb0 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb27fc640 1 -- 172.21.15.99:0/586947274 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4031c40 con 0x7f0cb415bbb0 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 -- 172.21.15.99:0/586947274 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 msgr2=0x7f0cb40d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/586947274 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40d0900 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4070790 tx=0x7f0ca402b910 comp rx=0 tx=0).stop 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 -- 172.21.15.99:0/586947274 shutdown_connections 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/586947274 >> v2:172.21.15.162:3300/0 conn(0x7f0cb40d0e40 0x7f0cb40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/586947274 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40d0900 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 -- 172.21.15.99:0/586947274 >> 172.21.15.99:0/586947274 conn(0x7f0cb40b3480 msgr2=0x7f0cb40b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 -- 172.21.15.99:0/586947274 shutdown_connections 2024-03-04T10:04:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.267+0000 7f0cb9aea640 1 -- 172.21.15.99:0/586947274 wait complete. 2024-03-04T10:04:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.268+0000 7f0cb9aea640 1 Processor -- start 2024-03-04T10:04:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 -- start start 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb40cc4c0 0x7f0cb40c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40d0e40 0x7f0cb4159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb415bf90 con 0x7f0cb415bbb0 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb40d03e0 con 0x7f0cb40cc4c0 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb9aea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb41f4fb0 con 0x7f0cb40d0e40 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.269+0000 7f0cb3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40ab0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35436/0 (socket says 172.21.15.99:35436) 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 -- 172.21.15.99:0/1226333008 learned_addr learned my addr 172.21.15.99:0/1226333008 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb37fe640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb40cc4c0 0x7f0cb40c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb2ffd640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40d0e40 0x7f0cb4159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 -- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40d0e40 msgr2=0x7f0cb4159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40d0e40 0x7f0cb4159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 -- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb40cc4c0 msgr2=0x7f0cb40c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb40cc4c0 0x7f0cb40c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 -- 172.21.15.99:0/1226333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb41f5200 con 0x7f0cb415bbb0 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb37fe640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb40cc4c0 0x7f0cb40c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb2ffd640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40d0e40 0x7f0cb4159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb3fff640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40ab0c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c780 tx=0x7f0cac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac004b60 con 0x7f0cb415bbb0 2024-03-04T10:04:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac004d00 con 0x7f0cb415bbb0 2024-03-04T10:04:40.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb41f5540 con 0x7f0cb415bbb0 2024-03-04T10:04:40.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00aa90 con 0x7f0cb415bbb0 2024-03-04T10:04:40.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.270+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb40c8100 con 0x7f0cb415bbb0 2024-03-04T10:04:40.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.271+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb4074f60 con 0x7f0cb415bbb0 2024-03-04T10:04:40.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.272+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0cac00ac30 con 0x7f0cb415bbb0 2024-03-04T10:04:40.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.272+0000 7f0c9bfff640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0ca006b4d0 0x7f0ca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.272+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f0cac08a940 con 0x7f0cb415bbb0 2024-03-04T10:04:40.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.272+0000 7f0cb37fe640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0ca006b4d0 0x7f0ca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.273+0000 7f0cb37fe640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0ca006b4d0 0x7f0ca006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4036000 tx=0x7f0ca403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.276+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0cac020070 con 0x7f0cb415bbb0 2024-03-04T10:04:40.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.481+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb415a9c0 con 0x7f0cb415bbb0 2024-03-04T10:04:40.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.482+0000 7f0c9bfff640 1 -- 172.21.15.99:0/1226333008 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f0cb41f5540 con 0x7f0cb415bbb0 2024-03-04T10:04:40.484 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:40.484 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/521591758","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":521591758},{"type":"v1","addr":"172.21.15.162:6836","nonce":521591758}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4250,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3041842401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3041842401},{"type":"v1","addr":"172.21.15.162:6837","nonce":3041842401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/2535703965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":2535703965},{"type":"v1","addr":"172.21.15.99:6834","nonce":2535703965}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:03:38.859750+0000","modified":"2024-03-04T10:04:00.946856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/593641651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":593641651},{"type":"v1","addr":"172.21.15.99:6835","nonce":593641651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T10:04:40.484 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 8 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.483+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0ca006b4d0 msgr2=0x7f0ca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.483+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0ca006b4d0 0x7f0ca006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4036000 tx=0x7f0ca403c040 comp rx=0 tx=0).stop 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 msgr2=0x7f0cb40ab0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40ab0c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c780 tx=0x7f0cac00cc40 comp rx=0 tx=0).stop 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 shutdown_connections 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0ca006b4d0 0x7f0ca006d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb415bbb0 0x7f0cb40ab0c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40d0e40 0x7f0cb4159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 --2- 172.21.15.99:0/1226333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb40cc4c0 0x7f0cb40c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 >> 172.21.15.99:0/1226333008 conn(0x7f0cb40b3480 msgr2=0x7f0cb40d1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:40.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 shutdown_connections 2024-03-04T10:04:40.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.484+0000 7f0cb9aea640 1 -- 172.21.15.99:0/1226333008 wait complete. 2024-03-04T10:04:40.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:40.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.597+0000 7fdbb4e43640 1 Processor -- start 2024-03-04T10:04:40.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 -- start start 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdbb015de00 0x7fdbb00b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fdbb00b9380 0x7fdbb00b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb00b70d0 0x7fdbb00b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00d2820 con 0x7fdbb015de00 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb015e1e0 con 0x7fdbb00b9380 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbb4e43640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00d1c10 con 0x7fdbb00b70d0 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbae59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdbb015de00 0x7fdbb00b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.598+0000 7fdbae59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdbb015de00 0x7fdbb00b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35438/0 (socket says 172.21.15.99:35438) 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbae59c640 1 -- 172.21.15.99:0/613600397 learned_addr learned my addr 172.21.15.99:0/613600397 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbaed9d640 1 --2- 172.21.15.99:0/613600397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb00b70d0 0x7fdbb00b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdba7fff640 1 --2- 172.21.15.99:0/613600397 >> v2:172.21.15.162:3300/0 conn(0x7fdbb00b9380 0x7fdbb00b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbae59c640 1 -- 172.21.15.99:0/613600397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb00b70d0 msgr2=0x7fdbb00b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbae59c640 1 --2- 172.21.15.99:0/613600397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb00b70d0 0x7fdbb00b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbae59c640 1 -- 172.21.15.99:0/613600397 >> v2:172.21.15.162:3300/0 conn(0x7fdbb00b9380 msgr2=0x7fdbb00b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbae59c640 1 --2- 172.21.15.99:0/613600397 >> v2:172.21.15.162:3300/0 conn(0x7fdbb00b9380 0x7fdbb00b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdbae59c640 1 -- 172.21.15.99:0/613600397 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00d1ef0 con 0x7fdbb015de00 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.599+0000 7fdba7fff640 1 --2- 172.21.15.99:0/613600397 >> v2:172.21.15.162:3300/0 conn(0x7fdbb00b9380 0x7fdbb00b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbae59c640 1 --2- 172.21.15.99:0/613600397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb015de00 0x7fdbb00b2e60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0161830 tx=0x7fdba002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9082d3a53820fc78 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbadd9b640 1 -- 172.21.15.99:0/613600397 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0035020 con 0x7fdbb015de00 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbadd9b640 1 -- 172.21.15.99:0/613600397 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0031870 con 0x7fdbb015de00 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbadd9b640 1 -- 172.21.15.99:0/613600397 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0031c40 con 0x7fdbb015de00 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 -- 172.21.15.99:0/613600397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb015de00 msgr2=0x7fdbb00b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/613600397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb015de00 0x7fdbb00b2e60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0161830 tx=0x7fdba002b910 comp rx=0 tx=0).stop 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 -- 172.21.15.99:0/613600397 shutdown_connections 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/613600397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb00b70d0 0x7fdbb00b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/613600397 >> v2:172.21.15.162:3300/0 conn(0x7fdbb00b9380 0x7fdbb00b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/613600397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb015de00 0x7fdbb00b2e60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.600+0000 7fdbb4e43640 1 -- 172.21.15.99:0/613600397 >> 172.21.15.99:0/613600397 conn(0x7fdbb00d0470 msgr2=0x7fdbb00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:40.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.601+0000 7fdbb4e43640 1 -- 172.21.15.99:0/613600397 shutdown_connections 2024-03-04T10:04:40.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.601+0000 7fdbb4e43640 1 -- 172.21.15.99:0/613600397 wait complete. 2024-03-04T10:04:40.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.602+0000 7fdbb4e43640 1 Processor -- start 2024-03-04T10:04:40.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.602+0000 7fdbb4e43640 1 -- start start 2024-03-04T10:04:40.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 0x7fdbb014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbb00b9380 0x7fdbb0158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb015de00 0x7fdbb015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00b97e0 con 0x7fdbb00b70d0 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00d24b0 con 0x7fdbb00b9380 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00b59b0 con 0x7fdbb015de00 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 0x7fdbb014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 0x7fdbb014ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35450/0 (socket says 172.21.15.99:35450) 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 -- 172.21.15.99:0/1876336780 learned_addr learned my addr 172.21.15.99:0/1876336780 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdba7fff640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbb00b9380 0x7fdbb0158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbaed9d640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb015de00 0x7fdbb015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 -- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb015de00 msgr2=0x7fdbb015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb015de00 0x7fdbb015f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 -- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbb00b9380 msgr2=0x7fdbb0158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbb00b9380 0x7fdbb0158210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 -- 172.21.15.99:0/1876336780 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb0155260 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbae59c640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 0x7fdbb014ddc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdba002b950 tx=0x7fdba00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0040070 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0030d00 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0155540 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.603+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0035040 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.604+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb0155b00 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.604+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb0074f60 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.605+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdba004b060 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.605+0000 7fdba6ffd640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb9006b500 0x7fdb9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.605+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7fdba00ac1f0 con 0x7fdbb00b70d0 2024-03-04T10:04:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.606+0000 7fdba7fff640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb9006b500 0x7fdb9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.606+0000 7fdba7fff640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb9006b500 0x7fdb9006d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdbb006d1e0 tx=0x7fdb98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.610+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdba007a630 con 0x7fdbb00b70d0 2024-03-04T10:04:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.783+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbb00b7a80 con 0x7fdbb00b70d0 2024-03-04T10:04:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.784+0000 7fdba6ffd640 1 -- 172.21.15.99:0/1876336780 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18839 (secure 0 0 0) 0x7fdba007e6d0 con 0x7fdbb00b70d0 2024-03-04T10:04:40.787 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:40.787 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":26,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:20.662485+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:03:37.439702+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T10:03:37.840772+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T10:03:41.564149+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2},{"from":7,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb9006b500 msgr2=0x7fdb9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb9006b500 0x7fdb9006d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdbb006d1e0 tx=0x7fdb98008040 comp rx=0 tx=0).stop 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 msgr2=0x7fdbb014ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 0x7fdbb014ddc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdba002b950 tx=0x7fdba00046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 shutdown_connections 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb9006b500 0x7fdb9006d9c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdbb015de00 0x7fdbb015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdbb00b9380 0x7fdbb0158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 --2- 172.21.15.99:0/1876336780 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdbb00b70d0 0x7fdbb014ddc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.786+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 >> 172.21.15.99:0/1876336780 conn(0x7fdbb00d0470 msgr2=0x7fdbb00ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.787+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 shutdown_connections 2024-03-04T10:04:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.787+0000 7fdbb4e43640 1 -- 172.21.15.99:0/1876336780 wait complete. 2024-03-04T10:04:40.800 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:04:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.898+0000 7f7f955b3640 1 Processor -- start 2024-03-04T10:04:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.898+0000 7f7f955b3640 1 -- start start 2024-03-04T10:04:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f955b3640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7f9015de00 0x7f7f900b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f955b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b9380 0x7f7f900b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f955b3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9015e1e0 con 0x7f7f900b70d0 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f955b3640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f900d2820 con 0x7f7f9015de00 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f955b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900d1c10 con 0x7f7f900b9380 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8effd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7f9015de00 0x7f7f900b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8f7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7f900b70d0 0x7f7f900b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34154/0 (socket says 172.21.15.99:34154) 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8f7fe640 1 -- 172.21.15.99:0/2381452165 learned_addr learned my addr 172.21.15.99:0/2381452165 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8e7fc640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b9380 0x7f7f900b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8effd640 1 -- 172.21.15.99:0/2381452165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b9380 msgr2=0x7f7f900b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8effd640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b9380 0x7f7f900b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8effd640 1 -- 172.21.15.99:0/2381452165 >> v1:172.21.15.99:6789/0 conn(0x7f7f900b70d0 legacy=0x7f7f900b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8effd640 1 -- 172.21.15.99:0/2381452165 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f900d1ef0 con 0x7f7f9015de00 2024-03-04T10:04:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.899+0000 7f7f8e7fc640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b9380 0x7f7f900b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.900+0000 7f7f8effd640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f9015de00 0x7f7f900b2e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f800097a0 tx=0x7f7f8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb83d39b297256d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.900+0000 7f7f8dffb640 1 -- 172.21.15.99:0/2381452165 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80035020 con 0x7f7f9015de00 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.900+0000 7f7f8dffb640 1 -- 172.21.15.99:0/2381452165 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f80031830 con 0x7f7f9015de00 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.900+0000 7f7f8dffb640 1 -- 172.21.15.99:0/2381452165 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f80031c00 con 0x7f7f9015de00 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 -- 172.21.15.99:0/2381452165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f9015de00 msgr2=0x7f7f900b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f9015de00 0x7f7f900b2e60 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7f800097a0 tx=0x7f7f8002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 -- 172.21.15.99:0/2381452165 shutdown_connections 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b9380 0x7f7f900b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2381452165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f9015de00 0x7f7f900b2e60 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 -- 172.21.15.99:0/2381452165 >> 172.21.15.99:0/2381452165 conn(0x7f7f900d0470 msgr2=0x7f7f900d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 -- 172.21.15.99:0/2381452165 shutdown_connections 2024-03-04T10:04:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.901+0000 7f7f955b3640 1 -- 172.21.15.99:0/2381452165 wait complete. 2024-03-04T10:04:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.902+0000 7f7f955b3640 1 Processor -- start 2024-03-04T10:04:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.902+0000 7f7f955b3640 1 -- start start 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f955b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b70d0 0x7f7f9015e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f955b3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 0x7f7f900cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f955b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f9015de00 0x7f7f901595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f955b3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900b97e0 con 0x7f7f900b9380 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f955b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900d24b0 con 0x7f7f9015de00 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f955b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f900b5b80 con 0x7f7f900b70d0 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 0x7f7f900cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 0x7f7f900cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35464/0 (socket says 172.21.15.99:35464) 2024-03-04T10:04:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 -- 172.21.15.99:0/2443078716 learned_addr learned my addr 172.21.15.99:0/2443078716 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8effd640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b70d0 0x7f7f9015e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8f7fe640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f9015de00 0x7f7f901595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 -- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b70d0 msgr2=0x7f7f9015e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b70d0 0x7f7f9015e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 -- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f9015de00 msgr2=0x7f7f901595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f9015de00 0x7f7f901595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.903+0000 7f7f8e7fc640 1 -- 172.21.15.99:0/2443078716 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f900cba10 con 0x7f7f900b9380 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.904+0000 7f7f8e7fc640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 0x7f7f900cb4d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400c7f0 tx=0x7f7f8400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.904+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f84015070 con 0x7f7f900b9380 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.904+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f84007a30 con 0x7f7f900b9380 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.904+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9015e9d0 con 0x7f7f900b9380 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.904+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f84007e00 con 0x7f7f900b9380 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.904+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9015ef90 con 0x7f7f900b9380 2024-03-04T10:04:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.905+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f90074f60 con 0x7f7f900b9380 2024-03-04T10:04:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.905+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7f84018070 con 0x7f7f900b9380 2024-03-04T10:04:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.906+0000 7f7f777fe640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.906+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6714+0+0 (secure 0 0 0) 0x7f7f8408a4b0 con 0x7f7f900b9380 2024-03-04T10:04:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.906+0000 7f7f8effd640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.906+0000 7f7f8effd640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f80036000 tx=0x7f7f8003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:40.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:40.910+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7f840588f0 con 0x7f7f900b9380 2024-03-04T10:04:41.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.115+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7f900b8ee0 con 0x7f7f900b9380 2024-03-04T10:04:41.118 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:04:41.116+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:04:41.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.120+0000 7f7f777fe640 1 -- 172.21.15.99:0/2443078716 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7f8405c990 con 0x7f7f900b9380 2024-03-04T10:04:41.121 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.122+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7f7c06b4d0 msgr2=0x7f7f7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.122+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7f80036000 tx=0x7f7f8003c040 comp rx=0 tx=0).stop 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.122+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 msgr2=0x7f7f900cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.122+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 0x7f7f900cb4d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400c7f0 tx=0x7f7f8400cbf0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.122+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 shutdown_connections 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7f7c06b4d0 0x7f7f7c06d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7f9015de00 0x7f7f901595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7f900b9380 0x7f7f900cb4d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 --2- 172.21.15.99:0/2443078716 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7f900b70d0 0x7f7f9015e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 >> 172.21.15.99:0/2443078716 conn(0x7f7f900d0470 msgr2=0x7f7f900d2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 shutdown_connections 2024-03-04T10:04:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.123+0000 7f7f955b3640 1 -- 172.21.15.99:0/2443078716 wait complete. 2024-03-04T10:04:41.127 INFO:tasks.ceph.mds.a.smithi099.stderr: -5053> 2024-03-04T10:03:43.043+0000 7fbe61960640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:04:41.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T10:04:41.157 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:04:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 Processor -- start 2024-03-04T10:04:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 -- start start 2024-03-04T10:04:41.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25c015eec0 0x7f25c0099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c00cf200 0x7f25c00ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f25c00cd240 0x7f25c00cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c00cc240 con 0x7f25c015eec0 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c009a110 con 0x7f25c00cd240 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25c519d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00cc500 con 0x7f25c00cf200 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.238+0000 7f25bffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25c015eec0 0x7f25c0099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25c015eec0 0x7f25c0099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35482/0 (socket says 172.21.15.99:35482) 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 -- 172.21.15.99:0/1171670905 learned_addr learned my addr 172.21.15.99:0/1171670905 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25c499c640 1 --2- 172.21.15.99:0/1171670905 >> v2:172.21.15.162:3300/0 conn(0x7f25c00cd240 0x7f25c00cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bf7fe640 1 --2- 172.21.15.99:0/1171670905 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c00cf200 0x7f25c00ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 -- 172.21.15.99:0/1171670905 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c00cf200 msgr2=0x7f25c00ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 --2- 172.21.15.99:0/1171670905 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c00cf200 0x7f25c00ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 -- 172.21.15.99:0/1171670905 >> v2:172.21.15.162:3300/0 conn(0x7f25c00cd240 msgr2=0x7f25c00cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 --2- 172.21.15.99:0/1171670905 >> v2:172.21.15.162:3300/0 conn(0x7f25c00cd240 0x7f25c00cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.239+0000 7f25bffff640 1 -- 172.21.15.99:0/1171670905 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00cc7e0 con 0x7f25c015eec0 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25bffff640 1 --2- 172.21.15.99:0/1171670905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c015eec0 0x7f25c0099a00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f25c0072d50 tx=0x7f25b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c27a8d28edabdf2e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25beffd640 1 -- 172.21.15.99:0/1171670905 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4035020 con 0x7f25c015eec0 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25beffd640 1 -- 172.21.15.99:0/1171670905 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b4031870 con 0x7f25c015eec0 2024-03-04T10:04:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25beffd640 1 -- 172.21.15.99:0/1171670905 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4031c40 con 0x7f25c015eec0 2024-03-04T10:04:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 -- 172.21.15.99:0/1171670905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c015eec0 msgr2=0x7f25c0099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 --2- 172.21.15.99:0/1171670905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c015eec0 0x7f25c0099a00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f25c0072d50 tx=0x7f25b402b910 comp rx=0 tx=0).stop 2024-03-04T10:04:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 -- 172.21.15.99:0/1171670905 shutdown_connections 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 --2- 172.21.15.99:0/1171670905 >> v2:172.21.15.162:3300/0 conn(0x7f25c00cd240 0x7f25c00cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 --2- 172.21.15.99:0/1171670905 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c00cf200 0x7f25c00ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 --2- 172.21.15.99:0/1171670905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c015eec0 0x7f25c0099a00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 -- 172.21.15.99:0/1171670905 >> 172.21.15.99:0/1171670905 conn(0x7f25c00b3120 msgr2=0x7f25c00b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 -- 172.21.15.99:0/1171670905 shutdown_connections 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.240+0000 7f25c519d640 1 -- 172.21.15.99:0/1171670905 wait complete. 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.241+0000 7f25c519d640 1 Processor -- start 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 -- start start 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25c00cd240 0x7f25c015a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 0x7f25c015d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c015eec0 0x7f25c015da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c015f420 con 0x7f25c00cf200 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c015f2a0 con 0x7f25c00cd240 2024-03-04T10:04:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25c519d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c00cfd70 con 0x7f25c015eec0 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25bf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 0x7f25c015d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25bf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 0x7f25c015d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35486/0 (socket says 172.21.15.99:35486) 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25bf7fe640 1 -- 172.21.15.99:0/216029682 learned_addr learned my addr 172.21.15.99:0/216029682 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.242+0000 7f25bffff640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25c00cd240 0x7f25c015a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25c499c640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c015eec0 0x7f25c015da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bf7fe640 1 -- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c015eec0 msgr2=0x7f25c015da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bf7fe640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c015eec0 0x7f25c015da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bf7fe640 1 -- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25c00cd240 msgr2=0x7f25c015a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bf7fe640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25c00cd240 0x7f25c015a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bf7fe640 1 -- 172.21.15.99:0/216029682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c015a5e0 con 0x7f25c00cf200 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bf7fe640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 0x7f25c015d4c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f25b000e9b0 tx=0x7f25b000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0017070 con 0x7f25c00cf200 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b000c9a0 con 0x7f25c00cf200 2024-03-04T10:04:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00b4ba0 con 0x7f25c00cf200 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b000cd90 con 0x7f25c00cf200 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.243+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c00b5160 con 0x7f25c00cf200 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.244+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c004f090 con 0x7f25c00cf200 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.245+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f25b001a070 con 0x7f25c00cf200 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.245+0000 7f25bcff9640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f25ac06b4d0 0x7f25ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.245+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f25b008a4d0 con 0x7f25c00cf200 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.245+0000 7f25bffff640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f25ac06b4d0 0x7f25ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.246+0000 7f25bffff640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f25ac06b4d0 0x7f25ac06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f25b4036000 tx=0x7f25b403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:41.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.249+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f25b00588c0 con 0x7f25c00cf200 2024-03-04T10:04:41.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:41.457+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f25c00cdb30 con 0x7f25c00cf200 2024-03-04T10:04:42.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.124+0000 7f25bcff9640 1 -- 172.21.15.99:0/216029682 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f25b005c960 con 0x7f25c00cf200 2024-03-04T10:04:42.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f25ac06b4d0 msgr2=0x7f25ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:42.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f25ac06b4d0 0x7f25ac06d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f25b4036000 tx=0x7f25b403c040 comp rx=0 tx=0).stop 2024-03-04T10:04:42.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 msgr2=0x7f25c015d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:42.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 0x7f25c015d4c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f25b000e9b0 tx=0x7f25b000ee70 comp rx=0 tx=0).stop 2024-03-04T10:04:42.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 shutdown_connections 2024-03-04T10:04:42.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f25ac06b4d0 0x7f25ac06d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25c015eec0 0x7f25c015da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25c00cf200 0x7f25c015d4c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 --2- 172.21.15.99:0/216029682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f25c00cd240 0x7f25c015a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 >> 172.21.15.99:0/216029682 conn(0x7f25c00b3120 msgr2=0x7f25c00737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:42.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 shutdown_connections 2024-03-04T10:04:42.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.127+0000 7f25c519d640 1 -- 172.21.15.99:0/216029682 wait complete. 2024-03-04T10:04:42.142 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T10:04:42.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.289+0000 7f633117e640 1 Processor -- start 2024-03-04T10:04:42.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.289+0000 7f633117e640 1 -- start start 2024-03-04T10:04:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f633117e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c06b200 0x7f632c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f633117e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f632c15ca00 0x7f632c0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f633117e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f632c0cfa40 0x7f632c0cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f633117e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c0cc860 con 0x7f632c15ca00 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f633117e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c15c570 con 0x7f632c0cfa40 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f633117e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c15bf80 con 0x7f632c06b200 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f632c15ca00 0x7f632c0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f632c15ca00 0x7f632c0cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35490/0 (socket says 172.21.15.99:35490) 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 -- 172.21.15.99:0/559275944 learned_addr learned my addr 172.21.15.99:0/559275944 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632ad9d640 1 --2- 172.21.15.99:0/559275944 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c06b200 0x7f632c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632b59e640 1 --2- 172.21.15.99:0/559275944 >> v2:172.21.15.162:3300/0 conn(0x7f632c0cfa40 0x7f632c0cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 -- 172.21.15.99:0/559275944 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c06b200 msgr2=0x7f632c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 --2- 172.21.15.99:0/559275944 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c06b200 0x7f632c0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 -- 172.21.15.99:0/559275944 >> v2:172.21.15.162:3300/0 conn(0x7f632c0cfa40 msgr2=0x7f632c0cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 --2- 172.21.15.99:0/559275944 >> v2:172.21.15.162:3300/0 conn(0x7f632c0cfa40 0x7f632c0cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.290+0000 7f632259c640 1 -- 172.21.15.99:0/559275944 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0cca90 con 0x7f632c15ca00 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.291+0000 7f632259c640 1 --2- 172.21.15.99:0/559275944 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c15ca00 0x7f632c0cc320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6314009870 tx=0x7f631402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e956337c8ffffc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.291+0000 7f632a59c640 1 -- 172.21.15.99:0/559275944 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314035020 con 0x7f632c15ca00 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.291+0000 7f632a59c640 1 -- 172.21.15.99:0/559275944 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314031640 con 0x7f632c15ca00 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.291+0000 7f632a59c640 1 -- 172.21.15.99:0/559275944 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631402d040 con 0x7f632c15ca00 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.291+0000 7f633117e640 1 -- 172.21.15.99:0/559275944 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c15ca00 msgr2=0x7f632c0cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.291+0000 7f633117e640 1 --2- 172.21.15.99:0/559275944 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c15ca00 0x7f632c0cc320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6314009870 tx=0x7f631402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 -- 172.21.15.99:0/559275944 shutdown_connections 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 --2- 172.21.15.99:0/559275944 >> v2:172.21.15.162:3300/0 conn(0x7f632c0cfa40 0x7f632c0cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 --2- 172.21.15.99:0/559275944 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c15ca00 0x7f632c0cc320 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 --2- 172.21.15.99:0/559275944 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c06b200 0x7f632c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 -- 172.21.15.99:0/559275944 >> 172.21.15.99:0/559275944 conn(0x7f632c099a00 msgr2=0x7f632c0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 -- 172.21.15.99:0/559275944 shutdown_connections 2024-03-04T10:04:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.292+0000 7f633117e640 1 -- 172.21.15.99:0/559275944 wait complete. 2024-03-04T10:04:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.293+0000 7f633117e640 1 Processor -- start 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.293+0000 7f633117e640 1 -- start start 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 0x7f632c149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f632c0cfa40 0x7f632c152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c15ca00 0x7f632c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c15ce40 con 0x7f632c06b200 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c15c200 con 0x7f632c0cfa40 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c06b5e0 con 0x7f632c15ca00 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 0x7f632c149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 0x7f632c149550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35500/0 (socket says 172.21.15.99:35500) 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 -- 172.21.15.99:0/2852758998 learned_addr learned my addr 172.21.15.99:0/2852758998 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632259c640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f632c0cfa40 0x7f632c152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632b59e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c15ca00 0x7f632c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 -- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c15ca00 msgr2=0x7f632c15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c15ca00 0x7f632c15ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 -- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f632c0cfa40 msgr2=0x7f632c152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f632c0cfa40 0x7f632c152bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 -- 172.21.15.99:0/2852758998 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c15efa0 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632259c640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f632c0cfa40 0x7f632c152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f632ad9d640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 0x7f632c149550 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c7f0 tx=0x7f631c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c015070 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c004220 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c007620 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.294+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c15f280 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.295+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c1e5370 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.295+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c074f60 con 0x7f632c06b200 2024-03-04T10:04:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.296+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f631c0077c0 con 0x7f632c06b200 2024-03-04T10:04:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.296+0000 7f6323fff640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f630c06b500 0x7f630c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.296+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6800+0+0 (secure 0 0 0) 0x7f631c08a4f0 con 0x7f632c06b200 2024-03-04T10:04:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.296+0000 7f632259c640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f630c06b500 0x7f630c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.297+0000 7f632259c640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f630c06b500 0x7f630c06d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6314036000 tx=0x7f631403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.300+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f631c058ea0 con 0x7f632c06b200 2024-03-04T10:04:42.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:42.472+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f632c0b66a0 con 0x7f632c06b200 2024-03-04T10:04:43.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.129+0000 7f6323fff640 1 -- 172.21.15.99:0/2852758998 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f631c0587a0 con 0x7f632c06b200 2024-03-04T10:04:43.131 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' removed 2024-03-04T10:04:43.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f630c06b500 msgr2=0x7f630c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f630c06b500 0x7f630c06d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6314036000 tx=0x7f631403c040 comp rx=0 tx=0).stop 2024-03-04T10:04:43.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 msgr2=0x7f632c149550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 0x7f632c149550 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c7f0 tx=0x7f631c00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 shutdown_connections 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f630c06b500 0x7f630c06d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f632c15ca00 0x7f632c15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f632c0cfa40 0x7f632c152bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 --2- 172.21.15.99:0/2852758998 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f632c06b200 0x7f632c149550 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 >> 172.21.15.99:0/2852758998 conn(0x7f632c099a00 msgr2=0x7f632c0ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 shutdown_connections 2024-03-04T10:04:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.131+0000 7f633117e640 1 -- 172.21.15.99:0/2852758998 wait complete. 2024-03-04T10:04:43.148 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T10:04:43.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.254+0000 7ffac224b640 1 Processor -- start 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.255+0000 7ffac224b640 1 -- start start 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.255+0000 7ffac224b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffabc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.255+0000 7ffac224b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc15e540 con 0x7ffabc15e7e0 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.255+0000 7ffac224b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffabc0b2e00 con 0x7ffabc15f370 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.255+0000 7ffac224b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0cdcd0 con 0x7ffabc15e160 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffabaffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ffabc15e7e0 0x7ffabc15ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:34170/0 (socket says 172.21.15.99:34170) 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffabaffd640 1 -- 172.21.15.99:0/1826065060 learned_addr learned my addr 172.21.15.99:0/1826065060 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffabb7fe640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffabc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299027170 0 0) 0x7ffabc15e540 con 0x7ffabc15e7e0 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa4005730 con 0x7ffabc15e7e0 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245757090 0 0) 0x7ffabc0b2e00 con 0x7ffabc15f370 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc15e540 con 0x7ffabc15f370 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733792891 0 0) 0x7ffaa4005730 con 0x7ffabc15e7e0 2024-03-04T10:04:43.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 msgr2=0x7ffabc0b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffabc0b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 >> v1:172.21.15.162:6789/0 conn(0x7ffabc15f370 legacy=0x7ffabc0cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0b30e0 con 0x7ffabc15e7e0 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7ffab0001670 con 0x7ffabc15e7e0 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 >> v1:172.21.15.99:6789/0 conn(0x7ffabc15e7e0 legacy=0x7ffabc15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffaa4005d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffaa4006330 0x7ffaa4006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaa4006f50 0x7ffaa40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0cdcd0 con 0x7ffaa4006330 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa400dd90 con 0x7ffaa4006f50 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa400df90 con 0x7ffabc15e160 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffabaffd640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffaa4006330 0x7ffaa4006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffabb7fe640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffaa4005d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.256+0000 7ffabbfff640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaa4006f50 0x7ffaa40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabaffd640 1 -- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 msgr2=0x7ffaa4005d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabaffd640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffaa4005d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabaffd640 1 -- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaa4006f50 msgr2=0x7ffaa40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabaffd640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaa4006f50 0x7ffaa40073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabaffd640 1 -- 172.21.15.99:0/1826065060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4005730 con 0x7ffaa4006330 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabbfff640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffaa4006f50 0x7ffaa40073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabb7fe640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffabc15e160 0x7ffaa4005d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffabaffd640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffaa4006330 0x7ffaa4006a10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c0e0 tx=0x7ffaa4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a874354e2a5253 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0007c00 con 0x7ffaa4006330 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab0030070 con 0x7ffaa4006330 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.257+0000 7ffaba7fc640 1 -- 172.21.15.99:0/1826065060 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab0005d10 con 0x7ffaa4006330 2024-03-04T10:04:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffac224b640 1 -- 172.21.15.99:0/1826065060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffaa4006330 msgr2=0x7ffaa4006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffac224b640 1 --2- 172.21.15.99:0/1826065060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffaa4006330 0x7ffaa4006a10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ffab000c0e0 tx=0x7ffaa4007a40 comp rx=0 tx=0).stop 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffabbfff640 1 -- 172.21.15.99:0/1826065060 reap_dead start 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffac224b640 1 -- 172.21.15.99:0/1826065060 shutdown_connections 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffac224b640 1 -- 172.21.15.99:0/1826065060 >> 172.21.15.99:0/1826065060 conn(0x7ffabc0b22e0 msgr2=0x7ffabc0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffac224b640 1 -- 172.21.15.99:0/1826065060 shutdown_connections 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.258+0000 7ffac224b640 1 -- 172.21.15.99:0/1826065060 wait complete. 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.259+0000 7ffac224b640 1 Processor -- start 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.259+0000 7ffac224b640 1 -- start start 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffac224b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaa4006330 0x7ffabc156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffac224b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffabc1f3c50 0x7ffabc15a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffac224b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 0x7ffabc15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffac224b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0b1e70 con 0x7ffabc0c7310 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffac224b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0cd960 con 0x7ffabc1f3c50 2024-03-04T10:04:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffac224b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc0b1ff0 con 0x7ffaa4006330 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 0x7ffabc15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 0x7ffabc15b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35514/0 (socket says 172.21.15.99:35514) 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 -- 172.21.15.99:0/1452370785 learned_addr learned my addr 172.21.15.99:0/1452370785 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabb7fe640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaa4006330 0x7ffabc156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabaffd640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffabc1f3c50 0x7ffabc15a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 -- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaa4006330 msgr2=0x7ffabc156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaa4006330 0x7ffabc156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 -- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffabc1f3c50 msgr2=0x7ffabc15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffabc1f3c50 0x7ffabc15a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.260+0000 7ffabbfff640 1 -- 172.21.15.99:0/1452370785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffabc0ad660 con 0x7ffabc0c7310 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.261+0000 7ffabbfff640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 0x7ffabc15b6c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ffabc15f880 tx=0x7ffab400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.261+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4003b80 con 0x7ffabc0c7310 2024-03-04T10:04:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.261+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab4003d40 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.261+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc15b0c0 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.261+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4006d90 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.261+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc15b3a0 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.262+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffabc074f60 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.262+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffab4005070 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.263+0000 7ffaa3fff640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.263+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6362+0+0 (secure 0 0 0) 0x7ffab4089f10 con 0x7ffabc0c7310 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.263+0000 7ffabb7fe640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.263+0000 7ffabb7fe640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009a60 tx=0x7ffaac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:43.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.267+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffab4058540 con 0x7ffabc0c7310 2024-03-04T10:04:43.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:43.436+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffabc1592b0 con 0x7ffabc0c7310 2024-03-04T10:04:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.136+0000 7ffaa3fff640 1 -- 172.21.15.99:0/1452370785 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7ffab405c5e0 con 0x7ffabc0c7310 2024-03-04T10:04:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' removed 2024-03-04T10:04:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaa806b4d0 msgr2=0x7ffaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009a60 tx=0x7ffaac008040 comp rx=0 tx=0).stop 2024-03-04T10:04:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 msgr2=0x7ffabc15b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 0x7ffabc15b6c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ffabc15f880 tx=0x7ffab400ec60 comp rx=0 tx=0).stop 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 shutdown_connections 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffabc0c7310 0x7ffabc15b6c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffabc1f3c50 0x7ffabc15a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 --2- 172.21.15.99:0/1452370785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffaa4006330 0x7ffabc156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 >> 172.21.15.99:0/1452370785 conn(0x7ffabc0b22e0 msgr2=0x7ffabc06acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 shutdown_connections 2024-03-04T10:04:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.138+0000 7ffac224b640 1 -- 172.21.15.99:0/1452370785 wait complete. 2024-03-04T10:04:44.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-04T10:04:44.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.258+0000 7f9bb853b640 1 Processor -- start 2024-03-04T10:04:44.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.258+0000 7f9bb853b640 1 -- start start 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb853b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb015dc80 0x7f9bb006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb853b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9bb00671c0 0x7f9bb00b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb853b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9bb00b2fe0 0x7f9bb00b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb853b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb006acd0 con 0x7f9bb00b2fe0 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb853b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb00729b0 con 0x7f9bb00671c0 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb853b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb006af20 con 0x7f9bb015dc80 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9bb00b2fe0 0x7f9bb00b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9bb00b2fe0 0x7f9bb00b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43714/0 (socket says 172.21.15.99:43714) 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 -- 172.21.15.99:0/1305618845 learned_addr learned my addr 172.21.15.99:0/1305618845 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb5ad6640 1 --2- 172.21.15.99:0/1305618845 >> v2:172.21.15.162:3300/0 conn(0x7f9bb00671c0 0x7f9bb00b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb62d7640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb015dc80 0x7f9bb006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 -- 172.21.15.99:0/1305618845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb015dc80 msgr2=0x7f9bb006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb015dc80 0x7f9bb006bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 -- 172.21.15.99:0/1305618845 >> v2:172.21.15.162:3300/0 conn(0x7f9bb00671c0 msgr2=0x7f9bb00b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 --2- 172.21.15.99:0/1305618845 >> v2:172.21.15.162:3300/0 conn(0x7f9bb00671c0 0x7f9bb00b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb6ad8640 1 -- 172.21.15.99:0/1305618845 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb006b200 con 0x7f9bb00b2fe0 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb62d7640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb015dc80 0x7f9bb006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.259+0000 7f9bb5ad6640 1 --2- 172.21.15.99:0/1305618845 >> v2:172.21.15.162:3300/0 conn(0x7f9bb00671c0 0x7f9bb00b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.260+0000 7f9bb6ad8640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb00b4e20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0070850 tx=0x7f9ba802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf6cf4a376e8c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.260+0000 7f9bb52d5640 1 -- 172.21.15.99:0/1305618845 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba802ca70 con 0x7f9bb00b2fe0 2024-03-04T10:04:44.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.260+0000 7f9bb52d5640 1 -- 172.21.15.99:0/1305618845 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba802cc10 con 0x7f9bb00b2fe0 2024-03-04T10:04:44.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.260+0000 7f9bb52d5640 1 -- 172.21.15.99:0/1305618845 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba80343f0 con 0x7f9bb00b2fe0 2024-03-04T10:04:44.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 -- 172.21.15.99:0/1305618845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00b2fe0 msgr2=0x7f9bb00b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb00b4e20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0070850 tx=0x7f9ba802bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 -- 172.21.15.99:0/1305618845 shutdown_connections 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb00b4e20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1305618845 >> v2:172.21.15.162:3300/0 conn(0x7f9bb00671c0 0x7f9bb00b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1305618845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb015dc80 0x7f9bb006bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 -- 172.21.15.99:0/1305618845 >> 172.21.15.99:0/1305618845 conn(0x7f9bb0071ca0 msgr2=0x7f9bb0072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 -- 172.21.15.99:0/1305618845 shutdown_connections 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.261+0000 7f9bb853b640 1 -- 172.21.15.99:0/1305618845 wait complete. 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.262+0000 7f9bb853b640 1 Processor -- start 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.262+0000 7f9bb853b640 1 -- start start 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 0x7f9bb00b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb014d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb014d8c0 0x7f9bb015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb006baa0 con 0x7f9bb00671c0 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb006b7c0 con 0x7f9bb00b2fe0 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb006c500 con 0x7f9bb014d8c0 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 0x7f9bb00b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 0x7f9bb00b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43716/0 (socket says 172.21.15.99:43716) 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 -- 172.21.15.99:0/1516235812 learned_addr learned my addr 172.21.15.99:0/1516235812 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb5ad6640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb014d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb6ad8640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb014d8c0 0x7f9bb015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 -- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb014d8c0 msgr2=0x7f9bb015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb014d8c0 0x7f9bb015c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 -- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bb00b2fe0 msgr2=0x7f9bb014d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb014d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:44.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 -- 172.21.15.99:0/1516235812 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb0156e80 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb6ad8640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb014d8c0 0x7f9bb015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb5ad6640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb014d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb62d7640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 0x7f9bb00b0900 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00e7e0 tx=0x7f9bac00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac00f040 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac004030 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb0157160 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac004400 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.263+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb015a5c0 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.264+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb015a0d0 con 0x7f9bb00671c0 2024-03-04T10:04:44.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.265+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9bac00c6d0 con 0x7f9bb00671c0 2024-03-04T10:04:44.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.265+0000 7f9ba6ffd640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:44.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.265+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f9bac08a0f0 con 0x7f9bb00671c0 2024-03-04T10:04:44.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.265+0000 7f9bb5ad6640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:44.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.266+0000 7f9bb5ad6640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0006fd0 tx=0x7f9ba0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:44.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.269+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9bac017020 con 0x7f9bb00671c0 2024-03-04T10:04:44.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:44.438+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9bb0074f60 con 0x7f9bb00671c0 2024-03-04T10:04:45.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.141+0000 7f9ba6ffd640 1 -- 172.21.15.99:0/1516235812 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7f9bac058ee0 con 0x7f9bb00671c0 2024-03-04T10:04:45.143 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data_ec' removed 2024-03-04T10:04:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.143+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9b9406b4d0 msgr2=0x7f9b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.143+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0006fd0 tx=0x7f9ba0008040 comp rx=0 tx=0).stop 2024-03-04T10:04:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.143+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 msgr2=0x7f9bb00b0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.143+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 0x7f9bb00b0900 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00e7e0 tx=0x7f9bac00eca0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 shutdown_connections 2024-03-04T10:04:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bb014d8c0 0x7f9bb015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bb00b2fe0 0x7f9bb014d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 --2- 172.21.15.99:0/1516235812 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bb00671c0 0x7f9bb00b0900 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 >> 172.21.15.99:0/1516235812 conn(0x7f9bb0071ca0 msgr2=0x7f9bb00b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 shutdown_connections 2024-03-04T10:04:45.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.144+0000 7f9bb853b640 1 -- 172.21.15.99:0/1516235812 wait complete. 2024-03-04T10:04:45.159 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:04:45.159 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T10:04:45.159 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:04:45.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:04:45.160 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:04:45.160 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T10:04:45.160 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:04:45.160 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:04:45.160 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T10:04:45.160 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T10:04:45.161 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T10:04:45.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:04:45.161 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T10:04:45.161 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T10:04:45.161 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T10:04:45.161 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:04:45.169 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:04:45.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:04:45.170 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:04:45.170 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:04:45.172 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T10:04:45.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T10:04:45.173 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T10:04:45.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T10:04:45.213 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:04:45.213 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:04:45.214 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:45.214 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:45.216 INFO:tasks.ceph.mds.b:Started 2024-03-04T10:04:45.216 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:45.217 INFO:tasks.ceph.mds.d:Started 2024-03-04T10:04:45.217 DEBUG:teuthology.parallel:result is None 2024-03-04T10:04:45.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T10:04:45.288 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:04:45.289 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:04:45.299 INFO:tasks.ceph.mds.b.smithi162.stdout:starting mds.b at 2024-03-04T10:04:45.299 INFO:tasks.ceph.mds.d.smithi162.stdout:starting mds.d at 2024-03-04T10:04:45.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.321+0000 7f2fcd3c4640 1 Processor -- start 2024-03-04T10:04:45.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.321+0000 7f2fcd3c4640 1 -- start start 2024-03-04T10:04:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.321+0000 7f2fcd3c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2fc815de00 0x7f2fc80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.321+0000 7f2fcd3c4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc815e1e0 con 0x7f2fc80b5470 2024-03-04T10:04:45.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.321+0000 7f2fcd3c4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80d1c10 con 0x7f2fc80b9380 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.321+0000 7f2fcd3c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc80d2820 con 0x7f2fc815de00 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc77fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2fc80b5470 0x7f2fc80d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60306/0 (socket says 172.21.15.99:60306) 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc77fe640 1 -- 172.21.15.99:0/2399928356 learned_addr learned my addr 172.21.15.99:0/2399928356 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc6ffd640 1 --2- 172.21.15.99:0/2399928356 >> v2:172.21.15.162:3300/0 conn(0x7f2fc815de00 0x7f2fc80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364051700 0 0) 0x7f2fc815e1e0 con 0x7f2fc80b5470 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb0003520 con 0x7f2fc80b5470 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352802150 0 0) 0x7f2fc80d1c10 con 0x7f2fc80b9380 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc815e1e0 con 0x7f2fc80b9380 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc6ffd640 1 -- 172.21.15.99:0/2399928356 >> v1:172.21.15.162:6789/0 conn(0x7f2fc80b9380 legacy=0x7f2fc80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387223497 0 0) 0x7f2fb0003520 con 0x7f2fc80b5470 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc6ffd640 1 -- 172.21.15.99:0/2399928356 >> v1:172.21.15.99:6789/0 conn(0x7f2fc80b5470 legacy=0x7f2fc80d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.322+0000 7f2fc6ffd640 1 -- 172.21.15.99:0/2399928356 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80d1ef0 con 0x7f2fc815de00 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fc6ffd640 1 --2- 172.21.15.99:0/2399928356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc80b2e60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80097a0 tx=0x7f2fb802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=762dc46d14c04ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8038070 con 0x7f2fc815de00 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8031500 con 0x7f2fc815de00 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fc5ffb640 1 -- 172.21.15.99:0/2399928356 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00019f0 con 0x7f2fc815de00 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/2399928356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 msgr2=0x7f2fc80b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/2399928356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc80b2e60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2fb80097a0 tx=0x7f2fb802b820 comp rx=0 tx=0).stop 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/2399928356 shutdown_connections 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/2399928356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc80b2e60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.323+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/2399928356 >> 172.21.15.99:0/2399928356 conn(0x7f2fc80d0470 msgr2=0x7f2fc80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.324+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/2399928356 shutdown_connections 2024-03-04T10:04:45.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.324+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/2399928356 wait complete. 2024-03-04T10:04:45.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.325+0000 7f2fcd3c4640 1 Processor -- start 2024-03-04T10:04:45.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.325+0000 7f2fcd3c4640 1 -- start start 2024-03-04T10:04:45.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.325+0000 7f2fcd3c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc80b5470 0x7f2fc81572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:45.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.325+0000 7f2fcd3c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 0x7f2fc80b8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.325+0000 7f2fcd3c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc8158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.325+0000 7f2fcd3c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80b97e0 con 0x7f2fc80b9380 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fcd3c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80d24b0 con 0x7f2fc815de00 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fcd3c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80b72a0 con 0x7f2fc80b5470 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 0x7f2fc80b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 0x7f2fc80b8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43754/0 (socket says 172.21.15.99:43754) 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 -- 172.21.15.99:0/3512282161 learned_addr learned my addr 172.21.15.99:0/3512282161 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc77fe640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc8158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc6ffd640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc80b5470 0x7f2fc81572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 -- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc80b5470 msgr2=0x7f2fc81572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc80b5470 0x7f2fc81572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 -- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 msgr2=0x7f2fc8158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:45.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc8158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 -- 172.21.15.99:0/3512282161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80b8950 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc6ffd640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc80b5470 0x7f2fc81572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc77fe640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc8158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fc67fc640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 0x7f2fc80b8410 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00c490 tx=0x7f2fbc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0052d0 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc005470 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc012850 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8159140 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.326+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc81596e0 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.328+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc8074f60 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.328+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2fbc0129f0 con 0x7f2fc80b9380 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.328+0000 7f2faf7fe640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb406b4d0 0x7f2fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.328+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2fbc089d70 con 0x7f2fc80b9380 2024-03-04T10:04:45.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.328+0000 7f2fc6ffd640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb406b4d0 0x7f2fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:45.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.329+0000 7f2fc6ffd640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb406b4d0 0x7f2fb406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8035000 tx=0x7f2fb8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:45.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.332+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2fbc058d60 con 0x7f2fc80b9380 2024-03-04T10:04:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:45.524+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2fc80b5230 con 0x7f2fc80b9380 2024-03-04T10:04:46.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.155+0000 7f2faf7fe640 1 -- 172.21.15.99:0/3512282161 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2fbc058660 con 0x7f2fc80b9380 2024-03-04T10:04:46.158 INFO:teuthology.orchestra.run.smithi099.stderr: removed all blocklist entries 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb406b4d0 msgr2=0x7f2fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb406b4d0 0x7f2fb406d990 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8035000 tx=0x7f2fb8008030 comp rx=0 tx=0).stop 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 msgr2=0x7f2fc80b8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 0x7f2fc80b8410 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00c490 tx=0x7f2fbc00c950 comp rx=0 tx=0).stop 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 shutdown_connections 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb406b4d0 0x7f2fb406d990 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc815de00 0x7f2fc8158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc80b9380 0x7f2fc80b8410 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 --2- 172.21.15.99:0/3512282161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc80b5470 0x7f2fc81572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 >> 172.21.15.99:0/3512282161 conn(0x7f2fc80d0470 msgr2=0x7f2fc80d2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 shutdown_connections 2024-03-04T10:04:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.158+0000 7f2fcd3c4640 1 -- 172.21.15.99:0/3512282161 wait complete. 2024-03-04T10:04:46.175 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.277+0000 7f77afd40640 1 Processor -- start 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.277+0000 7f77afd40640 1 -- start start 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77afd40640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a80b39c0 0x7f77a8161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77afd40640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f77a8161ca0 0x7f77a806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77afd40640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a806b500 con 0x7f77a8158410 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77afd40640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a806b7e0 con 0x7f77a8161ca0 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77afd40640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a806bac0 con 0x7f77a80b39c0 2024-03-04T10:04:46.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77ae2dd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f77a8161ca0 0x7f77a806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77adadc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f77a8158410 0x7f77a806acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60308/0 (socket says 172.21.15.99:60308) 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77adadc640 1 -- 172.21.15.99:0/3821526218 learned_addr learned my addr 172.21.15.99:0/3821526218 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77ad2db640 1 --2- 172.21.15.99:0/3821526218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a80b39c0 0x7f77a8161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77ae2dd640 1 -- 172.21.15.99:0/3821526218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a80b39c0 msgr2=0x7f77a8161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77ae2dd640 1 --2- 172.21.15.99:0/3821526218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a80b39c0 0x7f77a8161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77ae2dd640 1 -- 172.21.15.99:0/3821526218 >> v1:172.21.15.99:6789/0 conn(0x7f77a8158410 legacy=0x7f77a806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77acada640 1 -- 172.21.15.99:0/3821526218 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765508413 0 0) 0x7f77a806b500 con 0x7f77a8158410 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.278+0000 7f77ae2dd640 1 -- 172.21.15.99:0/3821526218 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a806bda0 con 0x7f77a8161ca0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.279+0000 7f77ae2dd640 1 --2- 172.21.15.99:0/3821526218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a8161ca0 0x7f77a806de50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f77a00097a0 tx=0x7f77a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88cb247f7e9bf04 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.279+0000 7f77acada640 1 -- 172.21.15.99:0/3821526218 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a0034070 con 0x7f77a8161ca0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.279+0000 7f77acada640 1 -- 172.21.15.99:0/3821526218 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a0030610 con 0x7f77a8161ca0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.279+0000 7f77acada640 1 -- 172.21.15.99:0/3821526218 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a00309e0 con 0x7f77a8161ca0 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 -- 172.21.15.99:0/3821526218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a8161ca0 msgr2=0x7f77a806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 --2- 172.21.15.99:0/3821526218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a8161ca0 0x7f77a806de50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f77a00097a0 tx=0x7f77a002b960 comp rx=0 tx=0).stop 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 -- 172.21.15.99:0/3821526218 shutdown_connections 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 --2- 172.21.15.99:0/3821526218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a8161ca0 0x7f77a806de50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 --2- 172.21.15.99:0/3821526218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a80b39c0 0x7f77a8161760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 -- 172.21.15.99:0/3821526218 >> 172.21.15.99:0/3821526218 conn(0x7f77a80cfef0 msgr2=0x7f77a80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 -- 172.21.15.99:0/3821526218 shutdown_connections 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.280+0000 7f77afd40640 1 -- 172.21.15.99:0/3821526218 wait complete. 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.281+0000 7f77afd40640 1 Processor -- start 2024-03-04T10:04:46.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.281+0000 7f77afd40640 1 -- start start 2024-03-04T10:04:46.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 0x7f77a8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77a8158410 0x7f77a815b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a8161ca0 0x7f77a80ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a806c5a0 con 0x7f77a80b39c0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80b3e20 con 0x7f77a8158410 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a806e390 con 0x7f77a8161ca0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 0x7f77a8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 0x7f77a8155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43768/0 (socket says 172.21.15.99:43768) 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 -- 172.21.15.99:0/2065537447 learned_addr learned my addr 172.21.15.99:0/2065537447 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 -- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a8161ca0 msgr2=0x7f77a80ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a8161ca0 0x7f77a80ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 -- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77a8158410 msgr2=0x7f77a815b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77a8158410 0x7f77a815b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 -- 172.21.15.99:0/2065537447 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a815a120 con 0x7f77a80b39c0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77adadc640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 0x7f77a8155cf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f77a400c890 tx=0x7f77a400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a4018070 con 0x7f77a80b39c0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a400a640 con 0x7f77a80b39c0 2024-03-04T10:04:46.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a815fa70 con 0x7f77a80b39c0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a400aa30 con 0x7f77a80b39c0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.282+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a8160030 con 0x7f77a80b39c0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.283+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a8074f60 con 0x7f77a80b39c0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.284+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f77a401b070 con 0x7f77a80b39c0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.284+0000 7f779e7fc640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f778c06b4d0 0x7f778c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.284+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f77a4089730 con 0x7f77a80b39c0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.284+0000 7f77ad2db640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f778c06b4d0 0x7f778c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.285+0000 7f77ad2db640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f778c06b4d0 0x7f778c06d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f77a815fa00 tx=0x7f7798008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:46.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.288+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f77a4058770 con 0x7f77a80b39c0 2024-03-04T10:04:46.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.522+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f77a806cba0 con 0x7f77a80b39c0 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.523+0000 7f779e7fc640 1 -- 172.21.15.99:0/2065537447 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f77a4058070 con 0x7f77a80b39c0 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_dump": [ 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.a", 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg==", 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.525 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.b", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw==", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.526 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.c", 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg==", 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.527 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.d", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw==", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.0", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg==", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.528 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.1", 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ==", 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.529 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.2", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw==", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.3", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA==", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.530 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.4", 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw==", 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.5", 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg==", 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.531 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.6", 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg==", 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.532 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.7", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA==", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.0", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ==", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:04:46.533 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.1", 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA==", 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.534 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.2", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg==", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.3", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ==", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:04:46.535 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.admin", 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBAnOVlV1MhNhAAJTJsIc9qGmeQBnHKjrkPvw==", 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.536 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.bootstrap-mds", 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBjnOVlS7zhCRAAj7ufnuyVsyOyWajwsRY9ew==", 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile bootstrap-mds" 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.bootstrap-mgr", 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBjnOVlf8ThCRAAbhxQs7W5+YhyzzPBvb2XpA==", 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.537 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.bootstrap-osd", 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBjnOVlFcvhCRAAwo9p5khxNmpxgdeVQq4vfg==", 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile bootstrap-osd" 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.bootstrap-rbd", 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBjnOVla9HhCRAAAZXuVwJRhgQ0OUi4Ryie4g==", 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-04T10:04:46.538 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBjnOVlLtnhCRAAWAGfNaVmPGO0OP06dUfjZg==", 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.bootstrap-rgw", 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBjnOVlDODhCRAAWUflXHuiWUXweMs4IEJgHw==", 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.x", 2024-03-04T10:04:46.539 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw==", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.y", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ==", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.540 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:46.541 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-03-04T10:04:46.541 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f778c06b4d0 msgr2=0x7f778c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f778c06b4d0 0x7f778c06d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f77a815fa00 tx=0x7f7798008040 comp rx=0 tx=0).stop 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 msgr2=0x7f77a8155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 0x7f77a8155cf0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f77a400c890 tx=0x7f77a400cd50 comp rx=0 tx=0).stop 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 shutdown_connections 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f778c06b4d0 0x7f778c06d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77a8161ca0 0x7f77a80ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77a8158410 0x7f77a815b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 --2- 172.21.15.99:0/2065537447 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77a80b39c0 0x7f77a8155cf0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 >> 172.21.15.99:0/2065537447 conn(0x7f77a80cfef0 msgr2=0x7f77a806acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 shutdown_connections 2024-03-04T10:04:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.525+0000 7f77afd40640 1 -- 172.21.15.99:0/2065537447 wait complete. 2024-03-04T10:04:46.546 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-04T10:04:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.645+0000 7fe659a61640 1 Processor -- start 2024-03-04T10:04:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.645+0000 7fe659a61640 1 -- start start 2024-03-04T10:04:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe659a61640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe654154a80 0x7fe65415e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe659a61640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6541624e0 con 0x7fe6540cfae0 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe659a61640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6540cd010 con 0x7fe6540cf650 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe659a61640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe654071df0 con 0x7fe654154a80 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe652ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe654154a80 0x7fe65415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe6537fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe6540cfae0 0x7fe654161dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60322/0 (socket says 172.21.15.99:60322) 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe6537fe640 1 -- 172.21.15.99:0/604268678 learned_addr learned my addr 172.21.15.99:0/604268678 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe652ffd640 1 -- 172.21.15.99:0/604268678 >> v1:172.21.15.162:6789/0 conn(0x7fe6540cf650 legacy=0x7fe6541616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe652ffd640 1 -- 172.21.15.99:0/604268678 >> v1:172.21.15.99:6789/0 conn(0x7fe6540cfae0 legacy=0x7fe654161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.646+0000 7fe652ffd640 1 -- 172.21.15.99:0/604268678 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540cff70 con 0x7fe654154a80 2024-03-04T10:04:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.647+0000 7fe652ffd640 1 --2- 172.21.15.99:0/604268678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe654154a80 0x7fe65415e910 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe644009d90 tx=0x7fe64402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0976bb7e7ff9b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.647+0000 7fe651ffb640 1 -- 172.21.15.99:0/604268678 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644039070 con 0x7fe654154a80 2024-03-04T10:04:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.647+0000 7fe651ffb640 1 -- 172.21.15.99:0/604268678 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64402cc60 con 0x7fe654154a80 2024-03-04T10:04:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.647+0000 7fe651ffb640 1 -- 172.21.15.99:0/604268678 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644034660 con 0x7fe654154a80 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 -- 172.21.15.99:0/604268678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe654154a80 msgr2=0x7fe65415e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 --2- 172.21.15.99:0/604268678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe654154a80 0x7fe65415e910 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe644009d90 tx=0x7fe64402beb0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 -- 172.21.15.99:0/604268678 shutdown_connections 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 --2- 172.21.15.99:0/604268678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe654154a80 0x7fe65415e910 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 -- 172.21.15.99:0/604268678 >> 172.21.15.99:0/604268678 conn(0x7fe65406f5f0 msgr2=0x7fe65406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 -- 172.21.15.99:0/604268678 shutdown_connections 2024-03-04T10:04:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.648+0000 7fe659a61640 1 -- 172.21.15.99:0/604268678 wait complete. 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.649+0000 7fe659a61640 1 Processor -- start 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.649+0000 7fe659a61640 1 -- start start 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe659a61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 0x7fe65414dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe659a61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6540cfae0 0x7fe6540c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe659a61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe654154a80 0x7fe6540c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe659a61640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ce060 con 0x7fe6540cf650 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe659a61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540cdd80 con 0x7fe6540cfae0 2024-03-04T10:04:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe659a61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540ce340 con 0x7fe654154a80 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 0x7fe65414dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 0x7fe65414dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43792/0 (socket says 172.21.15.99:43792) 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 -- 172.21.15.99:0/3481814329 learned_addr learned my addr 172.21.15.99:0/3481814329 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe6527fc640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6540cfae0 0x7fe6540c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe6537fe640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe654154a80 0x7fe6540c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 -- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe654154a80 msgr2=0x7fe6540c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe654154a80 0x7fe6540c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 -- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6540cfae0 msgr2=0x7fe6540c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6540cfae0 0x7fe6540c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 -- 172.21.15.99:0/3481814329 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540b58c0 con 0x7fe6540cf650 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe6537fe640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe654154a80 0x7fe6540c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe652ffd640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 0x7fe65414dc60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe644037040 tx=0x7fe644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.650+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644039050 con 0x7fe6540cf650 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.651+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644045070 con 0x7fe6540cf650 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.651+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6540b5ba0 con 0x7fe6540cf650 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.651+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64403ea50 con 0x7fe6540cf650 2024-03-04T10:04:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.651+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6540b6160 con 0x7fe6540cf650 2024-03-04T10:04:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.652+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6540d0670 con 0x7fe6540cf650 2024-03-04T10:04:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.652+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe64402cb20 con 0x7fe6540cf650 2024-03-04T10:04:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.652+0000 7fe63bfff640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe64006b4d0 0x7fe64006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.652+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe6440acf80 con 0x7fe6540cf650 2024-03-04T10:04:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.653+0000 7fe6527fc640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe64006b4d0 0x7fe64006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.653+0000 7fe6527fc640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe64006b4d0 0x7fe64006d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe6540cb9b0 tx=0x7fe648009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.657+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe6540d0670 con 0x7fe6540cf650 2024-03-04T10:04:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.894+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fe654074f60 con 0x7fe6540cf650 2024-03-04T10:04:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.896+0000 7fe63bfff640 1 -- 172.21.15.99:0/3481814329 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe64407b8e0 con 0x7fe6540cf650 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.898+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe64006b4d0 msgr2=0x7fe64006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.898+0000 7fe659a61640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe64006b4d0 0x7fe64006d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe6540cb9b0 tx=0x7fe648009040 comp rx=0 tx=0).stop 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.898+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 msgr2=0x7fe65414dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.898+0000 7fe659a61640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 0x7fe65414dc60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe644037040 tx=0x7fe644004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 shutdown_connections 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe64006b4d0 0x7fe64006d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe654154a80 0x7fe6540c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe6540cfae0 0x7fe6540c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 --2- 172.21.15.99:0/3481814329 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6540cf650 0x7fe65414dc60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 >> 172.21.15.99:0/3481814329 conn(0x7fe65406f5f0 msgr2=0x7fe6540d1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 shutdown_connections 2024-03-04T10:04:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:46.899+0000 7fe659a61640 1 -- 172.21.15.99:0/3481814329 wait complete. 2024-03-04T10:04:46.911 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-04T10:04:47.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.009+0000 7f69f7d59640 1 Processor -- start 2024-03-04T10:04:47.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.009+0000 7f69f7d59640 1 -- start start 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f7d59640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f69f00aeb20 0x7f69f0071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f7d59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f0150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f7d59640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f015da40 con 0x7f69f015d660 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f7d59640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f015cbc0 con 0x7f69f00aeb20 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f7d59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00af090 con 0x7f69f0071890 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f69f00aeb20 0x7f69f0071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f69f00aeb20 0x7f69f0071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43794/0 (socket says 172.21.15.99:43794) 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 -- 172.21.15.99:0/1670629698 learned_addr learned my addr 172.21.15.99:0/1670629698 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f62f6640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f0150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 -- 172.21.15.99:0/1670629698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 msgr2=0x7f69f0150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:47.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f0150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 -- 172.21.15.99:0/1670629698 >> v1:172.21.15.162:6789/0 conn(0x7f69f015d660 legacy=0x7f69f015e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:47.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 -- 172.21.15.99:0/1670629698 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f015a9f0 con 0x7f69f00aeb20 2024-03-04T10:04:47.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f62f6640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f0150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:47.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f52f4640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f00aeb20 0x7f69f0071350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009870 tx=0x7f69e002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3769c44f273e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f4af3640 1 -- 172.21.15.99:0/1670629698 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e0038070 con 0x7f69f00aeb20 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f4af3640 1 -- 172.21.15.99:0/1670629698 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e0031740 con 0x7f69f00aeb20 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.011+0000 7f69f4af3640 1 -- 172.21.15.99:0/1670629698 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e002f050 con 0x7f69f00aeb20 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 -- 172.21.15.99:0/1670629698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f00aeb20 msgr2=0x7f69f0071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f00aeb20 0x7f69f0071350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009870 tx=0x7f69e002b7e0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 -- 172.21.15.99:0/1670629698 shutdown_connections 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f0150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 --2- 172.21.15.99:0/1670629698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f00aeb20 0x7f69f0071350 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 -- 172.21.15.99:0/1670629698 >> 172.21.15.99:0/1670629698 conn(0x7f69f006acd0 msgr2=0x7f69f006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:47.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 -- 172.21.15.99:0/1670629698 shutdown_connections 2024-03-04T10:04:47.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.012+0000 7f69f7d59640 1 -- 172.21.15.99:0/1670629698 wait complete. 2024-03-04T10:04:47.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.013+0000 7f69f7d59640 1 Processor -- start 2024-03-04T10:04:47.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.013+0000 7f69f7d59640 1 -- start start 2024-03-04T10:04:47.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f7d59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f014baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f7d59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f00aeb20 0x7f69f00cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f7d59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 0x7f69f0148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f7d59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f015a660 con 0x7f69f015d660 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f7d59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f015afb0 con 0x7f69f00aeb20 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f7d59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f015b4d0 con 0x7f69f0071890 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 0x7f69f0148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 0x7f69f0148cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43796/0 (socket says 172.21.15.99:43796) 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 -- 172.21.15.99:0/2011427258 learned_addr learned my addr 172.21.15.99:0/2011427258 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f52f4640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f00aeb20 0x7f69f00cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f5af5640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f014baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 -- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 msgr2=0x7f69f014baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f014baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 -- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f00aeb20 msgr2=0x7f69f00cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f00aeb20 0x7f69f00cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 -- 172.21.15.99:0/2011427258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f00cf6b0 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f5af5640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f014baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.014+0000 7f69f62f6640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 0x7f69f0148cf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c7c0 tx=0x7f69e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.015+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8015020 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.015+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8004230 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.015+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f00b32b0 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.015+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8003610 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.015+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f0159bf0 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.016+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f00b37d0 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.016+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f69e8018020 con 0x7f69f015d660 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.016+0000 7f69e67fc640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f69d406b4d0 0x7f69d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.016+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f69e8089f70 con 0x7f69f015d660 2024-03-04T10:04:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.017+0000 7f69f5af5640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f69d406b4d0 0x7f69d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.017+0000 7f69f5af5640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f69d406b4d0 0x7f69d406d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f69f015c7d0 tx=0x7f69ec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.021+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f69f00b37d0 con 0x7f69f015d660 2024-03-04T10:04:47.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.257+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f69f0072460 con 0x7f69f015d660 2024-03-04T10:04:47.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.258+0000 7f69e67fc640 1 -- 172.21.15.99:0/2011427258 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f69e801f020 con 0x7f69f015d660 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f69d406b4d0 msgr2=0x7f69d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f69d406b4d0 0x7f69d406d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f69f015c7d0 tx=0x7f69ec009040 comp rx=0 tx=0).stop 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 msgr2=0x7f69f0148cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 0x7f69f0148cf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f69e800c7c0 tx=0x7f69e800cc80 comp rx=0 tx=0).stop 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 shutdown_connections 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f69d406b4d0 0x7f69d406d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f69f015d660 0x7f69f0148cf0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.260+0000 7f69f7d59640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f69f00aeb20 0x7f69f00cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.261+0000 7f69f7d59640 1 --2- 172.21.15.99:0/2011427258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f69f0071890 0x7f69f014baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.261+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 >> 172.21.15.99:0/2011427258 conn(0x7f69f006acd0 msgr2=0x7f69f006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.261+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 shutdown_connections 2024-03-04T10:04:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.261+0000 7f69f7d59640 1 -- 172.21.15.99:0/2011427258 wait complete. 2024-03-04T10:04:47.273 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-04T10:04:47.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.372+0000 7f8aa770a640 1 Processor -- start 2024-03-04T10:04:47.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.372+0000 7f8aa770a640 1 -- start start 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa770a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 0x7f8aa00cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa770a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8aa006d3b0 0x7f8aa00cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa770a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa00d0140 con 0x7f8aa00ce9c0 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa770a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa015cbb0 con 0x7f8aa006d3b0 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa770a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa015c280 con 0x7f8aa00cfd60 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa54a6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 0x7f8aa00cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa4ca5640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8aa006d3b0 0x7f8aa00cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa54a6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 0x7f8aa00cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:36116/0 (socket says 172.21.15.99:36116) 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa54a6640 1 -- 172.21.15.99:0/2574360278 learned_addr learned my addr 172.21.15.99:0/2574360278 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa5ca7640 1 --1- 172.21.15.99:0/2574360278 >> v1:172.21.15.162:6789/0 conn(0x7f8aa00ce9c0 0x7f8aa00cee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:39448/0 (socket says 172.21.15.99:39448) 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa4ca5640 1 -- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 msgr2=0x7f8aa00cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa4ca5640 1 --2- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 0x7f8aa00cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.373+0000 7f8aa4ca5640 1 -- 172.21.15.99:0/2574360278 >> v1:172.21.15.162:6789/0 conn(0x7f8aa00ce9c0 legacy=0x7f8aa00cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.374+0000 7f8a97fff640 1 -- 172.21.15.99:0/2574360278 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771520271 0 0) 0x7f8aa00d0140 con 0x7f8aa00ce9c0 2024-03-04T10:04:47.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.374+0000 7f8aa4ca5640 1 -- 172.21.15.99:0/2574360278 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa015c560 con 0x7f8aa006d3b0 2024-03-04T10:04:47.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.374+0000 7f8aa54a6640 1 --2- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 0x7f8aa00cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.374+0000 7f8aa4ca5640 1 --2- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa006d3b0 0x7f8aa00cc330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aa015cd30 tx=0x7f8a9002b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a28dbcf784c1b8f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.374+0000 7f8a97fff640 1 -- 172.21.15.99:0/2574360278 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90035020 con 0x7f8aa006d3b0 2024-03-04T10:04:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.374+0000 7f8a97fff640 1 -- 172.21.15.99:0/2574360278 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a90031800 con 0x7f8aa006d3b0 2024-03-04T10:04:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8a97fff640 1 -- 172.21.15.99:0/2574360278 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90031bd0 con 0x7f8aa006d3b0 2024-03-04T10:04:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 -- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa006d3b0 msgr2=0x7f8aa00cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 --2- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa006d3b0 0x7f8aa00cc330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aa015cd30 tx=0x7f8a9002b8a0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 -- 172.21.15.99:0/2574360278 shutdown_connections 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 --2- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa006d3b0 0x7f8aa00cc330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 --2- 172.21.15.99:0/2574360278 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00cfd60 0x7f8aa00cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 -- 172.21.15.99:0/2574360278 >> 172.21.15.99:0/2574360278 conn(0x7f8aa006acd0 msgr2=0x7f8aa006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 -- 172.21.15.99:0/2574360278 shutdown_connections 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.375+0000 7f8aa770a640 1 -- 172.21.15.99:0/2574360278 wait complete. 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.376+0000 7f8aa770a640 1 Processor -- start 2024-03-04T10:04:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.376+0000 7f8aa770a640 1 -- start start 2024-03-04T10:04:47.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa770a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 0x7f8aa0140120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa770a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00ce9c0 0x7f8aa015de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa770a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa00cfd60 0x7f8aa015e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa770a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa015f520 con 0x7f8aa006d3b0 2024-03-04T10:04:47.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa770a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa015bf30 con 0x7f8aa00cfd60 2024-03-04T10:04:47.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa770a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa015f6a0 con 0x7f8aa00ce9c0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 0x7f8aa0140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 0x7f8aa0140120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43802/0 (socket says 172.21.15.99:43802) 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 -- 172.21.15.99:0/1558666830 learned_addr learned my addr 172.21.15.99:0/1558666830 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa5ca7640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa00cfd60 0x7f8aa015e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa4ca5640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00ce9c0 0x7f8aa015de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 -- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00ce9c0 msgr2=0x7f8aa015de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00ce9c0 0x7f8aa015de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 -- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa00cfd60 msgr2=0x7f8aa015e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa00cfd60 0x7f8aa015e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.377+0000 7f8aa54a6640 1 -- 172.21.15.99:0/1558666830 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0140660 con 0x7f8aa006d3b0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.378+0000 7f8aa54a6640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 0x7f8aa0140120 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00e9b0 tx=0x7f8a9c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.378+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c017070 con 0x7f8aa006d3b0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.378+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c00c9a0 con 0x7f8aa006d3b0 2024-03-04T10:04:47.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.378+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa00b1c10 con 0x7f8aa006d3b0 2024-03-04T10:04:47.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.378+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c00cd50 con 0x7f8aa006d3b0 2024-03-04T10:04:47.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.378+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0156690 con 0x7f8aa006d3b0 2024-03-04T10:04:47.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.379+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa0074f60 con 0x7f8aa006d3b0 2024-03-04T10:04:47.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.379+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8a9c01a070 con 0x7f8aa006d3b0 2024-03-04T10:04:47.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.380+0000 7f8a95ffb640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a8406b4d0 0x7f8a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.380+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8a9c089e40 con 0x7f8aa006d3b0 2024-03-04T10:04:47.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.380+0000 7f8aa4ca5640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a8406b4d0 0x7f8a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.381+0000 7f8aa4ca5640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a8406b4d0 0x7f8a8406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a90036000 tx=0x7f8a90008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.385+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a9c058f10 con 0x7f8aa006d3b0 2024-03-04T10:04:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.620+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f8aa006ed90 con 0x7f8aa006d3b0 2024-03-04T10:04:47.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.622+0000 7f8a95ffb640 1 -- 172.21.15.99:0/1558666830 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8a9c058810 con 0x7f8aa006d3b0 2024-03-04T10:04:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a8406b4d0 msgr2=0x7f8a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a8406b4d0 0x7f8a8406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8a90036000 tx=0x7f8a90008030 comp rx=0 tx=0).stop 2024-03-04T10:04:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 msgr2=0x7f8aa0140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 0x7f8aa0140120 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00e9b0 tx=0x7f8a9c00ee70 comp rx=0 tx=0).stop 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 shutdown_connections 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a8406b4d0 0x7f8a8406d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa00cfd60 0x7f8aa015e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8aa00ce9c0 0x7f8aa015de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 --2- 172.21.15.99:0/1558666830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8aa006d3b0 0x7f8aa0140120 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 >> 172.21.15.99:0/1558666830 conn(0x7f8aa006acd0 msgr2=0x7f8aa006e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 shutdown_connections 2024-03-04T10:04:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.625+0000 7f8aa770a640 1 -- 172.21.15.99:0/1558666830 wait complete. 2024-03-04T10:04:47.638 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-04T10:04:47.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.738+0000 7ff8ba390640 1 Processor -- start 2024-03-04T10:04:47.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.738+0000 7ff8ba390640 1 -- start start 2024-03-04T10:04:47.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8ba390640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 0x7ff8b4071e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8ba390640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40ad560 con 0x7ff8b40acda0 2024-03-04T10:04:47.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8ba390640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b40ad840 con 0x7ff8b40cfb00 2024-03-04T10:04:47.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8ba390640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40adb20 con 0x7ff8b415db80 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b37fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff8b40acda0 0x7ff8b40cf3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60328/0 (socket says 172.21.15.99:60328) 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b37fe640 1 -- 172.21.15.99:0/2249955844 learned_addr learned my addr 172.21.15.99:0/2249955844 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b3fff640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 0x7ff8b4071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529504771 0 0) 0x7ff8b40ad560 con 0x7ff8b40acda0 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a0003520 con 0x7ff8b40acda0 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1681579029 0 0) 0x7ff8a0003520 con 0x7ff8b40acda0 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 msgr2=0x7ff8b4071e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 0x7ff8b4071e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 >> v1:172.21.15.162:6789/0 conn(0x7ff8b40cfb00 legacy=0x7ff8b406d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b40ade00 con 0x7ff8b40acda0 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7ff89c001670 con 0x7ff8b40acda0 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 >> v1:172.21.15.99:6789/0 conn(0x7ff8b40acda0 legacy=0x7ff8b40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 0x7ff8a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8a0003f60 0x7ff8a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8a0004d40 0x7ff8a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40adb20 con 0x7ff8a0004d40 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a000baa0 con 0x7ff8a0003f60 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.739+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a000bca0 con 0x7ff8b415db80 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b892d640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8a0004d40 0x7ff8a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b3fff640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 0x7ff8a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b37fe640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8a0003f60 0x7ff8a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b892d640 1 -- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 msgr2=0x7ff8a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b892d640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b415db80 0x7ff8a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b892d640 1 -- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8a0003f60 msgr2=0x7ff8a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b892d640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8a0003f60 0x7ff8a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.740+0000 7ff8b892d640 1 -- 172.21.15.99:0/2249955844 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0003520 con 0x7ff8a0004d40 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8b892d640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8a0004d40 0x7ff8a00051c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40cf1b0 tx=0x7ff8a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a770f44cb3ac7c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac001a70 con 0x7ff8a0004d40 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac003b70 con 0x7ff8a0004d40 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8b2ffd640 1 -- 172.21.15.99:0/2249955844 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0050d0 con 0x7ff8a0004d40 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8ba390640 1 -- 172.21.15.99:0/2249955844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8a0004d40 msgr2=0x7ff8a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8ba390640 1 --2- 172.21.15.99:0/2249955844 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8a0004d40 0x7ff8a00051c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff8b40cf1b0 tx=0x7ff8a0005830 comp rx=0 tx=0).stop 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8b892d640 1 -- 172.21.15.99:0/2249955844 reap_dead start 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8ba390640 1 -- 172.21.15.99:0/2249955844 shutdown_connections 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8ba390640 1 -- 172.21.15.99:0/2249955844 >> 172.21.15.99:0/2249955844 conn(0x7ff8b406acd0 msgr2=0x7ff8b406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8ba390640 1 -- 172.21.15.99:0/2249955844 shutdown_connections 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.741+0000 7ff8ba390640 1 -- 172.21.15.99:0/2249955844 wait complete. 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.742+0000 7ff8ba390640 1 Processor -- start 2024-03-04T10:04:47.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 -- start start 2024-03-04T10:04:47.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b4153e70 0x7ff8b413dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4154250 0x7ff8b4151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 0x7ff8b40b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b415df60 con 0x7ff8b40b0200 2024-03-04T10:04:47.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b40ad200 con 0x7ff8b4154250 2024-03-04T10:04:47.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8ba390640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4071e80 con 0x7ff8b4153e70 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8b892d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 0x7ff8b40b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8b892d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 0x7ff8b40b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43810/0 (socket says 172.21.15.99:43810) 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8b892d640 1 -- 172.21.15.99:0/4121036740 learned_addr learned my addr 172.21.15.99:0/4121036740 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8b3fff640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b4153e70 0x7ff8b413dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.743+0000 7ff8b37fe640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4154250 0x7ff8b4151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b892d640 1 -- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b4153e70 msgr2=0x7ff8b413dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b892d640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b4153e70 0x7ff8b413dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b892d640 1 -- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4154250 msgr2=0x7ff8b4151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b892d640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4154250 0x7ff8b4151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b892d640 1 -- 172.21.15.99:0/4121036740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b41520b0 con 0x7ff8b40b0200 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b3fff640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b4153e70 0x7ff8b413dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b37fe640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4154250 0x7ff8b4151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b892d640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 0x7ff8b40b06b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac02c020 tx=0x7ff8ac0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac001a70 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac003970 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b413e510 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac007db0 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.744+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b415b230 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.745+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b4074f60 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.745+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff8ac041030 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.746+0000 7ff8b0ff9640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff89406b500 0x7ff89406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.746+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff8ac0aaf50 con 0x7ff8b40b0200 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.746+0000 7ff8b3fff640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff89406b500 0x7ff89406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:47.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.747+0000 7ff8b3fff640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff89406b500 0x7ff89406d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8b415d460 tx=0x7ff8a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:47.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.750+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff8ac033020 con 0x7ff8b40b0200 2024-03-04T10:04:47.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.984+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ff8b406eaa0 con 0x7ff8b40b0200 2024-03-04T10:04:47.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.986+0000 7ff8b0ff9640 1 -- 172.21.15.99:0/4121036740 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff8ac003b40 con 0x7ff8b40b0200 2024-03-04T10:04:47.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff89406b500 msgr2=0x7ff89406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff89406b500 0x7ff89406d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff8b415d460 tx=0x7ff8a4008040 comp rx=0 tx=0).stop 2024-03-04T10:04:47.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 msgr2=0x7ff8b40b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:47.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 0x7ff8b40b06b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac02c020 tx=0x7ff8ac0014a0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 shutdown_connections 2024-03-04T10:04:47.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff89406b500 0x7ff89406d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8b40b0200 0x7ff8b40b06b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8b4154250 0x7ff8b4151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 --2- 172.21.15.99:0/4121036740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8b4153e70 0x7ff8b413dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:47.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 >> 172.21.15.99:0/4121036740 conn(0x7ff8b406acd0 msgr2=0x7ff8b4071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:47.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 shutdown_connections 2024-03-04T10:04:47.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:47.988+0000 7ff8ba390640 1 -- 172.21.15.99:0/4121036740 wait complete. 2024-03-04T10:04:48.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-04T10:04:48.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.099+0000 7f1ff45cd640 1 Processor -- start 2024-03-04T10:04:48.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.099+0000 7f1ff45cd640 1 -- start start 2024-03-04T10:04:48.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.100+0000 7f1ff45cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d01d0 0x7f1fec06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.100+0000 7f1ff45cd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1fec06b3e0 0x7f1fec162430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.100+0000 7f1ff45cd640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec0d05b0 con 0x7f1fec0d12a0 2024-03-04T10:04:48.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.100+0000 7f1ff45cd640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec06bbd0 con 0x7f1fec06b3e0 2024-03-04T10:04:48.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.100+0000 7f1ff45cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec070180 con 0x7f1fec0d01d0 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff2369640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d01d0 0x7f1fec06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff2b6a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1fec06b3e0 0x7f1fec162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff1b68640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f1fec0d12a0 0x7f1fec06acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:39456/0 (socket says 172.21.15.99:39456) 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff1b68640 1 -- 172.21.15.99:0/287625738 learned_addr learned my addr 172.21.15.99:0/287625738 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff1367640 1 -- 172.21.15.99:0/287625738 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082719493 0 0) 0x7f1fec0d05b0 con 0x7f1fec0d12a0 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff1367640 1 -- 172.21.15.99:0/287625738 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd8003520 con 0x7f1fec0d12a0 2024-03-04T10:04:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff2b6a640 1 -- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d01d0 msgr2=0x7f1fec06faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff2b6a640 1 --2- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d01d0 0x7f1fec06faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff2b6a640 1 -- 172.21.15.99:0/287625738 >> v1:172.21.15.162:6789/0 conn(0x7f1fec0d12a0 legacy=0x7f1fec06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.101+0000 7f1ff2b6a640 1 -- 172.21.15.99:0/287625738 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec070460 con 0x7f1fec06b3e0 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.102+0000 7f1ff2b6a640 1 --2- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec162430 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40097a0 tx=0x7f1fe402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d98f096b152de9a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.102+0000 7f1ff1367640 1 -- 172.21.15.99:0/287625738 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4035020 con 0x7f1fec06b3e0 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.102+0000 7f1ff1367640 1 -- 172.21.15.99:0/287625738 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4031830 con 0x7f1fec06b3e0 2024-03-04T10:04:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.102+0000 7f1ff1367640 1 -- 172.21.15.99:0/287625738 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4031c00 con 0x7f1fec06b3e0 2024-03-04T10:04:48.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 -- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 msgr2=0x7f1fec162430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec162430 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fe40097a0 tx=0x7f1fe402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 -- 172.21.15.99:0/287625738 shutdown_connections 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec162430 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/287625738 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d01d0 0x7f1fec06faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 -- 172.21.15.99:0/287625738 >> 172.21.15.99:0/287625738 conn(0x7f1fec0d0780 msgr2=0x7f1fec0d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 -- 172.21.15.99:0/287625738 shutdown_connections 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.103+0000 7f1ff45cd640 1 -- 172.21.15.99:0/287625738 wait complete. 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.104+0000 7f1ff45cd640 1 Processor -- start 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.104+0000 7f1ff45cd640 1 -- start start 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec0c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 0x7f1fec14bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d12a0 0x7f1fec15bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec161f10 con 0x7f1fec0d01d0 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec161c30 con 0x7f1fec06b3e0 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0cadd0 con 0x7f1fec0d12a0 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 0x7f1fec14bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 0x7f1fec14bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43822/0 (socket says 172.21.15.99:43822) 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 -- 172.21.15.99:0/1883388243 learned_addr learned my addr 172.21.15.99:0/1883388243 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff2369640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec0c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff2b6a640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d12a0 0x7f1fec15bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 -- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d12a0 msgr2=0x7f1fec15bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d12a0 0x7f1fec15bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 -- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 msgr2=0x7f1fec0c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec0c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 -- 172.21.15.99:0/1883388243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec14c0a0 con 0x7f1fec0d01d0 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff2369640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff1b68640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 0x7f1fec14bb60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00c390 tx=0x7f1fdc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc017070 con 0x7f1fec0d01d0 2024-03-04T10:04:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc005ce0 con 0x7f1fec0d01d0 2024-03-04T10:04:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0072f0 con 0x7f1fec0d01d0 2024-03-04T10:04:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.105+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec0c5ec0 con 0x7f1fec0d01d0 2024-03-04T10:04:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.106+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec0c6480 con 0x7f1fec0d01d0 2024-03-04T10:04:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.106+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec074f60 con 0x7f1fec0d01d0 2024-03-04T10:04:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.107+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1fdc01a070 con 0x7f1fec0d01d0 2024-03-04T10:04:48.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.107+0000 7f1fe2ffd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.107+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1fdc0893c0 con 0x7f1fec0d01d0 2024-03-04T10:04:48.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.107+0000 7f1ff2369640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.108+0000 7f1ff2369640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fd006b4d0 0x7f1fd006d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8007520 tx=0x7f1fe800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.112+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1fdc058490 con 0x7f1fec0d01d0 2024-03-04T10:04:48.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.349+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f1fec0c8f40 con 0x7f1fec0d01d0 2024-03-04T10:04:48.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.351+0000 7f1fe2ffd640 1 -- 172.21.15.99:0/1883388243 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1fdc057d90 con 0x7f1fec0d01d0 2024-03-04T10:04:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fd006b4d0 msgr2=0x7f1fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fd006b4d0 0x7f1fd006d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8007520 tx=0x7f1fe800b040 comp rx=0 tx=0).stop 2024-03-04T10:04:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 msgr2=0x7f1fec14bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 0x7f1fec14bb60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00c390 tx=0x7f1fdc00c850 comp rx=0 tx=0).stop 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 shutdown_connections 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fec0d12a0 0x7f1fec15bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fec0d01d0 0x7f1fec14bb60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 --2- 172.21.15.99:0/1883388243 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fec06b3e0 0x7f1fec0c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 >> 172.21.15.99:0/1883388243 conn(0x7f1fec0d0780 msgr2=0x7f1fec0ad530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 shutdown_connections 2024-03-04T10:04:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.353+0000 7f1ff45cd640 1 -- 172.21.15.99:0/1883388243 wait complete. 2024-03-04T10:04:48.365 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-04T10:04:48.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.465+0000 7f5d9a097640 1 Processor -- start 2024-03-04T10:04:48.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.465+0000 7f5d9a097640 1 -- start start 2024-03-04T10:04:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d9a097640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d9a097640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5d94070de0 0x7f5d94071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d9a097640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d9415f020 con 0x7f5d94154a80 2024-03-04T10:04:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d9a097640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d940cfff0 con 0x7f5d94070de0 2024-03-04T10:04:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d9a097640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940cd010 con 0x7f5d940cf650 2024-03-04T10:04:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5d94070de0 0x7f5d94071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5d94070de0 0x7f5d94071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43834/0 (socket says 172.21.15.99:43834) 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 -- 172.21.15.99:0/3972185363 learned_addr learned my addr 172.21.15.99:0/3972185363 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 -- 172.21.15.99:0/3972185363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 msgr2=0x7f5d940cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 --2- 172.21.15.99:0/3972185363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 -- 172.21.15.99:0/3972185363 >> v1:172.21.15.162:6789/0 conn(0x7f5d94154a80 legacy=0x7f5d9415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.466+0000 7f5d93fff640 1 -- 172.21.15.99:0/3972185363 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940cdaa0 con 0x7f5d94070de0 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.467+0000 7f5d93fff640 1 --2- 172.21.15.99:0/3972185363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94070de0 0x7f5d94071270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009e50 tx=0x7f5d8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b245802598acff64 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.467+0000 7f5d927fc640 1 -- 172.21.15.99:0/3972185363 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c02ca70 con 0x7f5d94070de0 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.467+0000 7f5d927fc640 1 -- 172.21.15.99:0/3972185363 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c02cc10 con 0x7f5d94070de0 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.467+0000 7f5d927fc640 1 -- 172.21.15.99:0/3972185363 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0343f0 con 0x7f5d94070de0 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 -- 172.21.15.99:0/3972185363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94070de0 msgr2=0x7f5d94071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3972185363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94070de0 0x7f5d94071270 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009e50 tx=0x7f5d8c02bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 -- 172.21.15.99:0/3972185363 shutdown_connections 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3972185363 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94070de0 0x7f5d94071270 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3972185363 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 -- 172.21.15.99:0/3972185363 >> 172.21.15.99:0/3972185363 conn(0x7f5d9406f5f0 msgr2=0x7f5d9406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 -- 172.21.15.99:0/3972185363 shutdown_connections 2024-03-04T10:04:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.468+0000 7f5d9a097640 1 -- 172.21.15.99:0/3972185363 wait complete. 2024-03-04T10:04:48.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.469+0000 7f5d9a097640 1 Processor -- start 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.469+0000 7f5d9a097640 1 -- start start 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d94070de0 0x7f5d940b53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 0x7f5d9415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ce340 con 0x7f5d94154a80 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ce060 con 0x7f5d94070de0 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d940ce620 con 0x7f5d940cf650 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 0x7f5d9415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 0x7f5d9415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43836/0 (socket says 172.21.15.99:43836) 2024-03-04T10:04:48.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 -- 172.21.15.99:0/3733986085 learned_addr learned my addr 172.21.15.99:0/3733986085 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d92ffd640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d937fe640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d94070de0 0x7f5d940b53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 -- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 msgr2=0x7f5d940b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 -- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d94070de0 msgr2=0x7f5d940b53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d94070de0 0x7f5d940b53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 -- 172.21.15.99:0/3733986085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940b4610 con 0x7f5d94154a80 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d92ffd640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d93fff640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 0x7f5d9415ae30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009e50 tx=0x7f5d8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c043020 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c03e450 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d940b48f0 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c009510 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.470+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d940b4eb0 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.471+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d94074f60 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.472+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5d8c046020 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.472+0000 7f5d7bfff640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d8006b4d0 0x7f5d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.472+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5d8c0ac8f0 con 0x7f5d94154a80 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.472+0000 7f5d937fe640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d8006b4d0 0x7f5d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.473+0000 7f5d937fe640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d8006b4d0 0x7f5d8006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5d84006fd0 tx=0x7f5d84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.477+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5d8c04d030 con 0x7f5d94154a80 2024-03-04T10:04:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.709+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f5d9415e6f0 con 0x7f5d94154a80 2024-03-04T10:04:48.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.711+0000 7f5d7bfff640 1 -- 172.21.15.99:0/3733986085 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5d9415f020 con 0x7f5d94154a80 2024-03-04T10:04:48.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d8006b4d0 msgr2=0x7f5d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d8006b4d0 0x7f5d8006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5d84006fd0 tx=0x7f5d84008040 comp rx=0 tx=0).stop 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 msgr2=0x7f5d9415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 0x7f5d9415ae30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009e50 tx=0x7f5d8c004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 shutdown_connections 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5d8006b4d0 0x7f5d8006d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d94154a80 0x7f5d9415ae30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5d940cf650 0x7f5d940b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 --2- 172.21.15.99:0/3733986085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5d94070de0 0x7f5d940b53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 >> 172.21.15.99:0/3733986085 conn(0x7f5d9406f5f0 msgr2=0x7f5d9406c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 shutdown_connections 2024-03-04T10:04:48.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.713+0000 7f5d9a097640 1 -- 172.21.15.99:0/3733986085 wait complete. 2024-03-04T10:04:48.725 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:48.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.825+0000 7fc329362640 1 Processor -- start 2024-03-04T10:04:48.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.825+0000 7fc329362640 1 -- start start 2024-03-04T10:04:48.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc329362640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc3240b2e60 0x7fc3240b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc329362640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc3240b5470 0x7fc3240b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc329362640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32415e1e0 con 0x7fc32415de00 2024-03-04T10:04:48.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc329362640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3240b98c0 con 0x7fc3240b2e60 2024-03-04T10:04:48.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc329362640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3240b32c0 con 0x7fc3240b5470 2024-03-04T10:04:48.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc3240b2e60 0x7fc3240b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc3240b2e60 0x7fc3240b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43846/0 (socket says 172.21.15.99:43846) 2024-03-04T10:04:48.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 -- 172.21.15.99:0/241474830 learned_addr learned my addr 172.21.15.99:0/241474830 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:48.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3237fe640 1 --2- 172.21.15.99:0/241474830 >> v2:172.21.15.162:3300/0 conn(0x7fc3240b5470 0x7fc3240b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 -- 172.21.15.99:0/241474830 >> v2:172.21.15.162:3300/0 conn(0x7fc3240b5470 msgr2=0x7fc3240b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 --2- 172.21.15.99:0/241474830 >> v2:172.21.15.162:3300/0 conn(0x7fc3240b5470 0x7fc3240b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 -- 172.21.15.99:0/241474830 >> v1:172.21.15.162:6789/0 conn(0x7fc32415de00 legacy=0x7fc3240b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.826+0000 7fc3227fc640 1 -- 172.21.15.99:0/241474830 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3240d1c10 con 0x7fc3240b2e60 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.827+0000 7fc3227fc640 1 --2- 172.21.15.99:0/241474830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b2e60 0x7fc3240b9380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc3240b2d40 tx=0x7fc31802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd96d20e7b19f9f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.827+0000 7fc321ffb640 1 -- 172.21.15.99:0/241474830 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318034070 con 0x7fc3240b2e60 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.827+0000 7fc321ffb640 1 -- 172.21.15.99:0/241474830 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318030860 con 0x7fc3240b2e60 2024-03-04T10:04:48.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.827+0000 7fc321ffb640 1 -- 172.21.15.99:0/241474830 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318030c30 con 0x7fc3240b2e60 2024-03-04T10:04:48.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.827+0000 7fc329362640 1 -- 172.21.15.99:0/241474830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b2e60 msgr2=0x7fc3240b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.827+0000 7fc329362640 1 --2- 172.21.15.99:0/241474830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b2e60 0x7fc3240b9380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc3240b2d40 tx=0x7fc31802b9a0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.828+0000 7fc329362640 1 -- 172.21.15.99:0/241474830 shutdown_connections 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.828+0000 7fc329362640 1 --2- 172.21.15.99:0/241474830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b2e60 0x7fc3240b9380 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.828+0000 7fc329362640 1 --2- 172.21.15.99:0/241474830 >> v2:172.21.15.162:3300/0 conn(0x7fc3240b5470 0x7fc3240b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.828+0000 7fc329362640 1 -- 172.21.15.99:0/241474830 >> 172.21.15.99:0/241474830 conn(0x7fc3240d0470 msgr2=0x7fc3240d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.828+0000 7fc329362640 1 -- 172.21.15.99:0/241474830 shutdown_connections 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.828+0000 7fc329362640 1 -- 172.21.15.99:0/241474830 wait complete. 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.829+0000 7fc329362640 1 Processor -- start 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.829+0000 7fc329362640 1 -- start start 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3240b2e60 0x7fc3241f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 0x7fc3241517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc32415de00 0x7fc3240b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240d21d0 con 0x7fc3240b5470 2024-03-04T10:04:48.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240d2820 con 0x7fc3240b2e60 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240d24b0 con 0x7fc32415de00 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 0x7fc3241517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 0x7fc3241517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43848/0 (socket says 172.21.15.99:43848) 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 -- 172.21.15.99:0/2556598500 learned_addr learned my addr 172.21.15.99:0/2556598500 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 -- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc32415de00 msgr2=0x7fc3240b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc32415de00 0x7fc3240b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 -- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3240b2e60 msgr2=0x7fc3241f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3240b2e60 0x7fc3241f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:48.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 -- 172.21.15.99:0/2556598500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3240c7700 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc3227fc640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 0x7fc3241517f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc31802b9e0 tx=0x7fc318004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318048070 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318030660 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3240c79e0 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318034040 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.830+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc324151d30 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.831+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc324074f60 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.832+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc31804b070 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.832+0000 7fc30b7fe640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.832+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc3180ac2f0 con 0x7fc3240b5470 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.833+0000 7fc322ffd640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.833+0000 7fc322ffd640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc31006b4d0 0x7fc31006d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc314000f60 tx=0x7fc3140079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:48.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:48.837+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc31807b4e0 con 0x7fc3240b5470 2024-03-04T10:04:49.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.011+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3240c7010 con 0x7fc3240b5470 2024-03-04T10:04:49.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.011+0000 7fc30b7fe640 1 -- 172.21.15.99:0/2556598500 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+13128 (secure 0 0 0) 0x7fc318030830 con 0x7fc3240b5470 2024-03-04T10:04:49.013 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:49.013 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":34,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:46.163728+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:49.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc31006b4d0 msgr2=0x7fc31006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc31006b4d0 0x7fc31006d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc314000f60 tx=0x7fc3140079d0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 msgr2=0x7fc3241517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 0x7fc3241517f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc31802b9e0 tx=0x7fc318004730 comp rx=0 tx=0).stop 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 shutdown_connections 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc31006b4d0 0x7fc31006d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc32415de00 0x7fc3240b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc3240b5470 0x7fc3241517f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 --2- 172.21.15.99:0/2556598500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3240b2e60 0x7fc3241f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 >> 172.21.15.99:0/2556598500 conn(0x7fc3240d0470 msgr2=0x7fc3240b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:49.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 shutdown_connections 2024-03-04T10:04:49.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.014+0000 7fc329362640 1 -- 172.21.15.99:0/2556598500 wait complete. 2024-03-04T10:04:49.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:04:49.027 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:49.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.129+0000 7fc9bc068640 1 Processor -- start 2024-03-04T10:04:49.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.130+0000 7fc9bc068640 1 -- start start 2024-03-04T10:04:49.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.130+0000 7fc9bc068640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b40ccbd0 0x7fc9b40b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.130+0000 7fc9bc068640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc9b415c2e0 0x7fc9b40ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.130+0000 7fc9bc068640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b415ca50 con 0x7fc9b40cecf0 2024-03-04T10:04:49.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.130+0000 7fc9bc068640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b415d2f0 con 0x7fc9b415c2e0 2024-03-04T10:04:49.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.130+0000 7fc9bc068640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b415d6d0 con 0x7fc9b40ccbd0 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9ba605640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc9b40cecf0 0x7fc9b40b0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60344/0 (socket says 172.21.15.99:60344) 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9ba605640 1 -- 172.21.15.99:0/304083686 learned_addr learned my addr 172.21.15.99:0/304083686 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b9603640 1 --2- 172.21.15.99:0/304083686 >> v2:172.21.15.99:3300/0 conn(0x7fc9b415c2e0 0x7fc9b40ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b9e04640 1 --2- 172.21.15.99:0/304083686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b40ccbd0 0x7fc9b40b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b8e02640 1 -- 172.21.15.99:0/304083686 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754793141 0 0) 0x7fc9b415ca50 con 0x7fc9b40cecf0 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b8e02640 1 -- 172.21.15.99:0/304083686 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a0003520 con 0x7fc9b40cecf0 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b9603640 1 -- 172.21.15.99:0/304083686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b40ccbd0 msgr2=0x7fc9b40b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b9603640 1 --2- 172.21.15.99:0/304083686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b40ccbd0 0x7fc9b40b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b9603640 1 -- 172.21.15.99:0/304083686 >> v1:172.21.15.99:6789/0 conn(0x7fc9b40cecf0 legacy=0x7fc9b40b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.131+0000 7fc9b9603640 1 -- 172.21.15.99:0/304083686 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b415cd30 con 0x7fc9b415c2e0 2024-03-04T10:04:49.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.132+0000 7fc9b9603640 1 --2- 172.21.15.99:0/304083686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b415c2e0 0x7fc9b40ce7b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40097a0 tx=0x7fc9a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287533f9ab7a5f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.132+0000 7fc9b8e02640 1 -- 172.21.15.99:0/304083686 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4035020 con 0x7fc9b415c2e0 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.132+0000 7fc9b8e02640 1 -- 172.21.15.99:0/304083686 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a4031830 con 0x7fc9b415c2e0 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.132+0000 7fc9b8e02640 1 -- 172.21.15.99:0/304083686 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4031c00 con 0x7fc9b415c2e0 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.132+0000 7fc9bc068640 1 -- 172.21.15.99:0/304083686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b415c2e0 msgr2=0x7fc9b40ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.132+0000 7fc9bc068640 1 --2- 172.21.15.99:0/304083686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b415c2e0 0x7fc9b40ce7b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40097a0 tx=0x7fc9a402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.133+0000 7fc9bc068640 1 -- 172.21.15.99:0/304083686 shutdown_connections 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.133+0000 7fc9bc068640 1 --2- 172.21.15.99:0/304083686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b415c2e0 0x7fc9b40ce7b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.133+0000 7fc9bc068640 1 --2- 172.21.15.99:0/304083686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b40ccbd0 0x7fc9b40b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.133+0000 7fc9bc068640 1 -- 172.21.15.99:0/304083686 >> 172.21.15.99:0/304083686 conn(0x7fc9b4066dc0 msgr2=0x7fc9b40671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.133+0000 7fc9bc068640 1 -- 172.21.15.99:0/304083686 shutdown_connections 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.133+0000 7fc9bc068640 1 -- 172.21.15.99:0/304083686 wait complete. 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.134+0000 7fc9bc068640 1 Processor -- start 2024-03-04T10:04:49.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.134+0000 7fc9bc068640 1 -- start start 2024-03-04T10:04:49.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.134+0000 7fc9bc068640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b40ccbd0 0x7fc9b40c7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.134+0000 7fc9bc068640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 0x7fc9b40d0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9bc068640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b415c2e0 0x7fc9b41558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9bc068640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b40ccfb0 con 0x7fc9b40cecf0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9bc068640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b415c760 con 0x7fc9b40ccbd0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9bc068640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b40b0c70 con 0x7fc9b415c2e0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 0x7fc9b40d0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 0x7fc9b40d0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43864/0 (socket says 172.21.15.99:43864) 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 -- 172.21.15.99:0/1154110085 learned_addr learned my addr 172.21.15.99:0/1154110085 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9e04640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b40ccbd0 0x7fc9b40c7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9ba605640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b415c2e0 0x7fc9b41558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 -- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b415c2e0 msgr2=0x7fc9b41558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b415c2e0 0x7fc9b41558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 -- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b40ccbd0 msgr2=0x7fc9b40c7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b40ccbd0 0x7fc9b40c7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 -- 172.21.15.99:0/1154110085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4157530 con 0x7fc9b40cecf0 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9ba605640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b415c2e0 0x7fc9b41558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9b9603640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 0x7fc9b40d0f10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40097a0 tx=0x7fc9a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a403a8f0 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a403aa90 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b4157810 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40443f0 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.135+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b4157dd0 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.136+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b4074f60 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.137+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc9a404b060 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.137+0000 7fc9aa7fc640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc99806b4d0 0x7fc99806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.137+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc9a40aba40 con 0x7fc9b40cecf0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.137+0000 7fc9b9e04640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc99806b4d0 0x7fc99806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.138+0000 7fc9b9e04640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc99806b4d0 0x7fc99806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40d19b0 tx=0x7fc9b0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.141+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc9a403f070 con 0x7fc9b40cecf0 2024-03-04T10:04:49.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.310+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9b4158200 con 0x7fc9b40cecf0 2024-03-04T10:04:49.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.311+0000 7fc9aa7fc640 1 -- 172.21.15.99:0/1154110085 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9a40abe20 con 0x7fc9b40cecf0 2024-03-04T10:04:49.312 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc99806b4d0 msgr2=0x7fc99806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc99806b4d0 0x7fc99806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40d19b0 tx=0x7fc9b0009190 comp rx=0 tx=0).stop 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 msgr2=0x7fc9b40d0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 0x7fc9b40d0f10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40097a0 tx=0x7fc9a40046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 shutdown_connections 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc99806b4d0 0x7fc99806d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc9b415c2e0 0x7fc9b41558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc9b40cecf0 0x7fc9b40d0f10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 --2- 172.21.15.99:0/1154110085 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9b40ccbd0 0x7fc9b40c7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 >> 172.21.15.99:0/1154110085 conn(0x7fc9b4066dc0 msgr2=0x7fc9b406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:49.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 shutdown_connections 2024-03-04T10:04:49.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.313+0000 7fc9bc068640 1 -- 172.21.15.99:0/1154110085 wait complete. 2024-03-04T10:04:49.325 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T10:04:49.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.424+0000 7f968ad09640 1 Processor -- start 2024-03-04T10:04:49.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.424+0000 7f968ad09640 1 -- start start 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f968ad09640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9684071a60 0x7f9684150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f968ad09640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968415da40 con 0x7f968415d660 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f968ad09640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96840af090 con 0x7f96840aeb20 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f968ad09640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968415cbc0 con 0x7f9684071a60 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f96892a6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9684071a60 0x7f9684150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f96892a6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9684071a60 0x7f9684150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43872/0 (socket says 172.21.15.99:43872) 2024-03-04T10:04:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f9688aa5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f968415d660 0x7f968415e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60360/0 (socket says 172.21.15.99:60360) 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f96892a6640 1 -- 172.21.15.99:0/3212651901 learned_addr learned my addr 172.21.15.99:0/3212651901 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f96892a6640 1 -- 172.21.15.99:0/3212651901 >> v1:172.21.15.162:6789/0 conn(0x7f96840aeb20 legacy=0x7f9684071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f96892a6640 1 -- 172.21.15.99:0/3212651901 >> v1:172.21.15.99:6789/0 conn(0x7f968415d660 legacy=0x7f968415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.425+0000 7f96892a6640 1 -- 172.21.15.99:0/3212651901 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968415a9f0 con 0x7f9684071a60 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.426+0000 7f96892a6640 1 --2- 172.21.15.99:0/3212651901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9684071a60 0x7f9684150960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f967c009e50 tx=0x7f967c02cff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3d88d012cd5f49 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.426+0000 7f96837fe640 1 -- 172.21.15.99:0/3212651901 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c02db90 con 0x7f9684071a60 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.426+0000 7f96837fe640 1 -- 172.21.15.99:0/3212651901 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c02dd30 con 0x7f9684071a60 2024-03-04T10:04:49.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.426+0000 7f96837fe640 1 -- 172.21.15.99:0/3212651901 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c035610 con 0x7f9684071a60 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 -- 172.21.15.99:0/3212651901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9684071a60 msgr2=0x7f9684150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 --2- 172.21.15.99:0/3212651901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9684071a60 0x7f9684150960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f967c009e50 tx=0x7f967c02cff0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 -- 172.21.15.99:0/3212651901 shutdown_connections 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 --2- 172.21.15.99:0/3212651901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9684071a60 0x7f9684150960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 -- 172.21.15.99:0/3212651901 >> 172.21.15.99:0/3212651901 conn(0x7f968406acd0 msgr2=0x7f968406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 -- 172.21.15.99:0/3212651901 shutdown_connections 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.427+0000 7f968ad09640 1 -- 172.21.15.99:0/3212651901 wait complete. 2024-03-04T10:04:49.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.428+0000 7f968ad09640 1 Processor -- start 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.428+0000 7f968ad09640 1 -- start start 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9684071a60 0x7f9684149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 0x7f96840cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f968415d660 0x7f96840b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968415a660 con 0x7f96840aeb20 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968415afb0 con 0x7f9684071a60 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968415b4d0 con 0x7f968415d660 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 0x7f96840cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 0x7f96840cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43884/0 (socket says 172.21.15.99:43884) 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 -- 172.21.15.99:0/3197120024 learned_addr learned my addr 172.21.15.99:0/3197120024 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9688aa5640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9684071a60 0x7f9684149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f96892a6640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f968415d660 0x7f96840b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 -- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f968415d660 msgr2=0x7f96840b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f968415d660 0x7f96840b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 -- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9684071a60 msgr2=0x7f9684149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9684071a60 0x7f9684149920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 -- 172.21.15.99:0/3197120024 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9684149e60 con 0x7f96840aeb20 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9688aa5640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9684071a60 0x7f9684149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f96892a6640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f968415d660 0x7f96840b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f9683fff640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 0x7f96840cf860 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f967000c430 tx=0x7f967000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96700052d0 con 0x7f96840aeb20 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670005490 con 0x7f96840aeb20 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96841f4360 con 0x7f96840aeb20 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670007260 con 0x7f96840aeb20 2024-03-04T10:04:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.429+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96841f4920 con 0x7f96840aeb20 2024-03-04T10:04:49.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.430+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9684074f60 con 0x7f96840aeb20 2024-03-04T10:04:49.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.431+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9670007400 con 0x7f96840aeb20 2024-03-04T10:04:49.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.431+0000 7f96817fa640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f966806b4d0 0x7f966806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.431+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f96700894a0 con 0x7f96840aeb20 2024-03-04T10:04:49.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.432+0000 7f9688aa5640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f966806b4d0 0x7f966806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.432+0000 7f9688aa5640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f966806b4d0 0x7f966806d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9678000f60 tx=0x7f9678009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.435+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9684149e60 con 0x7f96840aeb20 2024-03-04T10:04:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.609+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9684072460 con 0x7f96840aeb20 2024-03-04T10:04:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.609+0000 7f96817fa640 1 -- 172.21.15.99:0/3197120024 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9670058570 con 0x7f96840aeb20 2024-03-04T10:04:49.610 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:49.611 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:04:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:04:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2024-03-04T10:04:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2024-03-04T10:04:49.611 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:04:49.611 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:04:49.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f966806b4d0 msgr2=0x7f966806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f966806b4d0 0x7f966806d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9678000f60 tx=0x7f9678009040 comp rx=0 tx=0).stop 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 msgr2=0x7f96840cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 0x7f96840cf860 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f967000c430 tx=0x7f967000c8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 shutdown_connections 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f966806b4d0 0x7f966806d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f968415d660 0x7f96840b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96840aeb20 0x7f96840cf860 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 --2- 172.21.15.99:0/3197120024 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9684071a60 0x7f9684149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 >> 172.21.15.99:0/3197120024 conn(0x7f968406acd0 msgr2=0x7f968406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 shutdown_connections 2024-03-04T10:04:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.611+0000 7f968ad09640 1 -- 172.21.15.99:0/3197120024 wait complete. 2024-03-04T10:04:49.623 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T10:04:49.623 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T10:04:49.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.726+0000 7fdcc4d37640 1 Processor -- start 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.726+0000 7fdcc4d37640 1 -- start start 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcc4d37640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdcc00671c0 0x7fdcc00b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcc4d37640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc006af20 con 0x7fdcc015dc80 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcc4d37640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc006b200 con 0x7fdcc00b2fe0 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcc4d37640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc006acd0 con 0x7fdcc00671c0 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcb7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdcc00671c0 0x7fdcc00b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcbe59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdcc015dc80 0x7fdcc006bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60370/0 (socket says 172.21.15.99:60370) 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcbe59c640 1 -- 172.21.15.99:0/4067025139 learned_addr learned my addr 172.21.15.99:0/4067025139 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcb7fff640 1 -- 172.21.15.99:0/4067025139 >> v1:172.21.15.162:6789/0 conn(0x7fdcc00b2fe0 legacy=0x7fdcc00ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcb7fff640 1 -- 172.21.15.99:0/4067025139 >> v1:172.21.15.99:6789/0 conn(0x7fdcc015dc80 legacy=0x7fdcc006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.727+0000 7fdcb7fff640 1 -- 172.21.15.99:0/4067025139 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc006b4e0 con 0x7fdcc00671c0 2024-03-04T10:04:49.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.728+0000 7fdcb7fff640 1 --2- 172.21.15.99:0/4067025139 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc00b48e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdca80097a0 tx=0x7fdca802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c553fe4785697211 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.728+0000 7fdcbdd9b640 1 -- 172.21.15.99:0/4067025139 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8034070 con 0x7fdcc00671c0 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.728+0000 7fdcbdd9b640 1 -- 172.21.15.99:0/4067025139 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca8030610 con 0x7fdcc00671c0 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.728+0000 7fdcbdd9b640 1 -- 172.21.15.99:0/4067025139 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca80309e0 con 0x7fdcc00671c0 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 -- 172.21.15.99:0/4067025139 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 msgr2=0x7fdcc00b48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/4067025139 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc00b48e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdca80097a0 tx=0x7fdca802b960 comp rx=0 tx=0).stop 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 -- 172.21.15.99:0/4067025139 shutdown_connections 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/4067025139 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc00b48e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 -- 172.21.15.99:0/4067025139 >> 172.21.15.99:0/4067025139 conn(0x7fdcc0071ca0 msgr2=0x7fdcc0072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 -- 172.21.15.99:0/4067025139 shutdown_connections 2024-03-04T10:04:49.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.729+0000 7fdcc4d37640 1 -- 172.21.15.99:0/4067025139 wait complete. 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.730+0000 7fdcc4d37640 1 Processor -- start 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.730+0000 7fdcc4d37640 1 -- start start 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc014c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc00b2fe0 0x7fdcc015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcc015e150 0x7fdcc0160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc006baa0 con 0x7fdcc00671c0 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00729b0 con 0x7fdcc00b2fe0 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b4e20 con 0x7fdcc015e150 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc014c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc014c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43908/0 (socket says 172.21.15.99:43908) 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 -- 172.21.15.99:0/907451786 learned_addr learned my addr 172.21.15.99:0/907451786 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:49.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcb7fff640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc00b2fe0 0x7fdcc015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbed9d640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcc015e150 0x7fdcc0160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 -- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcc015e150 msgr2=0x7fdcc0160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcc015e150 0x7fdcc0160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 -- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc00b2fe0 msgr2=0x7fdcc015ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc00b2fe0 0x7fdcc015ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 -- 172.21.15.99:0/907451786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc015f900 con 0x7fdcc00671c0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbed9d640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcc015e150 0x7fdcc0160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcbe59c640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc014c5b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000c480 tx=0x7fdcb000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0017070 con 0x7fdcc00671c0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0005ce0 con 0x7fdcc00671c0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc015fbe0 con 0x7fdcc00671c0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.731+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00072f0 con 0x7fdcc00671c0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.732+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc015f4a0 con 0x7fdcc00671c0 2024-03-04T10:04:49.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.733+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc0074f60 con 0x7fdcc00671c0 2024-03-04T10:04:49.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.733+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdcb001a070 con 0x7fdcc00671c0 2024-03-04T10:04:49.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.733+0000 7fdcb6ffd640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdca006b4d0 0x7fdca006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:49.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.733+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fdcb0089240 con 0x7fdcc00671c0 2024-03-04T10:04:49.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.734+0000 7fdcb7fff640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdca006b4d0 0x7fdca006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:49.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.734+0000 7fdcb7fff640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdca006b4d0 0x7fdca006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdca8036000 tx=0x7fdca803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:49.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.738+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdcb0057cc0 con 0x7fdcc00671c0 2024-03-04T10:04:49.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:49.908+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdcc01533f0 con 0x7fdcc00671c0 2024-03-04T10:04:50.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.162+0000 7fdcb6ffd640 1 -- 172.21.15.99:0/907451786 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdcb005bd60 con 0x7fdcc00671c0 2024-03-04T10:04:50.163 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2024-03-04T10:04:50.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdca006b4d0 msgr2=0x7fdca006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdca006b4d0 0x7fdca006d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fdca8036000 tx=0x7fdca803b040 comp rx=0 tx=0).stop 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 msgr2=0x7fdcc014c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc014c5b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000c480 tx=0x7fdcb000c940 comp rx=0 tx=0).stop 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 shutdown_connections 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdca006b4d0 0x7fdca006d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcc015e150 0x7fdcc0160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcc00b2fe0 0x7fdcc015ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 --2- 172.21.15.99:0/907451786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcc00671c0 0x7fdcc014c5b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 >> 172.21.15.99:0/907451786 conn(0x7fdcc0071ca0 msgr2=0x7fdcc0072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 shutdown_connections 2024-03-04T10:04:50.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.164+0000 7fdcc4d37640 1 -- 172.21.15.99:0/907451786 wait complete. 2024-03-04T10:04:50.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T10:04:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.284+0000 7fc8d4759640 1 Processor -- start 2024-03-04T10:04:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.284+0000 7fc8d4759640 1 -- start start 2024-03-04T10:04:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d4759640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc8cc071320 0x7fc8cc071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d4759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d3c0 0x7fc8cc1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d4759640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc8cc06d7e0 0x7fc8cc150df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d4759640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc071c40 con 0x7fc8cc071320 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d4759640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8cc06dd10 con 0x7fc8cc06d7e0 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d4759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc06dff0 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc8cc071320 0x7fc8cc071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc8cc071320 0x7fc8cc071700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43922/0 (socket says 172.21.15.99:43922) 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 -- 172.21.15.99:0/1775742412 learned_addr learned my addr 172.21.15.99:0/1775742412 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 -- 172.21.15.99:0/1775742412 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d3c0 msgr2=0x7fc8cc1508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d1cf4640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d3c0 0x7fc8cc1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d2cf6640 1 --2- 172.21.15.99:0/1775742412 >> v2:172.21.15.162:3300/0 conn(0x7fc8cc06d7e0 0x7fc8cc150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d3c0 0x7fc8cc1508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 -- 172.21.15.99:0/1775742412 >> v2:172.21.15.162:3300/0 conn(0x7fc8cc06d7e0 msgr2=0x7fc8cc150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 --2- 172.21.15.99:0/1775742412 >> v2:172.21.15.162:3300/0 conn(0x7fc8cc06d7e0 0x7fc8cc150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d24f5640 1 -- 172.21.15.99:0/1775742412 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc06e2d0 con 0x7fc8cc071320 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d1cf4640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d3c0 0x7fc8cc1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.285+0000 7fc8d2cf6640 1 --2- 172.21.15.99:0/1775742412 >> v2:172.21.15.162:3300/0 conn(0x7fc8cc06d7e0 0x7fc8cc150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.286+0000 7fc8d24f5640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc071320 0x7fc8cc071700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc071e20 tx=0x7fc8c802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879ec2317145cd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.286+0000 7fc8d14f3640 1 -- 172.21.15.99:0/1775742412 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8035020 con 0x7fc8cc071320 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.286+0000 7fc8d14f3640 1 -- 172.21.15.99:0/1775742412 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c8031640 con 0x7fc8cc071320 2024-03-04T10:04:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.286+0000 7fc8d14f3640 1 -- 172.21.15.99:0/1775742412 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c802d040 con 0x7fc8cc071320 2024-03-04T10:04:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 -- 172.21.15.99:0/1775742412 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc071320 msgr2=0x7fc8cc071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc071320 0x7fc8cc071700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc071e20 tx=0x7fc8c802b8f0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 -- 172.21.15.99:0/1775742412 shutdown_connections 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 --2- 172.21.15.99:0/1775742412 >> v2:172.21.15.162:3300/0 conn(0x7fc8cc06d7e0 0x7fc8cc150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d3c0 0x7fc8cc1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 --2- 172.21.15.99:0/1775742412 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc071320 0x7fc8cc071700 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 -- 172.21.15.99:0/1775742412 >> 172.21.15.99:0/1775742412 conn(0x7fc8cc06acd0 msgr2=0x7fc8cc06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 -- 172.21.15.99:0/1775742412 shutdown_connections 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.287+0000 7fc8d4759640 1 -- 172.21.15.99:0/1775742412 wait complete. 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.288+0000 7fc8d4759640 1 Processor -- start 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.288+0000 7fc8d4759640 1 -- start start 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d4759640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 0x7fc8cc1481f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d4759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d7e0 0x7fc8cc148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d4759640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8cc071320 0x7fc8cc0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d4759640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0cf660 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d4759640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc151330 con 0x7fc8cc071320 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d4759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8cc0cf7e0 con 0x7fc8cc06d7e0 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 0x7fc8cc1481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 0x7fc8cc1481f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43930/0 (socket says 172.21.15.99:43930) 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 -- 172.21.15.99:0/3814298022 learned_addr learned my addr 172.21.15.99:0/3814298022 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d1cf4640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d7e0 0x7fc8cc148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d2cf6640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8cc071320 0x7fc8cc0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 -- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d7e0 msgr2=0x7fc8cc148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d7e0 0x7fc8cc148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 -- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8cc071320 msgr2=0x7fc8cc0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8cc071320 0x7fc8cc0b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 -- 172.21.15.99:0/3814298022 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8cc149050 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d2cf6640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8cc071320 0x7fc8cc0b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d1cf4640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d7e0 0x7fc8cc148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8d24f5640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 0x7fc8cc1481f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc8c802b930 tx=0x7fc8c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8040070 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.289+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c8039c10 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.290+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc148d70 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.290+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8035040 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.290+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc1f20a0 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.290+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8cc067640 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.291+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc8c804b060 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.291+0000 7fc8c2ffd640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc8b006b500 0x7fc8b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:50.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.292+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fc8c80abcc0 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.292+0000 7fc8d1cf4640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc8b006b500 0x7fc8b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:50.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.292+0000 7fc8d1cf4640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc8b006b500 0x7fc8b006d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc146110 tx=0x7fc8bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:50.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.296+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc8cc067640 con 0x7fc8cc06d3c0 2024-03-04T10:04:50.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:50.468+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc8cc074f60 con 0x7fc8cc06d3c0 2024-03-04T10:04:51.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.164+0000 7fc8c2ffd640 1 -- 172.21.15.99:0/3814298022 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc8c802f300 con 0x7fc8cc06d3c0 2024-03-04T10:04:51.166 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2024-03-04T10:04:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc8b006b500 msgr2=0x7fc8b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc8b006b500 0x7fc8b006d9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc146110 tx=0x7fc8bc008040 comp rx=0 tx=0).stop 2024-03-04T10:04:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 msgr2=0x7fc8cc1481f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 0x7fc8cc1481f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fc8c802b930 tx=0x7fc8c80046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 shutdown_connections 2024-03-04T10:04:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc8b006b500 0x7fc8b006d9c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8cc071320 0x7fc8cc0b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8cc06d7e0 0x7fc8cc148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 --2- 172.21.15.99:0/3814298022 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8cc06d3c0 0x7fc8cc1481f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 >> 172.21.15.99:0/3814298022 conn(0x7fc8cc06acd0 msgr2=0x7fc8cc150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 shutdown_connections 2024-03-04T10:04:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.167+0000 7fc8d4759640 1 -- 172.21.15.99:0/3814298022 wait complete. 2024-03-04T10:04:51.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T10:04:51.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.296+0000 7fef9e6a1640 1 Processor -- start 2024-03-04T10:04:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.296+0000 7fef9e6a1640 1 -- start start 2024-03-04T10:04:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.297+0000 7fef9e6a1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9815da20 0x7fef980ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.297+0000 7fef9e6a1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fef980cf660 0x7fef980cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.297+0000 7fef9e6a1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef98162450 con 0x7fef981616e0 2024-03-04T10:04:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.297+0000 7fef9e6a1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef980d0000 con 0x7fef980cf660 2024-03-04T10:04:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.297+0000 7fef9e6a1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef980cc920 con 0x7fef9815da20 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef977fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fef980cf660 0x7fef980cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef97fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9815da20 0x7fef980ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef9cc3e640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fef981616e0 0x7fef98161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:39494/0 (socket says 172.21.15.99:39494) 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef9cc3e640 1 -- 172.21.15.99:0/1907862661 learned_addr learned my addr 172.21.15.99:0/1907862661 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef97fff640 1 -- 172.21.15.99:0/1907862661 >> v2:172.21.15.162:3300/0 conn(0x7fef980cf660 msgr2=0x7fef980cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef97fff640 1 --2- 172.21.15.99:0/1907862661 >> v2:172.21.15.162:3300/0 conn(0x7fef980cf660 0x7fef980cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef97fff640 1 -- 172.21.15.99:0/1907862661 >> v1:172.21.15.162:6789/0 conn(0x7fef981616e0 legacy=0x7fef98161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef97fff640 1 -- 172.21.15.99:0/1907862661 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980cc2d0 con 0x7fef9815da20 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.298+0000 7fef977fe640 1 --2- 172.21.15.99:0/1907862661 >> v2:172.21.15.162:3300/0 conn(0x7fef980cf660 0x7fef980cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef97fff640 1 --2- 172.21.15.99:0/1907862661 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9815da20 0x7fef980ccdb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef880097a0 tx=0x7fef8802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bfbf02522b810b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef96ffd640 1 -- 172.21.15.99:0/1907862661 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88035020 con 0x7fef9815da20 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef96ffd640 1 -- 172.21.15.99:0/1907862661 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef88031830 con 0x7fef9815da20 2024-03-04T10:04:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef96ffd640 1 -- 172.21.15.99:0/1907862661 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88031c00 con 0x7fef9815da20 2024-03-04T10:04:51.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 -- 172.21.15.99:0/1907862661 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9815da20 msgr2=0x7fef980ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/1907862661 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9815da20 0x7fef980ccdb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef880097a0 tx=0x7fef8802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 -- 172.21.15.99:0/1907862661 shutdown_connections 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/1907862661 >> v2:172.21.15.162:3300/0 conn(0x7fef980cf660 0x7fef980cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/1907862661 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef9815da20 0x7fef980ccdb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 -- 172.21.15.99:0/1907862661 >> 172.21.15.99:0/1907862661 conn(0x7fef980b33a0 msgr2=0x7fef980b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 -- 172.21.15.99:0/1907862661 shutdown_connections 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.299+0000 7fef9e6a1640 1 -- 172.21.15.99:0/1907862661 wait complete. 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 Processor -- start 2024-03-04T10:04:51.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 -- start start 2024-03-04T10:04:51.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef980cf660 0x7fef9815ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef9815da20 0x7fef9815e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 0x7fef980c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980cbf80 con 0x7fef981616e0 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef980cd2f0 con 0x7fef9815da20 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.301+0000 7fef9e6a1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9815de00 con 0x7fef980cf660 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 0x7fef980c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 0x7fef980c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43940/0 (socket says 172.21.15.99:43940) 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 -- 172.21.15.99:0/3010691530 learned_addr learned my addr 172.21.15.99:0/3010691530 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef97fff640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef980cf660 0x7fef9815ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef977fe640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef9815da20 0x7fef9815e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 -- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef980cf660 msgr2=0x7fef9815ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef980cf660 0x7fef9815ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 -- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef9815da20 msgr2=0x7fef9815e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef9815da20 0x7fef9815e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 -- 172.21.15.99:0/3010691530 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9815b2d0 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9cc3e640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 0x7fef980c6a50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fef9000cab0 tx=0x7fef9000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90017070 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9000ac30 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9815fb20 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90012400 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.302+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9815ed60 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.303+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef98074f60 con 0x7fef981616e0 2024-03-04T10:04:51.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.304+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fef9001a070 con 0x7fef981616e0 2024-03-04T10:04:51.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.304+0000 7fef94ff9640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef8406b4d0 0x7fef8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:51.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.304+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fef90089e80 con 0x7fef981616e0 2024-03-04T10:04:51.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.305+0000 7fef97fff640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef8406b4d0 0x7fef8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:51.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.305+0000 7fef97fff640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef8406b4d0 0x7fef8406d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fef88036000 tx=0x7fef88008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.309+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fef9815b2d0 con 0x7fef981616e0 2024-03-04T10:04:51.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:51.515+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fef980d1c40 con 0x7fef981616e0 2024-03-04T10:04:52.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.188+0000 7fef94ff9640 1 -- 172.21.15.99:0/3010691530 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fef90058610 con 0x7fef981616e0 2024-03-04T10:04:52.190 INFO:teuthology.orchestra.run.smithi099.stderr:new fs with metadata pool 5 and data pool 6 2024-03-04T10:04:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.202+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef8406b4d0 msgr2=0x7fef8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.202+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef8406b4d0 0x7fef8406d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fef88036000 tx=0x7fef88008030 comp rx=0 tx=0).stop 2024-03-04T10:04:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.202+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 msgr2=0x7fef980c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.202+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 0x7fef980c6a50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fef9000cab0 tx=0x7fef9000cf70 comp rx=0 tx=0).stop 2024-03-04T10:04:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.204+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 shutdown_connections 2024-03-04T10:04:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.204+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fef8406b4d0 0x7fef8406d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.204+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fef981616e0 0x7fef980c6a50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.204+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fef9815da20 0x7fef9815e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.204+0000 7fef9e6a1640 1 --2- 172.21.15.99:0/3010691530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fef980cf660 0x7fef9815ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.204+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 >> 172.21.15.99:0/3010691530 conn(0x7fef980b33a0 msgr2=0x7fef980b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.206+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 shutdown_connections 2024-03-04T10:04:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.208+0000 7fef9e6a1640 1 -- 172.21.15.99:0/3010691530 wait complete. 2024-03-04T10:04:52.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:52.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.322+0000 7f4f6969a640 1 Processor -- start 2024-03-04T10:04:52.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.322+0000 7f4f6969a640 1 -- start start 2024-03-04T10:04:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.323+0000 7f4f6969a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f4f640b3e30 0x7f4f64168be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.323+0000 7f4f6969a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6406d860 con 0x7f4f6415b560 2024-03-04T10:04:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.323+0000 7f4f6969a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6415ed90 con 0x7f4f6415b940 2024-03-04T10:04:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.323+0000 7f4f6969a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6415f070 con 0x7f4f640b3e30 2024-03-04T10:04:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.323+0000 7f4f62ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f4f6415b560 0x7f4f6415e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60372/0 (socket says 172.21.15.99:60372) 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.323+0000 7f4f62ffd640 1 -- 172.21.15.99:0/137323073 learned_addr learned my addr 172.21.15.99:0/137323073 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037647893 0 0) 0x7f4f6406d860 con 0x7f4f6415b560 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f4c003520 con 0x7f4f6415b560 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 --2- 172.21.15.99:0/137323073 >> v2:172.21.15.162:3300/0 conn(0x7f4f640b3e30 0x7f4f64168be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1569728276 0 0) 0x7f4f4c003520 con 0x7f4f6415b560 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 >> v2:172.21.15.162:3300/0 conn(0x7f4f640b3e30 msgr2=0x7f4f64168be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 --2- 172.21.15.99:0/137323073 >> v2:172.21.15.162:3300/0 conn(0x7f4f640b3e30 0x7f4f64168be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 >> v1:172.21.15.162:6789/0 conn(0x7f4f6415b940 legacy=0x7f4f640b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6406db20 con 0x7f4f6415b560 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f4f540019f0 con 0x7f4f6415b560 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 >> v1:172.21.15.99:6789/0 conn(0x7f4f6415b560 legacy=0x7f4f6415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0039b0 0x7f4f4c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f4c004800 0x7f4f4c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f4c0051c0 0x7f4f4c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6415f070 con 0x7f4f4c0051c0 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c00c0f0 con 0x7f4f4c004800 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c00c2f0 con 0x7f4f4c0039b0 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f4c0051c0 0x7f4f4c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f62ffd640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0039b0 0x7f4f4c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f627fc640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f4c004800 0x7f4f4c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 -- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0039b0 msgr2=0x7f4f4c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0039b0 0x7f4f4c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 -- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f4c004800 msgr2=0x7f4f4c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f4c004800 0x7f4f4c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.324+0000 7f4f637fe640 1 -- 172.21.15.99:0/137323073 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c003520 con 0x7f4f4c0051c0 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.325+0000 7f4f637fe640 1 -- 172.21.15.99:0/137323073 reap_dead start 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.325+0000 7f4f637fe640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f4c0051c0 0x7f4f4c00b9c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00ba00 tx=0x7f4f4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39b923a7ecc40512 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.325+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c036030 con 0x7f4f4c0051c0 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.325+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c004020 con 0x7f4f4c0051c0 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.325+0000 7f4f61ffb640 1 -- 172.21.15.99:0/137323073 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00b600 con 0x7f4f4c0051c0 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 -- 172.21.15.99:0/137323073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f4c0051c0 msgr2=0x7f4f4c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f4c0051c0 0x7f4f4c00b9c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00ba00 tx=0x7f4f4c005830 comp rx=0 tx=0).stop 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 -- 172.21.15.99:0/137323073 shutdown_connections 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 --2- 172.21.15.99:0/137323073 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f4c0051c0 0x7f4f4c00b9c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 -- 172.21.15.99:0/137323073 >> 172.21.15.99:0/137323073 conn(0x7f4f640cfb40 msgr2=0x7f4f640cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 -- 172.21.15.99:0/137323073 shutdown_connections 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.326+0000 7f4f6969a640 1 -- 172.21.15.99:0/137323073 wait complete. 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.327+0000 7f4f6969a640 1 Processor -- start 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.327+0000 7f4f6969a640 1 -- start start 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f6969a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0051c0 0x7f4f640ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f6969a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f64142140 0x7f4f64153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f6969a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 0x7f4f64144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f6969a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640b3720 con 0x7f4f641448f0 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f6969a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64071a90 con 0x7f4f64142140 2024-03-04T10:04:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f6969a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640b38a0 con 0x7f4f4c0051c0 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 0x7f4f64144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 0x7f4f64144d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43958/0 (socket says 172.21.15.99:43958) 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 -- 172.21.15.99:0/2952188802 learned_addr learned my addr 172.21.15.99:0/2952188802 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f627fc640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f64142140 0x7f4f64153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f62ffd640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0051c0 0x7f4f640ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 -- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0051c0 msgr2=0x7f4f640ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0051c0 0x7f4f640ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 -- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f64142140 msgr2=0x7f4f64153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f64142140 0x7f4f64153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.328+0000 7f4f637fe640 1 -- 172.21.15.99:0/2952188802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f64162330 con 0x7f4f641448f0 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.329+0000 7f4f637fe640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 0x7f4f64144d70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c02bf90 tx=0x7f4f5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.329+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c040030 con 0x7f4f641448f0 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.329+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c003c10 con 0x7f4f641448f0 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.329+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f640ad8e0 con 0x7f4f641448f0 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.329+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c036040 con 0x7f4f641448f0 2024-03-04T10:04:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.329+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f64156060 con 0x7f4f641448f0 2024-03-04T10:04:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.330+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f64162710 con 0x7f4f641448f0 2024-03-04T10:04:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.330+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4f5c043030 con 0x7f4f641448f0 2024-03-04T10:04:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.331+0000 7f4f4b7fe640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f5006b4d0 0x7f4f5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.331+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f5c0abbd0 con 0x7f4f641448f0 2024-03-04T10:04:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.331+0000 7f4f62ffd640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f5006b4d0 0x7f4f5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.331+0000 7f4f62ffd640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f5006b4d0 0x7f4f5006d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4f54001f20 tx=0x7f4f54006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.335+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4f64162710 con 0x7f4f641448f0 2024-03-04T10:04:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.509+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f640b48d0 con 0x7f4f641448f0 2024-03-04T10:04:52.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.510+0000 7f4f4b7fe640 1 -- 172.21.15.99:0/2952188802 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f4f640b48d0 con 0x7f4f641448f0 2024-03-04T10:04:52.511 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:52.511 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:51.516476+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f5006b4d0 msgr2=0x7f4f5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f5006b4d0 0x7f4f5006d990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4f54001f20 tx=0x7f4f54006040 comp rx=0 tx=0).stop 2024-03-04T10:04:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 msgr2=0x7f4f64144d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 0x7f4f64144d70 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c02bf90 tx=0x7f4f5c004730 comp rx=0 tx=0).stop 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 shutdown_connections 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f5006b4d0 0x7f4f5006d990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4f641448f0 0x7f4f64144d70 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4f64142140 0x7f4f64153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 --2- 172.21.15.99:0/2952188802 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4f4c0051c0 0x7f4f640ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 >> 172.21.15.99:0/2952188802 conn(0x7f4f640cfb40 msgr2=0x7f4f6415bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 shutdown_connections 2024-03-04T10:04:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.512+0000 7f4f6969a640 1 -- 172.21.15.99:0/2952188802 wait complete. 2024-03-04T10:04:52.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:52.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.623+0000 7f2547b05640 1 Processor -- start 2024-03-04T10:04:52.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.623+0000 7f2547b05640 1 -- start start 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f2547b05640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f254015de00 0x7f25400b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f2547b05640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25400b9380 0x7f25400b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f2547b05640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254015e1e0 con 0x7f25400b70d0 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f2547b05640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25400d2820 con 0x7f25400b9380 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f2547b05640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400d1c10 con 0x7f254015de00 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f25450a0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f25400b9380 0x7f25400b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f25460a2640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f25400b70d0 0x7f25400b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60380/0 (socket says 172.21.15.99:60380) 2024-03-04T10:04:52.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.624+0000 7f25460a2640 1 -- 172.21.15.99:0/2320136409 learned_addr learned my addr 172.21.15.99:0/2320136409 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f25458a1640 1 --2- 172.21.15.99:0/2320136409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f254015de00 0x7f25400b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f254489f640 1 -- 172.21.15.99:0/2320136409 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037288025 0 0) 0x7f254015e1e0 con 0x7f25400b70d0 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f254489f640 1 -- 172.21.15.99:0/2320136409 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f252c003520 con 0x7f25400b70d0 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f25450a0640 1 -- 172.21.15.99:0/2320136409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f254015de00 msgr2=0x7f25400b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f25450a0640 1 --2- 172.21.15.99:0/2320136409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f254015de00 0x7f25400b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f25450a0640 1 -- 172.21.15.99:0/2320136409 >> v1:172.21.15.99:6789/0 conn(0x7f25400b70d0 legacy=0x7f25400b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f25450a0640 1 -- 172.21.15.99:0/2320136409 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400d1ef0 con 0x7f25400b9380 2024-03-04T10:04:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.625+0000 7f25450a0640 1 --2- 172.21.15.99:0/2320136409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f25400b6b90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f25300097a0 tx=0x7f253002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddea5ad3923c3167 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f254489f640 1 -- 172.21.15.99:0/2320136409 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530035020 con 0x7f25400b9380 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f254489f640 1 -- 172.21.15.99:0/2320136409 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2530031830 con 0x7f25400b9380 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f254489f640 1 -- 172.21.15.99:0/2320136409 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530031be0 con 0x7f25400b9380 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 -- 172.21.15.99:0/2320136409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 msgr2=0x7f25400b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 --2- 172.21.15.99:0/2320136409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f25400b6b90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f25300097a0 tx=0x7f253002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 -- 172.21.15.99:0/2320136409 shutdown_connections 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 --2- 172.21.15.99:0/2320136409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f25400b6b90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 --2- 172.21.15.99:0/2320136409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f254015de00 0x7f25400b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 -- 172.21.15.99:0/2320136409 >> 172.21.15.99:0/2320136409 conn(0x7f25400d0470 msgr2=0x7f25400d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 -- 172.21.15.99:0/2320136409 shutdown_connections 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.626+0000 7f2547b05640 1 -- 172.21.15.99:0/2320136409 wait complete. 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.627+0000 7f2547b05640 1 Processor -- start 2024-03-04T10:04:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.627+0000 7f2547b05640 1 -- start start 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f2547b05640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25400b70d0 0x7f25400c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f2547b05640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f254014cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f2547b05640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254015de00 0x7f254014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f2547b05640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b97e0 con 0x7f25400b9380 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f2547b05640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25400d24b0 con 0x7f254015de00 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f2547b05640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400b5b80 con 0x7f25400b70d0 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f254014cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f254014cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43980/0 (socket says 172.21.15.99:43980) 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 -- 172.21.15.99:0/2845932211 learned_addr learned my addr 172.21.15.99:0/2845932211 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25460a2640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254015de00 0x7f254014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25458a1640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25400b70d0 0x7f25400c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 -- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25400b70d0 msgr2=0x7f25400c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25400b70d0 0x7f25400c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 -- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254015de00 msgr2=0x7f254014d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.628+0000 7f25450a0640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254015de00 0x7f254014d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f25450a0640 1 -- 172.21.15.99:0/2845932211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2540160bc0 con 0x7f25400b9380 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f25458a1640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25400b70d0 0x7f25400c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f25450a0640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f254014cb40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f253002bee0 tx=0x7f25300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530035040 con 0x7f25400b9380 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253003f070 con 0x7f25400b9380 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2540160ea0 con 0x7f25400b9380 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253003bb40 con 0x7f25400b9380 2024-03-04T10:04:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.629+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25400c7500 con 0x7f25400b9380 2024-03-04T10:04:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.630+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2540074f60 con 0x7f25400b9380 2024-03-04T10:04:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.630+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2530038020 con 0x7f25400b9380 2024-03-04T10:04:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.631+0000 7f25367fc640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f252406b4d0 0x7f252406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.631+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f25300abec0 con 0x7f25400b9380 2024-03-04T10:04:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.631+0000 7f25458a1640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f252406b4d0 0x7f252406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.631+0000 7f25458a1640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f252406b4d0 0x7f252406d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f253c007440 tx=0x7f253c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.635+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f253007abb0 con 0x7f25400b9380 2024-03-04T10:04:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.806+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25400b3900 con 0x7f25400b9380 2024-03-04T10:04:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.807+0000 7f25367fc640 1 -- 172.21.15.99:0/2845932211 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f253007a4b0 con 0x7f25400b9380 2024-03-04T10:04:52.809 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:52.809 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":37,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:51.516476+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.809+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f252406b4d0 msgr2=0x7f252406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.809+0000 7f2547b05640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f252406b4d0 0x7f252406d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f253c007440 tx=0x7f253c00b040 comp rx=0 tx=0).stop 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.809+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 msgr2=0x7f254014cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.809+0000 7f2547b05640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f254014cb40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f253002bee0 tx=0x7f25300046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 shutdown_connections 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f252406b4d0 0x7f252406d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f254015de00 0x7f254014d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f25400b9380 0x7f254014cb40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 --2- 172.21.15.99:0/2845932211 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f25400b70d0 0x7f25400c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 >> 172.21.15.99:0/2845932211 conn(0x7f25400d0470 msgr2=0x7f254015bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 shutdown_connections 2024-03-04T10:04:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.810+0000 7f2547b05640 1 -- 172.21.15.99:0/2845932211 wait complete. 2024-03-04T10:04:52.822 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T10:04:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.925+0000 7f0d319a9640 1 Processor -- start 2024-03-04T10:04:52.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.925+0000 7f0d319a9640 1 -- start start 2024-03-04T10:04:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d319a9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b9410 0x7f0d2c1622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d319a9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c0d2400 con 0x7f0d2c0b5700 2024-03-04T10:04:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d319a9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d2c0d26e0 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d319a9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c0d29c0 con 0x7f0d2c0b9410 2024-03-04T10:04:52.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d2affd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0d2c0b5700 0x7f0d2c0d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60384/0 (socket says 172.21.15.99:60384) 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d2affd640 1 -- 172.21.15.99:0/524670711 learned_addr learned my addr 172.21.15.99:0/524670711 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d2b7fe640 1 --2- 172.21.15.99:0/524670711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b9410 0x7f0d2c1622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261968525 0 0) 0x7f0d2c0d2400 con 0x7f0d2c0b5700 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14003520 con 0x7f0d2c0b5700 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80640209 0 0) 0x7f0d2c0d26e0 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d2c0d2400 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916482388 0 0) 0x7f0d14003520 con 0x7f0d2c0b5700 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/524670711 >> v1:172.21.15.162:6789/0 conn(0x7f0d2c0d34a0 legacy=0x7f0d2c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/524670711 >> v1:172.21.15.99:6789/0 conn(0x7f0d2c0b5700 legacy=0x7f0d2c0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.926+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/524670711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c0d2ca0 con 0x7f0d2c0b9410 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.927+0000 7f0d2b7fe640 1 --2- 172.21.15.99:0/524670711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b9410 0x7f0d2c1622f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d240097d0 tx=0x7f0d2402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575adc88bf1f3ace server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.927+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24037070 con 0x7f0d2c0b9410 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.927+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d24030510 con 0x7f0d2c0b9410 2024-03-04T10:04:52.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.927+0000 7f0d29ffb640 1 -- 172.21.15.99:0/524670711 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c0019f0 con 0x7f0d2c0b9410 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 -- 172.21.15.99:0/524670711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b9410 msgr2=0x7f0d2c1622f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 --2- 172.21.15.99:0/524670711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b9410 0x7f0d2c1622f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d240097d0 tx=0x7f0d2402b770 comp rx=0 tx=0).stop 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 -- 172.21.15.99:0/524670711 shutdown_connections 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 --2- 172.21.15.99:0/524670711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b9410 0x7f0d2c1622f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 -- 172.21.15.99:0/524670711 >> 172.21.15.99:0/524670711 conn(0x7f0d2c0d0490 msgr2=0x7f0d2c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 -- 172.21.15.99:0/524670711 shutdown_connections 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.928+0000 7f0d319a9640 1 -- 172.21.15.99:0/524670711 wait complete. 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.929+0000 7f0d319a9640 1 Processor -- start 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.929+0000 7f0d319a9640 1 -- start start 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d319a9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b5700 0x7f0d2c14f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d319a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c0b9410 0x7f0d2c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d319a9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 0x7f0d2c1f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d319a9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c0b5c60 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d319a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c0b5ae0 con 0x7f0d2c0b9410 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d319a9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2c0b5de0 con 0x7f0d2c0b5700 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 0x7f0d2c1f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 0x7f0d2c1f6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43988/0 (socket says 172.21.15.99:43988) 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/2834235226 learned_addr learned my addr 172.21.15.99:0/2834235226 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2a7fc640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c0b9410 0x7f0d2c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2affd640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b5700 0x7f0d2c14f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b5700 msgr2=0x7f0d2c14f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b5700 0x7f0d2c14f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c0b9410 msgr2=0x7f0d2c0cad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c0b9410 0x7f0d2c0cad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:52.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 -- 172.21.15.99:0/2834235226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c0cb5d0 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2a7fc640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c0b9410 0x7f0d2c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.930+0000 7f0d2b7fe640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 0x7f0d2c1f6f30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009720 tx=0x7f0d240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.931+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24037040 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.931+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d24004230 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.931+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2c0cb8b0 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.931+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24030ea0 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.931+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2c0cbe70 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.932+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2c074f60 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.932+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0d2403a040 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.932+0000 7f0d13fff640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d1806b4d0 0x7f0d1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.932+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d240abcf0 con 0x7f0d2c0d34a0 2024-03-04T10:04:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.933+0000 7f0d2affd640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d1806b4d0 0x7f0d1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:52.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.933+0000 7f0d2affd640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d1806b4d0 0x7f0d1806d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c001560 tx=0x7f0d1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:52.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:52.937+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0d240b0070 con 0x7f0d2c0d34a0 2024-03-04T10:04:53.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.146+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0d2c160bc0 con 0x7f0d2c0d34a0 2024-03-04T10:04:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.187+0000 7f0d13fff640 1 -- 172.21.15.99:0/2834235226 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0d24042020 con 0x7f0d2c0d34a0 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.189+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d1806b4d0 msgr2=0x7f0d1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.189+0000 7f0d319a9640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d1806b4d0 0x7f0d1806d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c001560 tx=0x7f0d1c009040 comp rx=0 tx=0).stop 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.189+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 msgr2=0x7f0d2c1f6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.189+0000 7f0d319a9640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 0x7f0d2c1f6f30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009720 tx=0x7f0d240046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 shutdown_connections 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d1806b4d0 0x7f0d1806d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d2c0d34a0 0x7f0d2c1f6f30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c0b9410 0x7f0d2c0cad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 --2- 172.21.15.99:0/2834235226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d2c0b5700 0x7f0d2c14f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 >> 172.21.15.99:0/2834235226 conn(0x7f0d2c0d0490 msgr2=0x7f0d2c06c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 shutdown_connections 2024-03-04T10:04:53.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.190+0000 7f0d319a9640 1 -- 172.21.15.99:0/2834235226 wait complete. 2024-03-04T10:04:53.203 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T10:04:53.203 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T10:04:53.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.305+0000 7fe3e2bd4640 1 Processor -- start 2024-03-04T10:04:53.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.305+0000 7fe3e2bd4640 1 -- start start 2024-03-04T10:04:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e2bd4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3dc0cdd70 0x7fe3dc0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e2bd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e2bd4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc06ae00 con 0x7fe3dc0ccfe0 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e2bd4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3dc0ccde0 con 0x7fe3dc0cdd70 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e2bd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0d0480 con 0x7fe3dc0cc900 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e0970640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3dc0cdd70 0x7fe3dc0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e0970640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3dc0cdd70 0x7fe3dc0cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43994/0 (socket says 172.21.15.99:43994) 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.306+0000 7fe3e0970640 1 -- 172.21.15.99:0/1668239049 learned_addr learned my addr 172.21.15.99:0/1668239049 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3dbfff640 1 --2- 172.21.15.99:0/1668239049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3e0970640 1 -- 172.21.15.99:0/1668239049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 msgr2=0x7fe3dc0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3e0970640 1 --2- 172.21.15.99:0/1668239049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc0cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3e0970640 1 -- 172.21.15.99:0/1668239049 >> v1:172.21.15.162:6789/0 conn(0x7fe3dc0ccfe0 legacy=0x7fe3dc15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3e0970640 1 -- 172.21.15.99:0/1668239049 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc06b0e0 con 0x7fe3dc0cdd70 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3e0970640 1 --2- 172.21.15.99:0/1668239049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc0cdd70 0x7fe3dc0cfa00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4009e50 tx=0x7fe3d402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333057a608d3ffca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.307+0000 7fe3db7fe640 1 -- 172.21.15.99:0/1668239049 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d402c9e0 con 0x7fe3dc0cdd70 2024-03-04T10:04:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3db7fe640 1 -- 172.21.15.99:0/1668239049 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d402cb80 con 0x7fe3dc0cdd70 2024-03-04T10:04:53.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3db7fe640 1 -- 172.21.15.99:0/1668239049 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d40343f0 con 0x7fe3dc0cdd70 2024-03-04T10:04:53.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1668239049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc0cdd70 msgr2=0x7fe3dc0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1668239049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc0cdd70 0x7fe3dc0cfa00 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4009e50 tx=0x7fe3d402be40 comp rx=0 tx=0).stop 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1668239049 shutdown_connections 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1668239049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1668239049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc0cdd70 0x7fe3dc0cfa00 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1668239049 >> 172.21.15.99:0/1668239049 conn(0x7fe3dc0b22e0 msgr2=0x7fe3dc0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1668239049 shutdown_connections 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.308+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1668239049 wait complete. 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.309+0000 7fe3e2bd4640 1 Processor -- start 2024-03-04T10:04:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.309+0000 7fe3e2bd4640 1 -- start start 2024-03-04T10:04:53.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e2bd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc15c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:53.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e2bd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3dc0ccfe0 0x7fe3dc0c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e2bd4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 0x7fe3dc06f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e2bd4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0b2e90 con 0x7fe3dc06f280 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e2bd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc06b6a0 con 0x7fe3dc0ccfe0 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e2bd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc0b3010 con 0x7fe3dc0cc900 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 0x7fe3dc06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 0x7fe3dc06f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:44002/0 (socket says 172.21.15.99:44002) 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 -- 172.21.15.99:0/1779713273 learned_addr learned my addr 172.21.15.99:0/1779713273 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e0970640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc15c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3dbfff640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3dc0ccfe0 0x7fe3dc0c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 -- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 msgr2=0x7fe3dc15c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc15c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 -- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3dc0ccfe0 msgr2=0x7fe3dc0c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3dc0ccfe0 0x7fe3dc0c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.310+0000 7fe3e1171640 1 -- 172.21.15.99:0/1779713273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc0c9b30 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.311+0000 7fe3e1171640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 0x7fe3dc06f730 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c440 tx=0x7fe3cc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.311+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc011790 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.311+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc017070 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.311+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc0c8550 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.311+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc011b70 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.311+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc0c8b10 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.312+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc0b0860 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.312+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe3cc011d10 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.313+0000 7fe3d97fa640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c806b4d0 0x7fe3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.313+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe3cc08a130 con 0x7fe3dc06f280 2024-03-04T10:04:53.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.313+0000 7fe3e0970640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c806b4d0 0x7fe3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:53.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.313+0000 7fe3e0970640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c806b4d0 0x7fe3c806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4035000 tx=0x7fe3d4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:53.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.317+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3dc0b0860 con 0x7fe3dc06f280 2024-03-04T10:04:53.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:53.523+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe3dc074f60 con 0x7fe3dc06f280 2024-03-04T10:04:54.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.198+0000 7fe3d97fa640 1 -- 172.21.15.99:0/1779713273 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe3cc058740 con 0x7fe3dc06f280 2024-03-04T10:04:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c806b4d0 msgr2=0x7fe3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c806b4d0 0x7fe3c806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe3d4035000 tx=0x7fe3d4030000 comp rx=0 tx=0).stop 2024-03-04T10:04:54.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 msgr2=0x7fe3dc06f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 0x7fe3dc06f730 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc00c440 tx=0x7fe3cc00c900 comp rx=0 tx=0).stop 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 shutdown_connections 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c806b4d0 0x7fe3c806d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3dc06f280 0x7fe3dc06f730 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3dc0ccfe0 0x7fe3dc0c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 --2- 172.21.15.99:0/1779713273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3dc0cc900 0x7fe3dc15c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 >> 172.21.15.99:0/1779713273 conn(0x7fe3dc0b22e0 msgr2=0x7fe3dc0b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 shutdown_connections 2024-03-04T10:04:54.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.200+0000 7fe3e2bd4640 1 -- 172.21.15.99:0/1779713273 wait complete. 2024-03-04T10:04:54.214 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T10:04:54.214 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:54.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.314+0000 7fd03bcfc640 1 Processor -- start 2024-03-04T10:04:54.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.314+0000 7fd03bcfc640 1 -- start start 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd03bcfc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 0x7fd034161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd03bcfc640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03406b500 con 0x7fd034158410 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd03bcfc640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd03406b7e0 con 0x7fd034161ca0 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd03bcfc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03406bac0 con 0x7fd0340b39c0 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd039a98640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd034158410 0x7fd03406acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56950/0 (socket says 172.21.15.99:56950) 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd039a98640 1 -- 172.21.15.99:0/27775480 learned_addr learned my addr 172.21.15.99:0/27775480 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.315+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085625729 0 0) 0x7fd03406b500 con 0x7fd034158410 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd020003520 con 0x7fd034158410 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039297640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 0x7fd034161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3314065340 0 0) 0x7fd020003520 con 0x7fd034158410 2024-03-04T10:04:54.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 msgr2=0x7fd034161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 0x7fd034161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 >> v1:172.21.15.162:6789/0 conn(0x7fd034161ca0 legacy=0x7fd03406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03406bda0 con 0x7fd034158410 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fd030001670 con 0x7fd034158410 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 >> v1:172.21.15.99:6789/0 conn(0x7fd034158410 legacy=0x7fd03406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0200039e0 0x7fd020003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 0x7fd020004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd020004d40 0x7fd0200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03406bac0 con 0x7fd0200039e0 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02000b9c0 con 0x7fd020004d40 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02000bbc0 con 0x7fd0340b39c0 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039a98640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0200039e0 0x7fd020003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039297640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 0x7fd020004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd03a299640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd020004d40 0x7fd0200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039a98640 1 -- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 msgr2=0x7fd020004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039a98640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340b39c0 0x7fd020004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039a98640 1 -- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd020004d40 msgr2=0x7fd0200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039a98640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd020004d40 0x7fd0200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd039a98640 1 -- 172.21.15.99:0/27775480 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd020003520 con 0x7fd0200039e0 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.316+0000 7fd03a299640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd020004d40 0x7fd0200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.317+0000 7fd039a98640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0200039e0 0x7fd020003dc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd03000c240 tx=0x7fd020005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944eee86c2e26bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.317+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd030007d30 con 0x7fd0200039e0 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.317+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd030030070 con 0x7fd0200039e0 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.317+0000 7fd038a96640 1 -- 172.21.15.99:0/27775480 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd030005d30 con 0x7fd0200039e0 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03bcfc640 1 -- 172.21.15.99:0/27775480 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0200039e0 msgr2=0x7fd020003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/27775480 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0200039e0 0x7fd020003dc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd03000c240 tx=0x7fd020005830 comp rx=0 tx=0).stop 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03a299640 1 -- 172.21.15.99:0/27775480 reap_dead start 2024-03-04T10:04:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03bcfc640 1 -- 172.21.15.99:0/27775480 shutdown_connections 2024-03-04T10:04:54.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03bcfc640 1 -- 172.21.15.99:0/27775480 >> 172.21.15.99:0/27775480 conn(0x7fd0340cfef0 msgr2=0x7fd0340d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:54.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03bcfc640 1 -- 172.21.15.99:0/27775480 shutdown_connections 2024-03-04T10:04:54.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.318+0000 7fd03bcfc640 1 -- 172.21.15.99:0/27775480 wait complete. 2024-03-04T10:04:54.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.319+0000 7fd03bcfc640 1 Processor -- start 2024-03-04T10:04:54.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.319+0000 7fd03bcfc640 1 -- start start 2024-03-04T10:04:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0200039e0 0x7fd034158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 0x7fd034147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340cdea0 0x7fd034147a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03406c5a0 con 0x7fd0340cda60 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340b3e20 con 0x7fd0200039e0 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd034161760 con 0x7fd0340cdea0 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 0x7fd034147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 0x7fd034147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49976/0 (socket says 172.21.15.99:49976) 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 -- 172.21.15.99:0/2058295392 learned_addr learned my addr 172.21.15.99:0/2058295392 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039a98640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0200039e0 0x7fd034158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03a299640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340cdea0 0x7fd034147a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 -- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340cdea0 msgr2=0x7fd034147a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340cdea0 0x7fd034147a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 -- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0200039e0 msgr2=0x7fd034158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0200039e0 0x7fd034158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 -- 172.21.15.99:0/2058295392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0340ce440 con 0x7fd0340cda60 2024-03-04T10:04:54.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03a299640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340cdea0 0x7fd034147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd039297640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 0x7fd034147550 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd0240119a0 tx=0x7fd024011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02400faf0 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02400fc90 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0340ce720 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd024014440 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.320+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0340cece0 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.321+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd034073ef0 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.322+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd024012040 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.322+0000 7fd02a7fc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd01806b4d0 0x7fd01806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.322+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd02408aa60 con 0x7fd0340cda60 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.322+0000 7fd039a98640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd01806b4d0 0x7fd01806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.323+0000 7fd039a98640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd01806b4d0 0x7fd01806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd030034000 tx=0x7fd030009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.326+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd034073ef0 con 0x7fd0340cda60 2024-03-04T10:04:54.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.532+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd034074f60 con 0x7fd0340cda60 2024-03-04T10:04:54.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.533+0000 7fd02a7fc640 1 -- 172.21.15.99:0/2058295392 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fd024019030 con 0x7fd0340cda60 2024-03-04T10:04:54.534 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:54.534 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:04:53.524677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/434536114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":434536114},{"type":"v1","addr":"172.21.15.99:6835","nonce":434536114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:04:54.534 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 15 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.534+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd01806b4d0 msgr2=0x7fd01806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.534+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd01806b4d0 0x7fd01806d990 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd030034000 tx=0x7fd030009030 comp rx=0 tx=0).stop 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 msgr2=0x7fd034147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 0x7fd034147550 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd0240119a0 tx=0x7fd024011e60 comp rx=0 tx=0).stop 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 shutdown_connections 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd01806b4d0 0x7fd01806d990 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0340cdea0 0x7fd034147a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0340cda60 0x7fd034147550 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 --2- 172.21.15.99:0/2058295392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0200039e0 0x7fd034158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 >> 172.21.15.99:0/2058295392 conn(0x7fd0340cfef0 msgr2=0x7fd03406f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 shutdown_connections 2024-03-04T10:04:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.535+0000 7fd03bcfc640 1 -- 172.21.15.99:0/2058295392 wait complete. 2024-03-04T10:04:54.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:54.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.646+0000 7fd449d29640 1 Processor -- start 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.646+0000 7fd449d29640 1 -- start start 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd449d29640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd44406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd449d29640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4440ad6c0 con 0x7fd444066de0 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd449d29640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4440ad960 con 0x7fd44406c470 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd449d29640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44406ebc0 con 0x7fd44406c090 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd443fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd444066de0 0x7fd44415f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56952/0 (socket says 172.21.15.99:56952) 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd443fff640 1 -- 172.21.15.99:0/702521500 learned_addr learned my addr 172.21.15.99:0/702521500 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4437fe640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd44406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247356196 0 0) 0x7fd4440ad6c0 con 0x7fd444066de0 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42c003520 con 0x7fd444066de0 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411306247 0 0) 0x7fd4440ad960 con 0x7fd44406c470 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4440ad6c0 con 0x7fd44406c470 2024-03-04T10:04:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714596485 0 0) 0x7fd42c003520 con 0x7fd444066de0 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 msgr2=0x7fd44406f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd44406f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 >> v1:172.21.15.162:6789/0 conn(0x7fd44406c470 legacy=0x7fd4440af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44406eea0 con 0x7fd444066de0 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fd43c0019f0 con 0x7fd444066de0 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4437fe640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd44406f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 >> v1:172.21.15.99:6789/0 conn(0x7fd444066de0 legacy=0x7fd44415f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd42c003b70 0x7fd42c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c004800 0x7fd42c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd42c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44406ebc0 con 0x7fd42c003b70 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c00baa0 con 0x7fd42c004800 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.647+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42c00bca0 con 0x7fd44406c090 2024-03-04T10:04:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd4437fe640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd42c003b70 0x7fd42c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd442ffd640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c004800 0x7fd42c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd443fff640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd42c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd4437fe640 1 -- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 msgr2=0x7fd42c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd4437fe640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44406c090 0x7fd42c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd4437fe640 1 -- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c004800 msgr2=0x7fd42c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd4437fe640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c004800 0x7fd42c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd4437fe640 1 -- 172.21.15.99:0/702521500 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c003520 con 0x7fd42c003b70 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.648+0000 7fd442ffd640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c004800 0x7fd42c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd4437fe640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd42c003b70 0x7fd42c003f70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd434009c60 tx=0x7fd42c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a66495b5934f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434039070 con 0x7fd42c003b70 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43402c8a0 con 0x7fd42c003b70 2024-03-04T10:04:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd4427fc640 1 -- 172.21.15.99:0/702521500 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434009860 con 0x7fd42c003b70 2024-03-04T10:04:54.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd449d29640 1 -- 172.21.15.99:0/702521500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd42c003b70 msgr2=0x7fd42c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd449d29640 1 --2- 172.21.15.99:0/702521500 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd42c003b70 0x7fd42c003f70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd434009c60 tx=0x7fd42c005830 comp rx=0 tx=0).stop 2024-03-04T10:04:54.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd443fff640 1 -- 172.21.15.99:0/702521500 reap_dead start 2024-03-04T10:04:54.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd449d29640 1 -- 172.21.15.99:0/702521500 shutdown_connections 2024-03-04T10:04:54.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd449d29640 1 -- 172.21.15.99:0/702521500 >> 172.21.15.99:0/702521500 conn(0x7fd4440cfef0 msgr2=0x7fd4440d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:54.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd449d29640 1 -- 172.21.15.99:0/702521500 shutdown_connections 2024-03-04T10:04:54.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.649+0000 7fd449d29640 1 -- 172.21.15.99:0/702521500 wait complete. 2024-03-04T10:04:54.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.650+0000 7fd449d29640 1 Processor -- start 2024-03-04T10:04:54.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 -- start start 2024-03-04T10:04:54.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c003b70 0x7fd444147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 0x7fd44415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44415af80 0x7fd44409ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440addc0 con 0x7fd44409bf50 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4440adc40 con 0x7fd42c003b70 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd449d29640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44415fc70 con 0x7fd44415af80 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd442ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 0x7fd44415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd442ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 0x7fd44415aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49998/0 (socket says 172.21.15.99:49998) 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.651+0000 7fd442ffd640 1 -- 172.21.15.99:0/3676747060 learned_addr learned my addr 172.21.15.99:0/3676747060 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd4437fe640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c003b70 0x7fd444147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd443fff640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44415af80 0x7fd44409ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd442ffd640 1 -- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44415af80 msgr2=0x7fd44409ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd442ffd640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44415af80 0x7fd44409ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd442ffd640 1 -- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c003b70 msgr2=0x7fd444147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd442ffd640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c003b70 0x7fd444147a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd442ffd640 1 -- 172.21.15.99:0/3676747060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44409d740 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd442ffd640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 0x7fd44415aa40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd43800e7b0 tx=0x7fd43800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4380050d0 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd438005270 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44409c3d0 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd438005420 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.652+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44409b470 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.653+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd444074f60 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.653+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd43801a020 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.654+0000 7fd42bfff640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd43006b4d0 0x7fd43006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.654+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd438089b40 con 0x7fd44409bf50 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.654+0000 7fd4437fe640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd43006b4d0 0x7fd43006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.655+0000 7fd4437fe640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd43006b4d0 0x7fd43006d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd434036000 tx=0x7fd434030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.658+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd44409d740 con 0x7fd44409bf50 2024-03-04T10:04:54.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.829+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4440b0530 con 0x7fd44409bf50 2024-03-04T10:04:54.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.830+0000 7fd42bfff640 1 -- 172.21.15.99:0/3676747060 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16906 (secure 0 0 0) 0x7fd438058830 con 0x7fd44409bf50 2024-03-04T10:04:54.832 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:54.832 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":38,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:53.187527+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd43006b4d0 msgr2=0x7fd43006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd43006b4d0 0x7fd43006d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd434036000 tx=0x7fd434030000 comp rx=0 tx=0).stop 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 msgr2=0x7fd44415aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 0x7fd44415aa40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd43800e7b0 tx=0x7fd43800ec70 comp rx=0 tx=0).stop 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 shutdown_connections 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd43006b4d0 0x7fd43006d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd44415af80 0x7fd44409ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd44409bf50 0x7fd44415aa40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 --2- 172.21.15.99:0/3676747060 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd42c003b70 0x7fd444147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.832+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 >> 172.21.15.99:0/3676747060 conn(0x7fd4440cfef0 msgr2=0x7fd44415c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.833+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 shutdown_connections 2024-03-04T10:04:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.833+0000 7fd449d29640 1 -- 172.21.15.99:0/3676747060 wait complete. 2024-03-04T10:04:54.845 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T10:04:54.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:04:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.943+0000 7efd1e680640 1 Processor -- start 2024-03-04T10:04:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.943+0000 7efd1e680640 1 -- start start 2024-03-04T10:04:54.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1e680640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1806d3b0 0x7efd1815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1e680640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efd1815b410 0x7efd1815b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1e680640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1815be00 con 0x7efd1815b030 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1e680640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180b2720 con 0x7efd1815b410 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1e680640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806d8b0 con 0x7efd1806d3b0 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1cc1d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efd1815b410 0x7efd1815b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd17fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7efd1815b030 0x7efd1815dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56958/0 (socket says 172.21.15.99:56958) 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd17fff640 1 -- 172.21.15.99:0/3837300964 learned_addr learned my addr 172.21.15.99:0/3837300964 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd177fe640 1 --2- 172.21.15.99:0/3837300964 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1806d3b0 0x7efd1815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1cc1d640 1 -- 172.21.15.99:0/3837300964 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1806d3b0 msgr2=0x7efd1815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1cc1d640 1 --2- 172.21.15.99:0/3837300964 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1806d3b0 0x7efd1815c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd16ffd640 1 -- 172.21.15.99:0/3837300964 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830618026 0 0) 0x7efd1815be00 con 0x7efd1815b030 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1cc1d640 1 -- 172.21.15.99:0/3837300964 >> v1:172.21.15.99:6789/0 conn(0x7efd1815b030 legacy=0x7efd1815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.944+0000 7efd1cc1d640 1 -- 172.21.15.99:0/3837300964 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1806db90 con 0x7efd1815b410 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.945+0000 7efd1cc1d640 1 --2- 172.21.15.99:0/3837300964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd1815b8c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efd10009d90 tx=0x7efd1002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb977f83200ea8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.945+0000 7efd16ffd640 1 -- 172.21.15.99:0/3837300964 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10039070 con 0x7efd1815b410 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.945+0000 7efd16ffd640 1 -- 172.21.15.99:0/3837300964 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1002cbc0 con 0x7efd1815b410 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.945+0000 7efd16ffd640 1 -- 172.21.15.99:0/3837300964 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10034440 con 0x7efd1815b410 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 -- 172.21.15.99:0/3837300964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 msgr2=0x7efd1815b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 --2- 172.21.15.99:0/3837300964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd1815b8c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efd10009d90 tx=0x7efd1002be10 comp rx=0 tx=0).stop 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 -- 172.21.15.99:0/3837300964 shutdown_connections 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 --2- 172.21.15.99:0/3837300964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd1815b8c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 --2- 172.21.15.99:0/3837300964 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1806d3b0 0x7efd1815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 -- 172.21.15.99:0/3837300964 >> 172.21.15.99:0/3837300964 conn(0x7efd1806acd0 msgr2=0x7efd1806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:54.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 -- 172.21.15.99:0/3837300964 shutdown_connections 2024-03-04T10:04:54.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.946+0000 7efd1e680640 1 -- 172.21.15.99:0/3837300964 wait complete. 2024-03-04T10:04:54.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.947+0000 7efd1e680640 1 Processor -- start 2024-03-04T10:04:54.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.947+0000 7efd1e680640 1 -- start start 2024-03-04T10:04:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1e680640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1806d3b0 0x7efd181521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1e680640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1815b030 0x7efd180ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1e680640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd18154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1e680640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806e690 con 0x7efd1815b410 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1e680640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806e150 con 0x7efd1806d3b0 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1e680640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1806e810 con 0x7efd1815b030 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd18154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd18154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50028/0 (socket says 172.21.15.99:50028) 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 -- 172.21.15.99:0/2614967268 learned_addr learned my addr 172.21.15.99:0/2614967268 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd177fe640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1815b030 0x7efd180ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 -- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1815b030 msgr2=0x7efd180ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1815b030 0x7efd180ae5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 -- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1806d3b0 msgr2=0x7efd181521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd17fff640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1806d3b0 0x7efd181521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1806d3b0 0x7efd181521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 -- 172.21.15.99:0/2614967268 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd180ad500 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.948+0000 7efd1cc1d640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd18154d30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efd10009d90 tx=0x7efd10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.949+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10044020 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.949+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1003f470 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.949+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd180ad7e0 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.949+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10039040 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.949+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd180adda0 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.950+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd18074f60 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.950+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7efd10047020 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.950+0000 7efd14ff9640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efd0406b4d0 0x7efd0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.950+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efd100ad030 con 0x7efd1815b410 2024-03-04T10:04:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.951+0000 7efd17fff640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efd0406b4d0 0x7efd0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:54.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.951+0000 7efd17fff640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efd0406b4d0 0x7efd0406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efd08007fd0 tx=0x7efd08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:54.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:54.955+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efd1004e030 con 0x7efd1815b410 2024-03-04T10:04:55.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.120+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd180b0980 con 0x7efd0406b4d0 2024-03-04T10:04:55.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.127+0000 7efd14ff9640 1 -- 172.21.15.99:0/2614967268 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251886 (secure 0 0 0) 0x7efd180b0980 con 0x7efd0406b4d0 2024-03-04T10:04:55.131 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:55.133 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efd0406b4d0 msgr2=0x7efd0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efd0406b4d0 0x7efd0406d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efd08007fd0 tx=0x7efd08009040 comp rx=0 tx=0).stop 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 msgr2=0x7efd18154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd18154d30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7efd10009d90 tx=0x7efd10004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 shutdown_connections 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7efd0406b4d0 0x7efd0406d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efd1815b410 0x7efd18154d30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efd1815b030 0x7efd180ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 --2- 172.21.15.99:0/2614967268 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efd1806d3b0 0x7efd181521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 >> 172.21.15.99:0/2614967268 conn(0x7efd1806acd0 msgr2=0x7efd180af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:55.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 shutdown_connections 2024-03-04T10:04:55.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.130+0000 7efd1e680640 1 -- 172.21.15.99:0/2614967268 wait complete. 2024-03-04T10:04:55.142 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-03-04T10:04:53.455389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220592,"kb_used_data":4408,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754128,"statfs":{"total":773094113280,"available":772868227072,"internally_reserved":0,"allocated":4513792,"data_stored":2596369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000858"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206453+0000","last_change":"2024-03-04T10:04:52.199649+0000","last_active":"2024-03-04T10:04:52.206453+0000","last_peered":"2024-03-04T10:04:52.206453+0000","last_clean":"2024-03-04T10:04:52.206453+0000","last_became_active":"2024-03-04T10:04:52.199398+0000","last_became_peered":"2024-03-04T10:04:52.199398+0000","last_unstale":"2024-03-04T10:04:52.206453+0000","last_undegraded":"2024-03-04T10:04:52.206453+0000","last_fullsized":"2024-03-04T10:04:52.206453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208667+0000","last_change":"2024-03-04T10:04:52.202939+0000","last_active":"2024-03-04T10:04:52.208667+0000","last_peered":"2024-03-04T10:04:52.208667+0000","last_clean":"2024-03-04T10:04:52.208667+0000","last_became_active":"2024-03-04T10:04:52.200847+0000","last_became_peered":"2024-03-04T10:04:52.200847+0000","last_unstale":"2024-03-04T10:04:52.208667+0000","last_undegraded":"2024-03-04T10:04:52.208667+0000","last_fullsized":"2024-03-04T10:04:52.208667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:31.922962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198562+0000","last_change":"2024-03-04T10:04:52.194854+0000","last_active":"2024-03-04T10:04:52.198562+0000","last_peered":"2024-03-04T10:04:52.198562+0000","last_clean":"2024-03-04T10:04:52.198562+0000","last_became_active":"2024-03-04T10:04:52.194603+0000","last_became_peered":"2024-03-04T10:04:52.194603+0000","last_unstale":"2024-03-04T10:04:52.198562+0000","last_undegraded":"2024-03-04T10:04:52.198562+0000","last_fullsized":"2024-03-04T10:04:52.198562+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:14.100423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205997+0000","last_change":"2024-03-04T10:04:52.200439+0000","last_active":"2024-03-04T10:04:52.205997+0000","last_peered":"2024-03-04T10:04:52.205997+0000","last_clean":"2024-03-04T10:04:52.205997+0000","last_became_active":"2024-03-04T10:04:52.200194+0000","last_became_peered":"2024-03-04T10:04:52.200194+0000","last_unstale":"2024-03-04T10:04:52.205997+0000","last_undegraded":"2024-03-04T10:04:52.205997+0000","last_fullsized":"2024-03-04T10:04:52.205997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:48:01.778909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.201126+0000","last_change":"2024-03-04T10:04:52.199976+0000","last_active":"2024-03-04T10:04:52.201126+0000","last_peered":"2024-03-04T10:04:52.201126+0000","last_clean":"2024-03-04T10:04:52.201126+0000","last_became_active":"2024-03-04T10:04:52.199727+0000","last_became_peered":"2024-03-04T10:04:52.199727+0000","last_unstale":"2024-03-04T10:04:52.201126+0000","last_undegraded":"2024-03-04T10:04:52.201126+0000","last_fullsized":"2024-03-04T10:04:52.201126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:40.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206935+0000","last_change":"2024-03-04T10:04:52.199653+0000","last_active":"2024-03-04T10:04:52.206935+0000","last_peered":"2024-03-04T10:04:52.206935+0000","last_clean":"2024-03-04T10:04:52.206935+0000","last_became_active":"2024-03-04T10:04:52.199380+0000","last_became_peered":"2024-03-04T10:04:52.199380+0000","last_unstale":"2024-03-04T10:04:52.206935+0000","last_undegraded":"2024-03-04T10:04:52.206935+0000","last_fullsized":"2024-03-04T10:04:52.206935+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:01.728765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209073+0000","last_change":"2024-03-04T10:04:52.202562+0000","last_active":"2024-03-04T10:04:52.209073+0000","last_peered":"2024-03-04T10:04:52.209073+0000","last_clean":"2024-03-04T10:04:52.209073+0000","last_became_active":"2024-03-04T10:04:52.201480+0000","last_became_peered":"2024-03-04T10:04:52.201480+0000","last_unstale":"2024-03-04T10:04:52.209073+0000","last_undegraded":"2024-03-04T10:04:52.209073+0000","last_fullsized":"2024-03-04T10:04:52.209073+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208722+0000","last_change":"2024-03-04T10:04:52.204418+0000","last_active":"2024-03-04T10:04:52.208722+0000","last_peered":"2024-03-04T10:04:52.208722+0000","last_clean":"2024-03-04T10:04:52.208722+0000","last_became_active":"2024-03-04T10:04:52.204159+0000","last_became_peered":"2024-03-04T10:04:52.204159+0000","last_unstale":"2024-03-04T10:04:52.208722+0000","last_undegraded":"2024-03-04T10:04:52.208722+0000","last_fullsized":"2024-03-04T10:04:52.208722+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:21.848884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198620+0000","last_change":"2024-03-04T10:04:52.194939+0000","last_active":"2024-03-04T10:04:52.198620+0000","last_peered":"2024-03-04T10:04:52.198620+0000","last_clean":"2024-03-04T10:04:52.198620+0000","last_became_active":"2024-03-04T10:04:52.194653+0000","last_became_peered":"2024-03-04T10:04:52.194653+0000","last_unstale":"2024-03-04T10:04:52.198620+0000","last_undegraded":"2024-03-04T10:04:52.198620+0000","last_fullsized":"2024-03-04T10:04:52.198620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:41.181561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200374+0000","last_change":"2024-03-04T10:04:52.195037+0000","last_active":"2024-03-04T10:04:52.200374+0000","last_peered":"2024-03-04T10:04:52.200374+0000","last_clean":"2024-03-04T10:04:52.200374+0000","last_became_active":"2024-03-04T10:04:52.194750+0000","last_became_peered":"2024-03-04T10:04:52.194750+0000","last_unstale":"2024-03-04T10:04:52.200374+0000","last_undegraded":"2024-03-04T10:04:52.200374+0000","last_fullsized":"2024-03-04T10:04:52.200374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:14.100423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211929+0000","last_change":"2024-03-04T10:04:52.208932+0000","last_active":"2024-03-04T10:04:52.211929+0000","last_peered":"2024-03-04T10:04:52.211929+0000","last_clean":"2024-03-04T10:04:52.211929+0000","last_became_active":"2024-03-04T10:04:52.208650+0000","last_became_peered":"2024-03-04T10:04:52.208650+0000","last_unstale":"2024-03-04T10:04:52.211929+0000","last_undegraded":"2024-03-04T10:04:52.211929+0000","last_fullsized":"2024-03-04T10:04:52.211929+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:46:08.623375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206497+0000","last_change":"2024-03-04T10:04:52.195702+0000","last_active":"2024-03-04T10:04:52.206497+0000","last_peered":"2024-03-04T10:04:52.206497+0000","last_clean":"2024-03-04T10:04:52.206497+0000","last_became_active":"2024-03-04T10:04:52.195406+0000","last_became_peered":"2024-03-04T10:04:52.195406+0000","last_unstale":"2024-03-04T10:04:52.206497+0000","last_undegraded":"2024-03-04T10:04:52.206497+0000","last_fullsized":"2024-03-04T10:04:52.206497+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:50.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200199+0000","last_change":"2024-03-04T10:04:52.195347+0000","last_active":"2024-03-04T10:04:52.200199+0000","last_peered":"2024-03-04T10:04:52.200199+0000","last_clean":"2024-03-04T10:04:52.200199+0000","last_became_active":"2024-03-04T10:04:52.194696+0000","last_became_peered":"2024-03-04T10:04:52.194696+0000","last_unstale":"2024-03-04T10:04:52.200199+0000","last_undegraded":"2024-03-04T10:04:52.200199+0000","last_fullsized":"2024-03-04T10:04:52.200199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:50.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205124+0000","last_change":"2024-03-04T10:04:52.199326+0000","last_active":"2024-03-04T10:04:52.205124+0000","last_peered":"2024-03-04T10:04:52.205124+0000","last_clean":"2024-03-04T10:04:52.205124+0000","last_became_active":"2024-03-04T10:04:52.198978+0000","last_became_peered":"2024-03-04T10:04:52.198978+0000","last_unstale":"2024-03-04T10:04:52.205124+0000","last_undegraded":"2024-03-04T10:04:52.205124+0000","last_fullsized":"2024-03-04T10:04:52.205124+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:01.728765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.197731+0000","last_change":"2024-03-04T10:04:52.192798+0000","last_active":"2024-03-04T10:04:52.197731+0000","last_peered":"2024-03-04T10:04:52.197731+0000","last_clean":"2024-03-04T10:04:52.197731+0000","last_became_active":"2024-03-04T10:04:52.192512+0000","last_became_peered":"2024-03-04T10:04:52.192512+0000","last_unstale":"2024-03-04T10:04:52.197731+0000","last_undegraded":"2024-03-04T10:04:52.197731+0000","last_fullsized":"2024-03-04T10:04:52.197731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:02.066773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198278+0000","last_change":"2024-03-04T10:04:52.195037+0000","last_active":"2024-03-04T10:04:52.198278+0000","last_peered":"2024-03-04T10:04:52.198278+0000","last_clean":"2024-03-04T10:04:52.198278+0000","last_became_active":"2024-03-04T10:04:52.194751+0000","last_became_peered":"2024-03-04T10:04:52.194751+0000","last_unstale":"2024-03-04T10:04:52.198278+0000","last_undegraded":"2024-03-04T10:04:52.198278+0000","last_fullsized":"2024-03-04T10:04:52.198278+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:14.100423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.199319+0000","last_change":"2024-03-04T10:04:52.195633+0000","last_active":"2024-03-04T10:04:52.199319+0000","last_peered":"2024-03-04T10:04:52.199319+0000","last_clean":"2024-03-04T10:04:52.199319+0000","last_became_active":"2024-03-04T10:04:52.194911+0000","last_became_peered":"2024-03-04T10:04:52.194911+0000","last_unstale":"2024-03-04T10:04:52.199319+0000","last_undegraded":"2024-03-04T10:04:52.199319+0000","last_fullsized":"2024-03-04T10:04:52.199319+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:41.181561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202475+0000","last_change":"2024-03-04T10:04:52.198978+0000","last_active":"2024-03-04T10:04:52.202475+0000","last_peered":"2024-03-04T10:04:52.202475+0000","last_clean":"2024-03-04T10:04:52.202475+0000","last_became_active":"2024-03-04T10:04:52.198722+0000","last_became_peered":"2024-03-04T10:04:52.198722+0000","last_unstale":"2024-03-04T10:04:52.202475+0000","last_undegraded":"2024-03-04T10:04:52.202475+0000","last_fullsized":"2024-03-04T10:04:52.202475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:29:17.436926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208338+0000","last_change":"2024-03-04T10:04:52.201662+0000","last_active":"2024-03-04T10:04:52.208338+0000","last_peered":"2024-03-04T10:04:52.208338+0000","last_clean":"2024-03-04T10:04:52.208338+0000","last_became_active":"2024-03-04T10:04:52.201403+0000","last_became_peered":"2024-03-04T10:04:52.201403+0000","last_unstale":"2024-03-04T10:04:52.208338+0000","last_undegraded":"2024-03-04T10:04:52.208338+0000","last_fullsized":"2024-03-04T10:04:52.208338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:27:07.884258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.203260+0000","last_change":"2024-03-04T10:04:52.199707+0000","last_active":"2024-03-04T10:04:52.203260+0000","last_peered":"2024-03-04T10:04:52.203260+0000","last_clean":"2024-03-04T10:04:52.203260+0000","last_became_active":"2024-03-04T10:04:52.199344+0000","last_became_peered":"2024-03-04T10:04:52.199344+0000","last_unstale":"2024-03-04T10:04:52.203260+0000","last_undegraded":"2024-03-04T10:04:52.203260+0000","last_fullsized":"2024-03-04T10:04:52.203260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:30.103279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209153+0000","last_change":"2024-03-04T10:04:52.202972+0000","last_active":"2024-03-04T10:04:52.209153+0000","last_peered":"2024-03-04T10:04:52.209153+0000","last_clean":"2024-03-04T10:04:52.209153+0000","last_became_active":"2024-03-04T10:04:52.202414+0000","last_became_peered":"2024-03-04T10:04:52.202414+0000","last_unstale":"2024-03-04T10:04:52.209153+0000","last_undegraded":"2024-03-04T10:04:52.209153+0000","last_fullsized":"2024-03-04T10:04:52.209153+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:31.447689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208163+0000","last_change":"2024-03-04T10:04:52.200815+0000","last_active":"2024-03-04T10:04:52.208163+0000","last_peered":"2024-03-04T10:04:52.208163+0000","last_clean":"2024-03-04T10:04:52.208163+0000","last_became_active":"2024-03-04T10:04:52.200421+0000","last_became_peered":"2024-03-04T10:04:52.200421+0000","last_unstale":"2024-03-04T10:04:52.208163+0000","last_undegraded":"2024-03-04T10:04:52.208163+0000","last_fullsized":"2024-03-04T10:04:52.208163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:26.571217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.196112+0000","last_change":"2024-03-04T10:04:52.193528+0000","last_active":"2024-03-04T10:04:52.196112+0000","last_peered":"2024-03-04T10:04:52.196112+0000","last_clean":"2024-03-04T10:04:52.196112+0000","last_became_active":"2024-03-04T10:04:52.193244+0000","last_became_peered":"2024-03-04T10:04:52.193244+0000","last_unstale":"2024-03-04T10:04:52.196112+0000","last_undegraded":"2024-03-04T10:04:52.196112+0000","last_fullsized":"2024-03-04T10:04:52.196112+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:04.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198232+0000","last_change":"2024-03-04T10:04:52.194676+0000","last_active":"2024-03-04T10:04:52.198232+0000","last_peered":"2024-03-04T10:04:52.198232+0000","last_clean":"2024-03-04T10:04:52.198232+0000","last_became_active":"2024-03-04T10:04:52.194422+0000","last_became_peered":"2024-03-04T10:04:52.194422+0000","last_unstale":"2024-03-04T10:04:52.198232+0000","last_undegraded":"2024-03-04T10:04:52.198232+0000","last_fullsized":"2024-03-04T10:04:52.198232+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:06.699075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202509+0000","last_change":"2024-03-04T10:04:52.199912+0000","last_active":"2024-03-04T10:04:52.202509+0000","last_peered":"2024-03-04T10:04:52.202509+0000","last_clean":"2024-03-04T10:04:52.202509+0000","last_became_active":"2024-03-04T10:04:52.199610+0000","last_became_peered":"2024-03-04T10:04:52.199610+0000","last_unstale":"2024-03-04T10:04:52.202509+0000","last_undegraded":"2024-03-04T10:04:52.202509+0000","last_fullsized":"2024-03-04T10:04:52.202509+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:40.693258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207091+0000","last_change":"2024-03-04T10:04:52.200264+0000","last_active":"2024-03-04T10:04:52.207091+0000","last_peered":"2024-03-04T10:04:52.207091+0000","last_clean":"2024-03-04T10:04:52.207091+0000","last_became_active":"2024-03-04T10:04:52.199991+0000","last_became_peered":"2024-03-04T10:04:52.199991+0000","last_unstale":"2024-03-04T10:04:52.207091+0000","last_undegraded":"2024-03-04T10:04:52.207091+0000","last_fullsized":"2024-03-04T10:04:52.207091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:16.366185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.197275+0000","last_change":"2024-03-04T10:04:52.196527+0000","last_active":"2024-03-04T10:04:52.197275+0000","last_peered":"2024-03-04T10:04:52.197275+0000","last_clean":"2024-03-04T10:04:52.197275+0000","last_became_active":"2024-03-04T10:04:52.196261+0000","last_became_peered":"2024-03-04T10:04:52.196261+0000","last_unstale":"2024-03-04T10:04:52.197275+0000","last_undegraded":"2024-03-04T10:04:52.197275+0000","last_fullsized":"2024-03-04T10:04:52.197275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:52.780338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209033+0000","last_change":"2024-03-04T10:04:52.202193+0000","last_active":"2024-03-04T10:04:52.209033+0000","last_peered":"2024-03-04T10:04:52.209033+0000","last_clean":"2024-03-04T10:04:52.209033+0000","last_became_active":"2024-03-04T10:04:52.201708+0000","last_became_peered":"2024-03-04T10:04:52.201708+0000","last_unstale":"2024-03-04T10:04:52.209033+0000","last_undegraded":"2024-03-04T10:04:52.209033+0000","last_fullsized":"2024-03-04T10:04:52.209033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211631+0000","last_change":"2024-03-04T10:04:52.206905+0000","last_active":"2024-03-04T10:04:52.211631+0000","last_peered":"2024-03-04T10:04:52.211631+0000","last_clean":"2024-03-04T10:04:52.211631+0000","last_became_active":"2024-03-04T10:04:52.206655+0000","last_became_peered":"2024-03-04T10:04:52.206655+0000","last_unstale":"2024-03-04T10:04:52.211631+0000","last_undegraded":"2024-03-04T10:04:52.211631+0000","last_fullsized":"2024-03-04T10:04:52.211631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:01.728765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202447+0000","last_change":"2024-03-04T10:04:52.197798+0000","last_active":"2024-03-04T10:04:52.202447+0000","last_peered":"2024-03-04T10:04:52.202447+0000","last_clean":"2024-03-04T10:04:52.202447+0000","last_became_active":"2024-03-04T10:04:52.197487+0000","last_became_peered":"2024-03-04T10:04:52.197487+0000","last_unstale":"2024-03-04T10:04:52.202447+0000","last_undegraded":"2024-03-04T10:04:52.202447+0000","last_fullsized":"2024-03-04T10:04:52.202447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:36.564124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198140+0000","last_change":"2024-03-04T10:04:52.194073+0000","last_active":"2024-03-04T10:04:52.198140+0000","last_peered":"2024-03-04T10:04:52.198140+0000","last_clean":"2024-03-04T10:04:52.198140+0000","last_became_active":"2024-03-04T10:04:52.193742+0000","last_became_peered":"2024-03-04T10:04:52.193742+0000","last_unstale":"2024-03-04T10:04:52.198140+0000","last_undegraded":"2024-03-04T10:04:52.198140+0000","last_fullsized":"2024-03-04T10:04:52.198140+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.204889+0000","last_change":"2024-03-04T10:04:52.199105+0000","last_active":"2024-03-04T10:04:52.204889+0000","last_peered":"2024-03-04T10:04:52.204889+0000","last_clean":"2024-03-04T10:04:52.204889+0000","last_became_active":"2024-03-04T10:04:52.198825+0000","last_became_peered":"2024-03-04T10:04:52.198825+0000","last_unstale":"2024-03-04T10:04:52.204889+0000","last_undegraded":"2024-03-04T10:04:52.204889+0000","last_fullsized":"2024-03-04T10:04:52.204889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:40.693258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198098+0000","last_change":"2024-03-04T10:04:52.194930+0000","last_active":"2024-03-04T10:04:52.198098+0000","last_peered":"2024-03-04T10:04:52.198098+0000","last_clean":"2024-03-04T10:04:52.198098+0000","last_became_active":"2024-03-04T10:04:52.194637+0000","last_became_peered":"2024-03-04T10:04:52.194637+0000","last_unstale":"2024-03-04T10:04:52.198098+0000","last_undegraded":"2024-03-04T10:04:52.198098+0000","last_fullsized":"2024-03-04T10:04:52.198098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:02.066773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211936+0000","last_change":"2024-03-04T10:04:51.188914+0000","last_active":"2024-03-04T10:04:52.211936+0000","last_peered":"2024-03-04T10:04:52.211936+0000","last_clean":"2024-03-04T10:04:52.211936+0000","last_became_active":"2024-03-04T10:04:51.188650+0000","last_became_peered":"2024-03-04T10:04:51.188650+0000","last_unstale":"2024-03-04T10:04:52.211936+0000","last_undegraded":"2024-03-04T10:04:52.211936+0000","last_fullsized":"2024-03-04T10:04:52.211936+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:52.296428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207151+0000","last_change":"2024-03-04T10:04:52.200962+0000","last_active":"2024-03-04T10:04:52.207151+0000","last_peered":"2024-03-04T10:04:52.207151+0000","last_clean":"2024-03-04T10:04:52.207151+0000","last_became_active":"2024-03-04T10:04:52.200689+0000","last_became_peered":"2024-03-04T10:04:52.200689+0000","last_unstale":"2024-03-04T10:04:52.207151+0000","last_undegraded":"2024-03-04T10:04:52.207151+0000","last_fullsized":"2024-03-04T10:04:52.207151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:04.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205537+0000","last_change":"2024-03-04T10:04:51.182645+0000","last_active":"2024-03-04T10:04:52.205537+0000","last_peered":"2024-03-04T10:04:52.205537+0000","last_clean":"2024-03-04T10:04:52.205537+0000","last_became_active":"2024-03-04T10:04:51.182207+0000","last_became_peered":"2024-03-04T10:04:51.182207+0000","last_unstale":"2024-03-04T10:04:52.205537+0000","last_undegraded":"2024-03-04T10:04:52.205537+0000","last_fullsized":"2024-03-04T10:04:52.205537+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:03.804872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.201113+0000","last_change":"2024-03-04T10:04:52.196872+0000","last_active":"2024-03-04T10:04:52.201113+0000","last_peered":"2024-03-04T10:04:52.201113+0000","last_clean":"2024-03-04T10:04:52.201113+0000","last_became_active":"2024-03-04T10:04:52.196553+0000","last_became_peered":"2024-03-04T10:04:52.196553+0000","last_unstale":"2024-03-04T10:04:52.201113+0000","last_undegraded":"2024-03-04T10:04:52.201113+0000","last_fullsized":"2024-03-04T10:04:52.201113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:40.693258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206759+0000","last_change":"2024-03-04T10:04:51.183200+0000","last_active":"2024-03-04T10:04:52.206759+0000","last_peered":"2024-03-04T10:04:52.206759+0000","last_clean":"2024-03-04T10:04:52.206759+0000","last_became_active":"2024-03-04T10:04:51.182932+0000","last_became_peered":"2024-03-04T10:04:51.182932+0000","last_unstale":"2024-03-04T10:04:52.206759+0000","last_undegraded":"2024-03-04T10:04:52.206759+0000","last_fullsized":"2024-03-04T10:04:52.206759+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:29.099324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206429+0000","last_change":"2024-03-04T10:04:52.199095+0000","last_active":"2024-03-04T10:04:52.206429+0000","last_peered":"2024-03-04T10:04:52.206429+0000","last_clean":"2024-03-04T10:04:52.206429+0000","last_became_active":"2024-03-04T10:04:52.198835+0000","last_became_peered":"2024-03-04T10:04:52.198835+0000","last_unstale":"2024-03-04T10:04:52.206429+0000","last_undegraded":"2024-03-04T10:04:52.206429+0000","last_fullsized":"2024-03-04T10:04:52.206429+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210747+0000","last_change":"2024-03-04T10:04:51.186290+0000","last_active":"2024-03-04T10:04:52.210747+0000","last_peered":"2024-03-04T10:04:52.210747+0000","last_clean":"2024-03-04T10:04:52.210747+0000","last_became_active":"2024-03-04T10:04:51.186016+0000","last_became_peered":"2024-03-04T10:04:51.186016+0000","last_unstale":"2024-03-04T10:04:52.210747+0000","last_undegraded":"2024-03-04T10:04:52.210747+0000","last_fullsized":"2024-03-04T10:04:52.210747+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:29:16.432972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208465+0000","last_change":"2024-03-04T10:04:52.204984+0000","last_active":"2024-03-04T10:04:52.208465+0000","last_peered":"2024-03-04T10:04:52.208465+0000","last_clean":"2024-03-04T10:04:52.208465+0000","last_became_active":"2024-03-04T10:04:52.204490+0000","last_became_peered":"2024-03-04T10:04:52.204490+0000","last_unstale":"2024-03-04T10:04:52.208465+0000","last_undegraded":"2024-03-04T10:04:52.208465+0000","last_fullsized":"2024-03-04T10:04:52.208465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210630+0000","last_change":"2024-03-04T10:04:51.191689+0000","last_active":"2024-03-04T10:04:52.210630+0000","last_peered":"2024-03-04T10:04:52.210630+0000","last_clean":"2024-03-04T10:04:52.210630+0000","last_became_active":"2024-03-04T10:04:51.191150+0000","last_became_peered":"2024-03-04T10:04:51.191150+0000","last_unstale":"2024-03-04T10:04:52.210630+0000","last_undegraded":"2024-03-04T10:04:52.210630+0000","last_fullsized":"2024-03-04T10:04:52.210630+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.567262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209174+0000","last_change":"2024-03-04T10:04:52.203610+0000","last_active":"2024-03-04T10:04:52.209174+0000","last_peered":"2024-03-04T10:04:52.209174+0000","last_clean":"2024-03-04T10:04:52.209174+0000","last_became_active":"2024-03-04T10:04:52.203297+0000","last_became_peered":"2024-03-04T10:04:52.203297+0000","last_unstale":"2024-03-04T10:04:52.209174+0000","last_undegraded":"2024-03-04T10:04:52.209174+0000","last_fullsized":"2024-03-04T10:04:52.209174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:31.922962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210259+0000","last_change":"2024-03-04T10:04:51.179466+0000","last_active":"2024-03-04T10:04:52.210259+0000","last_peered":"2024-03-04T10:04:52.210259+0000","last_clean":"2024-03-04T10:04:52.210259+0000","last_became_active":"2024-03-04T10:04:51.179203+0000","last_became_peered":"2024-03-04T10:04:51.179203+0000","last_unstale":"2024-03-04T10:04:52.210259+0000","last_undegraded":"2024-03-04T10:04:52.210259+0000","last_fullsized":"2024-03-04T10:04:52.210259+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:00.724810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209487+0000","last_change":"2024-03-04T10:04:52.207971+0000","last_active":"2024-03-04T10:04:52.209487+0000","last_peered":"2024-03-04T10:04:52.209487+0000","last_clean":"2024-03-04T10:04:52.209487+0000","last_became_active":"2024-03-04T10:04:52.207695+0000","last_became_peered":"2024-03-04T10:04:52.207695+0000","last_unstale":"2024-03-04T10:04:52.209487+0000","last_undegraded":"2024-03-04T10:04:52.209487+0000","last_fullsized":"2024-03-04T10:04:52.209487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187762+0000","last_change":"2024-03-04T10:04:51.189127+0000","last_active":"2024-03-04T10:04:52.187762+0000","last_peered":"2024-03-04T10:04:52.187762+0000","last_clean":"2024-03-04T10:04:52.187762+0000","last_became_active":"2024-03-04T10:04:51.188873+0000","last_became_peered":"2024-03-04T10:04:51.188873+0000","last_unstale":"2024-03-04T10:04:52.187762+0000","last_undegraded":"2024-03-04T10:04:52.187762+0000","last_fullsized":"2024-03-04T10:04:52.187762+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:35.560169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208789+0000","last_change":"2024-03-04T10:04:52.204639+0000","last_active":"2024-03-04T10:04:52.208789+0000","last_peered":"2024-03-04T10:04:52.208789+0000","last_clean":"2024-03-04T10:04:52.208789+0000","last_became_active":"2024-03-04T10:04:52.204342+0000","last_became_peered":"2024-03-04T10:04:52.204342+0000","last_unstale":"2024-03-04T10:04:52.208789+0000","last_undegraded":"2024-03-04T10:04:52.208789+0000","last_fullsized":"2024-03-04T10:04:52.208789+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:58.274038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.190379+0000","last_change":"2024-03-04T10:04:51.184748+0000","last_active":"2024-03-04T10:04:52.190379+0000","last_peered":"2024-03-04T10:04:52.190379+0000","last_clean":"2024-03-04T10:04:52.190379+0000","last_became_active":"2024-03-04T10:04:51.184471+0000","last_became_peered":"2024-03-04T10:04:51.184471+0000","last_unstale":"2024-03-04T10:04:52.190379+0000","last_undegraded":"2024-03-04T10:04:52.190379+0000","last_fullsized":"2024-03-04T10:04:52.190379+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:52.296428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.201198+0000","last_change":"2024-03-04T10:04:52.200255+0000","last_active":"2024-03-04T10:04:52.201198+0000","last_peered":"2024-03-04T10:04:52.201198+0000","last_clean":"2024-03-04T10:04:52.201198+0000","last_became_active":"2024-03-04T10:04:52.199854+0000","last_became_peered":"2024-03-04T10:04:52.199854+0000","last_unstale":"2024-03-04T10:04:52.201198+0000","last_undegraded":"2024-03-04T10:04:52.201198+0000","last_fullsized":"2024-03-04T10:04:52.201198+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:07.065080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.193320+0000","last_change":"2024-03-04T10:04:51.186641+0000","last_active":"2024-03-04T10:04:52.193320+0000","last_peered":"2024-03-04T10:04:52.193320+0000","last_clean":"2024-03-04T10:04:52.193320+0000","last_became_active":"2024-03-04T10:04:51.186381+0000","last_became_peered":"2024-03-04T10:04:51.186381+0000","last_unstale":"2024-03-04T10:04:52.193320+0000","last_undegraded":"2024-03-04T10:04:52.193320+0000","last_fullsized":"2024-03-04T10:04:52.193320+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:13.096468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.199370+0000","last_change":"2024-03-04T10:04:52.197322+0000","last_active":"2024-03-04T10:04:52.199370+0000","last_peered":"2024-03-04T10:04:52.199370+0000","last_clean":"2024-03-04T10:04:52.199370+0000","last_became_active":"2024-03-04T10:04:52.197024+0000","last_became_peered":"2024-03-04T10:04:52.197024+0000","last_unstale":"2024-03-04T10:04:52.199370+0000","last_undegraded":"2024-03-04T10:04:52.199370+0000","last_fullsized":"2024-03-04T10:04:52.199370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.185090+0000","last_change":"2024-03-04T10:04:51.192066+0000","last_active":"2024-03-04T10:04:52.185090+0000","last_peered":"2024-03-04T10:04:52.185090+0000","last_clean":"2024-03-04T10:04:52.185090+0000","last_became_active":"2024-03-04T10:04:51.191793+0000","last_became_peered":"2024-03-04T10:04:51.191793+0000","last_unstale":"2024-03-04T10:04:52.185090+0000","last_undegraded":"2024-03-04T10:04:52.185090+0000","last_fullsized":"2024-03-04T10:04:52.185090+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:48.933510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198568+0000","last_change":"2024-03-04T10:04:52.193842+0000","last_active":"2024-03-04T10:04:52.198568+0000","last_peered":"2024-03-04T10:04:52.198568+0000","last_clean":"2024-03-04T10:04:52.198568+0000","last_became_active":"2024-03-04T10:04:52.193554+0000","last_became_peered":"2024-03-04T10:04:52.193554+0000","last_unstale":"2024-03-04T10:04:52.198568+0000","last_undegraded":"2024-03-04T10:04:52.198568+0000","last_fullsized":"2024-03-04T10:04:52.198568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211885+0000","last_change":"2024-03-04T10:04:51.191012+0000","last_active":"2024-03-04T10:04:52.211885+0000","last_peered":"2024-03-04T10:04:52.211885+0000","last_clean":"2024-03-04T10:04:52.211885+0000","last_became_active":"2024-03-04T10:04:51.190717+0000","last_became_peered":"2024-03-04T10:04:51.190717+0000","last_unstale":"2024-03-04T10:04:52.211885+0000","last_undegraded":"2024-03-04T10:04:52.211885+0000","last_fullsized":"2024-03-04T10:04:52.211885+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:00.724810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208135+0000","last_change":"2024-03-04T10:04:52.201515+0000","last_active":"2024-03-04T10:04:52.208135+0000","last_peered":"2024-03-04T10:04:52.208135+0000","last_clean":"2024-03-04T10:04:52.208135+0000","last_became_active":"2024-03-04T10:04:52.201230+0000","last_became_peered":"2024-03-04T10:04:52.201230+0000","last_unstale":"2024-03-04T10:04:52.208135+0000","last_undegraded":"2024-03-04T10:04:52.208135+0000","last_fullsized":"2024-03-04T10:04:52.208135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:16.366185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.217030+0000","last_change":"2024-03-04T10:04:51.192307+0000","last_active":"2024-03-04T10:04:52.217030+0000","last_peered":"2024-03-04T10:04:52.217030+0000","last_clean":"2024-03-04T10:04:52.217030+0000","last_became_active":"2024-03-04T10:04:51.191992+0000","last_became_peered":"2024-03-04T10:04:51.191992+0000","last_unstale":"2024-03-04T10:04:52.217030+0000","last_undegraded":"2024-03-04T10:04:52.217030+0000","last_fullsized":"2024-03-04T10:04:52.217030+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:49.953133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205153+0000","last_change":"2024-03-04T10:04:52.196294+0000","last_active":"2024-03-04T10:04:52.205153+0000","last_peered":"2024-03-04T10:04:52.205153+0000","last_clean":"2024-03-04T10:04:52.205153+0000","last_became_active":"2024-03-04T10:04:52.193706+0000","last_became_peered":"2024-03-04T10:04:52.193706+0000","last_unstale":"2024-03-04T10:04:52.205153+0000","last_undegraded":"2024-03-04T10:04:52.205153+0000","last_fullsized":"2024-03-04T10:04:52.205153+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211035+0000","last_change":"2024-03-04T10:04:51.185755+0000","last_active":"2024-03-04T10:04:52.211035+0000","last_peered":"2024-03-04T10:04:52.211035+0000","last_clean":"2024-03-04T10:04:52.211035+0000","last_became_active":"2024-03-04T10:04:51.185456+0000","last_became_peered":"2024-03-04T10:04:51.185456+0000","last_unstale":"2024-03-04T10:04:52.211035+0000","last_undegraded":"2024-03-04T10:04:52.211035+0000","last_fullsized":"2024-03-04T10:04:52.211035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:40.177606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206321+0000","last_change":"2024-03-04T10:04:52.199324+0000","last_active":"2024-03-04T10:04:52.206321+0000","last_peered":"2024-03-04T10:04:52.206321+0000","last_clean":"2024-03-04T10:04:52.206321+0000","last_became_active":"2024-03-04T10:04:52.199041+0000","last_became_peered":"2024-03-04T10:04:52.199041+0000","last_unstale":"2024-03-04T10:04:52.206321+0000","last_undegraded":"2024-03-04T10:04:52.206321+0000","last_fullsized":"2024-03-04T10:04:52.206321+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187966+0000","last_change":"2024-03-04T10:04:51.189200+0000","last_active":"2024-03-04T10:04:52.187966+0000","last_peered":"2024-03-04T10:04:52.187966+0000","last_clean":"2024-03-04T10:04:52.187966+0000","last_became_active":"2024-03-04T10:04:51.188968+0000","last_became_peered":"2024-03-04T10:04:51.188968+0000","last_unstale":"2024-03-04T10:04:52.187966+0000","last_undegraded":"2024-03-04T10:04:52.187966+0000","last_fullsized":"2024-03-04T10:04:52.187966+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:29:16.432972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202135+0000","last_change":"2024-03-04T10:04:52.198436+0000","last_active":"2024-03-04T10:04:52.202135+0000","last_peered":"2024-03-04T10:04:52.202135+0000","last_clean":"2024-03-04T10:04:52.202135+0000","last_became_active":"2024-03-04T10:04:52.197703+0000","last_became_peered":"2024-03-04T10:04:52.197703+0000","last_unstale":"2024-03-04T10:04:52.202135+0000","last_undegraded":"2024-03-04T10:04:52.202135+0000","last_fullsized":"2024-03-04T10:04:52.202135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.189930+0000","last_change":"2024-03-04T10:04:51.187701+0000","last_active":"2024-03-04T10:04:52.189930+0000","last_peered":"2024-03-04T10:04:52.189930+0000","last_clean":"2024-03-04T10:04:52.189930+0000","last_became_active":"2024-03-04T10:04:51.187403+0000","last_became_peered":"2024-03-04T10:04:51.187403+0000","last_unstale":"2024-03-04T10:04:52.189930+0000","last_undegraded":"2024-03-04T10:04:52.189930+0000","last_fullsized":"2024-03-04T10:04:52.189930+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:01.545571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206729+0000","last_change":"2024-03-04T10:04:52.200381+0000","last_active":"2024-03-04T10:04:52.206729+0000","last_peered":"2024-03-04T10:04:52.206729+0000","last_clean":"2024-03-04T10:04:52.206729+0000","last_became_active":"2024-03-04T10:04:52.200111+0000","last_became_peered":"2024-03-04T10:04:52.200111+0000","last_unstale":"2024-03-04T10:04:52.206729+0000","last_undegraded":"2024-03-04T10:04:52.206729+0000","last_fullsized":"2024-03-04T10:04:52.206729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:31.447689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":20,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.214790+0000","last_change":"2024-03-04T10:04:51.179406+0000","last_active":"2024-03-04T10:04:52.214790+0000","last_peered":"2024-03-04T10:04:52.214790+0000","last_clean":"2024-03-04T10:04:52.214790+0000","last_became_active":"2024-03-04T10:04:51.179008+0000","last_became_peered":"2024-03-04T10:04:51.179008+0000","last_unstale":"2024-03-04T10:04:52.214790+0000","last_undegraded":"2024-03-04T10:04:52.214790+0000","last_fullsized":"2024-03-04T10:04:52.214790+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:29.099324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209676+0000","last_change":"2024-03-04T10:04:52.203342+0000","last_active":"2024-03-04T10:04:52.209676+0000","last_peered":"2024-03-04T10:04:52.209676+0000","last_clean":"2024-03-04T10:04:52.209676+0000","last_became_active":"2024-03-04T10:04:52.203083+0000","last_became_peered":"2024-03-04T10:04:52.203083+0000","last_unstale":"2024-03-04T10:04:52.209676+0000","last_undegraded":"2024-03-04T10:04:52.209676+0000","last_fullsized":"2024-03-04T10:04:52.209676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:46:08.623375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.214508+0000","last_change":"2024-03-04T10:04:51.188593+0000","last_active":"2024-03-04T10:04:52.214508+0000","last_peered":"2024-03-04T10:04:52.214508+0000","last_clean":"2024-03-04T10:04:52.214508+0000","last_became_active":"2024-03-04T10:04:51.188320+0000","last_became_peered":"2024-03-04T10:04:51.188320+0000","last_unstale":"2024-03-04T10:04:52.214508+0000","last_undegraded":"2024-03-04T10:04:52.214508+0000","last_fullsized":"2024-03-04T10:04:52.214508+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:15.362230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209128+0000","last_change":"2024-03-04T10:04:52.203361+0000","last_active":"2024-03-04T10:04:52.209128+0000","last_peered":"2024-03-04T10:04:52.209128+0000","last_clean":"2024-03-04T10:04:52.209128+0000","last_became_active":"2024-03-04T10:04:52.203078+0000","last_became_peered":"2024-03-04T10:04:52.203078+0000","last_unstale":"2024-03-04T10:04:52.209128+0000","last_undegraded":"2024-03-04T10:04:52.209128+0000","last_fullsized":"2024-03-04T10:04:52.209128+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:42:06.313007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.212236+0000","last_change":"2024-03-04T10:04:51.186916+0000","last_active":"2024-03-04T10:04:52.212236+0000","last_peered":"2024-03-04T10:04:52.212236+0000","last_clean":"2024-03-04T10:04:52.212236+0000","last_became_active":"2024-03-04T10:04:51.186644+0000","last_became_peered":"2024-03-04T10:04:51.186644+0000","last_unstale":"2024-03-04T10:04:52.212236+0000","last_undegraded":"2024-03-04T10:04:52.212236+0000","last_fullsized":"2024-03-04T10:04:52.212236+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:35.560169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209416+0000","last_change":"2024-03-04T10:04:52.206479+0000","last_active":"2024-03-04T10:04:52.209416+0000","last_peered":"2024-03-04T10:04:52.209416+0000","last_clean":"2024-03-04T10:04:52.209416+0000","last_became_active":"2024-03-04T10:04:52.206198+0000","last_became_peered":"2024-03-04T10:04:52.206198+0000","last_unstale":"2024-03-04T10:04:52.209416+0000","last_undegraded":"2024-03-04T10:04:52.209416+0000","last_fullsized":"2024-03-04T10:04:52.209416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:04.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208408+0000","last_change":"2024-03-04T10:04:51.180364+0000","last_active":"2024-03-04T10:04:52.208408+0000","last_peered":"2024-03-04T10:04:52.208408+0000","last_clean":"2024-03-04T10:04:52.208408+0000","last_became_active":"2024-03-04T10:04:51.179274+0000","last_became_peered":"2024-03-04T10:04:51.179274+0000","last_unstale":"2024-03-04T10:04:52.208408+0000","last_undegraded":"2024-03-04T10:04:52.208408+0000","last_fullsized":"2024-03-04T10:04:52.208408+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:45.667918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200489+0000","last_change":"2024-03-04T10:04:52.196730+0000","last_active":"2024-03-04T10:04:52.200489+0000","last_peered":"2024-03-04T10:04:52.200489+0000","last_clean":"2024-03-04T10:04:52.200489+0000","last_became_active":"2024-03-04T10:04:52.196442+0000","last_became_peered":"2024-03-04T10:04:52.196442+0000","last_unstale":"2024-03-04T10:04:52.200489+0000","last_undegraded":"2024-03-04T10:04:52.200489+0000","last_fullsized":"2024-03-04T10:04:52.200489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:05.433228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187684+0000","last_change":"2024-03-04T10:04:51.183951+0000","last_active":"2024-03-04T10:04:52.187684+0000","last_peered":"2024-03-04T10:04:52.187684+0000","last_clean":"2024-03-04T10:04:52.187684+0000","last_became_active":"2024-03-04T10:04:51.183685+0000","last_became_peered":"2024-03-04T10:04:51.183685+0000","last_unstale":"2024-03-04T10:04:52.187684+0000","last_undegraded":"2024-03-04T10:04:52.187684+0000","last_fullsized":"2024-03-04T10:04:52.187684+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:51.776383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.195425+0000","last_change":"2024-03-04T10:04:52.192503+0000","last_active":"2024-03-04T10:04:52.195425+0000","last_peered":"2024-03-04T10:04:52.195425+0000","last_clean":"2024-03-04T10:04:52.195425+0000","last_became_active":"2024-03-04T10:04:52.192203+0000","last_became_peered":"2024-03-04T10:04:52.192203+0000","last_unstale":"2024-03-04T10:04:52.195425+0000","last_undegraded":"2024-03-04T10:04:52.195425+0000","last_fullsized":"2024-03-04T10:04:52.195425+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:16.366185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210655+0000","last_change":"2024-03-04T10:04:51.179891+0000","last_active":"2024-03-04T10:04:52.210655+0000","last_peered":"2024-03-04T10:04:52.210655+0000","last_clean":"2024-03-04T10:04:52.210655+0000","last_became_active":"2024-03-04T10:04:51.179634+0000","last_became_peered":"2024-03-04T10:04:51.179634+0000","last_unstale":"2024-03-04T10:04:52.210655+0000","last_undegraded":"2024-03-04T10:04:52.210655+0000","last_fullsized":"2024-03-04T10:04:52.210655+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:26:20.575398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198533+0000","last_change":"2024-03-04T10:04:52.193881+0000","last_active":"2024-03-04T10:04:52.198533+0000","last_peered":"2024-03-04T10:04:52.198533+0000","last_clean":"2024-03-04T10:04:52.198533+0000","last_became_active":"2024-03-04T10:04:52.193570+0000","last_became_peered":"2024-03-04T10:04:52.193570+0000","last_unstale":"2024-03-04T10:04:52.198533+0000","last_undegraded":"2024-03-04T10:04:52.198533+0000","last_fullsized":"2024-03-04T10:04:52.198533+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:02.066773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.182922+0000","last_change":"2024-03-04T10:04:51.175098+0000","last_active":"2024-03-04T10:04:52.182922+0000","last_peered":"2024-03-04T10:04:52.182922+0000","last_clean":"2024-03-04T10:04:52.182922+0000","last_became_active":"2024-03-04T10:04:51.174810+0000","last_became_peered":"2024-03-04T10:04:51.174810+0000","last_unstale":"2024-03-04T10:04:52.182922+0000","last_undegraded":"2024-03-04T10:04:52.182922+0000","last_fullsized":"2024-03-04T10:04:52.182922+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:58:17.348789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200148+0000","last_change":"2024-03-04T10:04:52.195195+0000","last_active":"2024-03-04T10:04:52.200148+0000","last_peered":"2024-03-04T10:04:52.200148+0000","last_clean":"2024-03-04T10:04:52.200148+0000","last_became_active":"2024-03-04T10:04:52.194862+0000","last_became_peered":"2024-03-04T10:04:52.194862+0000","last_unstale":"2024-03-04T10:04:52.200148+0000","last_undegraded":"2024-03-04T10:04:52.200148+0000","last_fullsized":"2024-03-04T10:04:52.200148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:41.181561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.185554+0000","last_change":"2024-03-04T10:04:51.186004+0000","last_active":"2024-03-04T10:04:52.185554+0000","last_peered":"2024-03-04T10:04:52.185554+0000","last_clean":"2024-03-04T10:04:52.185554+0000","last_became_active":"2024-03-04T10:04:51.185720+0000","last_became_peered":"2024-03-04T10:04:51.185720+0000","last_unstale":"2024-03-04T10:04:52.185554+0000","last_undegraded":"2024-03-04T10:04:52.185554+0000","last_fullsized":"2024-03-04T10:04:52.185554+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.567262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208771+0000","last_change":"2024-03-04T10:04:52.204444+0000","last_active":"2024-03-04T10:04:52.208771+0000","last_peered":"2024-03-04T10:04:52.208771+0000","last_clean":"2024-03-04T10:04:52.208771+0000","last_became_active":"2024-03-04T10:04:52.203964+0000","last_became_peered":"2024-03-04T10:04:52.203964+0000","last_unstale":"2024-03-04T10:04:52.208771+0000","last_undegraded":"2024-03-04T10:04:52.208771+0000","last_fullsized":"2024-03-04T10:04:52.208771+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:46.192992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.182902+0000","last_change":"2024-03-04T10:04:51.184634+0000","last_active":"2024-03-04T10:04:52.182902+0000","last_peered":"2024-03-04T10:04:52.182902+0000","last_clean":"2024-03-04T10:04:52.182902+0000","last_became_active":"2024-03-04T10:04:51.184282+0000","last_became_peered":"2024-03-04T10:04:51.184282+0000","last_unstale":"2024-03-04T10:04:52.182902+0000","last_undegraded":"2024-03-04T10:04:52.182902+0000","last_fullsized":"2024-03-04T10:04:52.182902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:22:35.616403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207123+0000","last_change":"2024-03-04T10:04:52.202450+0000","last_active":"2024-03-04T10:04:52.207123+0000","last_peered":"2024-03-04T10:04:52.207123+0000","last_clean":"2024-03-04T10:04:52.207123+0000","last_became_active":"2024-03-04T10:04:52.202155+0000","last_became_peered":"2024-03-04T10:04:52.202155+0000","last_unstale":"2024-03-04T10:04:52.207123+0000","last_undegraded":"2024-03-04T10:04:52.207123+0000","last_fullsized":"2024-03-04T10:04:52.207123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.216040+0000","last_change":"2024-03-04T10:04:51.190615+0000","last_active":"2024-03-04T10:04:52.216040+0000","last_peered":"2024-03-04T10:04:52.216040+0000","last_clean":"2024-03-04T10:04:52.216040+0000","last_became_active":"2024-03-04T10:04:51.190354+0000","last_became_peered":"2024-03-04T10:04:51.190354+0000","last_unstale":"2024-03-04T10:04:52.216040+0000","last_undegraded":"2024-03-04T10:04:52.216040+0000","last_fullsized":"2024-03-04T10:04:52.216040+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:03.804872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.199350+0000","last_change":"2024-03-04T10:04:52.197290+0000","last_active":"2024-03-04T10:04:52.199350+0000","last_peered":"2024-03-04T10:04:52.199350+0000","last_clean":"2024-03-04T10:04:52.199350+0000","last_became_active":"2024-03-04T10:04:52.196689+0000","last_became_peered":"2024-03-04T10:04:52.196689+0000","last_unstale":"2024-03-04T10:04:52.199350+0000","last_undegraded":"2024-03-04T10:04:52.199350+0000","last_fullsized":"2024-03-04T10:04:52.199350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:50.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.183410+0000","last_change":"2024-03-04T10:04:51.184885+0000","last_active":"2024-03-04T10:04:52.183410+0000","last_peered":"2024-03-04T10:04:52.183410+0000","last_clean":"2024-03-04T10:04:52.183410+0000","last_became_active":"2024-03-04T10:04:51.184505+0000","last_became_peered":"2024-03-04T10:04:51.184505+0000","last_unstale":"2024-03-04T10:04:52.183410+0000","last_undegraded":"2024-03-04T10:04:52.183410+0000","last_fullsized":"2024-03-04T10:04:52.183410+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:09:27.334889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206407+0000","last_change":"2024-03-04T10:04:52.201097+0000","last_active":"2024-03-04T10:04:52.206407+0000","last_peered":"2024-03-04T10:04:52.206407+0000","last_clean":"2024-03-04T10:04:52.206407+0000","last_became_active":"2024-03-04T10:04:52.198658+0000","last_became_peered":"2024-03-04T10:04:52.198658+0000","last_unstale":"2024-03-04T10:04:52.206407+0000","last_undegraded":"2024-03-04T10:04:52.206407+0000","last_fullsized":"2024-03-04T10:04:52.206407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:42:06.313007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.186681+0000","last_change":"2024-03-04T10:04:51.186045+0000","last_active":"2024-03-04T10:04:52.186681+0000","last_peered":"2024-03-04T10:04:52.186681+0000","last_clean":"2024-03-04T10:04:52.186681+0000","last_became_active":"2024-03-04T10:04:51.185760+0000","last_became_peered":"2024-03-04T10:04:51.185760+0000","last_unstale":"2024-03-04T10:04:52.186681+0000","last_undegraded":"2024-03-04T10:04:52.186681+0000","last_fullsized":"2024-03-04T10:04:52.186681+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:15.362230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208443+0000","last_change":"2024-03-04T10:04:52.204788+0000","last_active":"2024-03-04T10:04:52.208443+0000","last_peered":"2024-03-04T10:04:52.208443+0000","last_clean":"2024-03-04T10:04:52.208443+0000","last_became_active":"2024-03-04T10:04:52.204149+0000","last_became_peered":"2024-03-04T10:04:52.204149+0000","last_unstale":"2024-03-04T10:04:52.208443+0000","last_undegraded":"2024-03-04T10:04:52.208443+0000","last_fullsized":"2024-03-04T10:04:52.208443+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.186466+0000","last_change":"2024-03-04T10:04:51.185426+0000","last_active":"2024-03-04T10:04:52.186466+0000","last_peered":"2024-03-04T10:04:52.186466+0000","last_clean":"2024-03-04T10:04:52.186466+0000","last_became_active":"2024-03-04T10:04:51.185061+0000","last_became_peered":"2024-03-04T10:04:51.185061+0000","last_unstale":"2024-03-04T10:04:52.186466+0000","last_undegraded":"2024-03-04T10:04:52.186466+0000","last_fullsized":"2024-03-04T10:04:52.186466+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.567262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207017+0000","last_change":"2024-03-04T10:04:52.200084+0000","last_active":"2024-03-04T10:04:52.207017+0000","last_peered":"2024-03-04T10:04:52.207017+0000","last_clean":"2024-03-04T10:04:52.207017+0000","last_became_active":"2024-03-04T10:04:52.199819+0000","last_became_peered":"2024-03-04T10:04:52.199819+0000","last_unstale":"2024-03-04T10:04:52.207017+0000","last_undegraded":"2024-03-04T10:04:52.207017+0000","last_fullsized":"2024-03-04T10:04:52.207017+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:26.571217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187952+0000","last_change":"2024-03-04T10:04:51.184369+0000","last_active":"2024-03-04T10:04:52.187952+0000","last_peered":"2024-03-04T10:04:52.187952+0000","last_clean":"2024-03-04T10:04:52.187952+0000","last_became_active":"2024-03-04T10:04:51.184064+0000","last_became_peered":"2024-03-04T10:04:51.184064+0000","last_unstale":"2024-03-04T10:04:52.187952+0000","last_undegraded":"2024-03-04T10:04:52.187952+0000","last_fullsized":"2024-03-04T10:04:52.187952+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:29:05.996823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200947+0000","last_change":"2024-03-04T10:04:52.196848+0000","last_active":"2024-03-04T10:04:52.200947+0000","last_peered":"2024-03-04T10:04:52.200947+0000","last_clean":"2024-03-04T10:04:52.200947+0000","last_became_active":"2024-03-04T10:04:52.196580+0000","last_became_peered":"2024-03-04T10:04:52.196580+0000","last_unstale":"2024-03-04T10:04:52.200947+0000","last_undegraded":"2024-03-04T10:04:52.200947+0000","last_fullsized":"2024-03-04T10:04:52.200947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:30.103279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.190845+0000","last_change":"2024-03-04T10:04:51.178928+0000","last_active":"2024-03-04T10:04:52.190845+0000","last_peered":"2024-03-04T10:04:52.190845+0000","last_clean":"2024-03-04T10:04:52.190845+0000","last_became_active":"2024-03-04T10:04:51.178131+0000","last_became_peered":"2024-03-04T10:04:51.178131+0000","last_unstale":"2024-03-04T10:04:52.190845+0000","last_undegraded":"2024-03-04T10:04:52.190845+0000","last_fullsized":"2024-03-04T10:04:52.190845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:51.776383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202168+0000","last_change":"2024-03-04T10:04:52.197914+0000","last_active":"2024-03-04T10:04:52.202168+0000","last_peered":"2024-03-04T10:04:52.202168+0000","last_clean":"2024-03-04T10:04:52.202168+0000","last_became_active":"2024-03-04T10:04:52.197629+0000","last_became_peered":"2024-03-04T10:04:52.197629+0000","last_unstale":"2024-03-04T10:04:52.202168+0000","last_undegraded":"2024-03-04T10:04:52.202168+0000","last_fullsized":"2024-03-04T10:04:52.202168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.189467+0000","last_change":"2024-03-04T10:04:51.186643+0000","last_active":"2024-03-04T10:04:52.189467+0000","last_peered":"2024-03-04T10:04:52.189467+0000","last_clean":"2024-03-04T10:04:52.189467+0000","last_became_active":"2024-03-04T10:04:51.186424+0000","last_became_peered":"2024-03-04T10:04:51.186424+0000","last_unstale":"2024-03-04T10:04:52.189467+0000","last_undegraded":"2024-03-04T10:04:52.189467+0000","last_fullsized":"2024-03-04T10:04:52.189467+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:01.062818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209358+0000","last_change":"2024-03-04T10:04:52.201380+0000","last_active":"2024-03-04T10:04:52.209358+0000","last_peered":"2024-03-04T10:04:52.209358+0000","last_clean":"2024-03-04T10:04:52.209358+0000","last_became_active":"2024-03-04T10:04:52.201105+0000","last_became_peered":"2024-03-04T10:04:52.201105+0000","last_unstale":"2024-03-04T10:04:52.209358+0000","last_undegraded":"2024-03-04T10:04:52.209358+0000","last_fullsized":"2024-03-04T10:04:52.209358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.183086+0000","last_change":"2024-03-04T10:04:51.180914+0000","last_active":"2024-03-04T10:04:52.183086+0000","last_peered":"2024-03-04T10:04:52.183086+0000","last_clean":"2024-03-04T10:04:52.183086+0000","last_became_active":"2024-03-04T10:04:51.180615+0000","last_became_peered":"2024-03-04T10:04:51.180615+0000","last_unstale":"2024-03-04T10:04:52.183086+0000","last_undegraded":"2024-03-04T10:04:52.183086+0000","last_fullsized":"2024-03-04T10:04:52.183086+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:39.689303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":204,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.182457+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:04:52.182457+0000","last_peered":"2024-03-04T10:04:52.182457+0000","last_clean":"2024-03-04T10:04:52.182457+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:04:52.182457+0000","last_undegraded":"2024-03-04T10:04:52.182457+0000","last_fullsized":"2024-03-04T10:04:52.182457+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1007616,"data_stored":766415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:04:55.148 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:04:55.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.283+0000 7f7e4fc85640 1 Processor -- start 2024-03-04T10:04:55.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.284+0000 7f7e4fc85640 1 -- start start 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.284+0000 7f7e4fc85640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7e4806d3b0 0x7f7e4815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.284+0000 7f7e4fc85640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7e4815b410 0x7f7e4815b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.284+0000 7f7e4fc85640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4806d8b0 con 0x7f7e4815b030 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.284+0000 7f7e4fc85640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e480b2720 con 0x7f7e4806d3b0 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.284+0000 7f7e4fc85640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4806db90 con 0x7f7e4815b410 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4d220640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7e4806d3b0 0x7f7e4815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4da21640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7e4815b030 0x7f7e4815dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56962/0 (socket says 172.21.15.99:56962) 2024-03-04T10:04:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4da21640 1 -- 172.21.15.99:0/2749204788 learned_addr learned my addr 172.21.15.99:0/2749204788 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4d220640 1 -- 172.21.15.99:0/2749204788 >> v2:172.21.15.162:3300/0 conn(0x7f7e4815b410 msgr2=0x7f7e4815b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4e222640 1 --2- 172.21.15.99:0/2749204788 >> v2:172.21.15.162:3300/0 conn(0x7f7e4815b410 0x7f7e4815b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4ca1f640 1 -- 172.21.15.99:0/2749204788 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074458586 0 0) 0x7f7e4806d8b0 con 0x7f7e4815b030 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4d220640 1 --2- 172.21.15.99:0/2749204788 >> v2:172.21.15.162:3300/0 conn(0x7f7e4815b410 0x7f7e4815b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4d220640 1 -- 172.21.15.99:0/2749204788 >> v1:172.21.15.99:6789/0 conn(0x7f7e4815b030 legacy=0x7f7e4815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4d220640 1 -- 172.21.15.99:0/2749204788 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4806de70 con 0x7f7e4806d3b0 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.285+0000 7f7e4e222640 1 --2- 172.21.15.99:0/2749204788 >> v2:172.21.15.162:3300/0 conn(0x7f7e4815b410 0x7f7e4815b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4d220640 1 --2- 172.21.15.99:0/2749204788 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4806d3b0 0x7f7e4815c120 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7e480b28a0 tx=0x7f7e4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad77c3fa6e01eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4ca1f640 1 -- 172.21.15.99:0/2749204788 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e440315a0 con 0x7f7e4806d3b0 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4ca1f640 1 -- 172.21.15.99:0/2749204788 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e44037070 con 0x7f7e4806d3b0 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4ca1f640 1 -- 172.21.15.99:0/2749204788 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44031960 con 0x7f7e4806d3b0 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 -- 172.21.15.99:0/2749204788 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4806d3b0 msgr2=0x7f7e4815c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/2749204788 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4806d3b0 0x7f7e4815c120 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7e480b28a0 tx=0x7f7e4402b840 comp rx=0 tx=0).stop 2024-03-04T10:04:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 -- 172.21.15.99:0/2749204788 shutdown_connections 2024-03-04T10:04:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/2749204788 >> v2:172.21.15.162:3300/0 conn(0x7f7e4815b410 0x7f7e4815b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/2749204788 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4806d3b0 0x7f7e4815c120 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 -- 172.21.15.99:0/2749204788 >> 172.21.15.99:0/2749204788 conn(0x7f7e4806acd0 msgr2=0x7f7e4806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.286+0000 7f7e4fc85640 1 -- 172.21.15.99:0/2749204788 shutdown_connections 2024-03-04T10:04:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.287+0000 7f7e4fc85640 1 -- 172.21.15.99:0/2749204788 wait complete. 2024-03-04T10:04:55.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 Processor -- start 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 -- start start 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e4806d3b0 0x7f7e480cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 0x7f7e480acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e4815b410 0x7f7e480ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4806e690 con 0x7f7e4815b030 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4815bdc0 con 0x7f7e4815b410 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4fc85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4806e810 con 0x7f7e4806d3b0 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4d220640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 0x7f7e480acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4d220640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 0x7f7e480acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50034/0 (socket says 172.21.15.99:50034) 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.288+0000 7f7e4d220640 1 -- 172.21.15.99:0/544811651 learned_addr learned my addr 172.21.15.99:0/544811651 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4da21640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e4806d3b0 0x7f7e480cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4e222640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e4815b410 0x7f7e480ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4d220640 1 -- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e4806d3b0 msgr2=0x7f7e480cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4d220640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e4806d3b0 0x7f7e480cb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4d220640 1 -- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e4815b410 msgr2=0x7f7e480ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4d220640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e4815b410 0x7f7e480ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4d220640 1 -- 172.21.15.99:0/544811651 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e480a9510 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4d220640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 0x7f7e480acd90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7e44000c00 tx=0x7f7e440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44032840 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e44032a00 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e44032df0 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e480a97f0 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.289+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e480a9db0 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.290+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e48067680 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.290+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7e4404c020 con 0x7f7e4815b030 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.291+0000 7f7e3e7fc640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e2c06b4d0 0x7f7e2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.291+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e4402f020 con 0x7f7e4815b030 2024-03-04T10:04:55.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.291+0000 7f7e4da21640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e2c06b4d0 0x7f7e2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.291+0000 7f7e4da21640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e2c06b4d0 0x7f7e2c06d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7e38000f60 tx=0x7f7e38007b70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.295+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e48067680 con 0x7f7e4815b030 2024-03-04T10:04:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.461+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e48074f60 con 0x7f7e2c06b4d0 2024-03-04T10:04:55.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.468+0000 7f7e3e7fc640 1 -- 172.21.15.99:0/544811651 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251886 (secure 0 0 0) 0x7f7e48074f60 con 0x7f7e2c06b4d0 2024-03-04T10:04:55.472 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:55.474 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:04:55.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e2c06b4d0 msgr2=0x7f7e2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e2c06b4d0 0x7f7e2c06d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7e38000f60 tx=0x7f7e38007b70 comp rx=0 tx=0).stop 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 msgr2=0x7f7e480acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 0x7f7e480acd90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7e44000c00 tx=0x7f7e440046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 shutdown_connections 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e2c06b4d0 0x7f7e2c06d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e4815b410 0x7f7e480ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e4815b030 0x7f7e480acd90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.470+0000 7f7e4fc85640 1 --2- 172.21.15.99:0/544811651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e4806d3b0 0x7f7e480cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.471+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 >> 172.21.15.99:0/544811651 conn(0x7f7e4806acd0 msgr2=0x7f7e480b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.471+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 shutdown_connections 2024-03-04T10:04:55.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.471+0000 7f7e4fc85640 1 -- 172.21.15.99:0/544811651 wait complete. 2024-03-04T10:04:55.482 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-04T10:04:55.455872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220592,"kb_used_data":4408,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754128,"statfs":{"total":773094113280,"available":772868227072,"internally_reserved":0,"allocated":4513792,"data_stored":2596369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.291910"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206453+0000","last_change":"2024-03-04T10:04:52.199649+0000","last_active":"2024-03-04T10:04:52.206453+0000","last_peered":"2024-03-04T10:04:52.206453+0000","last_clean":"2024-03-04T10:04:52.206453+0000","last_became_active":"2024-03-04T10:04:52.199398+0000","last_became_peered":"2024-03-04T10:04:52.199398+0000","last_unstale":"2024-03-04T10:04:52.206453+0000","last_undegraded":"2024-03-04T10:04:52.206453+0000","last_fullsized":"2024-03-04T10:04:52.206453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208667+0000","last_change":"2024-03-04T10:04:52.202939+0000","last_active":"2024-03-04T10:04:52.208667+0000","last_peered":"2024-03-04T10:04:52.208667+0000","last_clean":"2024-03-04T10:04:52.208667+0000","last_became_active":"2024-03-04T10:04:52.200847+0000","last_became_peered":"2024-03-04T10:04:52.200847+0000","last_unstale":"2024-03-04T10:04:52.208667+0000","last_undegraded":"2024-03-04T10:04:52.208667+0000","last_fullsized":"2024-03-04T10:04:52.208667+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:31.922962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198562+0000","last_change":"2024-03-04T10:04:52.194854+0000","last_active":"2024-03-04T10:04:52.198562+0000","last_peered":"2024-03-04T10:04:52.198562+0000","last_clean":"2024-03-04T10:04:52.198562+0000","last_became_active":"2024-03-04T10:04:52.194603+0000","last_became_peered":"2024-03-04T10:04:52.194603+0000","last_unstale":"2024-03-04T10:04:52.198562+0000","last_undegraded":"2024-03-04T10:04:52.198562+0000","last_fullsized":"2024-03-04T10:04:52.198562+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:14.100423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205997+0000","last_change":"2024-03-04T10:04:52.200439+0000","last_active":"2024-03-04T10:04:52.205997+0000","last_peered":"2024-03-04T10:04:52.205997+0000","last_clean":"2024-03-04T10:04:52.205997+0000","last_became_active":"2024-03-04T10:04:52.200194+0000","last_became_peered":"2024-03-04T10:04:52.200194+0000","last_unstale":"2024-03-04T10:04:52.205997+0000","last_undegraded":"2024-03-04T10:04:52.205997+0000","last_fullsized":"2024-03-04T10:04:52.205997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:48:01.778909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.201126+0000","last_change":"2024-03-04T10:04:52.199976+0000","last_active":"2024-03-04T10:04:52.201126+0000","last_peered":"2024-03-04T10:04:52.201126+0000","last_clean":"2024-03-04T10:04:52.201126+0000","last_became_active":"2024-03-04T10:04:52.199727+0000","last_became_peered":"2024-03-04T10:04:52.199727+0000","last_unstale":"2024-03-04T10:04:52.201126+0000","last_undegraded":"2024-03-04T10:04:52.201126+0000","last_fullsized":"2024-03-04T10:04:52.201126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:40.813861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206935+0000","last_change":"2024-03-04T10:04:52.199653+0000","last_active":"2024-03-04T10:04:52.206935+0000","last_peered":"2024-03-04T10:04:52.206935+0000","last_clean":"2024-03-04T10:04:52.206935+0000","last_became_active":"2024-03-04T10:04:52.199380+0000","last_became_peered":"2024-03-04T10:04:52.199380+0000","last_unstale":"2024-03-04T10:04:52.206935+0000","last_undegraded":"2024-03-04T10:04:52.206935+0000","last_fullsized":"2024-03-04T10:04:52.206935+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:01.728765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209073+0000","last_change":"2024-03-04T10:04:52.202562+0000","last_active":"2024-03-04T10:04:52.209073+0000","last_peered":"2024-03-04T10:04:52.209073+0000","last_clean":"2024-03-04T10:04:52.209073+0000","last_became_active":"2024-03-04T10:04:52.201480+0000","last_became_peered":"2024-03-04T10:04:52.201480+0000","last_unstale":"2024-03-04T10:04:52.209073+0000","last_undegraded":"2024-03-04T10:04:52.209073+0000","last_fullsized":"2024-03-04T10:04:52.209073+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208722+0000","last_change":"2024-03-04T10:04:52.204418+0000","last_active":"2024-03-04T10:04:52.208722+0000","last_peered":"2024-03-04T10:04:52.208722+0000","last_clean":"2024-03-04T10:04:52.208722+0000","last_became_active":"2024-03-04T10:04:52.204159+0000","last_became_peered":"2024-03-04T10:04:52.204159+0000","last_unstale":"2024-03-04T10:04:52.208722+0000","last_undegraded":"2024-03-04T10:04:52.208722+0000","last_fullsized":"2024-03-04T10:04:52.208722+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:21.848884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198620+0000","last_change":"2024-03-04T10:04:52.194939+0000","last_active":"2024-03-04T10:04:52.198620+0000","last_peered":"2024-03-04T10:04:52.198620+0000","last_clean":"2024-03-04T10:04:52.198620+0000","last_became_active":"2024-03-04T10:04:52.194653+0000","last_became_peered":"2024-03-04T10:04:52.194653+0000","last_unstale":"2024-03-04T10:04:52.198620+0000","last_undegraded":"2024-03-04T10:04:52.198620+0000","last_fullsized":"2024-03-04T10:04:52.198620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:41.181561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200374+0000","last_change":"2024-03-04T10:04:52.195037+0000","last_active":"2024-03-04T10:04:52.200374+0000","last_peered":"2024-03-04T10:04:52.200374+0000","last_clean":"2024-03-04T10:04:52.200374+0000","last_became_active":"2024-03-04T10:04:52.194750+0000","last_became_peered":"2024-03-04T10:04:52.194750+0000","last_unstale":"2024-03-04T10:04:52.200374+0000","last_undegraded":"2024-03-04T10:04:52.200374+0000","last_fullsized":"2024-03-04T10:04:52.200374+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:14.100423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211929+0000","last_change":"2024-03-04T10:04:52.208932+0000","last_active":"2024-03-04T10:04:52.211929+0000","last_peered":"2024-03-04T10:04:52.211929+0000","last_clean":"2024-03-04T10:04:52.211929+0000","last_became_active":"2024-03-04T10:04:52.208650+0000","last_became_peered":"2024-03-04T10:04:52.208650+0000","last_unstale":"2024-03-04T10:04:52.211929+0000","last_undegraded":"2024-03-04T10:04:52.211929+0000","last_fullsized":"2024-03-04T10:04:52.211929+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:46:08.623375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206497+0000","last_change":"2024-03-04T10:04:52.195702+0000","last_active":"2024-03-04T10:04:52.206497+0000","last_peered":"2024-03-04T10:04:52.206497+0000","last_clean":"2024-03-04T10:04:52.206497+0000","last_became_active":"2024-03-04T10:04:52.195406+0000","last_became_peered":"2024-03-04T10:04:52.195406+0000","last_unstale":"2024-03-04T10:04:52.206497+0000","last_undegraded":"2024-03-04T10:04:52.206497+0000","last_fullsized":"2024-03-04T10:04:52.206497+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:50.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200199+0000","last_change":"2024-03-04T10:04:52.195347+0000","last_active":"2024-03-04T10:04:52.200199+0000","last_peered":"2024-03-04T10:04:52.200199+0000","last_clean":"2024-03-04T10:04:52.200199+0000","last_became_active":"2024-03-04T10:04:52.194696+0000","last_became_peered":"2024-03-04T10:04:52.194696+0000","last_unstale":"2024-03-04T10:04:52.200199+0000","last_undegraded":"2024-03-04T10:04:52.200199+0000","last_fullsized":"2024-03-04T10:04:52.200199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:50.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205124+0000","last_change":"2024-03-04T10:04:52.199326+0000","last_active":"2024-03-04T10:04:52.205124+0000","last_peered":"2024-03-04T10:04:52.205124+0000","last_clean":"2024-03-04T10:04:52.205124+0000","last_became_active":"2024-03-04T10:04:52.198978+0000","last_became_peered":"2024-03-04T10:04:52.198978+0000","last_unstale":"2024-03-04T10:04:52.205124+0000","last_undegraded":"2024-03-04T10:04:52.205124+0000","last_fullsized":"2024-03-04T10:04:52.205124+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:01.728765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.197731+0000","last_change":"2024-03-04T10:04:52.192798+0000","last_active":"2024-03-04T10:04:52.197731+0000","last_peered":"2024-03-04T10:04:52.197731+0000","last_clean":"2024-03-04T10:04:52.197731+0000","last_became_active":"2024-03-04T10:04:52.192512+0000","last_became_peered":"2024-03-04T10:04:52.192512+0000","last_unstale":"2024-03-04T10:04:52.197731+0000","last_undegraded":"2024-03-04T10:04:52.197731+0000","last_fullsized":"2024-03-04T10:04:52.197731+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:02.066773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198278+0000","last_change":"2024-03-04T10:04:52.195037+0000","last_active":"2024-03-04T10:04:52.198278+0000","last_peered":"2024-03-04T10:04:52.198278+0000","last_clean":"2024-03-04T10:04:52.198278+0000","last_became_active":"2024-03-04T10:04:52.194751+0000","last_became_peered":"2024-03-04T10:04:52.194751+0000","last_unstale":"2024-03-04T10:04:52.198278+0000","last_undegraded":"2024-03-04T10:04:52.198278+0000","last_fullsized":"2024-03-04T10:04:52.198278+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:14.100423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.199319+0000","last_change":"2024-03-04T10:04:52.195633+0000","last_active":"2024-03-04T10:04:52.199319+0000","last_peered":"2024-03-04T10:04:52.199319+0000","last_clean":"2024-03-04T10:04:52.199319+0000","last_became_active":"2024-03-04T10:04:52.194911+0000","last_became_peered":"2024-03-04T10:04:52.194911+0000","last_unstale":"2024-03-04T10:04:52.199319+0000","last_undegraded":"2024-03-04T10:04:52.199319+0000","last_fullsized":"2024-03-04T10:04:52.199319+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:41.181561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202475+0000","last_change":"2024-03-04T10:04:52.198978+0000","last_active":"2024-03-04T10:04:52.202475+0000","last_peered":"2024-03-04T10:04:52.202475+0000","last_clean":"2024-03-04T10:04:52.202475+0000","last_became_active":"2024-03-04T10:04:52.198722+0000","last_became_peered":"2024-03-04T10:04:52.198722+0000","last_unstale":"2024-03-04T10:04:52.202475+0000","last_undegraded":"2024-03-04T10:04:52.202475+0000","last_fullsized":"2024-03-04T10:04:52.202475+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:29:17.436926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208338+0000","last_change":"2024-03-04T10:04:52.201662+0000","last_active":"2024-03-04T10:04:52.208338+0000","last_peered":"2024-03-04T10:04:52.208338+0000","last_clean":"2024-03-04T10:04:52.208338+0000","last_became_active":"2024-03-04T10:04:52.201403+0000","last_became_peered":"2024-03-04T10:04:52.201403+0000","last_unstale":"2024-03-04T10:04:52.208338+0000","last_undegraded":"2024-03-04T10:04:52.208338+0000","last_fullsized":"2024-03-04T10:04:52.208338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:27:07.884258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.203260+0000","last_change":"2024-03-04T10:04:52.199707+0000","last_active":"2024-03-04T10:04:52.203260+0000","last_peered":"2024-03-04T10:04:52.203260+0000","last_clean":"2024-03-04T10:04:52.203260+0000","last_became_active":"2024-03-04T10:04:52.199344+0000","last_became_peered":"2024-03-04T10:04:52.199344+0000","last_unstale":"2024-03-04T10:04:52.203260+0000","last_undegraded":"2024-03-04T10:04:52.203260+0000","last_fullsized":"2024-03-04T10:04:52.203260+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:30.103279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209153+0000","last_change":"2024-03-04T10:04:52.202972+0000","last_active":"2024-03-04T10:04:52.209153+0000","last_peered":"2024-03-04T10:04:52.209153+0000","last_clean":"2024-03-04T10:04:52.209153+0000","last_became_active":"2024-03-04T10:04:52.202414+0000","last_became_peered":"2024-03-04T10:04:52.202414+0000","last_unstale":"2024-03-04T10:04:52.209153+0000","last_undegraded":"2024-03-04T10:04:52.209153+0000","last_fullsized":"2024-03-04T10:04:52.209153+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:31.447689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208163+0000","last_change":"2024-03-04T10:04:52.200815+0000","last_active":"2024-03-04T10:04:52.208163+0000","last_peered":"2024-03-04T10:04:52.208163+0000","last_clean":"2024-03-04T10:04:52.208163+0000","last_became_active":"2024-03-04T10:04:52.200421+0000","last_became_peered":"2024-03-04T10:04:52.200421+0000","last_unstale":"2024-03-04T10:04:52.208163+0000","last_undegraded":"2024-03-04T10:04:52.208163+0000","last_fullsized":"2024-03-04T10:04:52.208163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:26.571217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.196112+0000","last_change":"2024-03-04T10:04:52.193528+0000","last_active":"2024-03-04T10:04:52.196112+0000","last_peered":"2024-03-04T10:04:52.196112+0000","last_clean":"2024-03-04T10:04:52.196112+0000","last_became_active":"2024-03-04T10:04:52.193244+0000","last_became_peered":"2024-03-04T10:04:52.193244+0000","last_unstale":"2024-03-04T10:04:52.196112+0000","last_undegraded":"2024-03-04T10:04:52.196112+0000","last_fullsized":"2024-03-04T10:04:52.196112+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:04.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198232+0000","last_change":"2024-03-04T10:04:52.194676+0000","last_active":"2024-03-04T10:04:52.198232+0000","last_peered":"2024-03-04T10:04:52.198232+0000","last_clean":"2024-03-04T10:04:52.198232+0000","last_became_active":"2024-03-04T10:04:52.194422+0000","last_became_peered":"2024-03-04T10:04:52.194422+0000","last_unstale":"2024-03-04T10:04:52.198232+0000","last_undegraded":"2024-03-04T10:04:52.198232+0000","last_fullsized":"2024-03-04T10:04:52.198232+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:06.699075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202509+0000","last_change":"2024-03-04T10:04:52.199912+0000","last_active":"2024-03-04T10:04:52.202509+0000","last_peered":"2024-03-04T10:04:52.202509+0000","last_clean":"2024-03-04T10:04:52.202509+0000","last_became_active":"2024-03-04T10:04:52.199610+0000","last_became_peered":"2024-03-04T10:04:52.199610+0000","last_unstale":"2024-03-04T10:04:52.202509+0000","last_undegraded":"2024-03-04T10:04:52.202509+0000","last_fullsized":"2024-03-04T10:04:52.202509+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:40.693258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207091+0000","last_change":"2024-03-04T10:04:52.200264+0000","last_active":"2024-03-04T10:04:52.207091+0000","last_peered":"2024-03-04T10:04:52.207091+0000","last_clean":"2024-03-04T10:04:52.207091+0000","last_became_active":"2024-03-04T10:04:52.199991+0000","last_became_peered":"2024-03-04T10:04:52.199991+0000","last_unstale":"2024-03-04T10:04:52.207091+0000","last_undegraded":"2024-03-04T10:04:52.207091+0000","last_fullsized":"2024-03-04T10:04:52.207091+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:16.366185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.197275+0000","last_change":"2024-03-04T10:04:52.196527+0000","last_active":"2024-03-04T10:04:52.197275+0000","last_peered":"2024-03-04T10:04:52.197275+0000","last_clean":"2024-03-04T10:04:52.197275+0000","last_became_active":"2024-03-04T10:04:52.196261+0000","last_became_peered":"2024-03-04T10:04:52.196261+0000","last_unstale":"2024-03-04T10:04:52.197275+0000","last_undegraded":"2024-03-04T10:04:52.197275+0000","last_fullsized":"2024-03-04T10:04:52.197275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:52.780338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209033+0000","last_change":"2024-03-04T10:04:52.202193+0000","last_active":"2024-03-04T10:04:52.209033+0000","last_peered":"2024-03-04T10:04:52.209033+0000","last_clean":"2024-03-04T10:04:52.209033+0000","last_became_active":"2024-03-04T10:04:52.201708+0000","last_became_peered":"2024-03-04T10:04:52.201708+0000","last_unstale":"2024-03-04T10:04:52.209033+0000","last_undegraded":"2024-03-04T10:04:52.209033+0000","last_fullsized":"2024-03-04T10:04:52.209033+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211631+0000","last_change":"2024-03-04T10:04:52.206905+0000","last_active":"2024-03-04T10:04:52.211631+0000","last_peered":"2024-03-04T10:04:52.211631+0000","last_clean":"2024-03-04T10:04:52.211631+0000","last_became_active":"2024-03-04T10:04:52.206655+0000","last_became_peered":"2024-03-04T10:04:52.206655+0000","last_unstale":"2024-03-04T10:04:52.211631+0000","last_undegraded":"2024-03-04T10:04:52.211631+0000","last_fullsized":"2024-03-04T10:04:52.211631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:01.728765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202447+0000","last_change":"2024-03-04T10:04:52.197798+0000","last_active":"2024-03-04T10:04:52.202447+0000","last_peered":"2024-03-04T10:04:52.202447+0000","last_clean":"2024-03-04T10:04:52.202447+0000","last_became_active":"2024-03-04T10:04:52.197487+0000","last_became_peered":"2024-03-04T10:04:52.197487+0000","last_unstale":"2024-03-04T10:04:52.202447+0000","last_undegraded":"2024-03-04T10:04:52.202447+0000","last_fullsized":"2024-03-04T10:04:52.202447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:36.564124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198140+0000","last_change":"2024-03-04T10:04:52.194073+0000","last_active":"2024-03-04T10:04:52.198140+0000","last_peered":"2024-03-04T10:04:52.198140+0000","last_clean":"2024-03-04T10:04:52.198140+0000","last_became_active":"2024-03-04T10:04:52.193742+0000","last_became_peered":"2024-03-04T10:04:52.193742+0000","last_unstale":"2024-03-04T10:04:52.198140+0000","last_undegraded":"2024-03-04T10:04:52.198140+0000","last_fullsized":"2024-03-04T10:04:52.198140+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.204889+0000","last_change":"2024-03-04T10:04:52.199105+0000","last_active":"2024-03-04T10:04:52.204889+0000","last_peered":"2024-03-04T10:04:52.204889+0000","last_clean":"2024-03-04T10:04:52.204889+0000","last_became_active":"2024-03-04T10:04:52.198825+0000","last_became_peered":"2024-03-04T10:04:52.198825+0000","last_unstale":"2024-03-04T10:04:52.204889+0000","last_undegraded":"2024-03-04T10:04:52.204889+0000","last_fullsized":"2024-03-04T10:04:52.204889+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:40.693258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198098+0000","last_change":"2024-03-04T10:04:52.194930+0000","last_active":"2024-03-04T10:04:52.198098+0000","last_peered":"2024-03-04T10:04:52.198098+0000","last_clean":"2024-03-04T10:04:52.198098+0000","last_became_active":"2024-03-04T10:04:52.194637+0000","last_became_peered":"2024-03-04T10:04:52.194637+0000","last_unstale":"2024-03-04T10:04:52.198098+0000","last_undegraded":"2024-03-04T10:04:52.198098+0000","last_fullsized":"2024-03-04T10:04:52.198098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:02.066773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":16,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211936+0000","last_change":"2024-03-04T10:04:51.188914+0000","last_active":"2024-03-04T10:04:52.211936+0000","last_peered":"2024-03-04T10:04:52.211936+0000","last_clean":"2024-03-04T10:04:52.211936+0000","last_became_active":"2024-03-04T10:04:51.188650+0000","last_became_peered":"2024-03-04T10:04:51.188650+0000","last_unstale":"2024-03-04T10:04:52.211936+0000","last_undegraded":"2024-03-04T10:04:52.211936+0000","last_fullsized":"2024-03-04T10:04:52.211936+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:52.296428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207151+0000","last_change":"2024-03-04T10:04:52.200962+0000","last_active":"2024-03-04T10:04:52.207151+0000","last_peered":"2024-03-04T10:04:52.207151+0000","last_clean":"2024-03-04T10:04:52.207151+0000","last_became_active":"2024-03-04T10:04:52.200689+0000","last_became_peered":"2024-03-04T10:04:52.200689+0000","last_unstale":"2024-03-04T10:04:52.207151+0000","last_undegraded":"2024-03-04T10:04:52.207151+0000","last_fullsized":"2024-03-04T10:04:52.207151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:04.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205537+0000","last_change":"2024-03-04T10:04:51.182645+0000","last_active":"2024-03-04T10:04:52.205537+0000","last_peered":"2024-03-04T10:04:52.205537+0000","last_clean":"2024-03-04T10:04:52.205537+0000","last_became_active":"2024-03-04T10:04:51.182207+0000","last_became_peered":"2024-03-04T10:04:51.182207+0000","last_unstale":"2024-03-04T10:04:52.205537+0000","last_undegraded":"2024-03-04T10:04:52.205537+0000","last_fullsized":"2024-03-04T10:04:52.205537+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:03.804872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.201113+0000","last_change":"2024-03-04T10:04:52.196872+0000","last_active":"2024-03-04T10:04:52.201113+0000","last_peered":"2024-03-04T10:04:52.201113+0000","last_clean":"2024-03-04T10:04:52.201113+0000","last_became_active":"2024-03-04T10:04:52.196553+0000","last_became_peered":"2024-03-04T10:04:52.196553+0000","last_unstale":"2024-03-04T10:04:52.201113+0000","last_undegraded":"2024-03-04T10:04:52.201113+0000","last_fullsized":"2024-03-04T10:04:52.201113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:40.693258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206759+0000","last_change":"2024-03-04T10:04:51.183200+0000","last_active":"2024-03-04T10:04:52.206759+0000","last_peered":"2024-03-04T10:04:52.206759+0000","last_clean":"2024-03-04T10:04:52.206759+0000","last_became_active":"2024-03-04T10:04:51.182932+0000","last_became_peered":"2024-03-04T10:04:51.182932+0000","last_unstale":"2024-03-04T10:04:52.206759+0000","last_undegraded":"2024-03-04T10:04:52.206759+0000","last_fullsized":"2024-03-04T10:04:52.206759+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:29.099324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206429+0000","last_change":"2024-03-04T10:04:52.199095+0000","last_active":"2024-03-04T10:04:52.206429+0000","last_peered":"2024-03-04T10:04:52.206429+0000","last_clean":"2024-03-04T10:04:52.206429+0000","last_became_active":"2024-03-04T10:04:52.198835+0000","last_became_peered":"2024-03-04T10:04:52.198835+0000","last_unstale":"2024-03-04T10:04:52.206429+0000","last_undegraded":"2024-03-04T10:04:52.206429+0000","last_fullsized":"2024-03-04T10:04:52.206429+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210747+0000","last_change":"2024-03-04T10:04:51.186290+0000","last_active":"2024-03-04T10:04:52.210747+0000","last_peered":"2024-03-04T10:04:52.210747+0000","last_clean":"2024-03-04T10:04:52.210747+0000","last_became_active":"2024-03-04T10:04:51.186016+0000","last_became_peered":"2024-03-04T10:04:51.186016+0000","last_unstale":"2024-03-04T10:04:52.210747+0000","last_undegraded":"2024-03-04T10:04:52.210747+0000","last_fullsized":"2024-03-04T10:04:52.210747+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:29:16.432972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208465+0000","last_change":"2024-03-04T10:04:52.204984+0000","last_active":"2024-03-04T10:04:52.208465+0000","last_peered":"2024-03-04T10:04:52.208465+0000","last_clean":"2024-03-04T10:04:52.208465+0000","last_became_active":"2024-03-04T10:04:52.204490+0000","last_became_peered":"2024-03-04T10:04:52.204490+0000","last_unstale":"2024-03-04T10:04:52.208465+0000","last_undegraded":"2024-03-04T10:04:52.208465+0000","last_fullsized":"2024-03-04T10:04:52.208465+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210630+0000","last_change":"2024-03-04T10:04:51.191689+0000","last_active":"2024-03-04T10:04:52.210630+0000","last_peered":"2024-03-04T10:04:52.210630+0000","last_clean":"2024-03-04T10:04:52.210630+0000","last_became_active":"2024-03-04T10:04:51.191150+0000","last_became_peered":"2024-03-04T10:04:51.191150+0000","last_unstale":"2024-03-04T10:04:52.210630+0000","last_undegraded":"2024-03-04T10:04:52.210630+0000","last_fullsized":"2024-03-04T10:04:52.210630+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.567262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209174+0000","last_change":"2024-03-04T10:04:52.203610+0000","last_active":"2024-03-04T10:04:52.209174+0000","last_peered":"2024-03-04T10:04:52.209174+0000","last_clean":"2024-03-04T10:04:52.209174+0000","last_became_active":"2024-03-04T10:04:52.203297+0000","last_became_peered":"2024-03-04T10:04:52.203297+0000","last_unstale":"2024-03-04T10:04:52.209174+0000","last_undegraded":"2024-03-04T10:04:52.209174+0000","last_fullsized":"2024-03-04T10:04:52.209174+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:37:31.922962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210259+0000","last_change":"2024-03-04T10:04:51.179466+0000","last_active":"2024-03-04T10:04:52.210259+0000","last_peered":"2024-03-04T10:04:52.210259+0000","last_clean":"2024-03-04T10:04:52.210259+0000","last_became_active":"2024-03-04T10:04:51.179203+0000","last_became_peered":"2024-03-04T10:04:51.179203+0000","last_unstale":"2024-03-04T10:04:52.210259+0000","last_undegraded":"2024-03-04T10:04:52.210259+0000","last_fullsized":"2024-03-04T10:04:52.210259+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:00.724810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209487+0000","last_change":"2024-03-04T10:04:52.207971+0000","last_active":"2024-03-04T10:04:52.209487+0000","last_peered":"2024-03-04T10:04:52.209487+0000","last_clean":"2024-03-04T10:04:52.209487+0000","last_became_active":"2024-03-04T10:04:52.207695+0000","last_became_peered":"2024-03-04T10:04:52.207695+0000","last_unstale":"2024-03-04T10:04:52.209487+0000","last_undegraded":"2024-03-04T10:04:52.209487+0000","last_fullsized":"2024-03-04T10:04:52.209487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187762+0000","last_change":"2024-03-04T10:04:51.189127+0000","last_active":"2024-03-04T10:04:52.187762+0000","last_peered":"2024-03-04T10:04:52.187762+0000","last_clean":"2024-03-04T10:04:52.187762+0000","last_became_active":"2024-03-04T10:04:51.188873+0000","last_became_peered":"2024-03-04T10:04:51.188873+0000","last_unstale":"2024-03-04T10:04:52.187762+0000","last_undegraded":"2024-03-04T10:04:52.187762+0000","last_fullsized":"2024-03-04T10:04:52.187762+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:35.560169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208789+0000","last_change":"2024-03-04T10:04:52.204639+0000","last_active":"2024-03-04T10:04:52.208789+0000","last_peered":"2024-03-04T10:04:52.208789+0000","last_clean":"2024-03-04T10:04:52.208789+0000","last_became_active":"2024-03-04T10:04:52.204342+0000","last_became_peered":"2024-03-04T10:04:52.204342+0000","last_unstale":"2024-03-04T10:04:52.208789+0000","last_undegraded":"2024-03-04T10:04:52.208789+0000","last_fullsized":"2024-03-04T10:04:52.208789+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:31:58.274038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.190379+0000","last_change":"2024-03-04T10:04:51.184748+0000","last_active":"2024-03-04T10:04:52.190379+0000","last_peered":"2024-03-04T10:04:52.190379+0000","last_clean":"2024-03-04T10:04:52.190379+0000","last_became_active":"2024-03-04T10:04:51.184471+0000","last_became_peered":"2024-03-04T10:04:51.184471+0000","last_unstale":"2024-03-04T10:04:52.190379+0000","last_undegraded":"2024-03-04T10:04:52.190379+0000","last_fullsized":"2024-03-04T10:04:52.190379+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:55:52.296428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.201198+0000","last_change":"2024-03-04T10:04:52.200255+0000","last_active":"2024-03-04T10:04:52.201198+0000","last_peered":"2024-03-04T10:04:52.201198+0000","last_clean":"2024-03-04T10:04:52.201198+0000","last_became_active":"2024-03-04T10:04:52.199854+0000","last_became_peered":"2024-03-04T10:04:52.199854+0000","last_unstale":"2024-03-04T10:04:52.201198+0000","last_undegraded":"2024-03-04T10:04:52.201198+0000","last_fullsized":"2024-03-04T10:04:52.201198+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:07.065080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.193320+0000","last_change":"2024-03-04T10:04:51.186641+0000","last_active":"2024-03-04T10:04:52.193320+0000","last_peered":"2024-03-04T10:04:52.193320+0000","last_clean":"2024-03-04T10:04:52.193320+0000","last_became_active":"2024-03-04T10:04:51.186381+0000","last_became_peered":"2024-03-04T10:04:51.186381+0000","last_unstale":"2024-03-04T10:04:52.193320+0000","last_undegraded":"2024-03-04T10:04:52.193320+0000","last_fullsized":"2024-03-04T10:04:52.193320+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:13.096468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.199370+0000","last_change":"2024-03-04T10:04:52.197322+0000","last_active":"2024-03-04T10:04:52.199370+0000","last_peered":"2024-03-04T10:04:52.199370+0000","last_clean":"2024-03-04T10:04:52.199370+0000","last_became_active":"2024-03-04T10:04:52.197024+0000","last_became_peered":"2024-03-04T10:04:52.197024+0000","last_unstale":"2024-03-04T10:04:52.199370+0000","last_undegraded":"2024-03-04T10:04:52.199370+0000","last_fullsized":"2024-03-04T10:04:52.199370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.185090+0000","last_change":"2024-03-04T10:04:51.192066+0000","last_active":"2024-03-04T10:04:52.185090+0000","last_peered":"2024-03-04T10:04:52.185090+0000","last_clean":"2024-03-04T10:04:52.185090+0000","last_became_active":"2024-03-04T10:04:51.191793+0000","last_became_peered":"2024-03-04T10:04:51.191793+0000","last_unstale":"2024-03-04T10:04:52.185090+0000","last_undegraded":"2024-03-04T10:04:52.185090+0000","last_fullsized":"2024-03-04T10:04:52.185090+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:48.933510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198568+0000","last_change":"2024-03-04T10:04:52.193842+0000","last_active":"2024-03-04T10:04:52.198568+0000","last_peered":"2024-03-04T10:04:52.198568+0000","last_clean":"2024-03-04T10:04:52.198568+0000","last_became_active":"2024-03-04T10:04:52.193554+0000","last_became_peered":"2024-03-04T10:04:52.193554+0000","last_unstale":"2024-03-04T10:04:52.198568+0000","last_undegraded":"2024-03-04T10:04:52.198568+0000","last_fullsized":"2024-03-04T10:04:52.198568+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211885+0000","last_change":"2024-03-04T10:04:51.191012+0000","last_active":"2024-03-04T10:04:52.211885+0000","last_peered":"2024-03-04T10:04:52.211885+0000","last_clean":"2024-03-04T10:04:52.211885+0000","last_became_active":"2024-03-04T10:04:51.190717+0000","last_became_peered":"2024-03-04T10:04:51.190717+0000","last_unstale":"2024-03-04T10:04:52.211885+0000","last_undegraded":"2024-03-04T10:04:52.211885+0000","last_fullsized":"2024-03-04T10:04:52.211885+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:00.724810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208135+0000","last_change":"2024-03-04T10:04:52.201515+0000","last_active":"2024-03-04T10:04:52.208135+0000","last_peered":"2024-03-04T10:04:52.208135+0000","last_clean":"2024-03-04T10:04:52.208135+0000","last_became_active":"2024-03-04T10:04:52.201230+0000","last_became_peered":"2024-03-04T10:04:52.201230+0000","last_unstale":"2024-03-04T10:04:52.208135+0000","last_undegraded":"2024-03-04T10:04:52.208135+0000","last_fullsized":"2024-03-04T10:04:52.208135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:16.366185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.217030+0000","last_change":"2024-03-04T10:04:51.192307+0000","last_active":"2024-03-04T10:04:52.217030+0000","last_peered":"2024-03-04T10:04:52.217030+0000","last_clean":"2024-03-04T10:04:52.217030+0000","last_became_active":"2024-03-04T10:04:51.191992+0000","last_became_peered":"2024-03-04T10:04:51.191992+0000","last_unstale":"2024-03-04T10:04:52.217030+0000","last_undegraded":"2024-03-04T10:04:52.217030+0000","last_fullsized":"2024-03-04T10:04:52.217030+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:49.953133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.205153+0000","last_change":"2024-03-04T10:04:52.196294+0000","last_active":"2024-03-04T10:04:52.205153+0000","last_peered":"2024-03-04T10:04:52.205153+0000","last_clean":"2024-03-04T10:04:52.205153+0000","last_became_active":"2024-03-04T10:04:52.193706+0000","last_became_peered":"2024-03-04T10:04:52.193706+0000","last_unstale":"2024-03-04T10:04:52.205153+0000","last_undegraded":"2024-03-04T10:04:52.205153+0000","last_fullsized":"2024-03-04T10:04:52.205153+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.211035+0000","last_change":"2024-03-04T10:04:51.185755+0000","last_active":"2024-03-04T10:04:52.211035+0000","last_peered":"2024-03-04T10:04:52.211035+0000","last_clean":"2024-03-04T10:04:52.211035+0000","last_became_active":"2024-03-04T10:04:51.185456+0000","last_became_peered":"2024-03-04T10:04:51.185456+0000","last_unstale":"2024-03-04T10:04:52.211035+0000","last_undegraded":"2024-03-04T10:04:52.211035+0000","last_fullsized":"2024-03-04T10:04:52.211035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:40.177606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206321+0000","last_change":"2024-03-04T10:04:52.199324+0000","last_active":"2024-03-04T10:04:52.206321+0000","last_peered":"2024-03-04T10:04:52.206321+0000","last_clean":"2024-03-04T10:04:52.206321+0000","last_became_active":"2024-03-04T10:04:52.199041+0000","last_became_peered":"2024-03-04T10:04:52.199041+0000","last_unstale":"2024-03-04T10:04:52.206321+0000","last_undegraded":"2024-03-04T10:04:52.206321+0000","last_fullsized":"2024-03-04T10:04:52.206321+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187966+0000","last_change":"2024-03-04T10:04:51.189200+0000","last_active":"2024-03-04T10:04:52.187966+0000","last_peered":"2024-03-04T10:04:52.187966+0000","last_clean":"2024-03-04T10:04:52.187966+0000","last_became_active":"2024-03-04T10:04:51.188968+0000","last_became_peered":"2024-03-04T10:04:51.188968+0000","last_unstale":"2024-03-04T10:04:52.187966+0000","last_undegraded":"2024-03-04T10:04:52.187966+0000","last_fullsized":"2024-03-04T10:04:52.187966+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:29:16.432972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202135+0000","last_change":"2024-03-04T10:04:52.198436+0000","last_active":"2024-03-04T10:04:52.202135+0000","last_peered":"2024-03-04T10:04:52.202135+0000","last_clean":"2024-03-04T10:04:52.202135+0000","last_became_active":"2024-03-04T10:04:52.197703+0000","last_became_peered":"2024-03-04T10:04:52.197703+0000","last_unstale":"2024-03-04T10:04:52.202135+0000","last_undegraded":"2024-03-04T10:04:52.202135+0000","last_fullsized":"2024-03-04T10:04:52.202135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:28:36.414268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.189930+0000","last_change":"2024-03-04T10:04:51.187701+0000","last_active":"2024-03-04T10:04:52.189930+0000","last_peered":"2024-03-04T10:04:52.189930+0000","last_clean":"2024-03-04T10:04:52.189930+0000","last_became_active":"2024-03-04T10:04:51.187403+0000","last_became_peered":"2024-03-04T10:04:51.187403+0000","last_unstale":"2024-03-04T10:04:52.189930+0000","last_undegraded":"2024-03-04T10:04:52.189930+0000","last_fullsized":"2024-03-04T10:04:52.189930+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:01.545571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206729+0000","last_change":"2024-03-04T10:04:52.200381+0000","last_active":"2024-03-04T10:04:52.206729+0000","last_peered":"2024-03-04T10:04:52.206729+0000","last_clean":"2024-03-04T10:04:52.206729+0000","last_became_active":"2024-03-04T10:04:52.200111+0000","last_became_peered":"2024-03-04T10:04:52.200111+0000","last_unstale":"2024-03-04T10:04:52.206729+0000","last_undegraded":"2024-03-04T10:04:52.206729+0000","last_fullsized":"2024-03-04T10:04:52.206729+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:31.447689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":20,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.214790+0000","last_change":"2024-03-04T10:04:51.179406+0000","last_active":"2024-03-04T10:04:52.214790+0000","last_peered":"2024-03-04T10:04:52.214790+0000","last_clean":"2024-03-04T10:04:52.214790+0000","last_became_active":"2024-03-04T10:04:51.179008+0000","last_became_peered":"2024-03-04T10:04:51.179008+0000","last_unstale":"2024-03-04T10:04:52.214790+0000","last_undegraded":"2024-03-04T10:04:52.214790+0000","last_fullsized":"2024-03-04T10:04:52.214790+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:29.099324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209676+0000","last_change":"2024-03-04T10:04:52.203342+0000","last_active":"2024-03-04T10:04:52.209676+0000","last_peered":"2024-03-04T10:04:52.209676+0000","last_clean":"2024-03-04T10:04:52.209676+0000","last_became_active":"2024-03-04T10:04:52.203083+0000","last_became_peered":"2024-03-04T10:04:52.203083+0000","last_unstale":"2024-03-04T10:04:52.209676+0000","last_undegraded":"2024-03-04T10:04:52.209676+0000","last_fullsized":"2024-03-04T10:04:52.209676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:46:08.623375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.214508+0000","last_change":"2024-03-04T10:04:51.188593+0000","last_active":"2024-03-04T10:04:52.214508+0000","last_peered":"2024-03-04T10:04:52.214508+0000","last_clean":"2024-03-04T10:04:52.214508+0000","last_became_active":"2024-03-04T10:04:51.188320+0000","last_became_peered":"2024-03-04T10:04:51.188320+0000","last_unstale":"2024-03-04T10:04:52.214508+0000","last_undegraded":"2024-03-04T10:04:52.214508+0000","last_fullsized":"2024-03-04T10:04:52.214508+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:15.362230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209128+0000","last_change":"2024-03-04T10:04:52.203361+0000","last_active":"2024-03-04T10:04:52.209128+0000","last_peered":"2024-03-04T10:04:52.209128+0000","last_clean":"2024-03-04T10:04:52.209128+0000","last_became_active":"2024-03-04T10:04:52.203078+0000","last_became_peered":"2024-03-04T10:04:52.203078+0000","last_unstale":"2024-03-04T10:04:52.209128+0000","last_undegraded":"2024-03-04T10:04:52.209128+0000","last_fullsized":"2024-03-04T10:04:52.209128+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:42:06.313007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.212236+0000","last_change":"2024-03-04T10:04:51.186916+0000","last_active":"2024-03-04T10:04:52.212236+0000","last_peered":"2024-03-04T10:04:52.212236+0000","last_clean":"2024-03-04T10:04:52.212236+0000","last_became_active":"2024-03-04T10:04:51.186644+0000","last_became_peered":"2024-03-04T10:04:51.186644+0000","last_unstale":"2024-03-04T10:04:52.212236+0000","last_undegraded":"2024-03-04T10:04:52.212236+0000","last_fullsized":"2024-03-04T10:04:52.212236+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:56:35.560169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209416+0000","last_change":"2024-03-04T10:04:52.206479+0000","last_active":"2024-03-04T10:04:52.209416+0000","last_peered":"2024-03-04T10:04:52.209416+0000","last_clean":"2024-03-04T10:04:52.209416+0000","last_became_active":"2024-03-04T10:04:52.206198+0000","last_became_peered":"2024-03-04T10:04:52.206198+0000","last_unstale":"2024-03-04T10:04:52.209416+0000","last_undegraded":"2024-03-04T10:04:52.209416+0000","last_fullsized":"2024-03-04T10:04:52.209416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:04.808827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208408+0000","last_change":"2024-03-04T10:04:51.180364+0000","last_active":"2024-03-04T10:04:52.208408+0000","last_peered":"2024-03-04T10:04:52.208408+0000","last_clean":"2024-03-04T10:04:52.208408+0000","last_became_active":"2024-03-04T10:04:51.179274+0000","last_became_peered":"2024-03-04T10:04:51.179274+0000","last_unstale":"2024-03-04T10:04:52.208408+0000","last_undegraded":"2024-03-04T10:04:52.208408+0000","last_fullsized":"2024-03-04T10:04:52.208408+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:45.667918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200489+0000","last_change":"2024-03-04T10:04:52.196730+0000","last_active":"2024-03-04T10:04:52.200489+0000","last_peered":"2024-03-04T10:04:52.200489+0000","last_clean":"2024-03-04T10:04:52.200489+0000","last_became_active":"2024-03-04T10:04:52.196442+0000","last_became_peered":"2024-03-04T10:04:52.196442+0000","last_unstale":"2024-03-04T10:04:52.200489+0000","last_undegraded":"2024-03-04T10:04:52.200489+0000","last_fullsized":"2024-03-04T10:04:52.200489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:05.433228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187684+0000","last_change":"2024-03-04T10:04:51.183951+0000","last_active":"2024-03-04T10:04:52.187684+0000","last_peered":"2024-03-04T10:04:52.187684+0000","last_clean":"2024-03-04T10:04:52.187684+0000","last_became_active":"2024-03-04T10:04:51.183685+0000","last_became_peered":"2024-03-04T10:04:51.183685+0000","last_unstale":"2024-03-04T10:04:52.187684+0000","last_undegraded":"2024-03-04T10:04:52.187684+0000","last_fullsized":"2024-03-04T10:04:52.187684+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:51.776383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.195425+0000","last_change":"2024-03-04T10:04:52.192503+0000","last_active":"2024-03-04T10:04:52.195425+0000","last_peered":"2024-03-04T10:04:52.195425+0000","last_clean":"2024-03-04T10:04:52.195425+0000","last_became_active":"2024-03-04T10:04:52.192203+0000","last_became_peered":"2024-03-04T10:04:52.192203+0000","last_unstale":"2024-03-04T10:04:52.195425+0000","last_undegraded":"2024-03-04T10:04:52.195425+0000","last_fullsized":"2024-03-04T10:04:52.195425+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:16.366185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.210655+0000","last_change":"2024-03-04T10:04:51.179891+0000","last_active":"2024-03-04T10:04:52.210655+0000","last_peered":"2024-03-04T10:04:52.210655+0000","last_clean":"2024-03-04T10:04:52.210655+0000","last_became_active":"2024-03-04T10:04:51.179634+0000","last_became_peered":"2024-03-04T10:04:51.179634+0000","last_unstale":"2024-03-04T10:04:52.210655+0000","last_undegraded":"2024-03-04T10:04:52.210655+0000","last_fullsized":"2024-03-04T10:04:52.210655+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:26:20.575398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.198533+0000","last_change":"2024-03-04T10:04:52.193881+0000","last_active":"2024-03-04T10:04:52.198533+0000","last_peered":"2024-03-04T10:04:52.198533+0000","last_clean":"2024-03-04T10:04:52.198533+0000","last_became_active":"2024-03-04T10:04:52.193570+0000","last_became_peered":"2024-03-04T10:04:52.193570+0000","last_unstale":"2024-03-04T10:04:52.198533+0000","last_undegraded":"2024-03-04T10:04:52.198533+0000","last_fullsized":"2024-03-04T10:04:52.198533+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:02.066773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.182922+0000","last_change":"2024-03-04T10:04:51.175098+0000","last_active":"2024-03-04T10:04:52.182922+0000","last_peered":"2024-03-04T10:04:52.182922+0000","last_clean":"2024-03-04T10:04:52.182922+0000","last_became_active":"2024-03-04T10:04:51.174810+0000","last_became_peered":"2024-03-04T10:04:51.174810+0000","last_unstale":"2024-03-04T10:04:52.182922+0000","last_undegraded":"2024-03-04T10:04:52.182922+0000","last_fullsized":"2024-03-04T10:04:52.182922+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:58:17.348789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200148+0000","last_change":"2024-03-04T10:04:52.195195+0000","last_active":"2024-03-04T10:04:52.200148+0000","last_peered":"2024-03-04T10:04:52.200148+0000","last_clean":"2024-03-04T10:04:52.200148+0000","last_became_active":"2024-03-04T10:04:52.194862+0000","last_became_peered":"2024-03-04T10:04:52.194862+0000","last_unstale":"2024-03-04T10:04:52.200148+0000","last_undegraded":"2024-03-04T10:04:52.200148+0000","last_fullsized":"2024-03-04T10:04:52.200148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:41.181561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.185554+0000","last_change":"2024-03-04T10:04:51.186004+0000","last_active":"2024-03-04T10:04:52.185554+0000","last_peered":"2024-03-04T10:04:52.185554+0000","last_clean":"2024-03-04T10:04:52.185554+0000","last_became_active":"2024-03-04T10:04:51.185720+0000","last_became_peered":"2024-03-04T10:04:51.185720+0000","last_unstale":"2024-03-04T10:04:52.185554+0000","last_undegraded":"2024-03-04T10:04:52.185554+0000","last_fullsized":"2024-03-04T10:04:52.185554+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.567262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208771+0000","last_change":"2024-03-04T10:04:52.204444+0000","last_active":"2024-03-04T10:04:52.208771+0000","last_peered":"2024-03-04T10:04:52.208771+0000","last_clean":"2024-03-04T10:04:52.208771+0000","last_became_active":"2024-03-04T10:04:52.203964+0000","last_became_peered":"2024-03-04T10:04:52.203964+0000","last_unstale":"2024-03-04T10:04:52.208771+0000","last_undegraded":"2024-03-04T10:04:52.208771+0000","last_fullsized":"2024-03-04T10:04:52.208771+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:46.192992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.182902+0000","last_change":"2024-03-04T10:04:51.184634+0000","last_active":"2024-03-04T10:04:52.182902+0000","last_peered":"2024-03-04T10:04:52.182902+0000","last_clean":"2024-03-04T10:04:52.182902+0000","last_became_active":"2024-03-04T10:04:51.184282+0000","last_became_peered":"2024-03-04T10:04:51.184282+0000","last_unstale":"2024-03-04T10:04:52.182902+0000","last_undegraded":"2024-03-04T10:04:52.182902+0000","last_fullsized":"2024-03-04T10:04:52.182902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:22:35.616403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207123+0000","last_change":"2024-03-04T10:04:52.202450+0000","last_active":"2024-03-04T10:04:52.207123+0000","last_peered":"2024-03-04T10:04:52.207123+0000","last_clean":"2024-03-04T10:04:52.207123+0000","last_became_active":"2024-03-04T10:04:52.202155+0000","last_became_peered":"2024-03-04T10:04:52.202155+0000","last_unstale":"2024-03-04T10:04:52.207123+0000","last_undegraded":"2024-03-04T10:04:52.207123+0000","last_fullsized":"2024-03-04T10:04:52.207123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:49.937465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.216040+0000","last_change":"2024-03-04T10:04:51.190615+0000","last_active":"2024-03-04T10:04:52.216040+0000","last_peered":"2024-03-04T10:04:52.216040+0000","last_clean":"2024-03-04T10:04:52.216040+0000","last_became_active":"2024-03-04T10:04:51.190354+0000","last_became_peered":"2024-03-04T10:04:51.190354+0000","last_unstale":"2024-03-04T10:04:52.216040+0000","last_undegraded":"2024-03-04T10:04:52.216040+0000","last_fullsized":"2024-03-04T10:04:52.216040+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:03.804872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.199350+0000","last_change":"2024-03-04T10:04:52.197290+0000","last_active":"2024-03-04T10:04:52.199350+0000","last_peered":"2024-03-04T10:04:52.199350+0000","last_clean":"2024-03-04T10:04:52.199350+0000","last_became_active":"2024-03-04T10:04:52.196689+0000","last_became_peered":"2024-03-04T10:04:52.196689+0000","last_unstale":"2024-03-04T10:04:52.199350+0000","last_undegraded":"2024-03-04T10:04:52.199350+0000","last_fullsized":"2024-03-04T10:04:52.199350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:10:50.957088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.183410+0000","last_change":"2024-03-04T10:04:51.184885+0000","last_active":"2024-03-04T10:04:52.183410+0000","last_peered":"2024-03-04T10:04:52.183410+0000","last_clean":"2024-03-04T10:04:52.183410+0000","last_became_active":"2024-03-04T10:04:51.184505+0000","last_became_peered":"2024-03-04T10:04:51.184505+0000","last_unstale":"2024-03-04T10:04:52.183410+0000","last_undegraded":"2024-03-04T10:04:52.183410+0000","last_fullsized":"2024-03-04T10:04:52.183410+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:09:27.334889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.206407+0000","last_change":"2024-03-04T10:04:52.201097+0000","last_active":"2024-03-04T10:04:52.206407+0000","last_peered":"2024-03-04T10:04:52.206407+0000","last_clean":"2024-03-04T10:04:52.206407+0000","last_became_active":"2024-03-04T10:04:52.198658+0000","last_became_peered":"2024-03-04T10:04:52.198658+0000","last_unstale":"2024-03-04T10:04:52.206407+0000","last_undegraded":"2024-03-04T10:04:52.206407+0000","last_fullsized":"2024-03-04T10:04:52.206407+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:42:06.313007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.186681+0000","last_change":"2024-03-04T10:04:51.186045+0000","last_active":"2024-03-04T10:04:52.186681+0000","last_peered":"2024-03-04T10:04:52.186681+0000","last_clean":"2024-03-04T10:04:52.186681+0000","last_became_active":"2024-03-04T10:04:51.185760+0000","last_became_peered":"2024-03-04T10:04:51.185760+0000","last_unstale":"2024-03-04T10:04:52.186681+0000","last_undegraded":"2024-03-04T10:04:52.186681+0000","last_fullsized":"2024-03-04T10:04:52.186681+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:23:15.362230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.208443+0000","last_change":"2024-03-04T10:04:52.204788+0000","last_active":"2024-03-04T10:04:52.208443+0000","last_peered":"2024-03-04T10:04:52.208443+0000","last_clean":"2024-03-04T10:04:52.208443+0000","last_became_active":"2024-03-04T10:04:52.204149+0000","last_became_peered":"2024-03-04T10:04:52.204149+0000","last_unstale":"2024-03-04T10:04:52.208443+0000","last_undegraded":"2024-03-04T10:04:52.208443+0000","last_fullsized":"2024-03-04T10:04:52.208443+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.186466+0000","last_change":"2024-03-04T10:04:51.185426+0000","last_active":"2024-03-04T10:04:52.186466+0000","last_peered":"2024-03-04T10:04:52.186466+0000","last_clean":"2024-03-04T10:04:52.186466+0000","last_became_active":"2024-03-04T10:04:51.185061+0000","last_became_peered":"2024-03-04T10:04:51.185061+0000","last_unstale":"2024-03-04T10:04:52.186466+0000","last_undegraded":"2024-03-04T10:04:52.186466+0000","last_fullsized":"2024-03-04T10:04:52.186466+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:25.567262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.207017+0000","last_change":"2024-03-04T10:04:52.200084+0000","last_active":"2024-03-04T10:04:52.207017+0000","last_peered":"2024-03-04T10:04:52.207017+0000","last_clean":"2024-03-04T10:04:52.207017+0000","last_became_active":"2024-03-04T10:04:52.199819+0000","last_became_peered":"2024-03-04T10:04:52.199819+0000","last_unstale":"2024-03-04T10:04:52.207017+0000","last_undegraded":"2024-03-04T10:04:52.207017+0000","last_fullsized":"2024-03-04T10:04:52.207017+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:26.571217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.187952+0000","last_change":"2024-03-04T10:04:51.184369+0000","last_active":"2024-03-04T10:04:52.187952+0000","last_peered":"2024-03-04T10:04:52.187952+0000","last_clean":"2024-03-04T10:04:52.187952+0000","last_became_active":"2024-03-04T10:04:51.184064+0000","last_became_peered":"2024-03-04T10:04:51.184064+0000","last_unstale":"2024-03-04T10:04:52.187952+0000","last_undegraded":"2024-03-04T10:04:52.187952+0000","last_fullsized":"2024-03-04T10:04:52.187952+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:29:05.996823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.200947+0000","last_change":"2024-03-04T10:04:52.196848+0000","last_active":"2024-03-04T10:04:52.200947+0000","last_peered":"2024-03-04T10:04:52.200947+0000","last_clean":"2024-03-04T10:04:52.200947+0000","last_became_active":"2024-03-04T10:04:52.196580+0000","last_became_peered":"2024-03-04T10:04:52.196580+0000","last_unstale":"2024-03-04T10:04:52.200947+0000","last_undegraded":"2024-03-04T10:04:52.200947+0000","last_fullsized":"2024-03-04T10:04:52.200947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:30.103279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.190845+0000","last_change":"2024-03-04T10:04:51.178928+0000","last_active":"2024-03-04T10:04:52.190845+0000","last_peered":"2024-03-04T10:04:52.190845+0000","last_clean":"2024-03-04T10:04:52.190845+0000","last_became_active":"2024-03-04T10:04:51.178131+0000","last_became_peered":"2024-03-04T10:04:51.178131+0000","last_unstale":"2024-03-04T10:04:52.190845+0000","last_undegraded":"2024-03-04T10:04:52.190845+0000","last_fullsized":"2024-03-04T10:04:52.190845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:19:51.776383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.202168+0000","last_change":"2024-03-04T10:04:52.197914+0000","last_active":"2024-03-04T10:04:52.202168+0000","last_peered":"2024-03-04T10:04:52.202168+0000","last_clean":"2024-03-04T10:04:52.202168+0000","last_became_active":"2024-03-04T10:04:52.197629+0000","last_became_peered":"2024-03-04T10:04:52.197629+0000","last_unstale":"2024-03-04T10:04:52.202168+0000","last_undegraded":"2024-03-04T10:04:52.202168+0000","last_fullsized":"2024-03-04T10:04:52.202168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:35.896813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.189467+0000","last_change":"2024-03-04T10:04:51.186643+0000","last_active":"2024-03-04T10:04:52.189467+0000","last_peered":"2024-03-04T10:04:52.189467+0000","last_clean":"2024-03-04T10:04:52.189467+0000","last_became_active":"2024-03-04T10:04:51.186424+0000","last_became_peered":"2024-03-04T10:04:51.186424+0000","last_unstale":"2024-03-04T10:04:52.189467+0000","last_undegraded":"2024-03-04T10:04:52.189467+0000","last_fullsized":"2024-03-04T10:04:52.189467+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:17:01.062818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.209358+0000","last_change":"2024-03-04T10:04:52.201380+0000","last_active":"2024-03-04T10:04:52.209358+0000","last_peered":"2024-03-04T10:04:52.209358+0000","last_clean":"2024-03-04T10:04:52.209358+0000","last_became_active":"2024-03-04T10:04:52.201105+0000","last_became_peered":"2024-03-04T10:04:52.201105+0000","last_unstale":"2024-03-04T10:04:52.209358+0000","last_undegraded":"2024-03-04T10:04:52.209358+0000","last_fullsized":"2024-03-04T10:04:52.209358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:51.163178+0000","last_clean_scrub_stamp":"2024-03-04T10:04:51.163178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:48:04.258123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.183086+0000","last_change":"2024-03-04T10:04:51.180914+0000","last_active":"2024-03-04T10:04:52.183086+0000","last_peered":"2024-03-04T10:04:52.183086+0000","last_clean":"2024-03-04T10:04:52.183086+0000","last_became_active":"2024-03-04T10:04:51.180615+0000","last_became_peered":"2024-03-04T10:04:51.180615+0000","last_unstale":"2024-03-04T10:04:52.183086+0000","last_undegraded":"2024-03-04T10:04:52.183086+0000","last_fullsized":"2024-03-04T10:04:52.183086+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:04:50.159223+0000","last_clean_scrub_stamp":"2024-03-04T10:04:50.159223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:39.689303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":204,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-03-04T10:04:52.182457+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:04:52.182457+0000","last_peered":"2024-03-04T10:04:52.182457+0000","last_clean":"2024-03-04T10:04:52.182457+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:04:52.182457+0000","last_undegraded":"2024-03-04T10:04:52.182457+0000","last_fullsized":"2024-03-04T10:04:52.182457+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1007616,"data_stored":766415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":413696,"data_stored":176047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1007616,"data_stored":766415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:04:55.486 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T10:04:55.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T10:04:55.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.628+0000 7fa8cc1c4640 1 Processor -- start 2024-03-04T10:04:55.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.629+0000 7fa8cc1c4640 1 -- start start 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.629+0000 7fa8cc1c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 0x7fa8c406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.629+0000 7fa8cc1c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 0x7fa8c406b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.629+0000 7fa8cc1c4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c415cf50 con 0x7fa8c40b0150 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.629+0000 7fa8cc1c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c40cc390 con 0x7fa8c40c3e20 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8cc1c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40cc670 con 0x7fa8c415c7f0 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 0x7fa8c406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8ca761640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 0x7fa8c406b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 0x7fa8c406acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:36318/0 (socket says 172.21.15.99:36318) 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 -- 172.21.15.99:0/2237284743 learned_addr learned my addr 172.21.15.99:0/2237284743 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8ca761640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 0x7fa8c406b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:53076/0 (socket says 172.21.15.99:53076) 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c975f640 1 --1- 172.21.15.99:0/2237284743 >> v1:172.21.15.162:6789/0 conn(0x7fa8c40b0150 0x7fa8c415f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:33504/0 (socket says 172.21.15.99:33504) 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 -- 172.21.15.99:0/2237284743 >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 msgr2=0x7fa8c406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 --2- 172.21.15.99:0/2237284743 >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 0x7fa8c406b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 -- 172.21.15.99:0/2237284743 >> v1:172.21.15.162:6789/0 conn(0x7fa8c40b0150 legacy=0x7fa8c415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8c9f60640 1 -- 172.21.15.99:0/2237284743 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40cc950 con 0x7fa8c415c7f0 2024-03-04T10:04:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.630+0000 7fa8ca761640 1 --2- 172.21.15.99:0/2237284743 >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 0x7fa8c406b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.631+0000 7fa8c9f60640 1 --2- 172.21.15.99:0/2237284743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 0x7fa8c406acd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009870 tx=0x7fa8c002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abe48db3a4ad60a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.631+0000 7fa8c8f5e640 1 -- 172.21.15.99:0/2237284743 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00317c0 con 0x7fa8c415c7f0 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.631+0000 7fa8c8f5e640 1 -- 172.21.15.99:0/2237284743 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c0037070 con 0x7fa8c415c7f0 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.631+0000 7fa8c8f5e640 1 -- 172.21.15.99:0/2237284743 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0031b80 con 0x7fa8c415c7f0 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/2237284743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 msgr2=0x7fa8c406acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/2237284743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 0x7fa8c406acd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009870 tx=0x7fa8c002b860 comp rx=0 tx=0).stop 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/2237284743 shutdown_connections 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/2237284743 >> v2:172.21.15.162:3300/0 conn(0x7fa8c40c3e20 0x7fa8c406b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/2237284743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c415c7f0 0x7fa8c406acd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/2237284743 >> 172.21.15.99:0/2237284743 conn(0x7fa8c40b22e0 msgr2=0x7fa8c40b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/2237284743 shutdown_connections 2024-03-04T10:04:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.632+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/2237284743 wait complete. 2024-03-04T10:04:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.633+0000 7fa8cc1c4640 1 Processor -- start 2024-03-04T10:04:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.633+0000 7fa8cc1c4640 1 -- start start 2024-03-04T10:04:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 0x7fa8c40c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8c40c3e20 0x7fa8c41f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c40ca1d0 0x7fa8c40c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40b2e00 con 0x7fa8c40b0150 2024-03-04T10:04:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c415cbe0 con 0x7fa8c40c3e20 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c40b2f80 con 0x7fa8c40ca1d0 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 0x7fa8c40c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 0x7fa8c40c4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50048/0 (socket says 172.21.15.99:50048) 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 -- 172.21.15.99:0/1117752114 learned_addr learned my addr 172.21.15.99:0/1117752114 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 -- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c40ca1d0 msgr2=0x7fa8c40c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c975f640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8c40c3e20 0x7fa8c41f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8ca761640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c40ca1d0 0x7fa8c40c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c40ca1d0 0x7fa8c40c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 -- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8c40c3e20 msgr2=0x7fa8c41f3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8c40c3e20 0x7fa8c41f3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 -- 172.21.15.99:0/1117752114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c40c4a70 con 0x7fa8c40b0150 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c975f640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8c40c3e20 0x7fa8c41f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8c9f60640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 0x7fa8c40c4cb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa8c002b8a0 tx=0x7fa8c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0032650 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c00327f0 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c40ced30 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.634+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c0032be0 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.635+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c40cf2f0 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.635+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c4074f60 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.636+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa8c004c060 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.636+0000 7fa8ba7fc640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.636+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa8c00abe90 con 0x7fa8c40b0150 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.636+0000 7fa8c975f640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.637+0000 7fa8c975f640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8a806b4d0 0x7fa8a806d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40c6490 tx=0x7fa8b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.640+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa8c0040070 con 0x7fa8c40b0150 2024-03-04T10:04:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.876+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa8c4072f90 con 0x7fa8c40b0150 2024-03-04T10:04:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.878+0000 7fa8ba7fc640 1 -- 172.21.15.99:0/1117752114 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa8c007ab80 con 0x7fa8c40b0150 2024-03-04T10:04:55.879 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-04T10:04:55.879 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:04:55.880 INFO:teuthology.orchestra.run.smithi099.stdout: caps mds = "allow" 2024-03-04T10:04:55.880 INFO:teuthology.orchestra.run.smithi099.stdout: caps mon = "allow r" 2024-03-04T10:04:55.880 INFO:teuthology.orchestra.run.smithi099.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T10:04:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:updated caps for client.0 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8a806b4d0 msgr2=0x7fa8a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8a806b4d0 0x7fa8a806d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40c6490 tx=0x7fa8b4009040 comp rx=0 tx=0).stop 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 msgr2=0x7fa8c40c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 0x7fa8c40c4cb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa8c002b8a0 tx=0x7fa8c00046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 shutdown_connections 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8a806b4d0 0x7fa8a806d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8c40ca1d0 0x7fa8c40c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8c40c3e20 0x7fa8c41f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 --2- 172.21.15.99:0/1117752114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8c40b0150 0x7fa8c40c4cb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 >> 172.21.15.99:0/1117752114 conn(0x7fa8c40b22e0 msgr2=0x7fa8c40b26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 shutdown_connections 2024-03-04T10:04:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.880+0000 7fa8cc1c4640 1 -- 172.21.15.99:0/1117752114 wait complete. 2024-03-04T10:04:55.892 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:55.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.991+0000 7f210a075640 1 Processor -- start 2024-03-04T10:04:55.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.991+0000 7f210a075640 1 -- start start 2024-03-04T10:04:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f210a075640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f21040b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f210a075640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f21040b5470 0x7f21040b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f210a075640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210415e1e0 con 0x7f21040b9380 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f210a075640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21040d2820 con 0x7f21040b5470 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f210a075640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21040d1c10 con 0x7f210415de00 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2103fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f21040b5470 0x7f21040b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2102ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f21040b9380 0x7f21040b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56974/0 (socket says 172.21.15.99:56974) 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2102ffd640 1 -- 172.21.15.99:0/1709620525 learned_addr learned my addr 172.21.15.99:0/1709620525 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2103fff640 1 -- 172.21.15.99:0/1709620525 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 msgr2=0x7f21040b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2103fff640 1 --2- 172.21.15.99:0/1709620525 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f21040b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2103fff640 1 -- 172.21.15.99:0/1709620525 >> v1:172.21.15.99:6789/0 conn(0x7f21040b9380 legacy=0x7f21040b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f21027fc640 1 -- 172.21.15.99:0/1709620525 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823755804 0 0) 0x7f210415e1e0 con 0x7f21040b9380 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.992+0000 7f2103fff640 1 -- 172.21.15.99:0/1709620525 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21040d1ef0 con 0x7f21040b5470 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.993+0000 7f2103fff640 1 --2- 172.21.15.99:0/1709620525 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f21040b5900 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f20fc009d90 tx=0x7f20fc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f8e2311db17593 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.993+0000 7f21027fc640 1 -- 172.21.15.99:0/1709620525 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc039070 con 0x7f21040b5470 2024-03-04T10:04:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.993+0000 7f21027fc640 1 -- 172.21.15.99:0/1709620525 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc02cc50 con 0x7f21040b5470 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.993+0000 7f21027fc640 1 -- 172.21.15.99:0/1709620525 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc034440 con 0x7f21040b5470 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 -- 172.21.15.99:0/1709620525 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 msgr2=0x7f21040b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 --2- 172.21.15.99:0/1709620525 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f21040b5900 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f20fc009d90 tx=0x7f20fc02bea0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 -- 172.21.15.99:0/1709620525 shutdown_connections 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 --2- 172.21.15.99:0/1709620525 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f21040b5900 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 --2- 172.21.15.99:0/1709620525 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f21040b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 -- 172.21.15.99:0/1709620525 >> 172.21.15.99:0/1709620525 conn(0x7f21040d0470 msgr2=0x7f21040d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 -- 172.21.15.99:0/1709620525 shutdown_connections 2024-03-04T10:04:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.994+0000 7f210a075640 1 -- 172.21.15.99:0/1709620525 wait complete. 2024-03-04T10:04:55.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.995+0000 7f210a075640 1 Processor -- start 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.995+0000 7f210a075640 1 -- start start 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f210a075640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f2104156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f210a075640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21040b9380 0x7f21040b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f210a075640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f2104143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f210a075640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21040b97e0 con 0x7f21040b5470 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f210a075640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21040d24b0 con 0x7f21040b9380 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f210a075640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21040b72a0 con 0x7f210415de00 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f2104156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f2104156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50066/0 (socket says 172.21.15.99:50066) 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 -- 172.21.15.99:0/1002557659 learned_addr learned my addr 172.21.15.99:0/1002557659 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f2103fff640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f2104143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f2102ffd640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21040b9380 0x7f21040b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 -- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 msgr2=0x7f2104143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f2104143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 -- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21040b9380 msgr2=0x7f21040b28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21040b9380 0x7f21040b28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 -- 172.21.15.99:0/1002557659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21041555e0 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f21037fe640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f2104156680 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f20f800b410 tx=0x7f20f800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.996+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8015020 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.997+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f8011850 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.997+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21041558c0 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.997+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8011c40 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.997+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2104156160 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.998+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2104156c20 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.998+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f20f8018020 con 0x7f21040b5470 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.998+0000 7f20ebfff640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.999+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f20f808a260 con 0x7f21040b5470 2024-03-04T10:04:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.999+0000 7f2102ffd640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:55.999+0000 7f2102ffd640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21040d3700 tx=0x7f20f4007b70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.003+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f20f801f020 con 0x7f21040b5470 2024-03-04T10:04:56.159 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:04:56.158+0000 7f4c833bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:04:56.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.212+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2104074f60 con 0x7f21040b5470 2024-03-04T10:04:56.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.212+0000 7f20ebfff640 1 -- 172.21.15.99:0/1002557659 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f20f80172c0 con 0x7f21040b5470 2024-03-04T10:04:56.214 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:56.214 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:04:53.524677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/434536114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":434536114},{"type":"v1","addr":"172.21.15.99:6835","nonce":434536114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:04:56.215 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 15 2024-03-04T10:04:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.214+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 msgr2=0x7f20f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.214+0000 7f210a075640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21040d3700 tx=0x7f20f4007b70 comp rx=0 tx=0).stop 2024-03-04T10:04:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.214+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 msgr2=0x7f2104156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.214+0000 7f210a075640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f2104156680 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f20f800b410 tx=0x7f20f800b8d0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 shutdown_connections 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f20f006b4d0 0x7f20f006d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f210415de00 0x7f2104143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f21040b9380 0x7f21040b28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 --2- 172.21.15.99:0/1002557659 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f21040b5470 0x7f2104156680 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 >> 172.21.15.99:0/1002557659 conn(0x7f21040d0470 msgr2=0x7f21040b7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 shutdown_connections 2024-03-04T10:04:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.215+0000 7f210a075640 1 -- 172.21.15.99:0/1002557659 wait complete. 2024-03-04T10:04:56.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:04:51.516792+0000', 'modified': '2024-03-04T10:04:53.524677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5295}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5295': {'gid': 5295, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6835/434536114', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 434536114}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 434536114}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:04:56.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:04:56.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:04:56.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.731+0000 7ff793897640 1 Processor -- start 2024-03-04T10:04:56.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.731+0000 7ff793897640 1 -- start start 2024-03-04T10:04:56.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff793897640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0b7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff793897640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff78c15be90 0x7ff78c0cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff793897640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78c15f8c0 con 0x7ff78c0cfcd0 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff793897640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78c15c7f0 con 0x7ff78c15be90 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff793897640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c15cad0 con 0x7ff78c15bab0 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff790e32640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff78c15be90 0x7ff78c0cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff790e32640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff78c15be90 0x7ff78c0cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50076/0 (socket says 172.21.15.99:50076) 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.732+0000 7ff790e32640 1 -- 172.21.15.99:0/2567055266 learned_addr learned my addr 172.21.15.99:0/2567055266 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.733+0000 7ff791633640 1 --2- 172.21.15.99:0/2567055266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.733+0000 7ff790e32640 1 -- 172.21.15.99:0/2567055266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 msgr2=0x7ff78c0b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.733+0000 7ff790e32640 1 --2- 172.21.15.99:0/2567055266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.733+0000 7ff790e32640 1 -- 172.21.15.99:0/2567055266 >> v1:172.21.15.162:6789/0 conn(0x7ff78c0cfcd0 legacy=0x7ff78c0cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.733+0000 7ff790e32640 1 -- 172.21.15.99:0/2567055266 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c15cdb0 con 0x7ff78c15be90 2024-03-04T10:04:56.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.733+0000 7ff790e32640 1 --2- 172.21.15.99:0/2567055266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c0cf790 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009870 tx=0x7ff77c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d007bb941752687 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff783fff640 1 -- 172.21.15.99:0/2567055266 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c034070 con 0x7ff78c15be90 2024-03-04T10:04:56.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff783fff640 1 -- 172.21.15.99:0/2567055266 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c030840 con 0x7ff78c15be90 2024-03-04T10:04:56.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff783fff640 1 -- 172.21.15.99:0/2567055266 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c030c10 con 0x7ff78c15be90 2024-03-04T10:04:56.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 -- 172.21.15.99:0/2567055266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 msgr2=0x7ff78c0cf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 --2- 172.21.15.99:0/2567055266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c0cf790 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009870 tx=0x7ff77c02b980 comp rx=0 tx=0).stop 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 -- 172.21.15.99:0/2567055266 shutdown_connections 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 --2- 172.21.15.99:0/2567055266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c0cf790 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 --2- 172.21.15.99:0/2567055266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 -- 172.21.15.99:0/2567055266 >> 172.21.15.99:0/2567055266 conn(0x7ff78c06f6e0 msgr2=0x7ff78c06faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 -- 172.21.15.99:0/2567055266 shutdown_connections 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.734+0000 7ff793897640 1 -- 172.21.15.99:0/2567055266 wait complete. 2024-03-04T10:04:56.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.735+0000 7ff793897640 1 Processor -- start 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.735+0000 7ff793897640 1 -- start start 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff793897640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff78c0cfcd0 0x7ff78c0c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff793897640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff793897640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff793897640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c0b7920 con 0x7ff78c15be90 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff793897640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c0b77a0 con 0x7ff78c0cfcd0 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff793897640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c0cc8c0 con 0x7ff78c15bab0 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff791e34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff791e34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50080/0 (socket says 172.21.15.99:50080) 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff791e34640 1 -- 172.21.15.99:0/196646708 learned_addr learned my addr 172.21.15.99:0/196646708 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.736+0000 7ff791633640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff78c0cfcd0 0x7ff78c0c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791e34640 1 -- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 msgr2=0x7ff78c0d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791e34640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791e34640 1 -- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff78c0cfcd0 msgr2=0x7ff78c0c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791e34640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff78c0cfcd0 0x7ff78c0c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791e34640 1 -- 172.21.15.99:0/196646708 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c0caf90 con 0x7ff78c15be90 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791633640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff78c0cfcd0 0x7ff78c0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff791e34640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c148930 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff78400c3f0 tx=0x7ff78400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff781ffb640 1 -- 172.21.15.99:0/196646708 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7840052d0 con 0x7ff78c15be90 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff781ffb640 1 -- 172.21.15.99:0/196646708 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff784005490 con 0x7ff78c15be90 2024-03-04T10:04:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c0c95b0 con 0x7ff78c15be90 2024-03-04T10:04:56.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff781ffb640 1 -- 172.21.15.99:0/196646708 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784007260 con 0x7ff78c15be90 2024-03-04T10:04:56.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.737+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c0cb210 con 0x7ff78c15be90 2024-03-04T10:04:56.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.738+0000 7ff781ffb640 1 -- 172.21.15.99:0/196646708 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff784007400 con 0x7ff78c15be90 2024-03-04T10:04:56.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.739+0000 7ff781ffb640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff77006b4d0 0x7ff77006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.739+0000 7ff781ffb640 1 -- 172.21.15.99:0/196646708 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff784089820 con 0x7ff78c15be90 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.739+0000 7ff791633640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff77006b4d0 0x7ff77006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.740+0000 7ff791633640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff77006b4d0 0x7ff77006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff788007330 tx=0x7ff78800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.740+0000 7ff79647c740 1 Processor -- start 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.740+0000 7ff79647c740 1 -- start start 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674920 0x55a05c6725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674d00 0x55a05c62bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff79647c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 0x55a05c62c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff79647c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05c5917c0 con 0x55a05c654260 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05c53ab50 con 0x55a05c674d00 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a05c58d860 con 0x55a05c674920 2024-03-04T10:04:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 0x55a05c62c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 0x55a05c62c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50096/0 (socket says 172.21.15.99:50096) 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 -- 172.21.15.99:0/896621048 learned_addr learned my addr 172.21.15.99:0/896621048 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791633640 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674920 0x55a05c6725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff790e32640 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674d00 0x55a05c62bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 -- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674920 msgr2=0x55a05c6725e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674920 0x55a05c6725e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 -- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674d00 msgr2=0x55a05c62bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674d00 0x55a05c62bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 -- 172.21.15.99:0/896621048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a05c54e9a0 con 0x55a05c654260 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff790e32640 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674d00 0x55a05c62bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff791e34640 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 0x55a05c62c490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff784087b30 tx=0x7ff784000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242213e4e20855e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff776ffd640 1 -- 172.21.15.99:0/896621048 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784056110 con 0x55a05c654260 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff776ffd640 1 -- 172.21.15.99:0/896621048 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78405e0c0 con 0x55a05c654260 2024-03-04T10:04:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.741+0000 7ff776ffd640 1 -- 172.21.15.99:0/896621048 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784057260 con 0x55a05c654260 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 -- 172.21.15.99:0/896621048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 msgr2=0x55a05c62c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 0x55a05c62c490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff784087b30 tx=0x7ff784000bc0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 -- 172.21.15.99:0/896621048 shutdown_connections 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c654260 0x55a05c62c490 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674d00 0x55a05c62bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 --2- 172.21.15.99:0/896621048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674920 0x55a05c6725e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 -- 172.21.15.99:0/896621048 >> 172.21.15.99:0/896621048 conn(0x55a05c6290b0 msgr2=0x55a05c671c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 -- 172.21.15.99:0/896621048 shutdown_connections 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.742+0000 7ff79647c740 1 -- 172.21.15.99:0/896621048 wait complete. 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 -- start start 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 Processor -- start 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c676300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c64f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c659f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05c5917c0 con 0x55a05c674920 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05c54df90 con 0x55a05c650150 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.743+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a05c6276e0 con 0x55a05c674d00 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff79647c740 10 client.0 _ll_register_callbacks cb 0x55a05c56fd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c676300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c676300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50100/0 (socket says 172.21.15.99:50100) 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 -- 172.21.15.99:0/606479120 learned_addr learned my addr 172.21.15.99:0/606479120 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791e34640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c659f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff790e32640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c64f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 -- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 msgr2=0x55a05c64f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c64f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 -- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 msgr2=0x55a05c659f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c659f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 -- 172.21.15.99:0/606479120 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a05c2a62d0 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791e34640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c659f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff790e32640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c64f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff791633640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c676300 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff7880042e0 tx=0x7ff788012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6885657c79b69fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff76bfff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff76bfff640 1 -- 172.21.15.99:0/606479120 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788019b20 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff76bfff640 1 -- 172.21.15.99:0/606479120 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff788019cc0 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff78405e0c0 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff76bfff640 1 -- 172.21.15.99:0/606479120 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff788014410 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff79647c740 10 client.5481 fetch_fsmap learned FSMap version 15 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55a05c3706b0 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff76bfff640 1 -- 172.21.15.99:0/606479120 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78405e0c0 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.744+0000 7ff76bfff640 1 -- 172.21.15.99:0/606479120 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff78802d050 con 0x55a05c674920 2024-03-04T10:04:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff79647c740 10 client.5481 fetch_fsmap finished waiting for FSMap version 15 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff79647c740 10 client.5481 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff79647c740 20 client.5481 populate_metadata read hostname 'smithi099' 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff79647c740 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c692440 0x55a05c694830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff79647c740 4 client.5481 mds_command: new command op to 5295 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 --> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a05c59ed00 con 0x55a05c692440 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff790e32640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c692440 0x55a05c694830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff76bfff640 10 client.5481 ms_handle_connect on v2:172.21.15.99:6833/434536114 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff790e32640 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c692440 0x55a05c694830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdf0229869216d50 server_cookie=8998273ea212ac3f in_seq=0 out_seq=0 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff76bfff640 1 -- 172.21.15.99:0/606479120 <== mds.0 v2:172.21.15.99:6833/434536114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a05c59ed00 con 0x55a05c692440 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.745+0000 7ff76bfff640 10 client.5481 handle_command_reply: tid=0 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 client.5481 shutdown 2024-03-04T10:04:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 msgr2=0x55a05c676300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c676300 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff7880042e0 tx=0x7ff788012000 comp rx=0 tx=0).stop 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 shutdown_connections 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c692440 0x55a05c694830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c659f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c64f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 --2- 172.21.15.99:0/606479120 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c676300 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff76bfff640 0 client.5481 ms_handle_reset on v2:172.21.15.99:6833/434536114 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 >> 172.21.15.99:0/606479120 conn(0x55a05c654260 msgr2=0x55a05c673a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 shutdown_connections 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 1 -- 172.21.15.99:0/606479120 wait complete. 2024-03-04T10:04:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.746+0000 7ff79647c740 20 client.5481 trim_cache size 0 max 16384 2024-03-04T10:04:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.761+0000 7ff79647c740 1 Processor -- start 2024-03-04T10:04:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.761+0000 7ff79647c740 1 -- start start 2024-03-04T10:04:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c673a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff79647c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c673f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c625db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff79647c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c06dcc0 con 0x55a05c674920 2024-03-04T10:04:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c06d0c0 con 0x55a05c650150 2024-03-04T10:04:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c06d6c0 con 0x55a05c674d00 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c673f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c673f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50106/0 (socket says 172.21.15.99:50106) 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 -- 172.21.15.99:0/2253335338 learned_addr learned my addr 172.21.15.99:0/2253335338 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff791633640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c673a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff791e34640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c625db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 -- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 msgr2=0x55a05c625db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c625db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 -- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 msgr2=0x55a05c673a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c673a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff790e32640 1 -- 172.21.15.99:0/2253335338 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a05c654970 con 0x55a05c674920 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff791e34640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c625db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.762+0000 7ff791633640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c673a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff790e32640 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c673f80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0090b0 tx=0x7ff77c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ddb8da5bb1a5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff7767fc640 1 -- 172.21.15.99:0/2253335338 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c034040 con 0x55a05c674920 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff7767fc640 1 -- 172.21.15.99:0/2253335338 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c030d70 con 0x55a05c674920 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff7767fc640 1 -- 172.21.15.99:0/2253335338 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c0472d0 con 0x55a05c674920 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 -- 172.21.15.99:0/2253335338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 msgr2=0x55a05c673f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c673f80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0090b0 tx=0x7ff77c030000 comp rx=0 tx=0).stop 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 -- 172.21.15.99:0/2253335338 shutdown_connections 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c674d00 0x55a05c625db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674920 0x55a05c673f80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 --2- 172.21.15.99:0/2253335338 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c650150 0x55a05c673a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 -- 172.21.15.99:0/2253335338 >> 172.21.15.99:0/2253335338 conn(0x55a05c654260 msgr2=0x55a05c64f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 -- 172.21.15.99:0/2253335338 shutdown_connections 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.763+0000 7ff79647c740 1 -- 172.21.15.99:0/2253335338 wait complete. 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.764+0000 7ff79647c740 1 -- start start 2024-03-04T10:04:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.764+0000 7ff79647c740 1 Processor -- start 2024-03-04T10:04:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c650150 0x55a05c698a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674920 0x55a05c654b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 0x55a05c655090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05c659a20 con 0x55a05c674d00 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a05c659460 con 0x55a05c674920 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a05c656e90 con 0x55a05c650150 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff79647c740 10 client.0 _ll_register_callbacks cb 0x7ff784057be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 0x55a05c655090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 0x55a05c655090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50108/0 (socket says 172.21.15.99:50108) 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 -- 172.21.15.99:0/3851956122 learned_addr learned my addr 172.21.15.99:0/3851956122 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791633640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c650150 0x55a05c698a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff790e32640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674920 0x55a05c654b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 -- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c650150 msgr2=0x55a05c698a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c650150 0x55a05c698a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 -- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674920 msgr2=0x55a05c654b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674920 0x55a05c654b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 -- 172.21.15.99:0/3851956122 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a05c625e50 con 0x55a05c674d00 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff791e34640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 0x55a05c655090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff784014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca55dc566c95548a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.765+0000 7ff76bfff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff76bfff640 1 -- 172.21.15.99:0/3851956122 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784057950 con 0x55a05c674d00 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff76bfff640 1 -- 172.21.15.99:0/3851956122 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff784056110 con 0x55a05c674d00 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff77c030d70 con 0x55a05c674d00 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff76bfff640 1 -- 172.21.15.99:0/3851956122 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff78405e0c0 con 0x55a05c674d00 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 10 client.5487 fetch_fsmap learned FSMap version 15 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x55a05c6577b0 con 0x55a05c674d00 2024-03-04T10:04:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff76bfff640 1 -- 172.21.15.99:0/3851956122 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c030d70 con 0x55a05c674d00 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff76bfff640 1 -- 172.21.15.99:0/3851956122 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff784076ce0 con 0x55a05c674d00 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 10 client.5487 fetch_fsmap finished waiting for FSMap version 15 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 10 client.5487 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 20 client.5487 populate_metadata read hostname 'smithi099' 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c6534e0 0x55a05c6538c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 4 client.5487 mds_command: new command op to 5295 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 --> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a05c599050 con 0x55a05c6534e0 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff791633640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c6534e0 0x55a05c6538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff76bfff640 10 client.5487 ms_handle_connect on v2:172.21.15.99:6833/434536114 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.766+0000 7ff791633640 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c6534e0 0x55a05c6538c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=951117d9562795d2 server_cookie=147f2db981297acc in_seq=0 out_seq=0 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff76bfff640 1 -- 172.21.15.99:0/3851956122 <== mds.0 v2:172.21.15.99:6833/434536114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a05c599050 con 0x55a05c6534e0 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff76bfff640 10 client.5487 handle_command_reply: tid=0 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 client.5487 shutdown 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 msgr2=0x55a05c655090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 0x55a05c655090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff784000c00 tx=0x7ff784014040 comp rx=0 tx=0).stop 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 shutdown_connections 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:6833/434536114,v1:172.21.15.99:6835/434536114] conn(0x55a05c6534e0 0x55a05c6538c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a05c674d00 0x55a05c655090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff76bfff640 0 client.5487 ms_handle_reset on v2:172.21.15.99:6833/434536114 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a05c674920 0x55a05c654b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 --2- 172.21.15.99:0/3851956122 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a05c650150 0x55a05c698a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 >> 172.21.15.99:0/3851956122 conn(0x55a05c654260 msgr2=0x55a05c62b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 shutdown_connections 2024-03-04T10:04:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 1 -- 172.21.15.99:0/3851956122 wait complete. 2024-03-04T10:04:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff79647c740 20 client.5487 trim_cache size 0 max 16384 2024-03-04T10:04:56.776 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:04:56.776 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:04:56.776 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 5295, 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.5795449389999998, 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 15, 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 38, 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 38, 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 11.477602706000001 2024-03-04T10:04:56.777 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:04:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff77006b4d0 msgr2=0x7ff77006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff77006b4d0 0x7ff77006d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff788007330 tx=0x7ff78800b040 comp rx=0 tx=0).stop 2024-03-04T10:04:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 msgr2=0x7ff78c148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c148930 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff78400c3f0 tx=0x7ff78400c8b0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 shutdown_connections 2024-03-04T10:04:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff77006b4d0 0x7ff77006d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff78c15be90 0x7ff78c148930 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff78c15bab0 0x7ff78c0d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 --2- 172.21.15.99:0/196646708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff78c0cfcd0 0x7ff78c0c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 >> 172.21.15.99:0/196646708 conn(0x7ff78c06f6e0 msgr2=0x7ff78c06fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.767+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 shutdown_connections 2024-03-04T10:04:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.768+0000 7ff793897640 1 -- 172.21.15.99:0/196646708 wait complete. 2024-03-04T10:04:56.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:04:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.879+0000 7f070379e640 1 Processor -- start 2024-03-04T10:04:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.879+0000 7f070379e640 1 -- start start 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070379e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b2e60 0x7f06fc0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070379e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f06fc0b5470 0x7f06fc0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070379e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06fc15e1e0 con 0x7f06fc15de00 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070379e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06fc0b98c0 con 0x7f06fc0b5470 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070379e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0b32c0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070153a640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f06fc15de00 0x7f06fc0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56978/0 (socket says 172.21.15.99:56978) 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f070153a640 1 -- 172.21.15.99:0/3934425842 learned_addr learned my addr 172.21.15.99:0/3934425842 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f0701d3b640 1 --2- 172.21.15.99:0/3934425842 >> v2:172.21.15.162:3300/0 conn(0x7f06fc0b5470 0x7f06fc0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.880+0000 7f0700d39640 1 --2- 172.21.15.99:0/3934425842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b2e60 0x7f06fc0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f06f3fff640 1 -- 172.21.15.99:0/3934425842 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234577687 0 0) 0x7f06fc15e1e0 con 0x7f06fc15de00 2024-03-04T10:04:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f06f3fff640 1 -- 172.21.15.99:0/3934425842 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e8003520 con 0x7f06fc15de00 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f06f3fff640 1 -- 172.21.15.99:0/3934425842 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917857073 0 0) 0x7f06e8003520 con 0x7f06fc15de00 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f0700d39640 1 -- 172.21.15.99:0/3934425842 >> v2:172.21.15.162:3300/0 conn(0x7f06fc0b5470 msgr2=0x7f06fc0b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f0700d39640 1 --2- 172.21.15.99:0/3934425842 >> v2:172.21.15.162:3300/0 conn(0x7f06fc0b5470 0x7f06fc0b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f0700d39640 1 -- 172.21.15.99:0/3934425842 >> v1:172.21.15.99:6789/0 conn(0x7f06fc15de00 legacy=0x7f06fc0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f0700d39640 1 -- 172.21.15.99:0/3934425842 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc0d1c10 con 0x7f06fc0b2e60 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.881+0000 7f0701d3b640 1 --2- 172.21.15.99:0/3934425842 >> v2:172.21.15.162:3300/0 conn(0x7f06fc0b5470 0x7f06fc0b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f0700d39640 1 --2- 172.21.15.99:0/3934425842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b2e60 0x7f06fc0b9380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0097a0 tx=0x7f06ec02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8052eaf7863971c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f06f3fff640 1 -- 172.21.15.99:0/3934425842 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec037070 con 0x7f06fc0b2e60 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f06f3fff640 1 -- 172.21.15.99:0/3934425842 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec030510 con 0x7f06fc0b2e60 2024-03-04T10:04:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f06f3fff640 1 -- 172.21.15.99:0/3934425842 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f80019f0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 -- 172.21.15.99:0/3934425842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b2e60 msgr2=0x7f06fc0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 --2- 172.21.15.99:0/3934425842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b2e60 0x7f06fc0b9380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0097a0 tx=0x7f06ec02b790 comp rx=0 tx=0).stop 2024-03-04T10:04:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 -- 172.21.15.99:0/3934425842 shutdown_connections 2024-03-04T10:04:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 --2- 172.21.15.99:0/3934425842 >> v2:172.21.15.162:3300/0 conn(0x7f06fc0b5470 0x7f06fc0b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 --2- 172.21.15.99:0/3934425842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b2e60 0x7f06fc0b9380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 -- 172.21.15.99:0/3934425842 >> 172.21.15.99:0/3934425842 conn(0x7f06fc0d0470 msgr2=0x7f06fc0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 -- 172.21.15.99:0/3934425842 shutdown_connections 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.882+0000 7f070379e640 1 -- 172.21.15.99:0/3934425842 wait complete. 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 Processor -- start 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 -- start start 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 0x7f06fc0abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b5470 0x7f06fc15af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06fc15de00 0x7f06fc0ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0d21d0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0d2820 con 0x7f06fc15de00 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070379e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0d24b0 con 0x7f06fc0b5470 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070153a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 0x7f06fc0abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070153a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 0x7f06fc0abd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50118/0 (socket says 172.21.15.99:50118) 2024-03-04T10:04:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.884+0000 7f070153a640 1 -- 172.21.15.99:0/1471490919 learned_addr learned my addr 172.21.15.99:0/1471490919 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f0700d39640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b5470 0x7f06fc15af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f0701d3b640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06fc15de00 0x7f06fc0ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070153a640 1 -- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b5470 msgr2=0x7f06fc15af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070153a640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b5470 0x7f06fc15af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070153a640 1 -- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06fc15de00 msgr2=0x7f06fc0ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070153a640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06fc15de00 0x7f06fc0ac320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070153a640 1 -- 172.21.15.99:0/1471490919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc0ac860 con 0x7f06fc0b2e60 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f0700d39640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b5470 0x7f06fc15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070153a640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 0x7f06fc0abd20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f06f8001870 tx=0x7f06f800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8012400 con 0x7f06fc0b2e60 2024-03-04T10:04:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f80125c0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc0acb00 con 0x7f06fc0b2e60 2024-03-04T10:04:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f80129b0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.885+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06fc0cbfc0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.886+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06fc074f60 con 0x7f06fc0b2e60 2024-03-04T10:04:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.887+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f06f8012b50 con 0x7f06fc0b2e60 2024-03-04T10:04:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.887+0000 7f06f1ffb640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e006b4d0 0x7f06e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.887+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f06f808abc0 con 0x7f06fc0b2e60 2024-03-04T10:04:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.887+0000 7f0700d39640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e006b4d0 0x7f06e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.888+0000 7f0700d39640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e006b4d0 0x7f06e006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06ec034000 tx=0x7f06ec02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:56.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:56.891+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f06f80182b0 con 0x7f06fc0b2e60 2024-03-04T10:04:57.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.100+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f06fc0b6c80 con 0x7f06fc0b2e60 2024-03-04T10:04:57.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.100+0000 7f06f1ffb640 1 -- 172.21.15.99:0/1471490919 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7f06f80598a0 con 0x7f06fc0b2e60 2024-03-04T10:04:57.101 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:04:57.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e006b4d0 msgr2=0x7f06e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:57.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e006b4d0 0x7f06e006d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06ec034000 tx=0x7f06ec02f000 comp rx=0 tx=0).stop 2024-03-04T10:04:57.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 msgr2=0x7f06fc0abd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:57.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 0x7f06fc0abd20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f06f8001870 tx=0x7f06f800a430 comp rx=0 tx=0).stop 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 shutdown_connections 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f06e006b4d0 0x7f06e006d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06fc15de00 0x7f06fc0ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f06fc0b5470 0x7f06fc15af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 --2- 172.21.15.99:0/1471490919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f06fc0b2e60 0x7f06fc0abd20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 >> 172.21.15.99:0/1471490919 conn(0x7f06fc0d0470 msgr2=0x7f06fc0b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 shutdown_connections 2024-03-04T10:04:57.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:57.102+0000 7f070379e640 1 -- 172.21.15.99:0/1471490919 wait complete. 2024-03-04T10:04:57.113 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:04:57.113 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:04:57.114 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:04:57.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.114 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:04:57.130 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42688sec preferred_lft 42688sec 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:04:57.131 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:04:57.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.133 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:04:57.195 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:04:57.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.197 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:04:57.253 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:04:57.253 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:04:57.253 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:04:57.253 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:04:57.253 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:04:57.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.254 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:04:57.254 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:04:57.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:04:57.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:04:57.254 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:04:57.254 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:04:57.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.349 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:04:57.364 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:04:57.364 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:04:57.364 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:04:57.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:04:57.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:57.431 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:57.486 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:04:57.486 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:04:57.486 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:57.540 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:04:57.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.540 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:57.625 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:57.626 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:57.633 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:04:57.633 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:04:57.634 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:04:57.634 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:04:57.634 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:04:57.634 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:04:57.634 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:04:57.634 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:04:57.635 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:04:57.635 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:04:57.636 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:04:57.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.636 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:57.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.669 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:04:57.731 INFO:teuthology.orchestra.run.smithi099.stdout:client5179 2024-03-04T10:04:57.731 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:57.731 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:57.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.786 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-03-04T10:04:57.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.848 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-03-04T10:04:57.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:57.905 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-03-04T10:04:57.960 DEBUG:teuthology.orchestra.run.smithi099:> stdin-killer -- python3 -c ' 2024-03-04T10:04:57.960 DEBUG:teuthology.orchestra.run.smithi099:> import os 2024-03-04T10:04:57.960 DEBUG:teuthology.orchestra.run.smithi099:> import stat 2024-03-04T10:04:57.960 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:04:57.960 DEBUG:teuthology.orchestra.run.smithi099:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-03-04T10:04:57.960 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:04:58.064 INFO:teuthology.orchestra.run.smithi099.stdout:1099511627776 2024-03-04T10:04:58.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T10:04:58.073 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:58.073 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:58.129 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:04:58.129 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:04:58.185 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:04:58.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.324+0000 7fc193524640 1 Processor -- start 2024-03-04T10:04:58.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.324+0000 7fc193524640 1 -- start start 2024-03-04T10:04:58.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc193524640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc18c06d3b0 0x7fc18c0ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc193524640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc18c0cc500 0x7fc18c0cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc193524640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18c0d0140 con 0x7fc18c0cfd60 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc193524640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc18c0cf440 con 0x7fc18c0cc500 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc193524640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc18c15cbb0 con 0x7fc18c06d3b0 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc191ac1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc18c0cc500 0x7fc18c0cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc191ac1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc18c0cc500 0x7fc18c0cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50132/0 (socket says 172.21.15.99:50132) 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.325+0000 7fc191ac1640 1 -- 172.21.15.99:0/281897674 learned_addr learned my addr 172.21.15.99:0/281897674 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc190abf640 1 --2- 172.21.15.99:0/281897674 >> v2:172.21.15.162:3300/0 conn(0x7fc18c06d3b0 0x7fc18c0ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc191ac1640 1 -- 172.21.15.99:0/281897674 >> v2:172.21.15.162:3300/0 conn(0x7fc18c06d3b0 msgr2=0x7fc18c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc191ac1640 1 --2- 172.21.15.99:0/281897674 >> v2:172.21.15.162:3300/0 conn(0x7fc18c06d3b0 0x7fc18c0ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc191ac1640 1 -- 172.21.15.99:0/281897674 >> v1:172.21.15.162:6789/0 conn(0x7fc18c0cfd60 legacy=0x7fc18c0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc191ac1640 1 -- 172.21.15.99:0/281897674 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c15c280 con 0x7fc18c0cc500 2024-03-04T10:04:58.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc190abf640 1 --2- 172.21.15.99:0/281897674 >> v2:172.21.15.162:3300/0 conn(0x7fc18c06d3b0 0x7fc18c0ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc191ac1640 1 --2- 172.21.15.99:0/281897674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c0cc500 0x7fc18c0cef00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc18c15cd30 tx=0x7fc18402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5be76363fa2336 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.326+0000 7fc183fff640 1 -- 172.21.15.99:0/281897674 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184035020 con 0x7fc18c0cc500 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc183fff640 1 -- 172.21.15.99:0/281897674 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc184031870 con 0x7fc18c0cc500 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc183fff640 1 -- 172.21.15.99:0/281897674 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184031c40 con 0x7fc18c0cc500 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 -- 172.21.15.99:0/281897674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c0cc500 msgr2=0x7fc18c0cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 --2- 172.21.15.99:0/281897674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c0cc500 0x7fc18c0cef00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc18c15cd30 tx=0x7fc18402b910 comp rx=0 tx=0).stop 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 -- 172.21.15.99:0/281897674 shutdown_connections 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 --2- 172.21.15.99:0/281897674 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c0cc500 0x7fc18c0cef00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 --2- 172.21.15.99:0/281897674 >> v2:172.21.15.162:3300/0 conn(0x7fc18c06d3b0 0x7fc18c0ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 -- 172.21.15.99:0/281897674 >> 172.21.15.99:0/281897674 conn(0x7fc18c06acd0 msgr2=0x7fc18c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 -- 172.21.15.99:0/281897674 shutdown_connections 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.327+0000 7fc193524640 1 -- 172.21.15.99:0/281897674 wait complete. 2024-03-04T10:04:58.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.328+0000 7fc193524640 1 Processor -- start 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.328+0000 7fc193524640 1 -- start start 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc193524640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 0x7fc18c154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc193524640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c0cc500 0x7fc18c0aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc193524640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc18c0cfd60 0x7fc18c0ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc193524640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c15bf30 con 0x7fc18c06d3b0 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc193524640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c15c840 con 0x7fc18c0cc500 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc193524640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18c15f520 con 0x7fc18c0cfd60 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 0x7fc18c154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 0x7fc18c154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50142/0 (socket says 172.21.15.99:50142) 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 -- 172.21.15.99:0/2074321571 learned_addr learned my addr 172.21.15.99:0/2074321571 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc190abf640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c0cc500 0x7fc18c0aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc191ac1640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc18c0cfd60 0x7fc18c0ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 -- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc18c0cfd60 msgr2=0x7fc18c0ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc18c0cfd60 0x7fc18c0ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 -- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c0cc500 msgr2=0x7fc18c0aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c0cc500 0x7fc18c0aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.329+0000 7fc1912c0640 1 -- 172.21.15.99:0/2074321571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc18c158cf0 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.330+0000 7fc1912c0640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 0x7fc18c154150 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc18800ba50 tx=0x7fc18800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.330+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188019070 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.330+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18800cc10 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.330+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18c158fd0 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.330+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188014440 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.330+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18c0a8970 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.331+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18c074f60 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.331+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc188007370 con 0x7fc18c06d3b0 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.332+0000 7fc181ffb640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17006b500 0x7fc17006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.332+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc188089f60 con 0x7fc18c06d3b0 2024-03-04T10:04:58.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.332+0000 7fc190abf640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17006b500 0x7fc17006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.332+0000 7fc190abf640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17006b500 0x7fc17006d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0aa6d0 tx=0x7fc17c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.336+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc188058600 con 0x7fc18c06d3b0 2024-03-04T10:04:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.508+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc18c0acfb0 con 0x7fc18c06d3b0 2024-03-04T10:04:58.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.509+0000 7fc181ffb640 1 -- 172.21.15.99:0/2074321571 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16906 (secure 0 0 0) 0x7fc18805c6a0 con 0x7fc18c06d3b0 2024-03-04T10:04:58.511 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:04:58.511 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":38,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:04:53.187527+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17006b500 msgr2=0x7fc17006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17006b500 0x7fc17006d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0aa6d0 tx=0x7fc17c008040 comp rx=0 tx=0).stop 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 msgr2=0x7fc18c154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 0x7fc18c154150 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc18800ba50 tx=0x7fc18800bf10 comp rx=0 tx=0).stop 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 shutdown_connections 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17006b500 0x7fc17006d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc18c0cfd60 0x7fc18c0ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c0cc500 0x7fc18c0aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 --2- 172.21.15.99:0/2074321571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc18c06d3b0 0x7fc18c154150 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 >> 172.21.15.99:0/2074321571 conn(0x7fc18c06acd0 msgr2=0x7fc18c159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:58.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 shutdown_connections 2024-03-04T10:04:58.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.511+0000 7fc193524640 1 -- 172.21.15.99:0/2074321571 wait complete. 2024-03-04T10:04:58.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:04:49.926068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:04:50.486347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:04:58.524 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:04:58.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.625+0000 7fe52ef59640 1 Processor -- start 2024-03-04T10:04:58.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.625+0000 7fe52ef59640 1 -- start start 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52ef59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280cdd70 0x7fe5280cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52ef59640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5280ccfe0 0x7fe5280cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52ef59640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52806ae00 con 0x7fe5280cc900 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52ef59640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5280ccde0 con 0x7fe5280ccfe0 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52ef59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280d0480 con 0x7fe5280cdd70 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52d4f6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5280ccfe0 0x7fe5280cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52d4f6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5280ccfe0 0x7fe5280cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50158/0 (socket says 172.21.15.99:50158) 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52d4f6640 1 -- 172.21.15.99:0/4106509936 learned_addr learned my addr 172.21.15.99:0/4106509936 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.626+0000 7fe52ccf5640 1 --2- 172.21.15.99:0/4106509936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280cdd70 0x7fe5280cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe52d4f6640 1 -- 172.21.15.99:0/4106509936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280cdd70 msgr2=0x7fe5280cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe52d4f6640 1 --2- 172.21.15.99:0/4106509936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280cdd70 0x7fe5280cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe52d4f6640 1 -- 172.21.15.99:0/4106509936 >> v1:172.21.15.162:6789/0 conn(0x7fe5280cc900 legacy=0x7fe52815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe52d4f6640 1 -- 172.21.15.99:0/4106509936 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52806b0e0 con 0x7fe5280ccfe0 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe52d4f6640 1 --2- 172.21.15.99:0/4106509936 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280ccfe0 0x7fe5280cff40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe520009e50 tx=0x7fe52002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf79e066d712f815 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe5277fe640 1 -- 172.21.15.99:0/4106509936 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52002ca70 con 0x7fe5280ccfe0 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe5277fe640 1 -- 172.21.15.99:0/4106509936 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52002cc10 con 0x7fe5280ccfe0 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.627+0000 7fe5277fe640 1 -- 172.21.15.99:0/4106509936 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5200343f0 con 0x7fe5280ccfe0 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 -- 172.21.15.99:0/4106509936 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280ccfe0 msgr2=0x7fe5280cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 --2- 172.21.15.99:0/4106509936 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280ccfe0 0x7fe5280cff40 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fe520009e50 tx=0x7fe52002bed0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 -- 172.21.15.99:0/4106509936 shutdown_connections 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 --2- 172.21.15.99:0/4106509936 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280ccfe0 0x7fe5280cff40 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 --2- 172.21.15.99:0/4106509936 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280cdd70 0x7fe5280cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 -- 172.21.15.99:0/4106509936 >> 172.21.15.99:0/4106509936 conn(0x7fe5280b22e0 msgr2=0x7fe5280b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:58.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 -- 172.21.15.99:0/4106509936 shutdown_connections 2024-03-04T10:04:58.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.628+0000 7fe52ef59640 1 -- 172.21.15.99:0/4106509936 wait complete. 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.629+0000 7fe52ef59640 1 Processor -- start 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.629+0000 7fe52ef59640 1 -- start start 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ef59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280cc900 0x7fe5280b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ef59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280ccfe0 0x7fe5280b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ef59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 0x7fe5281588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ef59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b2e90 con 0x7fe5280cdd70 2024-03-04T10:04:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ef59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52806b6a0 con 0x7fe5280cc900 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ef59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b3010 con 0x7fe5280ccfe0 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 0x7fe5281588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 0x7fe5281588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50170/0 (socket says 172.21.15.99:50170) 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 -- 172.21.15.99:0/2193803400 learned_addr learned my addr 172.21.15.99:0/2193803400 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ccf5640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280cc900 0x7fe5280b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe527fff640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280ccfe0 0x7fe5280b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 -- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280ccfe0 msgr2=0x7fe5280b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280ccfe0 0x7fe5280b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 -- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280cc900 msgr2=0x7fe5280b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280cc900 0x7fe5280b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52d4f6640 1 -- 172.21.15.99:0/2193803400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5281f5860 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe527fff640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280ccfe0 0x7fe5280b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.630+0000 7fe52ccf5640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280cc900 0x7fe5280b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.631+0000 7fe52d4f6640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 0x7fe5281588a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe520036040 tx=0x7fe520004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.631+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520043020 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.631+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52003e450 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.631+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5281f5b40 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.631+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520009510 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.631+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5281f6100 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.632+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528074f60 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.632+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe520046020 con 0x7fe5280cdd70 2024-03-04T10:04:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.633+0000 7fe5257fa640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.633+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe5200aca40 con 0x7fe5280cdd70 2024-03-04T10:04:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.633+0000 7fe52ccf5640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.633+0000 7fe52ccf5640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe518009b90 tx=0x7fe518008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.637+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe52004d030 con 0x7fe5280cdd70 2024-03-04T10:04:58.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.806+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5280b1790 con 0x7fe5280cdd70 2024-03-04T10:04:58.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.807+0000 7fe5257fa640 1 -- 172.21.15.99:0/2193803400 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5200ace20 con 0x7fe5280cdd70 2024-03-04T10:04:58.808 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:04:58.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 msgr2=0x7fe51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe518009b90 tx=0x7fe518008040 comp rx=0 tx=0).stop 2024-03-04T10:04:58.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 msgr2=0x7fe5281588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 0x7fe5281588a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe520036040 tx=0x7fe520004a40 comp rx=0 tx=0).stop 2024-03-04T10:04:58.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 shutdown_connections 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5280cdd70 0x7fe5281588a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280ccfe0 0x7fe5280b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 --2- 172.21.15.99:0/2193803400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280cc900 0x7fe5280b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 >> 172.21.15.99:0/2193803400 conn(0x7fe5280b22e0 msgr2=0x7fe52806f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 shutdown_connections 2024-03-04T10:04:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.809+0000 7fe52ef59640 1 -- 172.21.15.99:0/2193803400 wait complete. 2024-03-04T10:04:58.821 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:04:58.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.922+0000 7f2d099ab640 1 Processor -- start 2024-03-04T10:04:58.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.922+0000 7f2d099ab640 1 -- start start 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d099ab640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2d0406acd0 0x7f2d040ad530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d099ab640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d040cc7a0 con 0x7f2d040ada70 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d099ab640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d040cca80 con 0x7f2d0415bc50 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d099ab640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d040adee0 con 0x7f2d0406acd0 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d027fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2d0406acd0 0x7f2d040ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d037fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2d040ada70 0x7f2d040cf580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56984/0 (socket says 172.21.15.99:56984) 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.923+0000 7f2d037fe640 1 -- 172.21.15.99:0/3273994839 learned_addr learned my addr 172.21.15.99:0/3273994839 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.924+0000 7f2d027fc640 1 -- 172.21.15.99:0/3273994839 >> v1:172.21.15.162:6789/0 conn(0x7f2d0415bc50 legacy=0x7f2d040cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.924+0000 7f2d027fc640 1 -- 172.21.15.99:0/3273994839 >> v1:172.21.15.99:6789/0 conn(0x7f2d040ada70 legacy=0x7f2d040cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.924+0000 7f2d027fc640 1 -- 172.21.15.99:0/3273994839 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040ccd60 con 0x7f2d0406acd0 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.924+0000 7f2d01ffb640 1 -- 172.21.15.99:0/3273994839 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8505972 0 0) 0x7f2d040cc7a0 con 0x7f2d040ada70 2024-03-04T10:04:58.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.924+0000 7f2d027fc640 1 --2- 172.21.15.99:0/3273994839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d0406acd0 0x7f2d040ad530 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80097a0 tx=0x7f2cf802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229f32d779099f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d01ffb640 1 -- 172.21.15.99:0/3273994839 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8034070 con 0x7f2d0406acd0 2024-03-04T10:04:58.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d01ffb640 1 -- 172.21.15.99:0/3273994839 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8030610 con 0x7f2d0406acd0 2024-03-04T10:04:58.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d01ffb640 1 -- 172.21.15.99:0/3273994839 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf80309e0 con 0x7f2d0406acd0 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 -- 172.21.15.99:0/3273994839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d0406acd0 msgr2=0x7f2d040ad530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 --2- 172.21.15.99:0/3273994839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d0406acd0 0x7f2d040ad530 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80097a0 tx=0x7f2cf802b960 comp rx=0 tx=0).stop 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 -- 172.21.15.99:0/3273994839 shutdown_connections 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 --2- 172.21.15.99:0/3273994839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d0406acd0 0x7f2d040ad530 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 -- 172.21.15.99:0/3273994839 >> 172.21.15.99:0/3273994839 conn(0x7f2d040b2250 msgr2=0x7f2d040b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 -- 172.21.15.99:0/3273994839 shutdown_connections 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.925+0000 7f2d099ab640 1 -- 172.21.15.99:0/3273994839 wait complete. 2024-03-04T10:04:58.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.926+0000 7f2d099ab640 1 Processor -- start 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 -- start start 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d0406acd0 0x7f2d04155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 0x7f2d040c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d0415bc50 0x7f2d040c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040cdf50 con 0x7f2d040ada70 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040cd320 con 0x7f2d0415bc50 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d099ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040cc330 con 0x7f2d0406acd0 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d027fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 0x7f2d040c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d027fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 0x7f2d040c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50186/0 (socket says 172.21.15.99:50186) 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.927+0000 7f2d027fc640 1 -- 172.21.15.99:0/1977091508 learned_addr learned my addr 172.21.15.99:0/1977091508 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d037fe640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d0415bc50 0x7f2d040c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d02ffd640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d0406acd0 0x7f2d04155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d027fc640 1 -- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d0406acd0 msgr2=0x7f2d04155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d027fc640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d0406acd0 0x7f2d04155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d027fc640 1 -- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d0415bc50 msgr2=0x7f2d040c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d027fc640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d0415bc50 0x7f2d040c9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d027fc640 1 -- 172.21.15.99:0/1977091508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040ca170 con 0x7f2d040ada70 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d037fe640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d0415bc50 0x7f2d040c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d027fc640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 0x7f2d040c95d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80098c0 tx=0x7f2cf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8034040 con 0x7f2d040ada70 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf803acc0 con 0x7f2d040ada70 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d040ca450 con 0x7f2d040ada70 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8044630 con 0x7f2d040ada70 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.928+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0406e5c0 con 0x7f2d040ada70 2024-03-04T10:04:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.929+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0406cd70 con 0x7f2d040ada70 2024-03-04T10:04:58.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.929+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2cf804b060 con 0x7f2d040ada70 2024-03-04T10:04:58.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.930+0000 7f2cebfff640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cf006b4d0 0x7f2cf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:58.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.930+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2cf80ac240 con 0x7f2d040ada70 2024-03-04T10:04:58.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.930+0000 7f2d02ffd640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cf006b4d0 0x7f2cf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:58.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.930+0000 7f2d02ffd640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cf006b4d0 0x7f2cf006d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40012e0 tx=0x7f2cf4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:58.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:58.934+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2cf803f070 con 0x7f2d040ada70 2024-03-04T10:04:59.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.102+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2d04074f60 con 0x7f2d040ada70 2024-03-04T10:04:59.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.103+0000 7f2cebfff640 1 -- 172.21.15.99:0/1977091508 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2cf807afc0 con 0x7f2d040ada70 2024-03-04T10:04:59.104 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.104+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cf006b4d0 msgr2=0x7f2cf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.104+0000 7f2d099ab640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cf006b4d0 0x7f2cf006d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40012e0 tx=0x7f2cf4009040 comp rx=0 tx=0).stop 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.104+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 msgr2=0x7f2d040c95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.104+0000 7f2d099ab640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 0x7f2d040c95d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80098c0 tx=0x7f2cf80046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 shutdown_connections 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2cf006b4d0 0x7f2cf006d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d0415bc50 0x7f2d040c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2d040ada70 0x7f2d040c95d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 --2- 172.21.15.99:0/1977091508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2d0406acd0 0x7f2d04155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 >> 172.21.15.99:0/1977091508 conn(0x7f2d040b2250 msgr2=0x7f2d040b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:59.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 shutdown_connections 2024-03-04T10:04:59.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.105+0000 7f2d099ab640 1 -- 172.21.15.99:0/1977091508 wait complete. 2024-03-04T10:04:59.117 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:04:59.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.221+0000 7fcd886fc640 1 Processor -- start 2024-03-04T10:04:59.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.221+0000 7fcd886fc640 1 -- start start 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd886fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8006c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd886fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcd800d20c0 0x7fcd800ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd886fc640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd800cf740 con 0x7fcd800cec20 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd886fc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8006c5a0 con 0x7fcd800d20c0 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd886fc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8006d3e0 con 0x7fcd80161390 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd85c97640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcd800d20c0 0x7fcd800ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd86c99640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fcd800cec20 0x7fcd80161aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56994/0 (socket says 172.21.15.99:56994) 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.222+0000 7fcd86c99640 1 -- 172.21.15.99:0/2075854037 learned_addr learned my addr 172.21.15.99:0/2075854037 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd86498640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8006c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd85c97640 1 -- 172.21.15.99:0/2075854037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 msgr2=0x7fcd8006c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd85c97640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8006c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd85c97640 1 -- 172.21.15.99:0/2075854037 >> v1:172.21.15.99:6789/0 conn(0x7fcd800cec20 legacy=0x7fcd80161aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd85c97640 1 -- 172.21.15.99:0/2075854037 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8006c7d0 con 0x7fcd800d20c0 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd85496640 1 -- 172.21.15.99:0/2075854037 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640129087 0 0) 0x7fcd800cf740 con 0x7fcd800cec20 2024-03-04T10:04:59.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd86498640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8006c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.223+0000 7fcd85c97640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd800ce6e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0097a0 tx=0x7fcd7c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c500a12845eafc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd85496640 1 -- 172.21.15.99:0/2075854037 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c035020 con 0x7fcd800d20c0 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd85496640 1 -- 172.21.15.99:0/2075854037 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c031810 con 0x7fcd800d20c0 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd85496640 1 -- 172.21.15.99:0/2075854037 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c031be0 con 0x7fcd800d20c0 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 -- 172.21.15.99:0/2075854037 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 msgr2=0x7fcd800ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd800ce6e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0097a0 tx=0x7fcd7c02b8b0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 -- 172.21.15.99:0/2075854037 shutdown_connections 2024-03-04T10:04:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd800ce6e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 --2- 172.21.15.99:0/2075854037 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8006c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 -- 172.21.15.99:0/2075854037 >> 172.21.15.99:0/2075854037 conn(0x7fcd80066dc0 msgr2=0x7fcd800671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 -- 172.21.15.99:0/2075854037 shutdown_connections 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.224+0000 7fcd886fc640 1 -- 172.21.15.99:0/2075854037 wait complete. 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 Processor -- start 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 -- start start 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd800cec20 0x7fcd8015c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd8015ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8015e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8006d070 con 0x7fcd800d20c0 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8006cd90 con 0x7fcd800cec20 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd886fc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80161770 con 0x7fcd80161390 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd85c97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd8015ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd85c97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd8015ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50202/0 (socket says 172.21.15.99:50202) 2024-03-04T10:04:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.226+0000 7fcd85c97640 1 -- 172.21.15.99:0/4130928523 learned_addr learned my addr 172.21.15.99:0/4130928523 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd86c99640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd86498640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd800cec20 0x7fcd8015c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd85c97640 1 -- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 msgr2=0x7fcd8015e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd85c97640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8015e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd85c97640 1 -- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd800cec20 msgr2=0x7fcd8015c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd85c97640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd800cec20 0x7fcd8015c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd85c97640 1 -- 172.21.15.99:0/4130928523 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8015d7e0 con 0x7fcd800d20c0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd85c97640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd8015ff00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02bec0 tx=0x7fcd7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c041070 con 0x7fcd800d20c0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c03ad00 con 0x7fcd800d20c0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8015dac0 con 0x7fcd800d20c0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c035040 con 0x7fcd800d20c0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.227+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8015e360 con 0x7fcd800d20c0 2024-03-04T10:04:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.228+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8015dc90 con 0x7fcd800d20c0 2024-03-04T10:04:59.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.228+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcd7c04c060 con 0x7fcd800d20c0 2024-03-04T10:04:59.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.229+0000 7fcd76ffd640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd6406b4d0 0x7fcd6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.229+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcd7c0abe60 con 0x7fcd800d20c0 2024-03-04T10:04:59.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.229+0000 7fcd86498640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd6406b4d0 0x7fcd6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.230+0000 7fcd86498640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd6406b4d0 0x7fcd6406d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd800b6b40 tx=0x7fcd70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.233+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcd7c07a500 con 0x7fcd800d20c0 2024-03-04T10:04:59.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.403+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcd8015e560 con 0x7fcd800d20c0 2024-03-04T10:04:59.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.403+0000 7fcd76ffd640 1 -- 172.21.15.99:0/4130928523 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd8015e560 con 0x7fcd800d20c0 2024-03-04T10:04:59.405 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd6406b4d0 msgr2=0x7fcd6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd6406b4d0 0x7fcd6406d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcd800b6b40 tx=0x7fcd70008040 comp rx=0 tx=0).stop 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 msgr2=0x7fcd8015ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd8015ff00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c02bec0 tx=0x7fcd7c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 shutdown_connections 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcd6406b4d0 0x7fcd6406d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcd80161390 0x7fcd8015e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcd800d20c0 0x7fcd8015ff00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 --2- 172.21.15.99:0/4130928523 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcd800cec20 0x7fcd8015c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 >> 172.21.15.99:0/4130928523 conn(0x7fcd80066dc0 msgr2=0x7fcd800cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 shutdown_connections 2024-03-04T10:04:59.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.405+0000 7fcd886fc640 1 -- 172.21.15.99:0/4130928523 wait complete. 2024-03-04T10:04:59.417 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:04:59.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.517+0000 7f3385061640 1 Processor -- start 2024-03-04T10:04:59.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.517+0000 7f3385061640 1 -- start start 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3385061640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f338006f440 0x7f33800cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3385061640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f33800cd380 0x7f33800ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3385061640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338015c160 con 0x7f33800cf410 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3385061640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33800cd7f0 con 0x7f33800cd380 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3385061640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338015c440 con 0x7f338006f440 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3377fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f33800cd380 0x7f33800ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f337f59e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f33800cf410 0x7f338015f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56998/0 (socket says 172.21.15.99:56998) 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f337f59e640 1 -- 172.21.15.99:0/570429669 learned_addr learned my addr 172.21.15.99:0/570429669 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f337ed9d640 1 --2- 172.21.15.99:0/570429669 >> v2:172.21.15.162:3300/0 conn(0x7f338006f440 0x7f33800cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3377fff640 1 -- 172.21.15.99:0/570429669 >> v2:172.21.15.162:3300/0 conn(0x7f338006f440 msgr2=0x7f33800cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3377fff640 1 --2- 172.21.15.99:0/570429669 >> v2:172.21.15.162:3300/0 conn(0x7f338006f440 0x7f33800cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3377fff640 1 -- 172.21.15.99:0/570429669 >> v1:172.21.15.99:6789/0 conn(0x7f33800cf410 legacy=0x7f338015f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f3377fff640 1 -- 172.21.15.99:0/570429669 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338015c720 con 0x7f33800cd380 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.518+0000 7f337e59c640 1 -- 172.21.15.99:0/570429669 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289819656 0 0) 0x7f338015c160 con 0x7f33800cf410 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.519+0000 7f3377fff640 1 --2- 172.21.15.99:0/570429669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cd380 0x7f33800ceed0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f33800713e0 tx=0x7f336802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e84cded47c42e758 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.519+0000 7f337e59c640 1 -- 172.21.15.99:0/570429669 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368038070 con 0x7f33800cd380 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.519+0000 7f337e59c640 1 -- 172.21.15.99:0/570429669 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3368031710 con 0x7f33800cd380 2024-03-04T10:04:59.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.519+0000 7f337e59c640 1 -- 172.21.15.99:0/570429669 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368031ae0 con 0x7f33800cd380 2024-03-04T10:04:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.519+0000 7f3385061640 1 -- 172.21.15.99:0/570429669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cd380 msgr2=0x7f33800ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.519+0000 7f3385061640 1 --2- 172.21.15.99:0/570429669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cd380 0x7f33800ceed0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f33800713e0 tx=0x7f336802b7b0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.520+0000 7f3385061640 1 -- 172.21.15.99:0/570429669 shutdown_connections 2024-03-04T10:04:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.520+0000 7f3385061640 1 --2- 172.21.15.99:0/570429669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cd380 0x7f33800ceed0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.520+0000 7f3385061640 1 --2- 172.21.15.99:0/570429669 >> v2:172.21.15.162:3300/0 conn(0x7f338006f440 0x7f33800cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.520+0000 7f3385061640 1 -- 172.21.15.99:0/570429669 >> 172.21.15.99:0/570429669 conn(0x7f33800cfb40 msgr2=0x7f33800cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.520+0000 7f3385061640 1 -- 172.21.15.99:0/570429669 shutdown_connections 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.520+0000 7f3385061640 1 -- 172.21.15.99:0/570429669 wait complete. 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.521+0000 7f3385061640 1 Processor -- start 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.521+0000 7f3385061640 1 -- start start 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f338006f440 0x7f3380155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33800cd380 0x7f3380151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 0x7f338014ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33800cf9c0 con 0x7f33800cf410 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33800cf840 con 0x7f338006f440 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33800728f0 con 0x7f33800cd380 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 0x7f338014ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 0x7f338014ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50222/0 (socket says 172.21.15.99:50222) 2024-03-04T10:04:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 -- 172.21.15.99:0/2213257838 learned_addr learned my addr 172.21.15.99:0/2213257838 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3377fff640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33800cd380 0x7f3380151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337ed9d640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f338006f440 0x7f3380155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 -- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33800cd380 msgr2=0x7f3380151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33800cd380 0x7f3380151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 -- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f338006f440 msgr2=0x7f3380155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f338006f440 0x7f3380155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 -- 172.21.15.99:0/2213257838 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3380157d70 con 0x7f33800cf410 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3377fff640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33800cd380 0x7f3380151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337ed9d640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f338006f440 0x7f3380155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f337f59e640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 0x7f338014ec30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f337800c730 tx=0x7f337800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3378018070 con 0x7f33800cf410 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337800ce40 con 0x7f33800cf410 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337800a860 con 0x7f33800cf410 2024-03-04T10:04:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3380157a90 con 0x7f33800cf410 2024-03-04T10:04:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.522+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33801552a0 con 0x7f33800cf410 2024-03-04T10:04:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.523+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3380074f60 con 0x7f33800cf410 2024-03-04T10:04:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.524+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f337800aa00 con 0x7f33800cf410 2024-03-04T10:04:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.524+0000 7f33777fe640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f336006b4d0 0x7f336006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.524+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3378089900 con 0x7f33800cf410 2024-03-04T10:04:59.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.524+0000 7f337ed9d640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f336006b4d0 0x7f336006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.525+0000 7f337ed9d640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f336006b4d0 0x7f336006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3370006fd0 tx=0x7f3370008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.528+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3378058680 con 0x7f33800cf410 2024-03-04T10:04:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.709+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f33800aed10 con 0x7f33800cf410 2024-03-04T10:04:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.710+0000 7f33777fe640 1 -- 172.21.15.99:0/2213257838 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3378057f80 con 0x7f33800cf410 2024-03-04T10:04:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:04:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f336006b4d0 msgr2=0x7f336006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f336006b4d0 0x7f336006d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3370006fd0 tx=0x7f3370008040 comp rx=0 tx=0).stop 2024-03-04T10:04:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 msgr2=0x7f338014ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 0x7f338014ec30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f337800c730 tx=0x7f337800cbf0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 shutdown_connections 2024-03-04T10:04:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f336006b4d0 0x7f336006d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33800cf410 0x7f338014ec30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33800cd380 0x7f3380151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 --2- 172.21.15.99:0/2213257838 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f338006f440 0x7f3380155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 >> 172.21.15.99:0/2213257838 conn(0x7f33800cfb40 msgr2=0x7f3380071800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:59.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 shutdown_connections 2024-03-04T10:04:59.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.712+0000 7f3385061640 1 -- 172.21.15.99:0/2213257838 wait complete. 2024-03-04T10:04:59.723 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:04:59.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:59.724 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:04:59.779 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:04:59.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:04:59.780 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:04:59.800 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:04:59.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.940+0000 7f0d759e1640 1 Processor -- start 2024-03-04T10:04:59.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.940+0000 7f0d759e1640 1 -- start start 2024-03-04T10:04:59.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d759e1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0d700b5b80 0x7f0d700b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d759e1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7015e1e0 con 0x7f0d700b2e60 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d759e1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d700d1c10 con 0x7f0d7015de00 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d759e1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d700b98c0 con 0x7f0d700b5b80 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0d700b5b80 0x7f0d700b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6e7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0d700b2e60 0x7f0d700b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57006/0 (socket says 172.21.15.99:57006) 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6e7fc640 1 -- 172.21.15.99:0/2503927689 learned_addr learned my addr 172.21.15.99:0/2503927689 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2503927689 >> v1:172.21.15.162:6789/0 conn(0x7f0d7015de00 legacy=0x7f0d700b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2503927689 >> v1:172.21.15.99:6789/0 conn(0x7f0d700b2e60 legacy=0x7f0d700b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2503927689 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700d1ef0 con 0x7f0d700b5b80 2024-03-04T10:04:59.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.941+0000 7f0d6dffb640 1 -- 172.21.15.99:0/2503927689 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554315739 0 0) 0x7f0d7015e1e0 con 0x7f0d700b2e60 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.942+0000 7f0d6f7fe640 1 --2- 172.21.15.99:0/2503927689 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d700b5b80 0x7f0d700b9380 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009dc0 tx=0x7f0d6802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa708899fa721d8a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.942+0000 7f0d6dffb640 1 -- 172.21.15.99:0/2503927689 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68039070 con 0x7f0d700b5b80 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.942+0000 7f0d6dffb640 1 -- 172.21.15.99:0/2503927689 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6802cc40 con 0x7f0d700b5b80 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.942+0000 7f0d6dffb640 1 -- 172.21.15.99:0/2503927689 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68034440 con 0x7f0d700b5b80 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 -- 172.21.15.99:0/2503927689 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d700b5b80 msgr2=0x7f0d700b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2503927689 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d700b5b80 0x7f0d700b9380 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009dc0 tx=0x7f0d6802be90 comp rx=0 tx=0).stop 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 -- 172.21.15.99:0/2503927689 shutdown_connections 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2503927689 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d700b5b80 0x7f0d700b9380 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 -- 172.21.15.99:0/2503927689 >> 172.21.15.99:0/2503927689 conn(0x7f0d700d0470 msgr2=0x7f0d700d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 -- 172.21.15.99:0/2503927689 shutdown_connections 2024-03-04T10:04:59.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.943+0000 7f0d759e1640 1 -- 172.21.15.99:0/2503927689 wait complete. 2024-03-04T10:04:59.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.944+0000 7f0d759e1640 1 Processor -- start 2024-03-04T10:04:59.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.944+0000 7f0d759e1640 1 -- start start 2024-03-04T10:04:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d700b2e60 0x7f0d7015fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d700b5b80 0x7f0d70160df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 0x7f0d701f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d700d2820 con 0x7f0d7015de00 2024-03-04T10:04:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d700d24b0 con 0x7f0d700b5b80 2024-03-04T10:04:59.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d700b72a0 con 0x7f0d700b2e60 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 0x7f0d701f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 0x7f0d701f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50236/0 (socket says 172.21.15.99:50236) 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2040250141 learned_addr learned my addr 172.21.15.99:0/2040250141 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6e7fc640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d700b5b80 0x7f0d70160df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6effd640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d700b2e60 0x7f0d7015fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d700b2e60 msgr2=0x7f0d7015fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d700b2e60 0x7f0d7015fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d700b5b80 msgr2=0x7f0d70160df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d700b5b80 0x7f0d70160df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 -- 172.21.15.99:0/2040250141 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d70160450 con 0x7f0d7015de00 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6effd640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d700b2e60 0x7f0d7015fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:04:59.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d6f7fe640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 0x7f0d701f6b50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0d68037040 tx=0x7f0d68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68044020 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d68034df0 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7015f290 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68039040 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.945+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7015f850 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.946+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d70074f60 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.947+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0d68047020 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.947+0000 7f0d57fff640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d5c06b4d0 0x7f0d5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.947+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d680acee0 con 0x7f0d7015de00 2024-03-04T10:04:59.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.947+0000 7f0d6effd640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d5c06b4d0 0x7f0d5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:04:59.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.948+0000 7f0d6effd640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d5c06b4d0 0x7f0d5c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d60003eb0 tx=0x7f0d60009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:04:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:04:59.951+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0d6804e030 con 0x7f0d7015de00 2024-03-04T10:05:00.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.157+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d70068050 con 0x7f0d7015de00 2024-03-04T10:05:00.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.158+0000 7f0d57fff640 1 -- 172.21.15.99:0/2040250141 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f0d6807bcf0 con 0x7f0d7015de00 2024-03-04T10:05:00.159 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:00.159 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:04:53.524677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/434536114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":434536114},{"type":"v1","addr":"172.21.15.99:6835","nonce":434536114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:00.159 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 15 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.159+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d5c06b4d0 msgr2=0x7f0d5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.159+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d5c06b4d0 0x7f0d5c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d60003eb0 tx=0x7f0d60009040 comp rx=0 tx=0).stop 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.159+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 msgr2=0x7f0d701f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.159+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 0x7f0d701f6b50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0d68037040 tx=0x7f0d68004a40 comp rx=0 tx=0).stop 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 shutdown_connections 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d5c06b4d0 0x7f0d5c06d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0d7015de00 0x7f0d701f6b50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d700b5b80 0x7f0d70160df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 --2- 172.21.15.99:0/2040250141 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0d700b2e60 0x7f0d7015fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 >> 172.21.15.99:0/2040250141 conn(0x7f0d700d0470 msgr2=0x7f0d700b8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:00.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 shutdown_connections 2024-03-04T10:05:00.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.160+0000 7f0d759e1640 1 -- 172.21.15.99:0/2040250141 wait complete. 2024-03-04T10:05:00.172 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-04T10:05:00.296 INFO:teuthology.orchestra.run.smithi099.stdout:{"message":"","return_code":0} 2024-03-04T10:05:00.296 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T10:05:00.297 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:05:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.396+0000 7f583b481640 1 Processor -- start 2024-03-04T10:05:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.396+0000 7f583b481640 1 -- start start 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f583b481640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f58340cfd60 0x7f58340cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f583b481640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 0x7f58340cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f583b481640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58340d0140 con 0x7f583406d3b0 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f583b481640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583415cbb0 con 0x7f58340cfd60 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f583b481640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583415c280 con 0x7f58340cf0d0 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f5838a1c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f583406d3b0 0x7f58340ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57022/0 (socket says 172.21.15.99:57022) 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f5838a1c640 1 -- 172.21.15.99:0/2306981033 learned_addr learned my addr 172.21.15.99:0/2306981033 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409782 0 0) 0x7f58340d0140 con 0x7f583406d3b0 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820003520 con 0x7f583406d3b0 2024-03-04T10:05:00.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f5839a1e640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 0x7f58340cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.397+0000 7f583921d640 1 --2- 172.21.15.99:0/2306981033 >> v2:172.21.15.162:3300/0 conn(0x7f58340cfd60 0x7f58340cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089676586 0 0) 0x7f5820003520 con 0x7f583406d3b0 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 msgr2=0x7f58340cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 0x7f58340cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 >> v2:172.21.15.162:3300/0 conn(0x7f58340cfd60 msgr2=0x7f58340cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 --2- 172.21.15.99:0/2306981033 >> v2:172.21.15.162:3300/0 conn(0x7f58340cfd60 0x7f58340cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583415c560 con 0x7f583406d3b0 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5824001670 con 0x7f583406d3b0 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 >> v1:172.21.15.99:6789/0 conn(0x7f583406d3b0 legacy=0x7f58340ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58200039b0 0x7f5820003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 0x7f5820004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5820004d40 0x7f58200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583415cbb0 con 0x7f5820004d40 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583415c280 con 0x7f58200039b0 2024-03-04T10:05:00.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582000bb50 con 0x7f58340cf0d0 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5839a1e640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5820004d40 0x7f58200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f583921d640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58200039b0 0x7f5820003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5838a1c640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 0x7f5820004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5839a1e640 1 -- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 msgr2=0x7f5820004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5839a1e640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340cf0d0 0x7f5820004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5839a1e640 1 -- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58200039b0 msgr2=0x7f5820003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5839a1e640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58200039b0 0x7f5820003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.398+0000 7f5839a1e640 1 -- 172.21.15.99:0/2306981033 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820003520 con 0x7f5820004d40 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f5839a1e640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5820004d40 0x7f58200051c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f582c00ba30 tx=0x7f5820005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81a8ae2ccc19c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c036020 con 0x7f5820004d40 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c032640 con 0x7f5820004d40 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f582bfff640 1 -- 172.21.15.99:0/2306981033 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c00b600 con 0x7f5820004d40 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f583b481640 1 -- 172.21.15.99:0/2306981033 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5820004d40 msgr2=0x7f58200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f583b481640 1 --2- 172.21.15.99:0/2306981033 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5820004d40 0x7f58200051c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f582c00ba30 tx=0x7f5820005830 comp rx=0 tx=0).stop 2024-03-04T10:05:00.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f5839a1e640 1 -- 172.21.15.99:0/2306981033 reap_dead start 2024-03-04T10:05:00.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f583b481640 1 -- 172.21.15.99:0/2306981033 shutdown_connections 2024-03-04T10:05:00.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.399+0000 7f583b481640 1 -- 172.21.15.99:0/2306981033 >> 172.21.15.99:0/2306981033 conn(0x7f583406acd0 msgr2=0x7f583406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:00.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.400+0000 7f583b481640 1 -- 172.21.15.99:0/2306981033 shutdown_connections 2024-03-04T10:05:00.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.400+0000 7f583b481640 1 -- 172.21.15.99:0/2306981033 wait complete. 2024-03-04T10:05:00.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 Processor -- start 2024-03-04T10:05:00.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 -- start start 2024-03-04T10:05:00.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5820004d40 0x7f58341457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58340b45f0 0x7f583415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 0x7f583415e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cc330 con 0x7f58341e4560 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583415bf30 con 0x7f58340b45f0 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.401+0000 7f583b481640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cc4b0 con 0x7f5820004d40 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 0x7f583415e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 0x7f583415e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50262/0 (socket says 172.21.15.99:50262) 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 -- 172.21.15.99:0/963271681 learned_addr learned my addr 172.21.15.99:0/963271681 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 -- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5820004d40 msgr2=0x7f58341457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5820004d40 0x7f58341457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 -- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58340b45f0 msgr2=0x7f583415acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58340b45f0 0x7f583415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 -- 172.21.15.99:0/963271681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583415b660 con 0x7f58341e4560 2024-03-04T10:05:00.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5839a1e640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 0x7f583415e0b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f582c035040 tx=0x7f582c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c041070 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c030950 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58340b3b10 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c036040 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.402+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58340b40d0 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.404+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834074f60 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.404+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f582c04c020 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.404+0000 7f5829ffb640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f581806b4d0 0x7f581806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.404+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f582c0ac320 con 0x7f58341e4560 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.405+0000 7f583921d640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f581806b4d0 0x7f581806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.405+0000 7f583921d640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f581806b4d0 0x7f581806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f58300070b0 tx=0x7f5830007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.409+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f582c07b010 con 0x7f58341e4560 2024-03-04T10:05:00.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.613+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f58340d0140 con 0x7f58341e4560 2024-03-04T10:05:00.615 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:05:00.614+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:05:00.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.615+0000 7f5829ffb640 1 -- 172.21.15.99:0/963271681 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f582c07a910 con 0x7f58341e4560 2024-03-04T10:05:00.616 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:05:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f581806b4d0 msgr2=0x7f581806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f581806b4d0 0x7f581806d990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f58300070b0 tx=0x7f5830007040 comp rx=0 tx=0).stop 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 msgr2=0x7f583415e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 0x7f583415e0b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f582c035040 tx=0x7f582c004730 comp rx=0 tx=0).stop 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 shutdown_connections 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f581806b4d0 0x7f581806d990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58341e4560 0x7f583415e0b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58340b45f0 0x7f583415acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 --2- 172.21.15.99:0/963271681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5820004d40 0x7f58341457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 >> 172.21.15.99:0/963271681 conn(0x7f583406acd0 msgr2=0x7f583406e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 shutdown_connections 2024-03-04T10:05:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.618+0000 7f583b481640 1 -- 172.21.15.99:0/963271681 wait complete. 2024-03-04T10:05:00.622 INFO:tasks.ceph.mds.c.smithi099.stderr: -4355> 2024-03-04T10:04:56.158+0000 7f4c833bd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:05:00.634 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-04T10:05:00.656 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:05:00.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 Processor -- start 2024-03-04T10:05:00.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 -- start start 2024-03-04T10:05:00.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cea030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cdbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x556157ce88c0 con 0x556157ce16e0 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x556157bb8b30 con 0x556157c59a20 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5f63af40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x556157ce8a40 con 0x556157c5aa10 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5d1df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5d1df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50290/0 (socket says 172.21.15.99:50290) 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.677+0000 7f3e5d1df640 1 -- 172.21.15.99:0/1779383711 learned_addr learned my addr 172.21.15.99:0/1779383711 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5c9de640 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e57fff640 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cdbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5d1df640 1 -- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 msgr2=0x556157cdbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5d1df640 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cdbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5d1df640 1 -- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 msgr2=0x556157cea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5d1df640 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5d1df640 1 -- 172.21.15.99:0/1779383711 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556157cd9dc0 con 0x556157ce16e0 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5c9de640 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5d1df640 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdc060 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3e58044850 tx=0x7f3e5804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863a7fc73c4e19c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e56ffd640 1 -- 172.21.15.99:0/1779383711 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58052860 con 0x556157ce16e0 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e56ffd640 1 -- 172.21.15.99:0/1779383711 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58062070 con 0x556157ce16e0 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e56ffd640 1 -- 172.21.15.99:0/1779383711 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e58052c20 con 0x556157ce16e0 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5f63af40 1 -- 172.21.15.99:0/1779383711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 msgr2=0x556157cdc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.678+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdc060 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3e58044850 tx=0x7f3e5804c000 comp rx=0 tx=0).stop 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 -- 172.21.15.99:0/1779383711 shutdown_connections 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdc060 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cdbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/1779383711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 -- 172.21.15.99:0/1779383711 >> 172.21.15.99:0/1779383711 conn(0x556157c123d0 msgr2=0x556157c418c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 -- 172.21.15.99:0/1779383711 shutdown_connections 2024-03-04T10:05:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 -- 172.21.15.99:0/1779383711 wait complete. 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 Processor -- start 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.679+0000 7f3e5f63af40 1 -- start start 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5f63af40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cdb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5f63af40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cd6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5f63af40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5f63af40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x556157ce9110 con 0x556157ce16e0 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5f63af40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x556157c42340 con 0x556157c59a20 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5f63af40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x556157ce9d50 con 0x556157c5aa10 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50294/0 (socket says 172.21.15.99:50294) 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 -- 172.21.15.99:0/3656511105 learned_addr learned my addr 172.21.15.99:0/3656511105 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5c9de640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cdb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e57fff640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cd6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 -- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 msgr2=0x556157cd6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556157c5aa10 0x556157cd6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 -- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 msgr2=0x556157cdb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556157c59a20 0x556157cdb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 -- 172.21.15.99:0/3656511105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556157c41e40 con 0x556157ce16e0 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e5d1df640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdbc60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3e58007000 tx=0x7f3e58003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.680+0000 7f3e55ffb640 1 -- 172.21.15.99:0/3656511105 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5805d620 con 0x556157ce16e0 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.681+0000 7f3e55ffb640 1 -- 172.21.15.99:0/3656511105 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58058040 con 0x556157ce16e0 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.681+0000 7f3e55ffb640 1 -- 172.21.15.99:0/3656511105 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5805da20 con 0x556157ce16e0 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.681+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556157cdfcd0 con 0x556157ce16e0 2024-03-04T10:05:00.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.681+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556157c41ad0 con 0x556157ce16e0 2024-03-04T10:05:00.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.682+0000 7f3e55ffb640 1 -- 172.21.15.99:0/3656511105 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3e58050070 con 0x556157ce16e0 2024-03-04T10:05:00.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.682+0000 7f3e55ffb640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e40069f00 0x7f3e4006c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.682+0000 7f3e55ffb640 1 -- 172.21.15.99:0/3656511105 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3e580cc5b0 con 0x556157ce16e0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.682+0000 7f3e5c9de640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e40069f00 0x7f3e4006c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.682+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x556157cdc1a0 0x556157cd56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.682+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) v9 -- 0x556157cd5c30 con 0x556157cdc1a0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.683+0000 7f3e57fff640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x556157cdc1a0 0x556157cd56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.683+0000 7f3e5c9de640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e40069f00 0x7f3e4006c3c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x556157ce71e0 tx=0x7f3e500037f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.683+0000 7f3e57fff640 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x556157cdc1a0 0x556157cd56f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e57fff640 1 -- 172.21.15.99:0/3656511105 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v39'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f3e480099b0 con 0x556157cdc1a0 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x556157cdc1a0 msgr2=0x556157cd56f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x556157cdc1a0 0x556157cd56f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e40069f00 msgr2=0x7f3e4006c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3e40069f00 0x7f3e4006c3c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x556157ce71e0 tx=0x7f3e500037f0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 msgr2=0x556157cdbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 --2- 172.21.15.99:0/3656511105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556157ce16e0 0x556157cdbc60 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3e58007000 tx=0x7f3e58003330 comp rx=0 tx=0).stop 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5d1df640 1 -- 172.21.15.99:0/3656511105 reap_dead start 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 shutdown_connections 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 >> 172.21.15.99:0/3656511105 conn(0x556157c123d0 msgr2=0x556157ce0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 shutdown_connections 2024-03-04T10:05:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.685+0000 7f3e5f63af40 1 -- 172.21.15.99:0/3656511105 wait complete. 2024-03-04T10:05:00.695 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:05:00.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.836+0000 7f22c1643640 1 Processor -- start 2024-03-04T10:05:00.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.836+0000 7f22c1643640 1 -- start start 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22c1643640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f22bc0b5700 0x7f22bc0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22c1643640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22bc0d34a0 0x7f22bc06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22c1643640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22bc0d2400 con 0x7f22bc0b9410 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22c1643640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc0d26e0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22c1643640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc0d29c0 con 0x7f22bc0b5700 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22bc0d34a0 0x7f22bc06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22bc0d34a0 0x7f22bc06c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50296/0 (socket says 172.21.15.99:50296) 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 -- 172.21.15.99:0/3493980568 learned_addr learned my addr 172.21.15.99:0/3493980568 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 -- 172.21.15.99:0/3493980568 >> v2:172.21.15.162:3300/0 conn(0x7f22bc0b5700 msgr2=0x7f22bc0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 --2- 172.21.15.99:0/3493980568 >> v2:172.21.15.162:3300/0 conn(0x7f22bc0b5700 0x7f22bc0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 -- 172.21.15.99:0/3493980568 >> v1:172.21.15.162:6789/0 conn(0x7f22bc0b9410 legacy=0x7f22bc06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.837+0000 7f22b3fff640 1 -- 172.21.15.99:0/3493980568 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc0d2ca0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.838+0000 7f22b3fff640 1 --2- 172.21.15.99:0/3493980568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc06c0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f22bc06dad0 tx=0x7f22a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd55b5115671bf15 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.838+0000 7f22ba7fc640 1 -- 172.21.15.99:0/3493980568 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a402e070 con 0x7f22bc0d34a0 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.838+0000 7f22ba7fc640 1 -- 172.21.15.99:0/3493980568 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a4037070 con 0x7f22bc0d34a0 2024-03-04T10:05:00.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.838+0000 7f22ba7fc640 1 -- 172.21.15.99:0/3493980568 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031a00 con 0x7f22bc0d34a0 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.838+0000 7f22c1643640 1 -- 172.21.15.99:0/3493980568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 msgr2=0x7f22bc06c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.838+0000 7f22c1643640 1 --2- 172.21.15.99:0/3493980568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc06c0e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f22bc06dad0 tx=0x7f22a402b880 comp rx=0 tx=0).stop 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.839+0000 7f22c1643640 1 -- 172.21.15.99:0/3493980568 shutdown_connections 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.839+0000 7f22c1643640 1 --2- 172.21.15.99:0/3493980568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc06c0e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.839+0000 7f22c1643640 1 --2- 172.21.15.99:0/3493980568 >> v2:172.21.15.162:3300/0 conn(0x7f22bc0b5700 0x7f22bc0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.839+0000 7f22c1643640 1 -- 172.21.15.99:0/3493980568 >> 172.21.15.99:0/3493980568 conn(0x7f22bc0d0490 msgr2=0x7f22bc0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.839+0000 7f22c1643640 1 -- 172.21.15.99:0/3493980568 shutdown_connections 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.839+0000 7f22c1643640 1 -- 172.21.15.99:0/3493980568 wait complete. 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 Processor -- start 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 -- start start 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22bc0b5700 0x7f22bc149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22bc0b9410 0x7f22bc1f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc06ceb0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc06cd30 con 0x7f22bc0b9410 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22c1643640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc0b5ae0 con 0x7f22bc0b5700 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.840+0000 7f22bb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc0cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50310/0 (socket says 172.21.15.99:50310) 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 -- 172.21.15.99:0/2825260764 learned_addr learned my addr 172.21.15.99:0/2825260764 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22b3fff640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22bc0b9410 0x7f22bc1f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22baffd640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22bc0b5700 0x7f22bc149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 -- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22bc0b5700 msgr2=0x7f22bc149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22bc0b5700 0x7f22bc149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 -- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22bc0b9410 msgr2=0x7f22bc1f7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22bc0b9410 0x7f22bc1f7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 -- 172.21.15.99:0/2825260764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc0ccce0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22bb7fe640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc0cc7a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c780 tx=0x7f22b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4017070 con 0x7f22bc0d34a0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b4003da0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc14a300 con 0x7f22bc0d34a0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b40052b0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.841+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc14a8c0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.842+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc074f60 con 0x7f22bc0d34a0 2024-03-04T10:05:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.843+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f22b4005450 con 0x7f22bc0d34a0 2024-03-04T10:05:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.843+0000 7f22b37fe640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f229c06b4d0 0x7f229c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.843+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f22b40899b0 con 0x7f22bc0d34a0 2024-03-04T10:05:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.843+0000 7f22baffd640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f229c06b4d0 0x7f229c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.844+0000 7f22baffd640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f229c06b4d0 0x7f229c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f22bc14e2f0 tx=0x7f22ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:00.847+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f22b4058660 con 0x7f22bc0d34a0 2024-03-04T10:05:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.054+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f22bc0b5070 con 0x7f22bc0d34a0 2024-03-04T10:05:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.616+0000 7f22b37fe640 1 -- 172.21.15.99:0/2825260764 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v17) v1 ==== 162+0+0 (secure 0 0 0) 0x7f22b4057f60 con 0x7f22bc0d34a0 2024-03-04T10:05:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.619+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f229c06b4d0 msgr2=0x7f229c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.619+0000 7f22c1643640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f229c06b4d0 0x7f229c06d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f22bc14e2f0 tx=0x7f22ac008040 comp rx=0 tx=0).stop 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.619+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 msgr2=0x7f22bc0cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.619+0000 7f22c1643640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc0cc7a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c780 tx=0x7f22b400cc40 comp rx=0 tx=0).stop 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 shutdown_connections 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f229c06b4d0 0x7f229c06d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22bc0d34a0 0x7f22bc0cc7a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22bc0b9410 0x7f22bc1f7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 --2- 172.21.15.99:0/2825260764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f22bc0b5700 0x7f22bc149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 >> 172.21.15.99:0/2825260764 conn(0x7f22bc0d0490 msgr2=0x7f22bc0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 shutdown_connections 2024-03-04T10:05:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.620+0000 7f22c1643640 1 -- 172.21.15.99:0/2825260764 wait complete. 2024-03-04T10:05:01.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:01.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.735+0000 7f2219cd1640 1 Processor -- start 2024-03-04T10:05:01.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.735+0000 7f2219cd1640 1 -- start start 2024-03-04T10:05:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2219cd1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2219cd1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f22140b9380 0x7f22140b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2219cd1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22140b70d0 0x7f22140b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2219cd1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22140d2820 con 0x7f22140b70d0 2024-03-04T10:05:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2219cd1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f221415e1e0 con 0x7f22140b9380 2024-03-04T10:05:01.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2219cd1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22140d1c10 con 0x7f221415de00 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2213fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22140b70d0 0x7f22140b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2213fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f22140b70d0 0x7f22140b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50318/0 (socket says 172.21.15.99:50318) 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2213fff640 1 -- 172.21.15.99:0/2635310970 learned_addr learned my addr 172.21.15.99:0/2635310970 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.736+0000 7f2212ffd640 1 --2- 172.21.15.99:0/2635310970 >> v2:172.21.15.162:3300/0 conn(0x7f22140b9380 0x7f22140b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2213fff640 1 -- 172.21.15.99:0/2635310970 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 msgr2=0x7f22140b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f22137fe640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2213fff640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2213fff640 1 -- 172.21.15.99:0/2635310970 >> v2:172.21.15.162:3300/0 conn(0x7f22140b9380 msgr2=0x7f22140b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2213fff640 1 --2- 172.21.15.99:0/2635310970 >> v2:172.21.15.162:3300/0 conn(0x7f22140b9380 0x7f22140b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2213fff640 1 -- 172.21.15.99:0/2635310970 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140d1ef0 con 0x7f22140b70d0 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2212ffd640 1 --2- 172.21.15.99:0/2635310970 >> v2:172.21.15.162:3300/0 conn(0x7f22140b9380 0x7f22140b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f22137fe640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:01.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f2213fff640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140b5470 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f22140d4a50 tx=0x7f220c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea0373e569761d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f22127fc640 1 -- 172.21.15.99:0/2635310970 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c02ca70 con 0x7f22140b70d0 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f22127fc640 1 -- 172.21.15.99:0/2635310970 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f220c02cc10 con 0x7f22140b70d0 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.737+0000 7f22127fc640 1 -- 172.21.15.99:0/2635310970 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220c0343f0 con 0x7f22140b70d0 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 -- 172.21.15.99:0/2635310970 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 msgr2=0x7f22140b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140b5470 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f22140d4a50 tx=0x7f220c02bed0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 -- 172.21.15.99:0/2635310970 shutdown_connections 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140b5470 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2635310970 >> v2:172.21.15.162:3300/0 conn(0x7f22140b9380 0x7f22140b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2635310970 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 -- 172.21.15.99:0/2635310970 >> 172.21.15.99:0/2635310970 conn(0x7f22140d0470 msgr2=0x7f22140d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 -- 172.21.15.99:0/2635310970 shutdown_connections 2024-03-04T10:05:01.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.738+0000 7f2219cd1640 1 -- 172.21.15.99:0/2635310970 wait complete. 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.739+0000 7f2219cd1640 1 Processor -- start 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.739+0000 7f2219cd1640 1 -- start start 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2219cd1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2219cd1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22140b9380 0x7f22140b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2219cd1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2219cd1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22140b97e0 con 0x7f22140b70d0 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2219cd1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22140d24b0 con 0x7f22140b9380 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2219cd1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22140b59b0 con 0x7f221415de00 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50326/0 (socket says 172.21.15.99:50326) 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 -- 172.21.15.99:0/2704319715 learned_addr learned my addr 172.21.15.99:0/2704319715 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2212ffd640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22140b9380 0x7f22140b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f2213fff640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 -- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 msgr2=0x7f22140b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 -- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22140b9380 msgr2=0x7f22140b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22140b9380 0x7f22140b2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.740+0000 7f22137fe640 1 -- 172.21.15.99:0/2704319715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22140cb600 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.741+0000 7f22137fe640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140cd0f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f220400c7f0 tx=0x7f220400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.741+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204015070 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.741+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2204007820 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.741+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22140cb8e0 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.741+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204007bd0 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.741+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22140cbea0 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.742+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2214074f60 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.742+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2204018070 con 0x7f22140b70d0 2024-03-04T10:05:01.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.743+0000 7f21fbfff640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f220006b4d0 0x7f220006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:01.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.743+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f220408a230 con 0x7f22140b70d0 2024-03-04T10:05:01.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.743+0000 7f2212ffd640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f220006b4d0 0x7f220006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:01.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.743+0000 7f2212ffd640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f220006b4d0 0x7f220006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f22140ccd90 tx=0x7f2208008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:01.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.747+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2204058f60 con 0x7f22140b70d0 2024-03-04T10:05:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.953+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f221415c6b0 con 0x7f22140b70d0 2024-03-04T10:05:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.953+0000 7f21fbfff640 1 -- 172.21.15.99:0/2704319715 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f2204058860 con 0x7f22140b70d0 2024-03-04T10:05:01.954 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:01.954 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:01.619857+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 18 2024-03-04T10:05:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f220006b4d0 msgr2=0x7f220006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f220006b4d0 0x7f220006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f22140ccd90 tx=0x7f2208008040 comp rx=0 tx=0).stop 2024-03-04T10:05:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 msgr2=0x7f22140cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140cd0f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f220400c7f0 tx=0x7f220400ccb0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 shutdown_connections 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f220006b4d0 0x7f220006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f221415de00 0x7f22140b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f22140b9380 0x7f22140b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 --2- 172.21.15.99:0/2704319715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f22140b70d0 0x7f22140cd0f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 >> 172.21.15.99:0/2704319715 conn(0x7f22140d0470 msgr2=0x7f22140ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 shutdown_connections 2024-03-04T10:05:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:01.955+0000 7f2219cd1640 1 -- 172.21.15.99:0/2704319715 wait complete. 2024-03-04T10:05:02.369 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:04:51.516792+0000', 'modified': '2024-03-04T10:05:01.619857+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5550': {'gid': 5550, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.99:6835/2085408316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 2085408316}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 2085408316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:05:02.370 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5550:up:replay 2024-03-04T10:05:03.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:03.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.483+0000 7fc011ff1640 1 Processor -- start 2024-03-04T10:05:03.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.483+0000 7fc011ff1640 1 -- start start 2024-03-04T10:05:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.483+0000 7fc011ff1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c15de00 0x7fc00c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.483+0000 7fc011ff1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc00c0b9380 0x7fc00c0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc011ff1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c15e1e0 con 0x7fc00c0b70d0 2024-03-04T10:05:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc011ff1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c0d2820 con 0x7fc00c0b9380 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc011ff1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0d1c10 con 0x7fc00c15de00 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00affd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc00c0b9380 0x7fc00c0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00bfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc00c0b70d0 0x7fc00c0b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51234/0 (socket says 172.21.15.99:51234) 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00bfff640 1 -- 172.21.15.99:0/870940498 learned_addr learned my addr 172.21.15.99:0/870940498 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00b7fe640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c15de00 0x7fc00c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00affd640 1 -- 172.21.15.99:0/870940498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c15de00 msgr2=0x7fc00c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00affd640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c15de00 0x7fc00c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00affd640 1 -- 172.21.15.99:0/870940498 >> v1:172.21.15.99:6789/0 conn(0x7fc00c0b70d0 legacy=0x7fc00c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00a7fc640 1 -- 172.21.15.99:0/870940498 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207132793 0 0) 0x7fc00c15e1e0 con 0x7fc00c0b70d0 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00affd640 1 -- 172.21.15.99:0/870940498 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0d1ef0 con 0x7fc00c0b9380 2024-03-04T10:05:03.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.484+0000 7fc00b7fe640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c15de00 0x7fc00c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.485+0000 7fc00affd640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c0b6b90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc0000097a0 tx=0x7fc00002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420aaf8f8eee7852 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.485+0000 7fc00a7fc640 1 -- 172.21.15.99:0/870940498 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000035020 con 0x7fc00c0b9380 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.485+0000 7fc00a7fc640 1 -- 172.21.15.99:0/870940498 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000031830 con 0x7fc00c0b9380 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.485+0000 7fc00a7fc640 1 -- 172.21.15.99:0/870940498 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000031c00 con 0x7fc00c0b9380 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.485+0000 7fc011ff1640 1 -- 172.21.15.99:0/870940498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 msgr2=0x7fc00c0b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.485+0000 7fc011ff1640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c0b6b90 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc0000097a0 tx=0x7fc00002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.486+0000 7fc011ff1640 1 -- 172.21.15.99:0/870940498 shutdown_connections 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.486+0000 7fc011ff1640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c0b6b90 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.486+0000 7fc011ff1640 1 --2- 172.21.15.99:0/870940498 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c15de00 0x7fc00c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.486+0000 7fc011ff1640 1 -- 172.21.15.99:0/870940498 >> 172.21.15.99:0/870940498 conn(0x7fc00c0d0470 msgr2=0x7fc00c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:03.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.486+0000 7fc011ff1640 1 -- 172.21.15.99:0/870940498 shutdown_connections 2024-03-04T10:05:03.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.486+0000 7fc011ff1640 1 -- 172.21.15.99:0/870940498 wait complete. 2024-03-04T10:05:03.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.487+0000 7fc011ff1640 1 Processor -- start 2024-03-04T10:05:03.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.487+0000 7fc011ff1640 1 -- start start 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c0b70d0 0x7fc00c0aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc00c15de00 0x7fc00c1f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0b97e0 con 0x7fc00c0b9380 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0d24b0 con 0x7fc00c15de00 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0b5b80 con 0x7fc00c0b70d0 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c1f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41744/0 (socket says 172.21.15.99:41744) 2024-03-04T10:05:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 -- 172.21.15.99:0/2202283538 learned_addr learned my addr 172.21.15.99:0/2202283538 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00b7fe640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c0b70d0 0x7fc00c0aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00bfff640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc00c15de00 0x7fc00c1f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 -- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c0b70d0 msgr2=0x7fc00c0aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c0b70d0 0x7fc00c0aa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 -- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc00c15de00 msgr2=0x7fc00c1f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc00c15de00 0x7fc00c1f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 -- 172.21.15.99:0/2202283538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c0c7aa0 con 0x7fc00c0b9380 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00bfff640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc00c15de00 0x7fc00c1f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc00affd640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c1f6870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc000000c00 tx=0x7fc0000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000035050 con 0x7fc00c0b9380 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000004230 con 0x7fc00c0b9380 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00c0c7d80 con 0x7fc00c0b9380 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.488+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00003b800 con 0x7fc00c0b9380 2024-03-04T10:05:03.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.489+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c0c8340 con 0x7fc00c0b9380 2024-03-04T10:05:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.490+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c074f60 con 0x7fc00c0b9380 2024-03-04T10:05:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.490+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc000038020 con 0x7fc00c0b9380 2024-03-04T10:05:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.491+0000 7fbff3fff640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff806b4d0 0x7fbff806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.491+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc0000ac210 con 0x7fc00c0b9380 2024-03-04T10:05:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.491+0000 7fc00b7fe640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff806b4d0 0x7fbff806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:03.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.491+0000 7fc00b7fe640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff806b4d0 0x7fbff806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc00c161460 tx=0x7fbffc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:03.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.495+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc000040070 con 0x7fc00c0b9380 2024-03-04T10:05:03.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.701+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc00c0cefb0 con 0x7fc00c0b9380 2024-03-04T10:05:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.702+0000 7fbff3fff640 1 -- 172.21.15.99:0/2202283538 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4945 (secure 0 0 0) 0x7fc00007aeb0 con 0x7fc00c0b9380 2024-03-04T10:05:03.704 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:03.704 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:02.623781+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 20 2024-03-04T10:05:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff806b4d0 msgr2=0x7fbff806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff806b4d0 0x7fbff806d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc00c161460 tx=0x7fbffc008040 comp rx=0 tx=0).stop 2024-03-04T10:05:03.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 msgr2=0x7fc00c1f6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c1f6870 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc000000c00 tx=0x7fc0000046c0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 shutdown_connections 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbff806b4d0 0x7fbff806d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc00c15de00 0x7fc00c1f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc00c0b9380 0x7fc00c1f6870 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 --2- 172.21.15.99:0/2202283538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc00c0b70d0 0x7fc00c0aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:03.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 >> 172.21.15.99:0/2202283538 conn(0x7fc00c0d0470 msgr2=0x7fc00c0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:03.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 shutdown_connections 2024-03-04T10:05:03.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:03.704+0000 7fc011ff1640 1 -- 172.21.15.99:0/2202283538 wait complete. 2024-03-04T10:05:04.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:04:51.516792+0000', 'modified': '2024-03-04T10:05:02.623781+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5550': {'gid': 5550, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.99:6835/2085408316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 2085408316}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 2085408316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:05:04.120 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5550:up:rejoin 2024-03-04T10:05:05.121 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:05.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 Processor -- start 2024-03-04T10:05:05.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 -- start start 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb8d415bbb0 0x7fb8d40d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb8d40d0e40 0x7fb8d40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d40d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d40d2880 con 0x7fb8d415bbb0 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d40cfb40 con 0x7fb8d40d0e40 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.233+0000 7fb8da374640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40cfe20 con 0x7fb8d40cc4c0 2024-03-04T10:05:05.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb8d415bbb0 0x7fb8d40d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb8d415bbb0 0x7fb8d40d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41748/0 (socket says 172.21.15.99:41748) 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 -- 172.21.15.99:0/3738985945 learned_addr learned my addr 172.21.15.99:0/3738985945 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 -- 172.21.15.99:0/3738985945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 msgr2=0x7fb8d40d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d37fe640 1 --2- 172.21.15.99:0/3738985945 >> v2:172.21.15.162:3300/0 conn(0x7fb8d40d0e40 0x7fb8d40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d8911640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d40d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d40d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 -- 172.21.15.99:0/3738985945 >> v2:172.21.15.162:3300/0 conn(0x7fb8d40d0e40 msgr2=0x7fb8d40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 --2- 172.21.15.99:0/3738985945 >> v2:172.21.15.162:3300/0 conn(0x7fb8d40d0e40 0x7fb8d40cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d3fff640 1 -- 172.21.15.99:0/3738985945 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d40d0100 con 0x7fb8d415bbb0 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d37fe640 1 --2- 172.21.15.99:0/3738985945 >> v2:172.21.15.162:3300/0 conn(0x7fb8d40d0e40 0x7fb8d40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.234+0000 7fb8d8911640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d40d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:05.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8d3fff640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40d0900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4070790 tx=0x7fb8c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4ef1972a757bae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8d2ffd640 1 -- 172.21.15.99:0/3738985945 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4035020 con 0x7fb8d415bbb0 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8d2ffd640 1 -- 172.21.15.99:0/3738985945 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c4031870 con 0x7fb8d415bbb0 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8d2ffd640 1 -- 172.21.15.99:0/3738985945 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4031c40 con 0x7fb8d415bbb0 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 -- 172.21.15.99:0/3738985945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d415bbb0 msgr2=0x7fb8d40d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40d0900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4070790 tx=0x7fb8c402b910 comp rx=0 tx=0).stop 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 -- 172.21.15.99:0/3738985945 shutdown_connections 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d40d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 --2- 172.21.15.99:0/3738985945 >> v2:172.21.15.162:3300/0 conn(0x7fb8d40d0e40 0x7fb8d40cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 --2- 172.21.15.99:0/3738985945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40d0900 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 -- 172.21.15.99:0/3738985945 >> 172.21.15.99:0/3738985945 conn(0x7fb8d40b3480 msgr2=0x7fb8d40b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.235+0000 7fb8da374640 1 -- 172.21.15.99:0/3738985945 shutdown_connections 2024-03-04T10:05:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.236+0000 7fb8da374640 1 -- 172.21.15.99:0/3738985945 wait complete. 2024-03-04T10:05:05.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.237+0000 7fb8da374640 1 Processor -- start 2024-03-04T10:05:05.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.237+0000 7fb8da374640 1 -- start start 2024-03-04T10:05:05.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.237+0000 7fb8da374640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d4159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 0x7fb8d40a91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d415bf90 con 0x7fb8d40d0e40 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d40d21e0 con 0x7fb8d415bbb0 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d41f4f20 con 0x7fb8d40cc4c0 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 0x7fb8d40a91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 0x7fb8d40a91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41764/0 (socket says 172.21.15.99:41764) 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 -- 172.21.15.99:0/307552183 learned_addr learned my addr 172.21.15.99:0/307552183 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d8911640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d3fff640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d4159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 -- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 msgr2=0x7fb8d4159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d4159560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 -- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d415bbb0 msgr2=0x7fb8d40b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40b13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 -- 172.21.15.99:0/307552183 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d41f2540 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d8911640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d37fe640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 0x7fb8d40a91d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800c820 tx=0x7fb8c800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8015070 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c8007a30 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d41f2260 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8007e00 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.238+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d40a89d0 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.239+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d4074f60 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.240+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb8c8018070 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.240+0000 7fb8d0ff9640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8c006b4d0 0x7fb8c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.240+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb8c808a220 con 0x7fb8d40d0e40 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.241+0000 7fb8d3fff640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8c006b4d0 0x7fb8c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.241+0000 7fb8d3fff640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8c006b4d0 0x7fb8c006d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4036000 tx=0x7fb8c4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.245+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb8c8058f50 con 0x7fb8d40d0e40 2024-03-04T10:05:05.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.451+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8d415a4e0 con 0x7fb8d40d0e40 2024-03-04T10:05:05.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.452+0000 7fb8d0ff9640 1 -- 172.21.15.99:0/307552183 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7fb8c8058850 con 0x7fb8d40d0e40 2024-03-04T10:05:05.453 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:05.453 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:05.454 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:05.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8c006b4d0 msgr2=0x7fb8c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8c006b4d0 0x7fb8c006d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4036000 tx=0x7fb8c4008030 comp rx=0 tx=0).stop 2024-03-04T10:05:05.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 msgr2=0x7fb8d40a91d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 0x7fb8d40a91d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800c820 tx=0x7fb8c800cce0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 shutdown_connections 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8c006b4d0 0x7fb8c006d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8d415bbb0 0x7fb8d40b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8d40d0e40 0x7fb8d40a91d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 --2- 172.21.15.99:0/307552183 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8d40cc4c0 0x7fb8d4159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 >> 172.21.15.99:0/307552183 conn(0x7fb8d40b3480 msgr2=0x7fb8d40b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 shutdown_connections 2024-03-04T10:05:05.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.454+0000 7fb8da374640 1 -- 172.21.15.99:0/307552183 wait complete. 2024-03-04T10:05:05.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:04:51.516792+0000', 'modified': '2024-03-04T10:05:03.628528+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5550': {'gid': 5550, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.99:6835/2085408316', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 2085408316}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 2085408316}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:05:05.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:05:05.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:05:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.969+0000 7f824fe81640 1 Processor -- start 2024-03-04T10:05:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.969+0000 7f824fe81640 1 -- start start 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824fe81640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8248154a80 0x7f824815e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824fe81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82480cf650 0x7f82480cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824fe81640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82480cdaa0 con 0x7f82481616c0 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824fe81640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82480cfff0 con 0x7f8248154a80 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824fe81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8248071df0 con 0x7f82480cf650 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824dc1d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8248154a80 0x7f824815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824e41e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f82481616c0 0x7f8248161b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51236/0 (socket says 172.21.15.99:51236) 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824e41e640 1 -- 172.21.15.99:0/387305497 learned_addr learned my addr 172.21.15.99:0/387305497 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824d41c640 1 --2- 172.21.15.99:0/387305497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82480cf650 0x7f82480cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824dc1d640 1 -- 172.21.15.99:0/387305497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82480cf650 msgr2=0x7f82480cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824dc1d640 1 --2- 172.21.15.99:0/387305497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82480cf650 0x7f82480cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824dc1d640 1 -- 172.21.15.99:0/387305497 >> v1:172.21.15.99:6789/0 conn(0x7f82481616c0 legacy=0x7f8248161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824dc1d640 1 -- 172.21.15.99:0/387305497 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480cd010 con 0x7f8248154a80 2024-03-04T10:05:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.970+0000 7f824cc1b640 1 -- 172.21.15.99:0/387305497 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176068472 0 0) 0x7f82480cdaa0 con 0x7f82481616c0 2024-03-04T10:05:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.971+0000 7f824dc1d640 1 --2- 172.21.15.99:0/387305497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f824815e910 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f82440097d0 tx=0x7f824402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df3d7b2efde59b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.971+0000 7f824cc1b640 1 -- 172.21.15.99:0/387305497 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244035020 con 0x7f8248154a80 2024-03-04T10:05:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.971+0000 7f824cc1b640 1 -- 172.21.15.99:0/387305497 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244031800 con 0x7f8248154a80 2024-03-04T10:05:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.971+0000 7f824cc1b640 1 -- 172.21.15.99:0/387305497 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244031bd0 con 0x7f8248154a80 2024-03-04T10:05:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 -- 172.21.15.99:0/387305497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 msgr2=0x7f824815e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 --2- 172.21.15.99:0/387305497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f824815e910 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f82440097d0 tx=0x7f824402b8a0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 -- 172.21.15.99:0/387305497 shutdown_connections 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 --2- 172.21.15.99:0/387305497 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82480cf650 0x7f82480cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 --2- 172.21.15.99:0/387305497 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f824815e910 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 -- 172.21.15.99:0/387305497 >> 172.21.15.99:0/387305497 conn(0x7f824806f5f0 msgr2=0x7f824806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 -- 172.21.15.99:0/387305497 shutdown_connections 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.972+0000 7f824fe81640 1 -- 172.21.15.99:0/387305497 wait complete. 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.973+0000 7f824fe81640 1 Processor -- start 2024-03-04T10:05:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.973+0000 7f824fe81640 1 -- start start 2024-03-04T10:05:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824fe81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82480cf650 0x7f8248149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824fe81640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f82480b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824fe81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82481616c0 0x7f824815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824fe81640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480ce340 con 0x7f8248154a80 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824fe81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480ce060 con 0x7f82480cf650 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824fe81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82480ce620 con 0x7f82481616c0 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f82480b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f82480b5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41786/0 (socket says 172.21.15.99:41786) 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 -- 172.21.15.99:0/580073921 learned_addr learned my addr 172.21.15.99:0/580073921 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824e41e640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82481616c0 0x7f824815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824dc1d640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82480cf650 0x7f8248149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 -- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82481616c0 msgr2=0x7f824815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82481616c0 0x7f824815f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 -- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82480cf650 msgr2=0x7f8248149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82480cf650 0x7f8248149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 -- 172.21.15.99:0/580073921 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82480b6200 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824e41e640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82481616c0 0x7f824815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824dc1d640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82480cf650 0x7f8248149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f824d41c640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f82480b5740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f823800d800 tx=0x7f823800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.974+0000 7f823e7fc640 1 -- 172.21.15.99:0/580073921 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823800b6c0 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.975+0000 7f823e7fc640 1 -- 172.21.15.99:0/580073921 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8238019070 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.975+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82480b64e0 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.975+0000 7f823e7fc640 1 -- 172.21.15.99:0/580073921 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823800ba80 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.975+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82480b6aa0 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.976+0000 7f823e7fc640 1 -- 172.21.15.99:0/580073921 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f823800bc20 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.976+0000 7f823e7fc640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.976+0000 7f823e7fc640 1 -- 172.21.15.99:0/580073921 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f823808a7b0 con 0x7f8248154a80 2024-03-04T10:05:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.977+0000 7f824dc1d640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.977+0000 7f824dc1d640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f822c06b500 0x7f822c06d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8244036000 tx=0x7f8244008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 Processor -- start 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 -- start start 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b4ea0 0x55e5b3987e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b3988380 0x55e5b39b8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b38f27c0 con 0x55e5b39b5280 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b389bb50 con 0x55e5b3988380 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5b38ee860 con 0x55e5b39b4ea0 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f824d41c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f824d41c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41796/0 (socket says 172.21.15.99:41796) 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f824d41c640 1 -- 172.21.15.99:0/3708207796 learned_addr learned my addr 172.21.15.99:0/3708207796 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f824e41e640 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b3988380 0x55e5b39b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.978+0000 7f824dc1d640 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b4ea0 0x55e5b3987e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824d41c640 1 -- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b4ea0 msgr2=0x55e5b3987e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824d41c640 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b4ea0 0x55e5b3987e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824d41c640 1 -- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b3988380 msgr2=0x55e5b39b8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824d41c640 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b3988380 0x55e5b39b8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824d41c640 1 -- 172.21.15.99:0/3708207796 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5b38af9a0 con 0x55e5b39b5280 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824dc1d640 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b4ea0 0x55e5b3987e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f824d41c640 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b7c30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8238088b90 tx=0x7f823801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a810f8cfc43cd410 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f822b7fe640 1 -- 172.21.15.99:0/3708207796 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238058de0 con 0x55e5b39b5280 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f822b7fe640 1 -- 172.21.15.99:0/3708207796 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823805de50 con 0x55e5b39b5280 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f822b7fe640 1 -- 172.21.15.99:0/3708207796 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823805cf30 con 0x55e5b39b5280 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f8252a5e740 1 -- 172.21.15.99:0/3708207796 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 msgr2=0x55e5b39b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.979+0000 7f8252a5e740 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b7c30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8238088b90 tx=0x7f823801c360 comp rx=0 tx=0).stop 2024-03-04T10:05:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 -- 172.21.15.99:0/3708207796 shutdown_connections 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b3988380 0x55e5b39b8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b7c30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 --2- 172.21.15.99:0/3708207796 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b4ea0 0x55e5b3987e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 -- 172.21.15.99:0/3708207796 >> 172.21.15.99:0/3708207796 conn(0x55e5b398a0b0 msgr2=0x55e5b39d2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 -- 172.21.15.99:0/3708207796 shutdown_connections 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 -- 172.21.15.99:0/3708207796 wait complete. 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 -- start start 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.980+0000 7f8252a5e740 1 Processor -- start 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39ada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b398c750 con 0x55e5b39b5280 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b38aef90 con 0x55e5b39b4ea0 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5b398ef90 con 0x55e5b39b6c30 2024-03-04T10:05:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f8252a5e740 10 client.0 _ll_register_callbacks cb 0x55e5b38d0d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41806/0 (socket says 172.21.15.99:41806) 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 -- 172.21.15.99:0/2948557092 learned_addr learned my addr 172.21.15.99:0/2948557092 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824e41e640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 -- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 msgr2=0x55e5b39ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39ada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 -- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b4ea0 msgr2=0x55e5b39d2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.981+0000 7f824d41c640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f824d41c640 1 -- 172.21.15.99:0/2948557092 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5b36072d0 con 0x55e5b39b5280 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f824d41c640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b66f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f823808aad0 tx=0x7f823808a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce836b4af20dbd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8228ff9640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8228ff9640 1 -- 172.21.15.99:0/2948557092 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8238074070 con 0x55e5b39b5280 2024-03-04T10:05:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8228ff9640 1 -- 172.21.15.99:0/2948557092 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8238055450 con 0x55e5b39b5280 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f823805de50 con 0x55e5b39b5280 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8228ff9640 1 -- 172.21.15.99:0/2948557092 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f823805cf30 con 0x55e5b39b5280 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 10 client.5601 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55e5b36d16b0 con 0x55e5b39b5280 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8228ff9640 1 -- 172.21.15.99:0/2948557092 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823805de50 con 0x55e5b39b5280 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8228ff9640 1 -- 172.21.15.99:0/2948557092 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8238013070 con 0x55e5b39b5280 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 10 client.5601 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 10 client.5601 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 20 client.5601 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39b9a70 0x55e5b39b9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 4 client.5601 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5b38ffd00 con 0x55e5b39b9a70 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.982+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39b9a70 0x55e5b39b9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39b9a70 0x55e5b39b9e50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b01069814adb800 server_cookie=8d6039699b33f18e in_seq=0 out_seq=0 2024-03-04T10:05:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8228ff9640 10 client.5601 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8228ff9640 1 -- 172.21.15.99:0/2948557092 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e5b38ffd00 con 0x55e5b39b9a70 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8228ff9640 10 client.5601 handle_command_reply: tid=0 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 client.5601 shutdown 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 msgr2=0x55e5b39b66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b66f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f823808aad0 tx=0x7f823808a040 comp rx=0 tx=0).stop 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 shutdown_connections 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b5280 0x55e5b39b66f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39b9a70 0x55e5b39b9e50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2948557092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8228ff9640 0 client.5601 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.983+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 >> 172.21.15.99:0/2948557092 conn(0x55e5b3988380 msgr2=0x55e5b39b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.984+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 shutdown_connections 2024-03-04T10:05:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.984+0000 7f8252a5e740 1 -- 172.21.15.99:0/2948557092 wait complete. 2024-03-04T10:05:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.984+0000 7f8252a5e740 20 client.5601 trim_cache size 0 max 16384 2024-03-04T10:05:06.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.999+0000 7f8252a5e740 1 Processor -- start 2024-03-04T10:05:06.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:05.999+0000 7f8252a5e740 1 -- start start 2024-03-04T10:05:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b5280 0x55e5b39b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b6c30 0x55e5b39b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f8252a5e740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82481ff720 con 0x55e5b39b4ea0 2024-03-04T10:05:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82480c5e60 con 0x55e5b39b6c30 2024-03-04T10:05:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82480c6460 con 0x55e5b39b5280 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41818/0 (socket says 172.21.15.99:41818) 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 -- 172.21.15.99:0/2672101595 learned_addr learned my addr 172.21.15.99:0/2672101595 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 -- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b5280 msgr2=0x55e5b39b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b5280 0x55e5b39b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824e41e640 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b6c30 0x55e5b39b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 -- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b6c30 msgr2=0x55e5b39b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b6c30 0x55e5b39b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 -- 172.21.15.99:0/2672101595 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5b39aff10 con 0x55e5b39b4ea0 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824e41e640 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b6c30 0x55e5b39b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f82440097d0 tx=0x7f8244030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefdf6cf0ab80586 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f822affd640 1 -- 172.21.15.99:0/2672101595 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244058070 con 0x55e5b39b4ea0 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f822affd640 1 -- 172.21.15.99:0/2672101595 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244053430 con 0x55e5b39b4ea0 2024-03-04T10:05:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.000+0000 7f822affd640 1 -- 172.21.15.99:0/2672101595 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244035040 con 0x55e5b39b4ea0 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 -- 172.21.15.99:0/2672101595 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 msgr2=0x55e5b39d2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f82440097d0 tx=0x7f8244030000 comp rx=0 tx=0).stop 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 -- 172.21.15.99:0/2672101595 shutdown_connections 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b6c30 0x55e5b39b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b5280 0x55e5b39b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2672101595 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b39d2f10 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 -- 172.21.15.99:0/2672101595 >> 172.21.15.99:0/2672101595 conn(0x55e5b3988380 msgr2=0x55e5b3986db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 -- 172.21.15.99:0/2672101595 shutdown_connections 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.001+0000 7f8252a5e740 1 -- 172.21.15.99:0/2672101595 wait complete. 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.002+0000 7f8252a5e740 1 -- start start 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.002+0000 7f8252a5e740 1 Processor -- start 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.002+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b3a3cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.002+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b5280 0x55e5b39e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b39b7c30 con 0x55e5b39b4ea0 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b398c750 con 0x55e5b39b5280 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5b39e6c50 con 0x55e5b39b6c30 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 10 client.0 _ll_register_callbacks cb 0x7f8238055080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b3a3cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b3a3cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41824/0 (socket says 172.21.15.99:41824) 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 -- 172.21.15.99:0/2253489202 learned_addr learned my addr 172.21.15.99:0/2253489202 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824e41e640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824d41c640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b5280 0x55e5b39e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 -- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 msgr2=0x55e5b39e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39e9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 -- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b5280 msgr2=0x55e5b39e94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b5280 0x55e5b39e94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 -- 172.21.15.99:0/2253489202 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5b39b38f0 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f824dc1d640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b3a3cf50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f824402beb0 tx=0x7f8244004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddb6fd42ba9da45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8228ff9640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8228ff9640 1 -- 172.21.15.99:0/2253489202 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244058040 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8228ff9640 1 -- 172.21.15.99:0/2253489202 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f824405d450 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8244053430 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8228ff9640 1 -- 172.21.15.99:0/2253489202 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f824406d050 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 10 client.5607 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55e5b39ec4e0 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.003+0000 7f8228ff9640 1 -- 172.21.15.99:0/2253489202 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244053430 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8228ff9640 1 -- 172.21.15.99:0/2253489202 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8244070020 con 0x55e5b39b4ea0 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 10 client.5607 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 10 client.5607 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 20 client.5607 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39ebfe0 0x55e5b3a40b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 4 client.5607 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e5b38fa050 con 0x55e5b39ebfe0 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f824d41c640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39ebfe0 0x55e5b3a40b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8228ff9640 10 client.5607 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f824d41c640 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39ebfe0 0x55e5b3a40b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0d2949a3f1eb7d server_cookie=14e3f2f1af366709 in_seq=0 out_seq=0 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8228ff9640 1 -- 172.21.15.99:0/2253489202 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e5b38fa050 con 0x55e5b39ebfe0 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8228ff9640 10 client.5607 handle_command_reply: tid=0 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 client.5607 shutdown 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 msgr2=0x55e5b3a3cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b3a3cf50 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f824402beb0 tx=0x7f8244004730 comp rx=0 tx=0).stop 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 shutdown_connections 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e5b39b6c30 0x55e5b39e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55e5b39ebfe0 0x55e5b3a40b30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e5b39b5280 0x55e5b39e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 --2- 172.21.15.99:0/2253489202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e5b39b4ea0 0x55e5b3a3cf50 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8228ff9640 0 client.5607 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.004+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 >> 172.21.15.99:0/2253489202 conn(0x55e5b3988380 msgr2=0x55e5b3986db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 shutdown_connections 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f8252a5e740 1 -- 172.21.15.99:0/2253489202 wait complete. 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f8252a5e740 20 client.5607 trim_cache size 0 max 16384 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:05:06.013 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 5550, 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.3814798680000004, 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 21, 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 39, 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 39, 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 5.3479740109999998 2024-03-04T10:05:06.014 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f822c06b500 msgr2=0x7f822c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f822c06b500 0x7f822c06d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8244036000 tx=0x7f8244008030 comp rx=0 tx=0).stop 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 msgr2=0x7f82480b5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f82480b5740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f823800d800 tx=0x7f823800dcc0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 shutdown_connections 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f822c06b500 0x7f822c06d9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f82481616c0 0x7f824815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8248154a80 0x7f82480b5740 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 --2- 172.21.15.99:0/580073921 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82480cf650 0x7f8248149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 >> 172.21.15.99:0/580073921 conn(0x7f824806f5f0 msgr2=0x7f82480d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 shutdown_connections 2024-03-04T10:05:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.005+0000 7f824fe81640 1 -- 172.21.15.99:0/580073921 wait complete. 2024-03-04T10:05:06.017 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:05:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.116+0000 7fa300d7e640 1 Processor -- start 2024-03-04T10:05:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.116+0000 7fa300d7e640 1 -- start start 2024-03-04T10:05:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa300d7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc15d660 0x7fa2fc15e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa300d7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2fc0aeb20 0x7fa2fc071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa300d7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa2fc071890 0x7fa2fc150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa300d7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc15da40 con 0x7fa2fc0aeb20 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa300d7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc15ebb0 con 0x7fa2fc071890 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa300d7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0af090 con 0x7fa2fc15d660 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2fc0aeb20 0x7fa2fc071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa2fc0aeb20 0x7fa2fc071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41834/0 (socket says 172.21.15.99:41834) 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 -- 172.21.15.99:0/4247984858 learned_addr learned my addr 172.21.15.99:0/4247984858 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2fad9d640 1 --2- 172.21.15.99:0/4247984858 >> v2:172.21.15.162:3300/0 conn(0x7fa2fc071890 0x7fa2fc150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 -- 172.21.15.99:0/4247984858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc15d660 msgr2=0x7fa2fc15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2fa59c640 1 --2- 172.21.15.99:0/4247984858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc15d660 0x7fa2fc15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 --2- 172.21.15.99:0/4247984858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc15d660 0x7fa2fc15e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 -- 172.21.15.99:0/4247984858 >> v2:172.21.15.162:3300/0 conn(0x7fa2fc071890 msgr2=0x7fa2fc150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 --2- 172.21.15.99:0/4247984858 >> v2:172.21.15.162:3300/0 conn(0x7fa2fc071890 0x7fa2fc150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2f1d9b640 1 -- 172.21.15.99:0/4247984858 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc15a9f0 con 0x7fa2fc0aeb20 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.117+0000 7fa2fad9d640 1 --2- 172.21.15.99:0/4247984858 >> v2:172.21.15.162:3300/0 conn(0x7fa2fc071890 0x7fa2fc150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa2f1d9b640 1 --2- 172.21.15.99:0/4247984858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc071350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc06f5a0 tx=0x7fa2e402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4fb18342e2b8e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa2f9d9b640 1 -- 172.21.15.99:0/4247984858 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4038070 con 0x7fa2fc0aeb20 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa2f9d9b640 1 -- 172.21.15.99:0/4247984858 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e402f050 con 0x7fa2fc0aeb20 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa2f9d9b640 1 -- 172.21.15.99:0/4247984858 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4031970 con 0x7fa2fc0aeb20 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 -- 172.21.15.99:0/4247984858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc0aeb20 msgr2=0x7fa2fc071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 --2- 172.21.15.99:0/4247984858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc071350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc06f5a0 tx=0x7fa2e402b7f0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 -- 172.21.15.99:0/4247984858 shutdown_connections 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 --2- 172.21.15.99:0/4247984858 >> v2:172.21.15.162:3300/0 conn(0x7fa2fc071890 0x7fa2fc150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 --2- 172.21.15.99:0/4247984858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc071350 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 --2- 172.21.15.99:0/4247984858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc15d660 0x7fa2fc15e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 -- 172.21.15.99:0/4247984858 >> 172.21.15.99:0/4247984858 conn(0x7fa2fc06acd0 msgr2=0x7fa2fc06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 -- 172.21.15.99:0/4247984858 shutdown_connections 2024-03-04T10:05:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.118+0000 7fa300d7e640 1 -- 172.21.15.99:0/4247984858 wait complete. 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.120+0000 7fa300d7e640 1 Processor -- start 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.120+0000 7fa300d7e640 1 -- start start 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc071890 0x7fa2fc1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 0x7fa2fc1f4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc15a660 con 0x7fa2fc15d660 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc15afb0 con 0x7fa2fc0aeb20 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc15b4d0 con 0x7fa2fc071890 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 0x7fa2fc1f4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 0x7fa2fc1f4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41844/0 (socket says 172.21.15.99:41844) 2024-03-04T10:05:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 -- 172.21.15.99:0/2762289293 learned_addr learned my addr 172.21.15.99:0/2762289293 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2f1d9b640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fa59c640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc071890 0x7fa2fc1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 -- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc071890 msgr2=0x7fa2fc1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc071890 0x7fa2fc1526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 -- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2fc0aeb20 msgr2=0x7fa2fc152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 -- 172.21.15.99:0/2762289293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc14f3e0 con 0x7fa2fc15d660 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2fad9d640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 0x7fa2fc1f4ff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400cd40 tx=0x7fa2f400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f400f680 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f400f840 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc14e620 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.121+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f400fc30 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.122+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc14ebe0 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.123+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc074f60 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.123+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa2f40213f0 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.123+0000 7fa2f37fe640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2dc06b4d0 0x7fa2dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.123+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa2f408afe0 con 0x7fa2fc15d660 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.124+0000 7fa2fa59c640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2dc06b4d0 0x7fa2dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.124+0000 7fa2fa59c640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2dc06b4d0 0x7fa2dc06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0af500 tx=0x7fa2ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:06.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.128+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa2f4014030 con 0x7fa2fc15d660 2024-03-04T10:05:06.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.334+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa2fc072460 con 0x7fa2fc15d660 2024-03-04T10:05:06.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.335+0000 7fa2f37fe640 1 -- 172.21.15.99:0/2762289293 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa2f4059630 con 0x7fa2fc15d660 2024-03-04T10:05:06.336 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.336+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2dc06b4d0 msgr2=0x7fa2dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.336+0000 7fa300d7e640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2dc06b4d0 0x7fa2dc06d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0af500 tx=0x7fa2ec008040 comp rx=0 tx=0).stop 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.336+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 msgr2=0x7fa2fc1f4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.336+0000 7fa300d7e640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 0x7fa2fc1f4ff0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400cd40 tx=0x7fa2f400a650 comp rx=0 tx=0).stop 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 shutdown_connections 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2dc06b4d0 0x7fa2dc06d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2fc15d660 0x7fa2fc1f4ff0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2fc0aeb20 0x7fa2fc152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 --2- 172.21.15.99:0/2762289293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2fc071890 0x7fa2fc1526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:06.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 >> 172.21.15.99:0/2762289293 conn(0x7fa2fc06acd0 msgr2=0x7fa2fc0b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:06.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 shutdown_connections 2024-03-04T10:05:06.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:06.337+0000 7fa300d7e640 1 -- 172.21.15.99:0/2762289293 wait complete. 2024-03-04T10:05:06.348 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:05:06.349 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:05:06.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.349 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:05:06.365 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:05:06.365 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:05:06.365 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:05:06.365 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:05:06.366 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42679sec preferred_lft 42679sec 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:05:06.367 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:05:06.368 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:05:06.368 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:06.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.368 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:05:06.431 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:05:06.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.435 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:05:06.488 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:05:06.489 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:05:06.489 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:05:06.489 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:05:06.489 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:05:06.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.489 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:05:06.489 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:05:06.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:05:06.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:05:06.490 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:05:06.490 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:05:06.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.585 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:05:06.599 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:05:06.599 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:05:06.600 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:05:06.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.600 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:05:06.625 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:05:06.624+0000 7f8c8a11f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:05:06.652 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:05:06.652 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:06.707 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:05:06.708 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:05:06.708 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:06.762 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:05:06.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:05:06.847 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:05:06.847 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:05:06.855 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:05:06.855 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:05:06.855 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:05:06.855 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:05:06.855 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:05:06.856 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:05:06.856 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:05:06.856 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:05:06.856 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:05:06.856 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:05:06.857 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:05:06.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:06.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.885 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:05:06.941 INFO:teuthology.orchestra.run.smithi099.stdout:client5616 2024-03-04T10:05:06.941 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:06.941 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:06.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:06.996 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-03-04T10:05:07.054 INFO:teuthology.orchestra.run.smithi099.stdout:file_undamaged 2024-03-04T10:05:07.054 DEBUG:teuthology.orchestra.run.smithi099:> stdin-killer -- python3 -c ' 2024-03-04T10:05:07.054 DEBUG:teuthology.orchestra.run.smithi099:> import os 2024-03-04T10:05:07.054 DEBUG:teuthology.orchestra.run.smithi099:> import stat 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> import json 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> import sys 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> try: 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> except OSError as e: 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> sys.exit(e.errno) 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> print(json.dumps( 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> dict([(a, getattr(s, a)) for a in attrs]), 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> indent=2)) 2024-03-04T10:05:07.055 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:05:07.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-03-04T10:05:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-04T10:05:07.169 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:07.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.307+0000 7f1fdfeb2640 1 Processor -- start 2024-03-04T10:05:07.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.308+0000 7f1fdfeb2640 1 -- start start 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.308+0000 7f1fdfeb2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1fd815de00 0x7f1fd80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.308+0000 7f1fdfeb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd80b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.308+0000 7f1fdfeb2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd815e1e0 con 0x7f1fd80b9380 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.308+0000 7f1fdfeb2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd80d2820 con 0x7f1fd815de00 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.308+0000 7f1fdfeb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80d1c10 con 0x7f1fd80b5470 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1fd815de00 0x7f1fd80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1fd815de00 0x7f1fd80b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41854/0 (socket says 172.21.15.99:41854) 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 -- 172.21.15.99:0/927431499 learned_addr learned my addr 172.21.15.99:0/927431499 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fde44f640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd80b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 -- 172.21.15.99:0/927431499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 msgr2=0x7f1fd80b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd80b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 -- 172.21.15.99:0/927431499 >> v1:172.21.15.162:6789/0 conn(0x7f1fd80b9380 legacy=0x7f1fd80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fddc4e640 1 -- 172.21.15.99:0/927431499 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80d1ef0 con 0x7f1fd815de00 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.309+0000 7f1fde44f640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd80b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fddc4e640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd815de00 0x7f1fd80b2e60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009e50 tx=0x7f1fd002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df07c744733b1c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdcc4c640 1 -- 172.21.15.99:0/927431499 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd002ca70 con 0x7f1fd815de00 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdcc4c640 1 -- 172.21.15.99:0/927431499 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd002cc10 con 0x7f1fd815de00 2024-03-04T10:05:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdcc4c640 1 -- 172.21.15.99:0/927431499 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00343f0 con 0x7f1fd815de00 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/927431499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd815de00 msgr2=0x7f1fd80b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd815de00 0x7f1fd80b2e60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009e50 tx=0x7f1fd002bed0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/927431499 shutdown_connections 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd80b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/927431499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd815de00 0x7f1fd80b2e60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/927431499 >> 172.21.15.99:0/927431499 conn(0x7f1fd80d0470 msgr2=0x7f1fd80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/927431499 shutdown_connections 2024-03-04T10:05:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.310+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/927431499 wait complete. 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.311+0000 7f1fdfeb2640 1 Processor -- start 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 -- start start 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd8157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 0x7f1fd81f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd815de00 0x7f1fd81f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80b97e0 con 0x7f1fd80b9380 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80d24b0 con 0x7f1fd815de00 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdfeb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd80b72a0 con 0x7f1fd80b5470 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdd44d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 0x7f1fd81f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdd44d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 0x7f1fd81f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41864/0 (socket says 172.21.15.99:41864) 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.312+0000 7f1fdd44d640 1 -- 172.21.15.99:0/1330899612 learned_addr learned my addr 172.21.15.99:0/1330899612 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fddc4e640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd8157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fde44f640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd815de00 0x7f1fd81f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdd44d640 1 -- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 msgr2=0x7f1fd8157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdd44d640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd8157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdd44d640 1 -- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd815de00 msgr2=0x7f1fd81f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdd44d640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd815de00 0x7f1fd81f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdd44d640 1 -- 172.21.15.99:0/1330899612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8157bc0 con 0x7f1fd80b9380 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fde44f640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd815de00 0x7f1fd81f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdd44d640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 0x7f1fd81f47a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800c430 tx=0x7f1fc800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8011790 con 0x7f1fd80b9380 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc8017070 con 0x7f1fd80b9380 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd81f4010 con 0x7f1fd80b9380 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8011b70 con 0x7f1fd80b9380 2024-03-04T10:05:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.313+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd8159290 con 0x7f1fd80b9380 2024-03-04T10:05:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.314+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd8074f60 con 0x7f1fd80b9380 2024-03-04T10:05:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.314+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1fc8011d10 con 0x7f1fd80b9380 2024-03-04T10:05:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.315+0000 7f1fce7fc640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fbc06b4d0 0x7f1fbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.315+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1fc808a1f0 con 0x7f1fd80b9380 2024-03-04T10:05:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.315+0000 7f1fddc4e640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fbc06b4d0 0x7f1fbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.316+0000 7f1fddc4e640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fbc06b4d0 0x7f1fbc06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0035000 tx=0x7f1fd0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.319+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1fc8058e90 con 0x7f1fd80b9380 2024-03-04T10:05:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.524+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fd815d3f0 con 0x7f1fd80b9380 2024-03-04T10:05:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.525+0000 7f1fce7fc640 1 -- 172.21.15.99:0/1330899612 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f1fc8058790 con 0x7f1fd80b9380 2024-03-04T10:05:07.526 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:07.526 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.526+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fbc06b4d0 msgr2=0x7f1fbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.526+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fbc06b4d0 0x7f1fbc06d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0035000 tx=0x7f1fd0030000 comp rx=0 tx=0).stop 2024-03-04T10:05:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.526+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 msgr2=0x7f1fd81f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.526+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 0x7f1fd81f47a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800c430 tx=0x7f1fc800c8f0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 shutdown_connections 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1fbc06b4d0 0x7f1fbc06d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1fd815de00 0x7f1fd81f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1fd80b9380 0x7f1fd81f47a0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 --2- 172.21.15.99:0/1330899612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1fd80b5470 0x7f1fd8157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 >> 172.21.15.99:0/1330899612 conn(0x7f1fd80d0470 msgr2=0x7f1fd80b7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 shutdown_connections 2024-03-04T10:05:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.527+0000 7f1fdfeb2640 1 -- 172.21.15.99:0/1330899612 wait complete. 2024-03-04T10:05:07.538 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-03-04T10:05:07.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.639+0000 7ffb7355b640 1 Processor -- start 2024-03-04T10:05:07.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.640+0000 7ffb7355b640 1 -- start start 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.640+0000 7ffb7355b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 0x7ffb6c06c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.640+0000 7ffb7355b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ffb6c0cedf0 0x7ffb6c0cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.640+0000 7ffb7355b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0cf740 con 0x7ffb6c0d20c0 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.640+0000 7ffb7355b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6c06c5a0 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.640+0000 7ffb7355b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c06d3e0 con 0x7ffb6c161390 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb712f7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 0x7ffb6c06c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb712f7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 0x7ffb6c06c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:56026/0 (socket says 172.21.15.99:56026) 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb712f7640 1 -- 172.21.15.99:0/1081982147 learned_addr learned my addr 172.21.15.99:0/1081982147 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb71af8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ffb6c0cedf0 0x7ffb6c0cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb71af8640 1 -- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 msgr2=0x7ffb6c06c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb71af8640 1 --2- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 0x7ffb6c06c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb71af8640 1 -- 172.21.15.99:0/1081982147 >> v1:172.21.15.162:6789/0 conn(0x7ffb6c0d20c0 legacy=0x7ffb6c0ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb71af8640 1 -- 172.21.15.99:0/1081982147 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c06c7d0 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.641+0000 7ffb712f7640 1 --2- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 0x7ffb6c06c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.642+0000 7ffb71af8640 1 --2- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c0cc0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009d90 tx=0x7ffb6402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b122a969f21ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.642+0000 7ffb63fff640 1 -- 172.21.15.99:0/1081982147 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64039070 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.642+0000 7ffb63fff640 1 -- 172.21.15.99:0/1081982147 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6402cc50 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.642+0000 7ffb63fff640 1 -- 172.21.15.99:0/1081982147 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64034440 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 -- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c0cedf0 msgr2=0x7ffb6c0cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 --2- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c0cc0d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009d90 tx=0x7ffb6402bea0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 -- 172.21.15.99:0/1081982147 shutdown_connections 2024-03-04T10:05:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 --2- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c0cc0d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 --2- 172.21.15.99:0/1081982147 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c161390 0x7ffb6c06c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 -- 172.21.15.99:0/1081982147 >> 172.21.15.99:0/1081982147 conn(0x7ffb6c066dc0 msgr2=0x7ffb6c0671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 -- 172.21.15.99:0/1081982147 shutdown_connections 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.643+0000 7ffb7355b640 1 -- 172.21.15.99:0/1081982147 wait complete. 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.644+0000 7ffb7355b640 1 Processor -- start 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.644+0000 7ffb7355b640 1 -- start start 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c15cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c0d20c0 0x7ffb6c1e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c161390 0x7ffb6c1f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c06d070 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c06cd90 con 0x7ffb6c161390 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c161770 con 0x7ffb6c0d20c0 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c15cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c15cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41872/0 (socket says 172.21.15.99:41872) 2024-03-04T10:05:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 -- 172.21.15.99:0/605586450 learned_addr learned my addr 172.21.15.99:0/605586450 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb71af8640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c161390 0x7ffb6c1f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb70af6640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c0d20c0 0x7ffb6c1e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 -- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c0d20c0 msgr2=0x7ffb6c1e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c0d20c0 0x7ffb6c1e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 -- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c161390 msgr2=0x7ffb6c1f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c161390 0x7ffb6c1f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 -- 172.21.15.99:0/605586450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c1f5050 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb712f7640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c15cc40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800e9b0 tx=0x7ffb6800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb61ffb640 1 -- 172.21.15.99:0/605586450 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68017070 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb61ffb640 1 -- 172.21.15.99:0/605586450 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6800c9a0 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.645+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c1f5330 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.646+0000 7ffb61ffb640 1 -- 172.21.15.99:0/605586450 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6800cd50 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.646+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c1e8a20 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.647+0000 7ffb61ffb640 1 -- 172.21.15.99:0/605586450 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffb6801a070 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.647+0000 7ffb61ffb640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb5006b4d0 0x7ffb5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.647+0000 7ffb61ffb640 1 -- 172.21.15.99:0/605586450 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffb6808a8b0 con 0x7ffb6c0cedf0 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.648+0000 7ffb70af6640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb5006b4d0 0x7ffb5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.648+0000 7ffb70af6640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb5006b4d0 0x7ffb5006d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0b15f0 tx=0x7ffb5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 Processor -- start 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 -- start start 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c36ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c39b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c36bf90 0x55ea6c39bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c338c00 con 0x55ea6c398e90 2024-03-04T10:05:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c2d5c50 con 0x55ea6c398ab0 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c274c60 con 0x55ea6c36bf90 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb70af6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c39b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb70af6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c39b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41888/0 (socket says 172.21.15.99:41888) 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb70af6640 1 -- 172.21.15.99:0/3014958168 learned_addr learned my addr 172.21.15.99:0/3014958168 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.649+0000 7ffb712f7640 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c36ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb71af8640 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c36bf90 0x55ea6c39bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb70af6640 1 -- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c36bf90 msgr2=0x55ea6c39bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb70af6640 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c36bf90 0x55ea6c39bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb70af6640 1 -- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 msgr2=0x55ea6c36ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb70af6640 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c36ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb70af6640 1 -- 172.21.15.99:0/3014958168 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea6c269310 con 0x55ea6c398e90 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb71af8640 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c36bf90 0x55ea6c39bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb70af6640 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c39b840 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00fbd0 tx=0x7ffb5c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155536804585da2e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb56ffd640 1 -- 172.21.15.99:0/3014958168 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c026070 con 0x55ea6c398e90 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb56ffd640 1 -- 172.21.15.99:0/3014958168 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c007030 con 0x55ea6c398e90 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.650+0000 7ffb56ffd640 1 -- 172.21.15.99:0/3014958168 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c0073e0 con 0x55ea6c398e90 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 -- 172.21.15.99:0/3014958168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 msgr2=0x55ea6c39b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c39b840 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00fbd0 tx=0x7ffb5c012000 comp rx=0 tx=0).stop 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 -- 172.21.15.99:0/3014958168 shutdown_connections 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c36bf90 0x55ea6c39bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c39b840 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 --2- 172.21.15.99:0/3014958168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c36ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 -- 172.21.15.99:0/3014958168 >> 172.21.15.99:0/3014958168 conn(0x55ea6c36dcc0 msgr2=0x55ea6c3b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 -- 172.21.15.99:0/3014958168 shutdown_connections 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 -- 172.21.15.99:0/3014958168 wait complete. 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 -- start start 2024-03-04T10:05:07.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.651+0000 7ffb76219740 1 Processor -- start 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c38a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c38ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c389040 0x55ea6c38b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c2725f0 con 0x55ea6c398e90 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c2d1cf0 con 0x55ea6c398ab0 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c370360 con 0x55ea6c389040 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb76219740 10 client.0 _ll_register_callbacks cb 0x55ea6bfe9ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb70af6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c38ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb70af6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c38ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41892/0 (socket says 172.21.15.99:41892) 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb70af6640 1 -- 172.21.15.99:0/187059170 learned_addr learned my addr 172.21.15.99:0/187059170 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb712f7640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c38a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.652+0000 7ffb71af8640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c389040 0x55ea6c38b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb70af6640 1 -- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c389040 msgr2=0x55ea6c38b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb70af6640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c389040 0x55ea6c38b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb70af6640 1 -- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 msgr2=0x55ea6c38a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb70af6640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c38a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb70af6640 1 -- 172.21.15.99:0/187059170 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea6c0b67c0 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb71af8640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c389040 0x55ea6c38b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb70af6640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c38ac30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00fba0 tx=0x7ffb5c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5709b5f9234f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb4bfff640 1 -- 172.21.15.99:0/187059170 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c026040 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb4bfff640 1 -- 172.21.15.99:0/187059170 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffb5c020620 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb5c007030 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb4bfff640 1 -- 172.21.15.99:0/187059170 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb5c038050 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 10 client.5631 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55ea6c116f50 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb4bfff640 1 -- 172.21.15.99:0/187059170 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c007030 con 0x55ea6c398e90 2024-03-04T10:05:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb4bfff640 1 -- 172.21.15.99:0/187059170 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ffb5c03b020 con 0x55ea6c398e90 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 10 client.5631 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 10 client.5631 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 20 client.5631 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c3d0c00 0x55ea6c3d4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 4 client.5631 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea6c2e3190 con 0x55ea6c3d0c00 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.653+0000 7ffb712f7640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c3d0c00 0x55ea6c3d4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb4bfff640 10 client.5631 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb712f7640 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c3d0c00 0x55ea6c3d4dc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7100f587a7ed0bd3 server_cookie=5ead2a19fa9ae13a in_seq=0 out_seq=0 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb4bfff640 1 -- 172.21.15.99:0/187059170 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ea6c2e3190 con 0x55ea6c3d0c00 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb4bfff640 10 client.5631 handle_command_reply: tid=0 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 client.5631 shutdown 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 msgr2=0x55ea6c38ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c38ac30 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00fba0 tx=0x7ffb5c000ed0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 shutdown_connections 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c389040 0x55ea6c38b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398e90 0x55ea6c38ac30 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c3d0c00 0x55ea6c3d4dc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 --2- 172.21.15.99:0/187059170 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c38a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb4bfff640 0 client.5631 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.654+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 >> 172.21.15.99:0/187059170 conn(0x55ea6c36bf90 msgr2=0x55ea6c372880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.655+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 shutdown_connections 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.655+0000 7ffb76219740 1 -- 172.21.15.99:0/187059170 wait complete. 2024-03-04T10:05:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.655+0000 7ffb76219740 20 client.5631 trim_cache size 0 max 16384 2024-03-04T10:05:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.666+0000 7ffb76219740 1 Processor -- start 2024-03-04T10:05:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.666+0000 7ffb76219740 1 -- start start 2024-03-04T10:05:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c389040 0x55ea6c36f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb76219740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 0x55ea6c3b9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c3ba480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb76219740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0b5ae0 con 0x55ea6c398ab0 2024-03-04T10:05:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0ca130 con 0x55ea6c389040 2024-03-04T10:05:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c0b54e0 con 0x55ea6c398e90 2024-03-04T10:05:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb70af6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 0x55ea6c3b9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb70af6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 0x55ea6c3b9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41908/0 (socket says 172.21.15.99:41908) 2024-03-04T10:05:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb70af6640 1 -- 172.21.15.99:0/4093479333 learned_addr learned my addr 172.21.15.99:0/4093479333 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb712f7640 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c389040 0x55ea6c36f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.667+0000 7ffb71af8640 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c3ba480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb70af6640 1 -- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 msgr2=0x55ea6c3ba480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb70af6640 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c3ba480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb70af6640 1 -- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c389040 msgr2=0x55ea6c36f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb70af6640 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c389040 0x55ea6c36f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb70af6640 1 -- 172.21.15.99:0/4093479333 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea6bfaa0b0 con 0x55ea6c398ab0 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb70af6640 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 0x55ea6c3b9f40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00f9b0 tx=0x7ffb5c02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf1a2fd24414588 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb567fc640 1 -- 172.21.15.99:0/4093479333 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c026040 con 0x55ea6c398ab0 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb567fc640 1 -- 172.21.15.99:0/4093479333 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c007030 con 0x55ea6c398ab0 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb567fc640 1 -- 172.21.15.99:0/4093479333 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c007740 con 0x55ea6c398ab0 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 -- 172.21.15.99:0/4093479333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 msgr2=0x55ea6c3b9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 0x55ea6c3b9f40 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00f9b0 tx=0x7ffb5c02a000 comp rx=0 tx=0).stop 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 -- 172.21.15.99:0/4093479333 shutdown_connections 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c3ba480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c398ab0 0x55ea6c3b9f40 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 --2- 172.21.15.99:0/4093479333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c389040 0x55ea6c36f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 -- 172.21.15.99:0/4093479333 >> 172.21.15.99:0/4093479333 conn(0x55ea6c36bf90 msgr2=0x55ea6c38a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 -- 172.21.15.99:0/4093479333 shutdown_connections 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.668+0000 7ffb76219740 1 -- 172.21.15.99:0/4093479333 wait complete. 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.669+0000 7ffb76219740 1 -- start start 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.669+0000 7ffb76219740 1 Processor -- start 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 0x55ea6c41bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c385320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c4223a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c3d0a80 con 0x55ea6c389040 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c3d06f0 con 0x55ea6c398ab0 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea6c3d1080 con 0x55ea6c398e90 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb76219740 10 client.0 _ll_register_callbacks cb 0x7ffb5c024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 0x55ea6c41bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 0x55ea6c41bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41922/0 (socket says 172.21.15.99:41922) 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 -- 172.21.15.99:0/2620981377 learned_addr learned my addr 172.21.15.99:0/2620981377 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb70af6640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c385320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb71af8640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c4223a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 -- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 msgr2=0x55ea6c4223a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c4223a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 -- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 msgr2=0x55ea6c385320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c385320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 -- 172.21.15.99:0/2620981377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea6c396b80 con 0x55ea6c389040 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.670+0000 7ffb712f7640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 0x55ea6c41bba0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ffb68058c90 tx=0x7ffb6800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4be5267a3329599 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb4bfff640 1 -- 172.21.15.99:0/2620981377 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68075060 con 0x55ea6c389040 2024-03-04T10:05:07.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb4bfff640 1 -- 172.21.15.99:0/2620981377 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffb680591f0 con 0x55ea6c389040 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb5c007030 con 0x55ea6c389040 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb4bfff640 1 -- 172.21.15.99:0/2620981377 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb68021040 con 0x55ea6c389040 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 10 client.5637 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55ea6c385920 con 0x55ea6c389040 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb4bfff640 1 -- 172.21.15.99:0/2620981377 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c007030 con 0x55ea6c389040 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb4bfff640 1 -- 172.21.15.99:0/2620981377 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ffb68061160 con 0x55ea6c389040 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 10 client.5637 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 10 client.5637 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 20 client.5637 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c38a8b0 0x55ea6c3970f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 4 client.5637 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x55ea6c2dd4e0 con 0x55ea6c38a8b0 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb70af6640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c38a8b0 0x55ea6c3970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb4bfff640 10 client.5637 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.671+0000 7ffb70af6640 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c38a8b0 0x55ea6c3970f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56336b53baca614d server_cookie=baa13cb774b0402b in_seq=0 out_seq=0 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb4bfff640 1 -- 172.21.15.99:0/2620981377 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+237 (crc 0 0 0) 0x55ea6c2dd4e0 con 0x55ea6c38a8b0 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb4bfff640 10 client.5637 handle_command_reply: tid=0 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 client.5637 shutdown 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 msgr2=0x55ea6c41bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 0x55ea6c41bba0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ffb68058c90 tx=0x7ffb6800cf70 comp rx=0 tx=0).stop 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 shutdown_connections 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea6c398e90 0x55ea6c4223a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55ea6c38a8b0 0x55ea6c3970f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea6c398ab0 0x55ea6c385320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 --2- 172.21.15.99:0/2620981377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea6c389040 0x55ea6c41bba0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 >> 172.21.15.99:0/2620981377 conn(0x55ea6c36bf90 msgr2=0x55ea6c3b9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb4bfff640 0 client.5637 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 shutdown_connections 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 1 -- 172.21.15.99:0/2620981377 wait complete. 2024-03-04T10:05:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.672+0000 7ffb76219740 20 client.5637 trim_cache size 0 max 16384 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "damage_type": "dentry", 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 3253863887, 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "ino": 1099511627776, 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "frag": "*", 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "dname": "file_to_be_damaged", 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "snap_id": "head", 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: "path": "/subdir/file_to_be_damaged" 2024-03-04T10:05:07.683 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:07.684 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:05:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb5006b4d0 msgr2=0x7ffb5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb5006b4d0 0x7ffb5006d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c0b15f0 tx=0x7ffb5c009040 comp rx=0 tx=0).stop 2024-03-04T10:05:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 msgr2=0x7ffb6c15cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c15cc40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800e9b0 tx=0x7ffb6800ee70 comp rx=0 tx=0).stop 2024-03-04T10:05:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 shutdown_connections 2024-03-04T10:05:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb5006b4d0 0x7ffb5006d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb6c161390 0x7ffb6c1f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb6c0d20c0 0x7ffb6c1e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 --2- 172.21.15.99:0/605586450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb6c0cedf0 0x7ffb6c15cc40 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 >> 172.21.15.99:0/605586450 conn(0x7ffb6c066dc0 msgr2=0x7ffb6c0b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 shutdown_connections 2024-03-04T10:05:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07.673+0000 7ffb7355b640 1 -- 172.21.15.99:0/605586450 wait complete. 2024-03-04T10:05:07.686 DEBUG:teuthology.orchestra.run.smithi099:> stdin-killer -- python3 -c ' 2024-03-04T10:05:07.686 DEBUG:teuthology.orchestra.run.smithi099:> import sys 2024-03-04T10:05:07.686 DEBUG:teuthology.orchestra.run.smithi099:> import errno 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> try: 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> f.close() 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> except IOError as e: 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> sys.exit(errno.EIO) 2024-03-04T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:05:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-03-04T10:05:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> stdin-killer -- python3 -c ' 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> import os 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> import stat 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> import json 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> import sys 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:05:07.761 DEBUG:teuthology.orchestra.run.smithi099:> try: 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> except OSError as e: 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> sys.exit(e.errno) 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> print(json.dumps( 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> dict([(a, getattr(s, a)) for a in attrs]), 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> indent=2)) 2024-03-04T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:05:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:07 stdin-killer INFO: command exited with status 5: exiting normally with same code! 2024-03-04T10:05:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-03-04T10:05:07.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:07.877 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-04T10:05:07.933 INFO:teuthology.orchestra.run.smithi099.stdout:2 2024-03-04T10:05:07.934 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:07.934 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:07.989 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:07.989 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:08.044 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:08.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.182+0000 7f240b369640 1 Processor -- start 2024-03-04T10:05:08.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.182+0000 7f240b369640 1 -- start start 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f240b369640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2404158410 0x7f24040b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f240b369640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f240406acd0 0x7f2404161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f240b369640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f240406b500 con 0x7f2404161ca0 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f240b369640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24040b3f00 con 0x7f2404158410 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f240b369640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240406b7e0 con 0x7f240406acd0 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f2409105640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2404158410 0x7f24040b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f2409906640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2404161ca0 0x7f240415e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51250/0 (socket says 172.21.15.99:51250) 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f2409906640 1 -- 172.21.15.99:0/2441699462 learned_addr learned my addr 172.21.15.99:0/2441699462 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.183+0000 7f2408904640 1 --2- 172.21.15.99:0/2441699462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f240406acd0 0x7f2404161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f2409105640 1 -- 172.21.15.99:0/2441699462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f240406acd0 msgr2=0x7f2404161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f2409105640 1 --2- 172.21.15.99:0/2441699462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f240406acd0 0x7f2404161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f2409105640 1 -- 172.21.15.99:0/2441699462 >> v1:172.21.15.99:6789/0 conn(0x7f2404161ca0 legacy=0x7f240415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f2409105640 1 -- 172.21.15.99:0/2441699462 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240406bac0 con 0x7f2404158410 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f2409105640 1 --2- 172.21.15.99:0/2441699462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2404158410 0x7f24040b39c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f24000097a0 tx=0x7f240002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae5f2f011cabf45 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f23fbfff640 1 -- 172.21.15.99:0/2441699462 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400034070 con 0x7f2404158410 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f23fbfff640 1 -- 172.21.15.99:0/2441699462 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2400030800 con 0x7f2404158410 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.184+0000 7f23fbfff640 1 -- 172.21.15.99:0/2441699462 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400030bd0 con 0x7f2404158410 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 -- 172.21.15.99:0/2441699462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2404158410 msgr2=0x7f24040b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 --2- 172.21.15.99:0/2441699462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2404158410 0x7f24040b39c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f24000097a0 tx=0x7f240002b940 comp rx=0 tx=0).stop 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 -- 172.21.15.99:0/2441699462 shutdown_connections 2024-03-04T10:05:08.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 --2- 172.21.15.99:0/2441699462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f240406acd0 0x7f2404161760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 --2- 172.21.15.99:0/2441699462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2404158410 0x7f24040b39c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 -- 172.21.15.99:0/2441699462 >> 172.21.15.99:0/2441699462 conn(0x7f24040cfef0 msgr2=0x7f24040d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:08.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 -- 172.21.15.99:0/2441699462 shutdown_connections 2024-03-04T10:05:08.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.185+0000 7f240b369640 1 -- 172.21.15.99:0/2441699462 wait complete. 2024-03-04T10:05:08.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.186+0000 7f240b369640 1 Processor -- start 2024-03-04T10:05:08.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.186+0000 7f240b369640 1 -- start start 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f240b369640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 0x7f24040cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f240b369640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2404158410 0x7f24040cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f240b369640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2404161ca0 0x7f2404147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f240b369640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240406c5a0 con 0x7f240406acd0 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f240b369640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240406c080 con 0x7f2404158410 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f240b369640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240406ee60 con 0x7f2404161ca0 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 0x7f24040cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 0x7f24040cefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41926/0 (socket says 172.21.15.99:41926) 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 -- 172.21.15.99:0/2955561885 learned_addr learned my addr 172.21.15.99:0/2955561885 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2408904640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2404158410 0x7f24040cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409906640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2404161ca0 0x7f2404147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 -- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2404161ca0 msgr2=0x7f2404147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2404161ca0 0x7f2404147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 -- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2404158410 msgr2=0x7f24040cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2404158410 0x7f24040cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 -- 172.21.15.99:0/2955561885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2404147d90 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2408904640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2404158410 0x7f24040cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.187+0000 7f2409105640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 0x7f24040cefe0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f240002bf50 tx=0x7f24000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.188+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2400034040 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.188+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2400030600 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.188+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24041e8c40 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.188+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24000423f0 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.188+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24041e9200 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.189+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2404074f60 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.189+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2400037070 con 0x7f240406acd0 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.190+0000 7f23f9ffb640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23e806b500 0x7f23e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.190+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f24000ac550 con 0x7f240406acd0 2024-03-04T10:05:08.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.190+0000 7f2408904640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23e806b500 0x7f23e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.190+0000 7f2408904640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23e806b500 0x7f23e806d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2404148100 tx=0x7f23f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.194+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f24000307d0 con 0x7f240406acd0 2024-03-04T10:05:08.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.366+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2404157f80 con 0x7f240406acd0 2024-03-04T10:05:08.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.367+0000 7f23f9ffb640 1 -- 172.21.15.99:0/2955561885 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+17033 (secure 0 0 0) 0x7f240007b1f0 con 0x7f240406acd0 2024-03-04T10:05:08.368 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:08.369 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":39,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:00.614530+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/434536114":"2024-03-05T10:05:00.614507+0000","172.21.15.99:6833/434536114":"2024-03-05T10:05:00.614507+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:08.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23e806b500 msgr2=0x7f23e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23e806b500 0x7f23e806d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2404148100 tx=0x7f23f4008040 comp rx=0 tx=0).stop 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 msgr2=0x7f24040cefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 0x7f24040cefe0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f240002bf50 tx=0x7f24000046c0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 shutdown_connections 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23e806b500 0x7f23e806d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2404161ca0 0x7f2404147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2404158410 0x7f24040cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 --2- 172.21.15.99:0/2955561885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f240406acd0 0x7f24040cefe0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 >> 172.21.15.99:0/2955561885 conn(0x7f24040cfef0 msgr2=0x7f240415ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 shutdown_connections 2024-03-04T10:05:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.369+0000 7f240b369640 1 -- 172.21.15.99:0/2955561885 wait complete. 2024-03-04T10:05:08.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:04:49.926068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:04:50.486347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:05:08.381 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:05:08.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.483+0000 7fb9d7e2b640 1 Processor -- start 2024-03-04T10:05:08.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.483+0000 7fb9d7e2b640 1 -- start start 2024-03-04T10:05:08.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d7e2b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb9d006d3c0 0x7fb9d00b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d7e2b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d006d8a0 con 0x7fb9d00b48a0 2024-03-04T10:05:08.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d7e2b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9d006db80 con 0x7fb9d015a810 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d7e2b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9d006de60 con 0x7fb9d006d3c0 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d63c8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb9d006d3c0 0x7fb9d00b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d63c8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb9d006d3c0 0x7fb9d00b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41930/0 (socket says 172.21.15.99:41930) 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d63c8640 1 -- 172.21.15.99:0/2214284507 learned_addr learned my addr 172.21.15.99:0/2214284507 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d63c8640 1 -- 172.21.15.99:0/2214284507 >> v1:172.21.15.162:6789/0 conn(0x7fb9d015a810 legacy=0x7fb9d00b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d63c8640 1 -- 172.21.15.99:0/2214284507 >> v1:172.21.15.99:6789/0 conn(0x7fb9d00b48a0 legacy=0x7fb9d00b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d63c8640 1 -- 172.21.15.99:0/2214284507 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d006e140 con 0x7fb9d006d3c0 2024-03-04T10:05:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.484+0000 7fb9d4bc5640 1 -- 172.21.15.99:0/2214284507 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029992218 0 0) 0x7fb9d006d8a0 con 0x7fb9d00b48a0 2024-03-04T10:05:08.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.485+0000 7fb9d63c8640 1 --2- 172.21.15.99:0/2214284507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d006d3c0 0x7fb9d00b3100 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009e50 tx=0x7fb9c802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a36e9142d47d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.485+0000 7fb9d4bc5640 1 -- 172.21.15.99:0/2214284507 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c802cb20 con 0x7fb9d006d3c0 2024-03-04T10:05:08.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.485+0000 7fb9d4bc5640 1 -- 172.21.15.99:0/2214284507 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c802ccc0 con 0x7fb9d006d3c0 2024-03-04T10:05:08.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.485+0000 7fb9d4bc5640 1 -- 172.21.15.99:0/2214284507 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8034610 con 0x7fb9d006d3c0 2024-03-04T10:05:08.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/2214284507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d006d3c0 msgr2=0x7fb9d00b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/2214284507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d006d3c0 0x7fb9d00b3100 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8009e50 tx=0x7fb9c802bf80 comp rx=0 tx=0).stop 2024-03-04T10:05:08.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/2214284507 shutdown_connections 2024-03-04T10:05:08.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/2214284507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d006d3c0 0x7fb9d00b3100 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/2214284507 >> 172.21.15.99:0/2214284507 conn(0x7fb9d006acd0 msgr2=0x7fb9d006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:08.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/2214284507 shutdown_connections 2024-03-04T10:05:08.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.486+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/2214284507 wait complete. 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.487+0000 7fb9d7e2b640 1 Processor -- start 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.487+0000 7fb9d7e2b640 1 -- start start 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.487+0000 7fb9d7e2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9d006d3c0 0x7fb9d01539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 0x7fb9d01549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9d015a810 0x7fb9d0150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d00cc130 con 0x7fb9d00b48a0 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d00cbfb0 con 0x7fb9d015a810 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d0150bb0 con 0x7fb9d006d3c0 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 0x7fb9d01549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 0x7fb9d01549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41940/0 (socket says 172.21.15.99:41940) 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 -- 172.21.15.99:0/1738545597 learned_addr learned my addr 172.21.15.99:0/1738545597 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d63c8640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9d015a810 0x7fb9d0150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d5bc7640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9d006d3c0 0x7fb9d01539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 -- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9d006d3c0 msgr2=0x7fb9d01539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9d006d3c0 0x7fb9d01539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 -- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9d015a810 msgr2=0x7fb9d0150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9d015a810 0x7fb9d0150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 -- 172.21.15.99:0/1738545597 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d01e46e0 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d63c8640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9d015a810 0x7fb9d0150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d53c6640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 0x7fb9d01549a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c4b0 tx=0x7fb9c000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0017070 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c0005480 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9d0152ee0 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0003830 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.488+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9d01534a0 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.489+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9d0074f60 con 0x7fb9d00b48a0 2024-03-04T10:05:08.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.490+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb9c001a070 con 0x7fb9d00b48a0 2024-03-04T10:05:08.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.490+0000 7fb9c67fc640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb9b406b4d0 0x7fb9b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.490+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb9c0089790 con 0x7fb9d00b48a0 2024-03-04T10:05:08.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.490+0000 7fb9d5bc7640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb9b406b4d0 0x7fb9b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.491+0000 7fb9d5bc7640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb9b406b4d0 0x7fb9b406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006fd0 tx=0x7fb9cc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.494+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb9c0058430 con 0x7fb9d00b48a0 2024-03-04T10:05:08.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.665+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb9d006ec30 con 0x7fb9d00b48a0 2024-03-04T10:05:08.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.666+0000 7fb9c67fc640 1 -- 172.21.15.99:0/1738545597 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb9c0057d30 con 0x7fb9d00b48a0 2024-03-04T10:05:08.667 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:05:08.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb9b406b4d0 msgr2=0x7fb9b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb9b406b4d0 0x7fb9b406d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc006fd0 tx=0x7fb9cc008040 comp rx=0 tx=0).stop 2024-03-04T10:05:08.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 msgr2=0x7fb9d01549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 0x7fb9d01549a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb9c000c4b0 tx=0x7fb9c000c970 comp rx=0 tx=0).stop 2024-03-04T10:05:08.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 shutdown_connections 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb9b406b4d0 0x7fb9b406d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9d015a810 0x7fb9d0150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9d00b48a0 0x7fb9d01549a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 --2- 172.21.15.99:0/1738545597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9d006d3c0 0x7fb9d01539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 >> 172.21.15.99:0/1738545597 conn(0x7fb9d006acd0 msgr2=0x7fb9d00b4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 shutdown_connections 2024-03-04T10:05:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.668+0000 7fb9d7e2b640 1 -- 172.21.15.99:0/1738545597 wait complete. 2024-03-04T10:05:08.680 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:05:08.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.781+0000 7f77806e8640 1 Processor -- start 2024-03-04T10:05:08.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.781+0000 7f77806e8640 1 -- start start 2024-03-04T10:05:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.782+0000 7f77806e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780b9410 0x7f77781622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.782+0000 7f77806e8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77780d2400 con 0x7f77780d34a0 2024-03-04T10:05:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.782+0000 7f77806e8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77780d26e0 con 0x7f77780b5700 2024-03-04T10:05:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.782+0000 7f77806e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780d29c0 con 0x7f77780b9410 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.782+0000 7f777dc83640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f77780d34a0 0x7f7778161be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51276/0 (socket says 172.21.15.99:51276) 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777dc83640 1 -- 172.21.15.99:0/4110427315 learned_addr learned my addr 172.21.15.99:0/4110427315 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777ec85640 1 --2- 172.21.15.99:0/4110427315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780b9410 0x7f77781622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134791671 0 0) 0x7f77780d2400 con 0x7f77780d34a0 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7764003520 con 0x7f77780d34a0 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222936708 0 0) 0x7f77780d26e0 con 0x7f77780b5700 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77780d2400 con 0x7f77780b5700 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3585012260 0 0) 0x7f7764003520 con 0x7f77780d34a0 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777ec85640 1 -- 172.21.15.99:0/4110427315 >> v1:172.21.15.162:6789/0 conn(0x7f77780b5700 legacy=0x7f77780d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777ec85640 1 -- 172.21.15.99:0/4110427315 >> v1:172.21.15.99:6789/0 conn(0x7f77780d34a0 legacy=0x7f7778161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.783+0000 7f777ec85640 1 -- 172.21.15.99:0/4110427315 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780d2ca0 con 0x7f77780b9410 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.784+0000 7f777ec85640 1 --2- 172.21.15.99:0/4110427315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780b9410 0x7f77781622f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7770009dc0 tx=0x7f777002bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7c5ad50c6d7561c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.784+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770039070 con 0x7f77780b9410 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.784+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777002c880 con 0x7f77780b9410 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.784+0000 7f777d482640 1 -- 172.21.15.99:0/4110427315 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768001670 con 0x7f77780b9410 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 -- 172.21.15.99:0/4110427315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780b9410 msgr2=0x7f77781622f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 --2- 172.21.15.99:0/4110427315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780b9410 0x7f77781622f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7770009dc0 tx=0x7f777002bd50 comp rx=0 tx=0).stop 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 -- 172.21.15.99:0/4110427315 shutdown_connections 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 --2- 172.21.15.99:0/4110427315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780b9410 0x7f77781622f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 -- 172.21.15.99:0/4110427315 >> 172.21.15.99:0/4110427315 conn(0x7f77780d0490 msgr2=0x7f77780d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 -- 172.21.15.99:0/4110427315 shutdown_connections 2024-03-04T10:05:08.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.785+0000 7f77806e8640 1 -- 172.21.15.99:0/4110427315 wait complete. 2024-03-04T10:05:08.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 Processor -- start 2024-03-04T10:05:08.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 -- start start 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 0x7f777814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77780b9410 0x7f77780c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780d34a0 0x7f777815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b5c60 con 0x7f77780b5700 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b5ae0 con 0x7f77780b9410 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.786+0000 7f77806e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77780b5de0 con 0x7f77780d34a0 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 0x7f777814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 0x7f777814ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41942/0 (socket says 172.21.15.99:41942) 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 -- 172.21.15.99:0/3575555461 learned_addr learned my addr 172.21.15.99:0/3575555461 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777dc83640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77780b9410 0x7f77780c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777ec85640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780d34a0 0x7f777815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 -- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780d34a0 msgr2=0x7f777815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780d34a0 0x7f777815ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 -- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77780b9410 msgr2=0x7f77780c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77780b9410 0x7f77780c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 -- 172.21.15.99:0/3575555461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777815b470 con 0x7f77780b5700 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777dc83640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77780b9410 0x7f77780c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777ec85640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780d34a0 0x7f777815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f777e484640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 0x7f777814ed30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f777400c460 tx=0x7f777400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740052d0 con 0x7f77780b5700 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774005490 con 0x7f77780b5700 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777815b750 con 0x7f77780b5700 2024-03-04T10:05:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774012870 con 0x7f77780b5700 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.787+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777815bd10 con 0x7f77780b5700 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.788+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7778074f60 con 0x7f77780b5700 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.789+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7774012a10 con 0x7f77780b5700 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.789+0000 7f776effd640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.789+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7774089fa0 con 0x7f77780b5700 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.789+0000 7f777dc83640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.790+0000 7f777dc83640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f775c06b4d0 0x7f775c06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f77780b4fd0 tx=0x7f7768009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.793+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7774058cd0 con 0x7f77780b5700 2024-03-04T10:05:08.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.964+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f77780d1660 con 0x7f77780b5700 2024-03-04T10:05:08.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.964+0000 7f776effd640 1 -- 172.21.15.99:0/3575555461 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) v1 ==== 104+0+11 (secure 0 0 0) 0x7f77740585d0 con 0x7f77780b5700 2024-03-04T10:05:08.966 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f775c06b4d0 msgr2=0x7f775c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f775c06b4d0 0x7f775c06d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f77780b4fd0 tx=0x7f7768009040 comp rx=0 tx=0).stop 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 msgr2=0x7f777814ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 0x7f777814ed30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f777400c460 tx=0x7f777400c920 comp rx=0 tx=0).stop 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 shutdown_connections 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f775c06b4d0 0x7f775c06d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f77780d34a0 0x7f777815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f77780b9410 0x7f77780c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 --2- 172.21.15.99:0/3575555461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f77780b5700 0x7f777814ed30 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 >> 172.21.15.99:0/3575555461 conn(0x7f77780d0490 msgr2=0x7f777806c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 shutdown_connections 2024-03-04T10:05:08.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:08.966+0000 7f77806e8640 1 -- 172.21.15.99:0/3575555461 wait complete. 2024-03-04T10:05:08.978 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:05:09.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.080+0000 7fbd47c81640 1 Processor -- start 2024-03-04T10:05:09.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.080+0000 7fbd47c81640 1 -- start start 2024-03-04T10:05:09.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd47c81640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbd4015b700 0x7fbd400ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd47c81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 0x7fbd4006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd47c81640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4015bae0 con 0x7fbd4006bce0 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd47c81640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4006c680 con 0x7fbd4015b700 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd47c81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400adb90 con 0x7fbd4015e810 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd4521c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbd4006bce0 0x7fbd4015e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51282/0 (socket says 172.21.15.99:51282) 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd4521c640 1 -- 172.21.15.99:0/2686326464 learned_addr learned my addr 172.21.15.99:0/2686326464 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd45a1d640 1 --2- 172.21.15.99:0/2686326464 >> v2:172.21.15.162:3300/0 conn(0x7fbd4015b700 0x7fbd400ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd4621e640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 0x7fbd4006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983849552 0 0) 0x7fbd4015bae0 con 0x7fbd4006bce0 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2c003520 con 0x7fbd4006bce0 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036806277 0 0) 0x7fbd2c003520 con 0x7fbd4006bce0 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 msgr2=0x7fbd4006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 0x7fbd4006c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 >> v2:172.21.15.162:3300/0 conn(0x7fbd4015b700 msgr2=0x7fbd400ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 --2- 172.21.15.99:0/2686326464 >> v2:172.21.15.162:3300/0 conn(0x7fbd4015b700 0x7fbd400ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4015ee60 con 0x7fbd4006bce0 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fbd30001670 con 0x7fbd4006bce0 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 >> v1:172.21.15.99:6789/0 conn(0x7fbd4006bce0 legacy=0x7fbd4015e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 0x7fbd2c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd2c003f30 0x7fbd2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400adb90 con 0x7fbd2c003f30 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c00b9c0 con 0x7fbd2c004d40 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c00bb40 con 0x7fbd4015e810 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.081+0000 7fbd4521c640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd2c003f30 0x7fbd2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4621e640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd45a1d640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 0x7fbd2c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4521c640 1 -- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 msgr2=0x7fbd2c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4521c640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd4015e810 0x7fbd2c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4521c640 1 -- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd2c004d40 msgr2=0x7fbd2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4521c640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4521c640 1 -- 172.21.15.99:0/2686326464 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c003520 con 0x7fbd2c003f30 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4621e640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd2c004d40 0x7fbd2c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd4521c640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd2c003f30 0x7fbd2c004800 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbd30007fd0 tx=0x7fbd2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a059781c240075 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.082+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30007d50 con 0x7fbd2c003f30 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd30030070 con 0x7fbd2c003f30 2024-03-04T10:05:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd44a1b640 1 -- 172.21.15.99:0/2686326464 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3002cd20 con 0x7fbd2c003f30 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd47c81640 1 -- 172.21.15.99:0/2686326464 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd2c003f30 msgr2=0x7fbd2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd47c81640 1 --2- 172.21.15.99:0/2686326464 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd2c003f30 0x7fbd2c004800 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbd30007fd0 tx=0x7fbd2c005830 comp rx=0 tx=0).stop 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd4621e640 1 -- 172.21.15.99:0/2686326464 reap_dead start 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd47c81640 1 -- 172.21.15.99:0/2686326464 shutdown_connections 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd47c81640 1 -- 172.21.15.99:0/2686326464 >> 172.21.15.99:0/2686326464 conn(0x7fbd40071bc0 msgr2=0x7fbd40071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd47c81640 1 -- 172.21.15.99:0/2686326464 shutdown_connections 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.083+0000 7fbd47c81640 1 -- 172.21.15.99:0/2686326464 wait complete. 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.084+0000 7fbd47c81640 1 Processor -- start 2024-03-04T10:05:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.084+0000 7fbd47c81640 1 -- start start 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd47c81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd2c003f30 0x7fbd40148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd47c81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd40149730 0x7fbd400cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd47c81640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 0x7fbd4006f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd47c81640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400ae560 con 0x7fbd4006ec70 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd47c81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400ad820 con 0x7fbd40149730 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd47c81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400ae6e0 con 0x7fbd2c003f30 2024-03-04T10:05:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 0x7fbd4006f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 0x7fbd4006f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41962/0 (socket says 172.21.15.99:41962) 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 -- 172.21.15.99:0/1805075303 learned_addr learned my addr 172.21.15.99:0/1805075303 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd45a1d640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd2c003f30 0x7fbd40148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4521c640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd40149730 0x7fbd400cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 -- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd2c003f30 msgr2=0x7fbd40148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd2c003f30 0x7fbd40148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 -- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd40149730 msgr2=0x7fbd400cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd40149730 0x7fbd400cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 -- 172.21.15.99:0/1805075303 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd400cdd10 con 0x7fbd4006ec70 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4521c640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd40149730 0x7fbd400cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.085+0000 7fbd4621e640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 0x7fbd4006f100 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbd400add10 tx=0x7fbd38010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.086+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3801a030 con 0x7fbd4006ec70 2024-03-04T10:05:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.086+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd38010d80 con 0x7fbd4006ec70 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.086+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38003bd0 con 0x7fbd4006ec70 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.086+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd401e4180 con 0x7fbd4006ec70 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.086+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd400ce290 con 0x7fbd4006ec70 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.087+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd40074f60 con 0x7fbd4006ec70 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.087+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbd38005d10 con 0x7fbd4006ec70 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.087+0000 7fbd367fc640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd2406b500 0x7fbd2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.087+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fbd3808a920 con 0x7fbd4006ec70 2024-03-04T10:05:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.088+0000 7fbd45a1d640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd2406b500 0x7fbd2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.088+0000 7fbd45a1d640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd2406b500 0x7fbd2406d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009a90 tx=0x7fbd3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.092+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbd38025020 con 0x7fbd4006ec70 2024-03-04T10:05:09.106 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:09.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:09.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.263+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd400722b0 con 0x7fbd4006ec70 2024-03-04T10:05:09.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.264+0000 7fbd367fc640 1 -- 172.21.15.99:0/1805075303 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd38059650 con 0x7fbd4006ec70 2024-03-04T10:05:09.265 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:05:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd2406b500 msgr2=0x7fbd2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd2406b500 0x7fbd2406d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009a90 tx=0x7fbd3c008040 comp rx=0 tx=0).stop 2024-03-04T10:05:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 msgr2=0x7fbd4006f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 0x7fbd4006f100 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbd400add10 tx=0x7fbd38010590 comp rx=0 tx=0).stop 2024-03-04T10:05:09.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 shutdown_connections 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbd2406b500 0x7fbd2406d9c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbd4006ec70 0x7fbd4006f100 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbd40149730 0x7fbd400cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 --2- 172.21.15.99:0/1805075303 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbd2c003f30 0x7fbd40148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 >> 172.21.15.99:0/1805075303 conn(0x7fbd40071bc0 msgr2=0x7fbd40073910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 shutdown_connections 2024-03-04T10:05:09.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.266+0000 7fbd47c81640 1 -- 172.21.15.99:0/1805075303 wait complete. 2024-03-04T10:05:09.277 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:05:09.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.377+0000 7fb3d34b8640 1 Processor -- start 2024-03-04T10:05:09.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.377+0000 7fb3d34b8640 1 -- start start 2024-03-04T10:05:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d34b8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb3cc15b030 0x7fb3cc15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d34b8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3cc06d3b0 0x7fb3cc15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d34b8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d34b8640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc0b2720 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d34b8640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3cc15e2d0 con 0x7fb3cc15b030 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d34b8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc15bde0 con 0x7fb3cc15b410 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3cc06d3b0 0x7fb3cc15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb3cc06d3b0 0x7fb3cc15dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41966/0 (socket says 172.21.15.99:41966) 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 -- 172.21.15.99:0/3673382398 learned_addr learned my addr 172.21.15.99:0/3673382398 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d1254640 1 --2- 172.21.15.99:0/3673382398 >> v2:172.21.15.162:3300/0 conn(0x7fb3cc15b030 0x7fb3cc15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d1a55640 1 --2- 172.21.15.99:0/3673382398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 -- 172.21.15.99:0/3673382398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 msgr2=0x7fb3cc15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/3673382398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc15b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 -- 172.21.15.99:0/3673382398 >> v2:172.21.15.162:3300/0 conn(0x7fb3cc15b030 msgr2=0x7fb3cc15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/3673382398 >> v2:172.21.15.162:3300/0 conn(0x7fb3cc15b030 0x7fb3cc15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.378+0000 7fb3d0a53640 1 -- 172.21.15.99:0/3673382398 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc06d8b0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.379+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/3673382398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc15dd90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0725c0 tx=0x7fb3bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258a67d2ad236427 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.379+0000 7fb3c3fff640 1 -- 172.21.15.99:0/3673382398 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc02e070 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.379+0000 7fb3c3fff640 1 -- 172.21.15.99:0/3673382398 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc037070 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.379+0000 7fb3c3fff640 1 -- 172.21.15.99:0/3673382398 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc031a00 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 -- 172.21.15.99:0/3673382398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 msgr2=0x7fb3cc15dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/3673382398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc15dd90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0725c0 tx=0x7fb3bc02b880 comp rx=0 tx=0).stop 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 -- 172.21.15.99:0/3673382398 shutdown_connections 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/3673382398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/3673382398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc15dd90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/3673382398 >> v2:172.21.15.162:3300/0 conn(0x7fb3cc15b030 0x7fb3cc15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 -- 172.21.15.99:0/3673382398 >> 172.21.15.99:0/3673382398 conn(0x7fb3cc06acd0 msgr2=0x7fb3cc06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 -- 172.21.15.99:0/3673382398 shutdown_connections 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.380+0000 7fb3d34b8640 1 -- 172.21.15.99:0/3673382398 wait complete. 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.381+0000 7fb3d34b8640 1 Processor -- start 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.381+0000 7fb3d34b8640 1 -- start start 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3cc15b030 0x7fb3cc071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc06e150 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc06de70 con 0x7fb3cc15b030 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3cc06e690 con 0x7fb3cc15b410 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc14e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41968/0 (socket says 172.21.15.99:41968) 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 -- 172.21.15.99:0/733389727 learned_addr learned my addr 172.21.15.99:0/733389727 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3cc15b030 0x7fb3cc071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1a55640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 -- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 msgr2=0x7fb3cc071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 -- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3cc15b030 msgr2=0x7fb3cc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3cc15b030 0x7fb3cc071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 -- 172.21.15.99:0/733389727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0c9150 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3cc15b030 0x7fb3cc071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d1254640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc14e4b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400d800 tx=0x7fb3c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4018030 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c40096a0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4009870 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.382+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3cc0c93f0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.383+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3cc0c99b0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.384+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3cc074f60 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.384+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb3c401b070 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.384+0000 7fb3c1ffb640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b006b4d0 0x7fb3b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.384+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb3c408a5b0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.385+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b006b4d0 0x7fb3b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.385+0000 7fb3d0a53640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b006b4d0 0x7fb3b006d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc036000 tx=0x7fb3bc03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.389 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.389+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb3c40592e0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.574+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb3cc0b0b70 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.574+0000 7fb3c1ffb640 1 -- 172.21.15.99:0/733389727 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v39) v1 ==== 80+0+120 (secure 0 0 0) 0x7fb3c4058be0 con 0x7fb3cc06d3b0 2024-03-04T10:05:09.576 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6835/434536114 2024-03-05T10:05:00.614507+0000 2024-03-04T10:05:09.576 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6833/434536114 2024-03-05T10:05:00.614507+0000 2024-03-04T10:05:09.576 INFO:teuthology.orchestra.run.smithi099.stderr:listed 2 entries 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b006b4d0 msgr2=0x7fb3b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b006b4d0 0x7fb3b006d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc036000 tx=0x7fb3bc03d040 comp rx=0 tx=0).stop 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 msgr2=0x7fb3cc14e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc14e4b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400d800 tx=0x7fb3c400dcc0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 shutdown_connections 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b006b4d0 0x7fb3b006d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3cc15b410 0x7fb3cc071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3cc15b030 0x7fb3cc071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 --2- 172.21.15.99:0/733389727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb3cc06d3b0 0x7fb3cc14e4b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 >> 172.21.15.99:0/733389727 conn(0x7fb3cc06acd0 msgr2=0x7fb3cc06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 shutdown_connections 2024-03-04T10:05:09.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.576+0000 7fb3d34b8640 1 -- 172.21.15.99:0/733389727 wait complete. 2024-03-04T10:05:09.588 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:05:09.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:09.588 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:09.618 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:05:09.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:09.619 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:05:09.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:09.674 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub start /subdir repair 2024-03-04T10:05:09.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.816+0000 7f89fe4f1640 1 Processor -- start 2024-03-04T10:05:09.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.816+0000 7f89fe4f1640 1 -- start start 2024-03-04T10:05:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89fe4f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89f815de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89fe4f1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80cf090 con 0x7f89f80cc9b0 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89fe4f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80cb910 con 0x7f89f813cf30 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89fe4f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80cc520 con 0x7f89f8072300 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89fca8e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f89f80cc9b0 0x7f89f815e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51284/0 (socket says 172.21.15.99:51284) 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89fca8e640 1 -- 172.21.15.99:0/1053184481 learned_addr learned my addr 172.21.15.99:0/1053184481 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89f77fe640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89f815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117696697 0 0) 0x7f89f80cf090 con 0x7f89f80cc9b0 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.817+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e0003520 con 0x7f89f80cc9b0 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710341901 0 0) 0x7f89f80cb910 con 0x7f89f813cf30 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f80cf090 con 0x7f89f813cf30 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 524598060 0 0) 0x7f89e0003520 con 0x7f89f80cc9b0 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 msgr2=0x7f89f815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89f815de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 >> v1:172.21.15.162:6789/0 conn(0x7f89f813cf30 legacy=0x7f89f8071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f80cbbf0 con 0x7f89f80cc9b0 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f89f00019f0 con 0x7f89f80cc9b0 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 >> v1:172.21.15.99:6789/0 conn(0x7f89f80cc9b0 legacy=0x7f89f815e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89e0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89e0004120 0x7f89e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e0004d40 0x7f89e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80cc520 con 0x7f89e0004120 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e000bb00 con 0x7f89e0004d40 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e000bd00 con 0x7f89f8072300 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f77fe640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89e0004120 0x7f89e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f7fff640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89e0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89fca8e640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e0004d40 0x7f89e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f77fe640 1 -- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 msgr2=0x7f89e0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f77fe640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89e0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f77fe640 1 -- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e0004d40 msgr2=0x7f89e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f77fe640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89e0004d40 0x7f89e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f77fe640 1 -- 172.21.15.99:0/1053184481 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e0003520 con 0x7f89e0004120 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.818+0000 7f89f7fff640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89f8072300 0x7f89e0003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.819+0000 7f89f77fe640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89e0004120 0x7f89e0004800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009ac0 tx=0x7f89e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7155479f664f0f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.819+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec039070 con 0x7f89e0004120 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.819+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec02c600 con 0x7f89e0004120 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.819+0000 7f89f6ffd640 1 -- 172.21.15.99:0/1053184481 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec009690 con 0x7f89e0004120 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.819+0000 7f89fe4f1640 1 -- 172.21.15.99:0/1053184481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89e0004120 msgr2=0x7f89e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.819+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/1053184481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89e0004120 0x7f89e0004800 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f89ec009ac0 tx=0x7f89e0005830 comp rx=0 tx=0).stop 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.820+0000 7f89fca8e640 1 -- 172.21.15.99:0/1053184481 reap_dead start 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.820+0000 7f89fe4f1640 1 -- 172.21.15.99:0/1053184481 shutdown_connections 2024-03-04T10:05:09.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.820+0000 7f89fe4f1640 1 -- 172.21.15.99:0/1053184481 >> 172.21.15.99:0/1053184481 conn(0x7f89f80b1600 msgr2=0x7f89f80b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.820+0000 7f89fe4f1640 1 -- 172.21.15.99:0/1053184481 shutdown_connections 2024-03-04T10:05:09.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.820+0000 7f89fe4f1640 1 -- 172.21.15.99:0/1053184481 wait complete. 2024-03-04T10:05:09.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.821+0000 7f89fe4f1640 1 Processor -- start 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.821+0000 7f89fe4f1640 1 -- start start 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89e0004120 0x7f89f81f7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 0x7f89f8159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89f814dd90 0x7f89f814e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f813d310 con 0x7f89f8132480 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80cc1b0 con 0x7f89f814dd90 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f813d490 con 0x7f89e0004120 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 0x7f89f8159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 0x7f89f8159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41974/0 (socket says 172.21.15.99:41974) 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 -- 172.21.15.99:0/3936839420 learned_addr learned my addr 172.21.15.99:0/3936839420 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 -- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89e0004120 msgr2=0x7f89f81f7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89e0004120 0x7f89f81f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 -- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89f814dd90 msgr2=0x7f89f814e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89f814dd90 0x7f89f814e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 -- 172.21.15.99:0/3936839420 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f81f8090 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f77fe640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 0x7f89f8159a90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f89ec037040 tx=0x7f89ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f4ff9640 1 -- 172.21.15.99:0/3936839420 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec042020 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f4ff9640 1 -- 172.21.15.99:0/3936839420 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec03a040 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f814e750 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89f4ff9640 1 -- 172.21.15.99:0/3936839420 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec039070 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.822+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f81f7ad0 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.824+0000 7f89f4ff9640 1 -- 172.21.15.99:0/3936839420 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f89ec045020 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.824+0000 7f89f4ff9640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89e406b500 0x7f89e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.824+0000 7f89f4ff9640 1 -- 172.21.15.99:0/3936839420 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f89ec0ab740 con 0x7f89f8132480 2024-03-04T10:05:09.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.824+0000 7f89f7fff640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89e406b500 0x7f89e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.825+0000 7f89f7fff640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89e406b500 0x7f89e406d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f89e8007ae0 tx=0x7f89e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.825+0000 7f8a011ac740 1 Processor -- start 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.825+0000 7f8a011ac740 1 -- start start 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e386c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 0x563de1e38c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f8a011ac740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de1d721e0 con 0x563de1e08e10 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de1d1aca0 con 0x563de1e35d10 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de1d6e280 con 0x563de1e35930 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 0x563de1e38c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 0x563de1e38c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41990/0 (socket says 172.21.15.99:41990) 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 -- 172.21.15.99:0/2174580811 learned_addr learned my addr 172.21.15.99:0/2174580811 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89f7fff640 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89f77fe640 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 -- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 msgr2=0x563de1e088d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e088d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 -- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 msgr2=0x563de1e386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e386c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 -- 172.21.15.99:0/2174580811 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de1b52a20 con 0x563de1e08e10 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89fca8e640 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 0x563de1e38c00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f89f815ec00 tx=0x7f89f0005f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cd51751c08b580 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89ddffb640 1 -- 172.21.15.99:0/2174580811 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0007760 con 0x563de1e08e10 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89ddffb640 1 -- 172.21.15.99:0/2174580811 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0007900 con 0x563de1e08e10 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.826+0000 7f89ddffb640 1 -- 172.21.15.99:0/2174580811 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0014670 con 0x563de1e08e10 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 -- 172.21.15.99:0/2174580811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 msgr2=0x563de1e38c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 0x563de1e38c00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f89f815ec00 tx=0x7f89f0005f40 comp rx=0 tx=0).stop 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 -- 172.21.15.99:0/2174580811 shutdown_connections 2024-03-04T10:05:09.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e08e10 0x563de1e38c00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2174580811 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 -- 172.21.15.99:0/2174580811 >> 172.21.15.99:0/2174580811 conn(0x563de1e0ab40 msgr2=0x563de1e536c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 -- 172.21.15.99:0/2174580811 shutdown_connections 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.827+0000 7f8a011ac740 1 -- 172.21.15.99:0/2174580811 wait complete. 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.828+0000 7f8a011ac740 1 -- start start 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.828+0000 7f8a011ac740 1 Processor -- start 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 0x563de1e33170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e53840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e336b0 0x563de1e53d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de1e0d1e0 con 0x563de1e35930 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de1d2e260 con 0x563de1e35d10 2024-03-04T10:05:09.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de1e0fa20 con 0x563de1e336b0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 10 client.0 _ll_register_callbacks cb 0x563de1d6b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 0x563de1e33170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 0x563de1e33170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41994/0 (socket says 172.21.15.99:41994) 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 -- 172.21.15.99:0/2245310983 learned_addr learned my addr 172.21.15.99:0/2245310983 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f77fe640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e53840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89fca8e640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e336b0 0x563de1e53d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 -- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e336b0 msgr2=0x563de1e53d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e336b0 0x563de1e53d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 -- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 msgr2=0x563de1e53840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e53840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 -- 172.21.15.99:0/2245310983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de1a46310 con 0x563de1e35930 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f77fe640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e53840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89f7fff640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 0x563de1e33170 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f89e800ea60 tx=0x7f89e8007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75ded6511a41779 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:09.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89d6ffd640 1 -- 172.21.15.99:0/2245310983 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8027060 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89d6ffd640 1 -- 172.21.15.99:0/2245310983 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f89e8010040 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89f0007900 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f89d6ffd640 1 -- 172.21.15.99:0/2245310983 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89e8003d80 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 10 client.5685 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.829+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x563de1bb31b0 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f89d6ffd640 1 -- 172.21.15.99:0/2245310983 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0007900 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f89d6ffd640 1 -- 172.21.15.99:0/2245310983 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f89e8013420 con 0x563de1e35930 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f8a011ac740 10 client.5685 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f8a011ac740 10 client.5685 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f8a011ac740 20 client.5685 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:09.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e6d9c0 0x563de1e6fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f8a011ac740 4 client.5685 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563de1d7f720 con 0x563de1e6d9c0 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f89f77fe640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e6d9c0 0x563de1e6fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f89f77fe640 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e6d9c0 0x563de1e6fdb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d5f6014abf605 server_cookie=2b3a49fc52b6067b in_seq=0 out_seq=0 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.830+0000 7f89d6ffd640 10 client.5685 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f89d6ffd640 1 -- 172.21.15.99:0/2245310983 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563de1d7f720 con 0x563de1e6d9c0 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f89d6ffd640 10 client.5685 handle_command_reply: tid=0 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 client.5685 shutdown 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 msgr2=0x563de1e33170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 0x563de1e33170 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f89e800ea60 tx=0x7f89e8007a50 comp rx=0 tx=0).stop 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 shutdown_connections 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e336b0 0x563de1e53d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e6d9c0 0x563de1e6fdb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e35d10 0x563de1e53840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 --2- 172.21.15.99:0/2245310983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35930 0x563de1e33170 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 >> 172.21.15.99:0/2245310983 conn(0x563de1e08e10 msgr2=0x563de1e55aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f89d6ffd640 0 client.5685 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 shutdown_connections 2024-03-04T10:05:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 1 -- 172.21.15.99:0/2245310983 wait complete. 2024-03-04T10:05:09.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.831+0000 7f8a011ac740 20 client.5685 trim_cache size 0 max 16384 2024-03-04T10:05:09.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.846+0000 7f8a011ac740 1 Processor -- start 2024-03-04T10:05:09.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.846+0000 7f8a011ac740 1 -- start start 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1e09ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e74490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e56b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f8a011ac740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f8200f40 con 0x563de1e35d10 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f8200060 con 0x563de1e336b0 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f81ff4c0 con 0x563de1e35930 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e56b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e56b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42000/0 (socket says 172.21.15.99:42000) 2024-03-04T10:05:09.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 -- 172.21.15.99:0/3110682720 learned_addr learned my addr 172.21.15.99:0/3110682720 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89f7fff640 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1e09ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89f77fe640 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e74490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 -- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 msgr2=0x563de1e74490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e74490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 -- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 msgr2=0x563de1e09ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1e09ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 -- 172.21.15.99:0/3110682720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563de1a1f490 con 0x563de1e35d10 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89f7fff640 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1e09ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:09.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89fca8e640 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e56b80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f89f00078f0 tx=0x7f89f000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a926ccdfc384c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89dd7fa640 1 -- 172.21.15.99:0/3110682720 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00079d0 con 0x563de1e35d10 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89dd7fa640 1 -- 172.21.15.99:0/3110682720 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0014430 con 0x563de1e35d10 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.847+0000 7f89dd7fa640 1 -- 172.21.15.99:0/3110682720 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0014820 con 0x563de1e35d10 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 -- 172.21.15.99:0/3110682720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 msgr2=0x563de1e56b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e56b80 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f89f00078f0 tx=0x7f89f000e000 comp rx=0 tx=0).stop 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 -- 172.21.15.99:0/3110682720 shutdown_connections 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e56b80 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e74490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 --2- 172.21.15.99:0/3110682720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1e09ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 -- 172.21.15.99:0/3110682720 >> 172.21.15.99:0/3110682720 conn(0x563de1e08e10 msgr2=0x563de1e0e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 -- 172.21.15.99:0/3110682720 shutdown_connections 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.848+0000 7f8a011ac740 1 -- 172.21.15.99:0/3110682720 wait complete. 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.849+0000 7f8a011ac740 1 -- start start 2024-03-04T10:05:09.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.849+0000 7f8a011ac740 1 Processor -- start 2024-03-04T10:05:09.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1ea76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e33d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e6b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563de1e66090 con 0x563de1e35d10 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f81ffa60 con 0x563de1e336b0 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563de1d7e860 con 0x563de1e35930 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 10 client.0 _ll_register_callbacks cb 0x7f89f0001350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e6b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e6b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42006/0 (socket says 172.21.15.99:42006) 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 -- 172.21.15.99:0/769852270 learned_addr learned my addr 172.21.15.99:0/769852270 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89f7fff640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1ea76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89f77fe640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e33d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 -- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 msgr2=0x563de1e33d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e33d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 -- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 msgr2=0x563de1ea76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1ea76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 -- 172.21.15.99:0/769852270 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563de1e6aad0 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89fca8e640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e6b570 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f89f0007920 tx=0x7f89f0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd155213d13206c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89d6ffd640 1 -- 172.21.15.99:0/769852270 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0014b70 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89d6ffd640 1 -- 172.21.15.99:0/769852270 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f89f0014d10 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89f0014430 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f89d6ffd640 1 -- 172.21.15.99:0/769852270 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89f0030050 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 10 client.5691 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.850+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x563de1e6b050 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f89d6ffd640 1 -- 172.21.15.99:0/769852270 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f0014430 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f89d6ffd640 1 -- 172.21.15.99:0/769852270 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f89f0033020 con 0x563de1e35d10 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f8a011ac740 10 client.5691 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f8a011ac740 10 client.5691 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-03-04T10:05:09.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f8a011ac740 20 client.5691 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f8a011ac740 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e50df0 0x563de1e0e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f8a011ac740 4 client.5691 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}] 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair"]}) v1 -- 0x563de1d79a70 con 0x563de1e50df0 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f89f7fff640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e50df0 0x563de1e0e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f89d6ffd640 10 client.5691 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.851+0000 7f89f7fff640 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e50df0 0x563de1e0e5e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddf27705bb01ede3 server_cookie=72a372a4c6eb5088 in_seq=0 out_seq=0 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89d6ffd640 1 -- 172.21.15.99:0/769852270 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563de1d79a70 con 0x563de1e50df0 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89d6ffd640 10 client.5691 handle_command_reply: tid=0 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 client.5691 shutdown 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 msgr2=0x563de1e6b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e6b570 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f89f0007920 tx=0x7f89f0001e80 comp rx=0 tx=0).stop 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 shutdown_connections 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563de1e35d10 0x563de1e6b570 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563de1e35930 0x563de1e33d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x563de1e50df0 0x563de1e0e5e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 --2- 172.21.15.99:0/769852270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563de1e336b0 0x563de1ea76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 >> 172.21.15.99:0/769852270 conn(0x563de1e08e10 msgr2=0x563de1e38100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89d6ffd640 0 client.5691 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 shutdown_connections 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 1 -- 172.21.15.99:0/769852270 wait complete. 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f8a011ac740 20 client.5691 trim_cache size 0 max 16384 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0, 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stdout: "scrub_tag": "a60bad1a-9109-4ae3-bcb4-b7aed5c2d66a", 2024-03-04T10:05:09.860 INFO:teuthology.orchestra.run.smithi099.stdout: "mode": "asynchronous" 2024-03-04T10:05:09.861 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:05:09.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89e406b500 msgr2=0x7f89e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89e406b500 0x7f89e406d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f89e8007ae0 tx=0x7f89e8009040 comp rx=0 tx=0).stop 2024-03-04T10:05:09.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 msgr2=0x7f89f8159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.852+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 0x7f89f8159a90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f89ec037040 tx=0x7f89ec004730 comp rx=0 tx=0).stop 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 shutdown_connections 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89e406b500 0x7f89e406d9c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89f814dd90 0x7f89f814e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f89f8132480 0x7f89f8159a90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 --2- 172.21.15.99:0/3936839420 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89e0004120 0x7f89f81f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 >> 172.21.15.99:0/3936839420 conn(0x7f89f80b1600 msgr2=0x7f89f815dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 shutdown_connections 2024-03-04T10:05:09.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.853+0000 7f89fe4f1640 1 -- 172.21.15.99:0/3936839420 wait complete. 2024-03-04T10:05:09.864 INFO:tasks.cephfs.test_damage:{ "return_code": 0, "scrub_tag": "a60bad1a-9109-4ae3-bcb4-b7aed5c2d66a", "mode": "asynchronous" } 2024-03-04T10:05:09.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:09.864 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-03-04T10:05:09.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.965+0000 7f0a51da2640 1 Processor -- start 2024-03-04T10:05:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.965+0000 7f0a51da2640 1 -- start start 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.966+0000 7f0a51da2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0a4c0d00f0 0x7f0a4c0cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.966+0000 7f0a51da2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4c0cc650 con 0x7f0a4c099a00 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.966+0000 7f0a51da2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4c0cc930 con 0x7f0a4c0cdc30 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.966+0000 7f0a51da2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4c0cc3f0 con 0x7f0a4c0d00f0 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.966+0000 7f0a50da0640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0a4c099a00 0x7f0a4c099de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:51286/0 (socket says 172.21.15.99:51286) 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.966+0000 7f0a50da0640 1 -- 172.21.15.99:0/926323699 learned_addr learned my addr 172.21.15.99:0/926323699 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/926323699 >> v2:172.21.15.162:3300/0 conn(0x7f0a4c0d00f0 0x7f0a4c0cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4b7fe640 1 -- 172.21.15.99:0/926323699 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215495860 0 0) 0x7f0a4c0cc650 con 0x7f0a4c099a00 2024-03-04T10:05:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4b7fe640 1 -- 172.21.15.99:0/926323699 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a34003520 con 0x7f0a4c099a00 2024-03-04T10:05:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4b7fe640 1 -- 172.21.15.99:0/926323699 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528858535 0 0) 0x7f0a34003520 con 0x7f0a4c099a00 2024-03-04T10:05:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4bfff640 1 -- 172.21.15.99:0/926323699 >> v1:172.21.15.162:6789/0 conn(0x7f0a4c0cdc30 legacy=0x7f0a4c0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4bfff640 1 -- 172.21.15.99:0/926323699 >> v1:172.21.15.99:6789/0 conn(0x7f0a4c099a00 legacy=0x7f0a4c099de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.967+0000 7f0a4bfff640 1 -- 172.21.15.99:0/926323699 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0ccc10 con 0x7f0a4c0d00f0 2024-03-04T10:05:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/926323699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0cd6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097a0 tx=0x7f0a4002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f3c2a9ad132037f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a4b7fe640 1 -- 172.21.15.99:0/926323699 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40038070 con 0x7f0a4c0d00f0 2024-03-04T10:05:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a4b7fe640 1 -- 172.21.15.99:0/926323699 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a40031500 con 0x7f0a4c0d00f0 2024-03-04T10:05:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a4b7fe640 1 -- 172.21.15.99:0/926323699 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c001670 con 0x7f0a4c0d00f0 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 -- 172.21.15.99:0/926323699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 msgr2=0x7f0a4c0cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 --2- 172.21.15.99:0/926323699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0cd6f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097a0 tx=0x7f0a4002b820 comp rx=0 tx=0).stop 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 -- 172.21.15.99:0/926323699 shutdown_connections 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 --2- 172.21.15.99:0/926323699 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0cd6f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 -- 172.21.15.99:0/926323699 >> 172.21.15.99:0/926323699 conn(0x7f0a4c0b30f0 msgr2=0x7f0a4c0b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 -- 172.21.15.99:0/926323699 shutdown_connections 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.968+0000 7f0a51da2640 1 -- 172.21.15.99:0/926323699 wait complete. 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 Processor -- start 2024-03-04T10:05:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 -- start start 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 0x7f0a4c0b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a4c0cdc30 0x7f0a4c0c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0cf360 con 0x7f0a4c099a00 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0cd1d0 con 0x7f0a4c0d00f0 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.970+0000 7f0a51da2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c0cf4e0 con 0x7f0a4c0cdc30 2024-03-04T10:05:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 0x7f0a4c0b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 0x7f0a4c0b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42016/0 (socket says 172.21.15.99:42016) 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 -- 172.21.15.99:0/3662984655 learned_addr learned my addr 172.21.15.99:0/3662984655 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a515a1640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 -- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a4c0cdc30 msgr2=0x7f0a4c0c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a4c0cdc30 0x7f0a4c0c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 -- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 msgr2=0x7f0a4c0c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 -- 172.21.15.99:0/3662984655 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c0b5fd0 con 0x7f0a4c099a00 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a50da0640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 0x7f0a4c0b59d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00c760 tx=0x7f0a3c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a497fa640 1 -- 172.21.15.99:0/3662984655 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c017070 con 0x7f0a4c099a00 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a497fa640 1 -- 172.21.15.99:0/3662984655 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c005af0 con 0x7f0a4c099a00 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4c0b62b0 con 0x7f0a4c099a00 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a497fa640 1 -- 172.21.15.99:0/3662984655 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c007070 con 0x7f0a4c099a00 2024-03-04T10:05:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.971+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4c0c7e80 con 0x7f0a4c099a00 2024-03-04T10:05:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.973+0000 7f0a497fa640 1 -- 172.21.15.99:0/3662984655 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0a3c01a070 con 0x7f0a4c099a00 2024-03-04T10:05:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.973+0000 7f0a497fa640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a3806b4d0 0x7f0a3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.973+0000 7f0a497fa640 1 -- 172.21.15.99:0/3662984655 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0a3c0899a0 con 0x7f0a4c099a00 2024-03-04T10:05:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.974+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a3806b4d0 0x7f0a3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.974+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a3806b4d0 0x7f0a3806d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0a40035000 tx=0x7f0a40030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.974+0000 7f0a55be3740 1 Processor -- start 2024-03-04T10:05:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.974+0000 7f0a55be3740 1 -- start start 2024-03-04T10:05:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c0fa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c12a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c0faf10 0x55d15c12ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a55be3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d15c063e00 con 0x55d15c127e10 2024-03-04T10:05:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d15c00cb10 con 0x55d15c127a30 2024-03-04T10:05:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d15c05fea0 con 0x55d15c0faf10 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c12a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c12a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42028/0 (socket says 172.21.15.99:42028) 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 -- 172.21.15.99:0/3114835460 learned_addr learned my addr 172.21.15.99:0/3114835460 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a50da0640 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c0fa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a515a1640 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c0faf10 0x55d15c12ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 -- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c0faf10 msgr2=0x55d15c12ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c0faf10 0x55d15c12ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 -- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 msgr2=0x55d15c0fa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c0fa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.975+0000 7f0a4bfff640 1 -- 172.21.15.99:0/3114835460 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d15be427d0 con 0x55d15c127e10 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c12a7c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0a400098c0 tx=0x7f0a40031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc589fe3cb99ace server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a327fc640 1 -- 172.21.15.99:0/3114835460 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40038040 con 0x55d15c127e10 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a327fc640 1 -- 172.21.15.99:0/3114835460 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4004d070 con 0x55d15c127e10 2024-03-04T10:05:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a327fc640 1 -- 172.21.15.99:0/3114835460 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4003dbb0 con 0x55d15c127e10 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 -- 172.21.15.99:0/3114835460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 msgr2=0x55d15c12a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c12a7c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0a400098c0 tx=0x7f0a40031000 comp rx=0 tx=0).stop 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 -- 172.21.15.99:0/3114835460 shutdown_connections 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c0faf10 0x55d15c12ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c12a7c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 --2- 172.21.15.99:0/3114835460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c0fa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 -- 172.21.15.99:0/3114835460 >> 172.21.15.99:0/3114835460 conn(0x55d15c0fcc40 msgr2=0x55d15c1457c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 -- 172.21.15.99:0/3114835460 shutdown_connections 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.976+0000 7f0a55be3740 1 -- 172.21.15.99:0/3114835460 wait complete. 2024-03-04T10:05:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.977+0000 7f0a55be3740 1 -- start start 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.977+0000 7f0a55be3740 1 Processor -- start 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c145db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c117ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c117ff0 0x55d15c116910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d15c0ff1e0 con 0x55d15c127e10 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d15c0200a0 con 0x55d15c127a30 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d15c0fe5d0 con 0x55d15c117ff0 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a55be3740 10 client.0 _ll_register_callbacks cb 0x55d15c01fa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c117ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c117ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42038/0 (socket says 172.21.15.99:42038) 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 -- 172.21.15.99:0/1622478505 learned_addr learned my addr 172.21.15.99:0/1622478505 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a515a1640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c117ff0 0x55d15c116910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 -- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c117ff0 msgr2=0x55d15c116910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c117ff0 0x55d15c116910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 -- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 msgr2=0x55d15c145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 -- 172.21.15.99:0/1622478505 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d15bd783f0 con 0x55d15c127e10 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a515a1640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c117ff0 0x55d15c116910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c117ab0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0a4002be30 tx=0x7f0a40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343f15866240c545 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.978+0000 7f0a2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1622478505 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40038040 con 0x55d15c127e10 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1622478505 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0a4005d480 con 0x55d15c127e10 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a4004d070 con 0x55d15c127e10 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1622478505 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a40065050 con 0x55d15c127e10 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 10 client.5703 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55d15bd380f0 con 0x55d15c127e10 2024-03-04T10:05:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1622478505 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4004d070 con 0x55d15c127e10 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1622478505 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0a40068020 con 0x55d15c127e10 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 10 client.5703 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 10 client.5703 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 20 client.5703 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c12d980 0x55d15c117320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 4 client.5703 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d15c071340 con 0x55d15c12d980 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c12d980 0x55d15c117320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a2b7fe640 10 client.5703 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.979+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c12d980 0x55d15c117320 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f76898c30a6bc148 server_cookie=59f05072f8a37d18 in_seq=0 out_seq=0 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1622478505 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d15c071340 con 0x55d15c12d980 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a2b7fe640 10 client.5703 handle_command_reply: tid=0 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 client.5703 shutdown 2024-03-04T10:05:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 msgr2=0x55d15c117ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c117ab0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0a4002be30 tx=0x7f0a40004730 comp rx=0 tx=0).stop 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 shutdown_connections 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c117ff0 0x55d15c116910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c117ab0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c12d980 0x55d15c117320 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1622478505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127a30 0x55d15c145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a2b7fe640 0 client.5703 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 >> 172.21.15.99:0/1622478505 conn(0x55d15c0faf10 msgr2=0x55d15c12a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 shutdown_connections 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 1 -- 172.21.15.99:0/1622478505 wait complete. 2024-03-04T10:05:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.980+0000 7f0a55be3740 20 client.5703 trim_cache size 0 max 16384 2024-03-04T10:05:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.995+0000 7f0a55be3740 1 Processor -- start 2024-03-04T10:05:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.995+0000 7f0a55be3740 1 -- start start 2024-03-04T10:05:09.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 0x55d15c1268a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c100b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127e10 0x55d15c0f9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a55be3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c2002b0 con 0x55d15c117ff0 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c06d880 con 0x55d15c127e10 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c06d280 con 0x55d15c127a30 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 0x55d15c1268a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 0x55d15c1268a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42040/0 (socket says 172.21.15.99:42040) 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 -- 172.21.15.99:0/2605718625 learned_addr learned my addr 172.21.15.99:0/2605718625 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a515a1640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127e10 0x55d15c0f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 -- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 msgr2=0x55d15c100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 -- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127e10 msgr2=0x55d15c0f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:09.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127e10 0x55d15c0f9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 -- 172.21.15.99:0/2605718625 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d15c145c60 con 0x55d15c117ff0 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a515a1640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127e10 0x55d15c0f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a4bfff640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a50da0640 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 0x55d15c1268a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c054340 tx=0x7f0a3c05e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c29c947861630e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a31ffb640 1 -- 172.21.15.99:0/2605718625 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c056230 con 0x55d15c117ff0 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a31ffb640 1 -- 172.21.15.99:0/2605718625 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c086070 con 0x55d15c117ff0 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.996+0000 7f0a31ffb640 1 -- 172.21.15.99:0/2605718625 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c0601b0 con 0x55d15c117ff0 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 -- 172.21.15.99:0/2605718625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 msgr2=0x55d15c1268a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 0x55d15c1268a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c054340 tx=0x7f0a3c05e040 comp rx=0 tx=0).stop 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 -- 172.21.15.99:0/2605718625 shutdown_connections 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c127e10 0x55d15c0f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 --2- 172.21.15.99:0/2605718625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c117ff0 0x55d15c1268a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 -- 172.21.15.99:0/2605718625 >> 172.21.15.99:0/2605718625 conn(0x55d15c0faf10 msgr2=0x55d15c0fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 -- 172.21.15.99:0/2605718625 shutdown_connections 2024-03-04T10:05:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 -- 172.21.15.99:0/2605718625 wait complete. 2024-03-04T10:05:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.997+0000 7f0a55be3740 1 -- start start 2024-03-04T10:05:10.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 Processor -- start 2024-03-04T10:05:10.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c117ff0 0x55d15c198940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:10.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c0f9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c0f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d15c0fc3a0 con 0x55d15c127e10 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d15c196b00 con 0x55d15c117ff0 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d15c1173b0 con 0x55d15c127a30 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a55be3740 10 client.0 _ll_register_callbacks cb 0x7f0a3c054290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a515a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c0f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a515a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c0f9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:42050/0 (socket says 172.21.15.99:42050) 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.998+0000 7f0a515a1640 1 -- 172.21.15.99:0/1892073592 learned_addr learned my addr 172.21.15.99:0/1892073592 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c117ff0 0x55d15c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a515a1640 1 -- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 msgr2=0x55d15c0f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a515a1640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c0f9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a515a1640 1 -- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c117ff0 msgr2=0x55d15c198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a515a1640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c117ff0 0x55d15c198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a515a1640 1 -- 172.21.15.99:0/1892073592 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d15c117600 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c117ff0 0x55d15c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a515a1640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c0f9e60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0a44005950 tx=0x7f0a44001910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16579323459c0b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a2b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1892073592 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a44006b50 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1892073592 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0a44006cf0 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a3c086070 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1892073592 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a44016440 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 10 client.5709 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55d15c198f70 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1892073592 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c086070 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1892073592 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0a440165e0 con 0x55d15c127e10 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 10 client.5709 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 10 client.5709 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 20 client.5709 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:10.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c195d00 0x55d15c11aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 4 client.5709 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:09.999+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d15c06b690 con 0x55d15c195d00 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c195d00 0x55d15c11aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a2b7fe640 10 client.5709 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a50da0640 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c195d00 0x55d15c11aec0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d467229a2c8ee3c server_cookie=5fdc181555f35fa4 in_seq=0 out_seq=0 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a2b7fe640 1 -- 172.21.15.99:0/1892073592 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+256 (crc 0 0 0) 0x55d15c06b690 con 0x55d15c195d00 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a2b7fe640 10 client.5709 handle_command_reply: tid=0 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 client.5709 shutdown 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 msgr2=0x55d15c0f9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c0f9e60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0a44005950 tx=0x7f0a44001910 comp rx=0 tx=0).stop 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 shutdown_connections 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d15c127e10 0x55d15c0f9e60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d15c127a30 0x55d15c0f9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55d15c195d00 0x55d15c11aec0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 --2- 172.21.15.99:0/1892073592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d15c117ff0 0x55d15c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 >> 172.21.15.99:0/1892073592 conn(0x55d15c0faf10 msgr2=0x55d15c13f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a2b7fe640 0 client.5709 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 shutdown_connections 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 1 -- 172.21.15.99:0/1892073592 wait complete. 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.000+0000 7f0a55be3740 20 client.5709 trim_cache size 0 max 16384 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stdout: "scrubs": { 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stdout: "a60bad1a-9109-4ae3-bcb4-b7aed5c2d66a": { 2024-03-04T10:05:10.008 INFO:teuthology.orchestra.run.smithi099.stdout: "path": "/subdir", 2024-03-04T10:05:10.009 INFO:teuthology.orchestra.run.smithi099.stdout: "tag": "a60bad1a-9109-4ae3-bcb4-b7aed5c2d66a", 2024-03-04T10:05:10.009 INFO:teuthology.orchestra.run.smithi099.stdout: "options": "repair" 2024-03-04T10:05:10.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:10.009 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:10.009 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a3806b4d0 msgr2=0x7f0a3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a3806b4d0 0x7f0a3806d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0a40035000 tx=0x7f0a40030000 comp rx=0 tx=0).stop 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 msgr2=0x7f0a4c0b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 0x7f0a4c0b59d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c00c760 tx=0x7f0a3c00cc20 comp rx=0 tx=0).stop 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 shutdown_connections 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a3806b4d0 0x7f0a3806d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a4c0d00f0 0x7f0a4c0c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a4c0cdc30 0x7f0a4c0c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 --2- 172.21.15.99:0/3662984655 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a4c099a00 0x7f0a4c0b59d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 >> 172.21.15.99:0/3662984655 conn(0x7f0a4c0b30f0 msgr2=0x7f0a4c0b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 shutdown_connections 2024-03-04T10:05:10.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:10.001+0000 7f0a51da2640 1 -- 172.21.15.99:0/3662984655 wait complete. 2024-03-04T10:05:10.012 INFO:tasks.cephfs.filesystem:scrub status for tag:a60bad1a-9109-4ae3-bcb4-b7aed5c2d66a - {'path': '/subdir', 'tag': 'a60bad1a-9109-4ae3-bcb4-b7aed5c2d66a', 'options': 'repair'} 2024-03-04T10:05:39.278 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:39.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:05:40.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.014 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-03-04T10:05:40.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.127+0000 7fefe0473640 1 Processor -- start 2024-03-04T10:05:40.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.127+0000 7fefe0473640 1 -- start start 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefe0473640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fefd80ccfe0 0x7fefd815db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefe0473640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd806ae00 con 0x7fefd80cc900 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefe0473640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd806b0e0 con 0x7fefd80cdd70 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefe0473640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd80ccde0 con 0x7fefd80ccfe0 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdda0e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fefd80cc900 0x7fefd815d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46714/0 (socket says 172.21.15.99:46714) 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdda0e640 1 -- 172.21.15.99:0/3611276075 learned_addr learned my addr 172.21.15.99:0/3611276075 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781708752 0 0) 0x7fefd806ae00 con 0x7fefd80cc900 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc4003520 con 0x7fefd80cc900 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdea10640 1 --2- 172.21.15.99:0/3611276075 >> v2:172.21.15.162:3300/0 conn(0x7fefd80ccfe0 0x7fefd815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3386191419 0 0) 0x7fefc4003520 con 0x7fefd80cc900 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 >> v2:172.21.15.162:3300/0 conn(0x7fefd80ccfe0 msgr2=0x7fefd815db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 --2- 172.21.15.99:0/3611276075 >> v2:172.21.15.162:3300/0 conn(0x7fefd80ccfe0 0x7fefd815db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 >> v1:172.21.15.162:6789/0 conn(0x7fefd80cdd70 legacy=0x7fefd80cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd806b3c0 con 0x7fefd80cc900 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fefc8001670 con 0x7fefd80cc900 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 >> v1:172.21.15.99:6789/0 conn(0x7fefd80cc900 legacy=0x7fefd815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc40039b0 0x7fefc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefc4004800 0x7fefc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefc400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80ccde0 con 0x7fefc40051c0 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc400c160 con 0x7fefc40039b0 2024-03-04T10:05:40.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.128+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc400c360 con 0x7fefc4004800 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefc400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefde20f640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc40039b0 0x7fefc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefc4004800 0x7fefc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 -- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefc4004800 msgr2=0x7fefc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefc4004800 0x7fefc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 -- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc40039b0 msgr2=0x7fefc4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefc40039b0 0x7fefc4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 -- 172.21.15.99:0/3611276075 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc4003520 con 0x7fefc40051c0 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefc4004800 0x7fefc4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 -- 172.21.15.99:0/3611276075 reap_dead start 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.129+0000 7fefdea10640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefc400b9c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009ac0 tx=0x7fefc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7431ac390263b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd0009690 con 0x7fefc40051c0 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd0004020 con 0x7fefc40051c0 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefdd20d640 1 -- 172.21.15.99:0/3611276075 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd0004410 con 0x7fefc40051c0 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 -- 172.21.15.99:0/3611276075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 msgr2=0x7fefc400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefc400b9c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009ac0 tx=0x7fefc4005830 comp rx=0 tx=0).stop 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 -- 172.21.15.99:0/3611276075 shutdown_connections 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 --2- 172.21.15.99:0/3611276075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefc400b9c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 -- 172.21.15.99:0/3611276075 >> 172.21.15.99:0/3611276075 conn(0x7fefd80b22e0 msgr2=0x7fefd80b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 -- 172.21.15.99:0/3611276075 shutdown_connections 2024-03-04T10:05:40.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.130+0000 7fefe0473640 1 -- 172.21.15.99:0/3611276075 wait complete. 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.131+0000 7fefe0473640 1 Processor -- start 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.131+0000 7fefe0473640 1 -- start start 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefe0473640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefd8150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefe0473640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefd80c4330 0x7fefd8150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefe0473640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd80b3cc0 0x7fefd80b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefe0473640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80b3010 con 0x7fefc40051c0 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefe0473640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80b2e90 con 0x7fefd80b3cc0 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefe0473640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd80af950 con 0x7fefd80c4330 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefde20f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefd8150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefde20f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefd8150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54676/0 (socket says 172.21.15.99:54676) 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefde20f640 1 -- 172.21.15.99:0/3191452385 learned_addr learned my addr 172.21.15.99:0/3191452385 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefd80c4330 0x7fefd8150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.132+0000 7fefdea10640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd80b3cc0 0x7fefd80b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefde20f640 1 -- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefd80c4330 msgr2=0x7fefd8150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefde20f640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefd80c4330 0x7fefd8150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefde20f640 1 -- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd80b3cc0 msgr2=0x7fefd80b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefde20f640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd80b3cc0 0x7fefd80b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefde20f640 1 -- 172.21.15.99:0/3191452385 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd80c7460 con 0x7fefc40051c0 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefde20f640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefd8150800 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fefd400f990 tx=0x7fefd400fe50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefceffd640 1 -- 172.21.15.99:0/3191452385 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4003b80 con 0x7fefc40051c0 2024-03-04T10:05:40.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefceffd640 1 -- 172.21.15.99:0/3191452385 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd4018070 con 0x7fefc40051c0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd815c140 con 0x7fefc40051c0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefceffd640 1 -- 172.21.15.99:0/3191452385 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd400d7d0 con 0x7fefc40051c0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.133+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd80b4680 con 0x7fefc40051c0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.134+0000 7fefceffd640 1 -- 172.21.15.99:0/3191452385 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fefd400d970 con 0x7fefc40051c0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.135+0000 7fefceffd640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefbc06b500 0x7fefbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.135+0000 7fefceffd640 1 -- 172.21.15.99:0/3191452385 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fefd408a560 con 0x7fefc40051c0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.135+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefbc06b500 0x7fefbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.135+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefbc06b500 0x7fefbc06d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fefd815bf30 tx=0x7fefc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.136+0000 7fefe3051740 1 Processor -- start 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.136+0000 7fefe3051740 1 -- start start 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefe3051740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 0x55fdb567a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7c40 0x55fdb56aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb567ad40 0x55fdb56aab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefe3051740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb55e3e00 con 0x55fdb56a7860 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb558cb10 con 0x55fdb56a7c40 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdb55dfea0 con 0x55fdb567ad40 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 0x55fdb567a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 0x55fdb567a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54688/0 (socket says 172.21.15.99:54688) 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 -- 172.21.15.99:0/2686004960 learned_addr learned my addr 172.21.15.99:0/2686004960 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7c40 0x55fdb56aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 -- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb567ad40 msgr2=0x55fdb56aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefdea10640 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb567ad40 0x55fdb56aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb567ad40 0x55fdb56aab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 -- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7c40 msgr2=0x55fdb56aa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7c40 0x55fdb56aa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 -- 172.21.15.99:0/2686004960 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdb53c27d0 con 0x55fdb56a7860 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7c40 0x55fdb56aa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefde20f640 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 0x55fdb567a800 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fefd4088830 tx=0x7fefd4058440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c999071f982d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefbbfff640 1 -- 172.21.15.99:0/2686004960 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd4070020 con 0x55fdb56a7860 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.137+0000 7fefbbfff640 1 -- 172.21.15.99:0/2686004960 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd4058ae0 con 0x55fdb56a7860 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefbbfff640 1 -- 172.21.15.99:0/2686004960 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd405cc30 con 0x55fdb56a7860 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 -- 172.21.15.99:0/2686004960 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 msgr2=0x55fdb567a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 0x55fdb567a800 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fefd4088830 tx=0x7fefd4058440 comp rx=0 tx=0).stop 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 -- 172.21.15.99:0/2686004960 shutdown_connections 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb567ad40 0x55fdb56aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7c40 0x55fdb56aa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 --2- 172.21.15.99:0/2686004960 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7860 0x55fdb567a800 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 -- 172.21.15.99:0/2686004960 >> 172.21.15.99:0/2686004960 conn(0x55fdb567ca70 msgr2=0x55fdb56c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 -- 172.21.15.99:0/2686004960 shutdown_connections 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.138+0000 7fefe3051740 1 -- 172.21.15.99:0/2686004960 wait complete. 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.139+0000 7fefe3051740 1 -- start start 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.139+0000 7fefe3051740 1 Processor -- start 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56a6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56bc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56bec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb567f010 con 0x55fdb56a7c40 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb55a00a0 con 0x55fdb56a7860 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdb567e400 con 0x55fdb56bddf0 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefe3051740 10 client.0 _ll_register_callbacks cb 0x55fdb559fa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56bc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56bc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54692/0 (socket says 172.21.15.99:54692) 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 -- 172.21.15.99:0/2043204783 learned_addr learned my addr 172.21.15.99:0/2043204783 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:40.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefde20f640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdea10640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56bec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 -- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 msgr2=0x55fdb56bec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56bec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 -- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 msgr2=0x55fdb56a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 -- 172.21.15.99:0/2043204783 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdb52f83f0 con 0x55fdb56a7c40 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdea10640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56bec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56bc470 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fefc8005330 tx=0x7fefc8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e870df74f300099 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.140+0000 7fefb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefb97fa640 1 -- 172.21.15.99:0/2043204783 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8027060 con 0x55fdb56a7c40 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefb97fa640 1 -- 172.21.15.99:0/2043204783 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fefc8022ca0 con 0x55fdb56a7c40 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fefd4058ae0 con 0x55fdb56a7c40 2024-03-04T10:05:40.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefb97fa640 1 -- 172.21.15.99:0/2043204783 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefc8010e70 con 0x55fdb56a7c40 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 10 client.5721 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55fdb52b80f0 con 0x55fdb56a7c40 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefb97fa640 1 -- 172.21.15.99:0/2043204783 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd4058ae0 con 0x55fdb56a7c40 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefb97fa640 1 -- 172.21.15.99:0/2043204783 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fefc8020070 con 0x55fdb56a7c40 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 10 client.5721 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 10 client.5721 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 20 client.5721 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a6d50 0x55fdb56d4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 4 client.5721 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fdb55f1340 con 0x55fdb56a6d50 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.141+0000 7fefde20f640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a6d50 0x55fdb56d4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefb97fa640 10 client.5721 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefde20f640 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a6d50 0x55fdb56d4e90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89166ce91a390e83 server_cookie=32fb5618d9b665d2 in_seq=0 out_seq=0 2024-03-04T10:05:40.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefb97fa640 1 -- 172.21.15.99:0/2043204783 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fdb55f1340 con 0x55fdb56a6d50 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefb97fa640 10 client.5721 handle_command_reply: tid=0 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 client.5721 shutdown 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 msgr2=0x55fdb56bc470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56bc470 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fefc8005330 tx=0x7fefc8001a40 comp rx=0 tx=0).stop 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 shutdown_connections 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56bec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56bc470 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a6d50 0x55fdb56d4e90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 --2- 172.21.15.99:0/2043204783 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 >> 172.21.15.99:0/2043204783 conn(0x55fdb567ad40 msgr2=0x55fdb56c8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefb97fa640 0 client.5721 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 shutdown_connections 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 1 -- 172.21.15.99:0/2043204783 wait complete. 2024-03-04T10:05:40.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.142+0000 7fefe3051740 20 client.5721 trim_cache size 0 max 16384 2024-03-04T10:05:40.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.157+0000 7fefe3051740 1 Processor -- start 2024-03-04T10:05:40.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.157+0000 7fefe3051740 1 -- start start 2024-03-04T10:05:40.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefe3051740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb567e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefe3051740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd8200dd0 con 0x55fdb56a7c40 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd8200000 con 0x55fdb56a7860 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd81ff480 con 0x55fdb56bddf0 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefdda0e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefdda0e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54698/0 (socket says 172.21.15.99:54698) 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefdda0e640 1 -- 172.21.15.99:0/3989889982 learned_addr learned my addr 172.21.15.99:0/3989889982 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.158+0000 7fefde20f640 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdea10640 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb567e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdda0e640 1 -- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 msgr2=0x55fdb567e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb567e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdda0e640 1 -- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 msgr2=0x55fdb56be5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56be5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdda0e640 1 -- 172.21.15.99:0/3989889982 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdb56c0030 con 0x55fdb56a7c40 2024-03-04T10:05:40.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefde20f640 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56be5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefdda0e640 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fefc8024f40 tx=0x7fefc8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2325033e195cd4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefbb7fe640 1 -- 172.21.15.99:0/3989889982 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc801a070 con 0x55fdb56a7c40 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefbb7fe640 1 -- 172.21.15.99:0/3989889982 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc801b030 con 0x55fdb56a7c40 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.159+0000 7fefbb7fe640 1 -- 172.21.15.99:0/3989889982 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc8014860 con 0x55fdb56a7c40 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 -- 172.21.15.99:0/3989889982 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 msgr2=0x55fdb56ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fefc8024f40 tx=0x7fefc8020000 comp rx=0 tx=0).stop 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 -- 172.21.15.99:0/3989889982 shutdown_connections 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb567e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 --2- 172.21.15.99:0/3989889982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb56be5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 -- 172.21.15.99:0/3989889982 >> 172.21.15.99:0/3989889982 conn(0x55fdb567ad40 msgr2=0x55fdb5675df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 -- 172.21.15.99:0/3989889982 shutdown_connections 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 -- 172.21.15.99:0/3989889982 wait complete. 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 -- start start 2024-03-04T10:05:40.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.160+0000 7fefe3051740 1 Processor -- start 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb57188a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd81ffa00 con 0x55fdb56a7c40 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdb56d5490 con 0x55fdb56a7860 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdb567a630 con 0x55fdb56bddf0 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefe3051740 10 client.0 _ll_register_callbacks cb 0x7fefc8010da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdda0e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdda0e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54706/0 (socket says 172.21.15.99:54706) 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdda0e640 1 -- 172.21.15.99:0/2897866058 learned_addr learned my addr 172.21.15.99:0/2897866058 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefde20f640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb57188a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdea10640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdda0e640 1 -- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 msgr2=0x55fdb56adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.161+0000 7fefdda0e640 1 -- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 msgr2=0x55fdb57188a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb57188a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefdda0e640 1 -- 172.21.15.99:0/2897866058 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdb5679870 con 0x55fdb56a7c40 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefdda0e640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fefc8011010 tx=0x7fefc8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214e104712e5aefc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefb97fa640 1 -- 172.21.15.99:0/2897866058 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc801a040 con 0x55fdb56a7c40 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefb97fa640 1 -- 172.21.15.99:0/2897866058 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fefc802c420 con 0x55fdb56a7c40 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fefc801b030 con 0x55fdb56a7c40 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefb97fa640 1 -- 172.21.15.99:0/2897866058 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefc804b050 con 0x55fdb56a7c40 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 10 client.5727 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55fdb567a110 con 0x55fdb56a7c40 2024-03-04T10:05:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefb97fa640 1 -- 172.21.15.99:0/2897866058 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc801b030 con 0x55fdb56a7c40 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefb97fa640 1 -- 172.21.15.99:0/2897866058 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fefc804e020 con 0x55fdb56a7c40 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 10 client.5727 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 10 client.5727 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 20 client.5727 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a5a10 0x55fdb572f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 4 client.5727 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.162+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fdb55eb690 con 0x55fdb56a5a10 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefde20f640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a5a10 0x55fdb572f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefb97fa640 10 client.5727 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefde20f640 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a5a10 0x55fdb572f6c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39d643575c65fc6b server_cookie=48011cc8620e3bea in_seq=0 out_seq=0 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefb97fa640 1 -- 172.21.15.99:0/2897866058 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fdb55eb690 con 0x55fdb56a5a10 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefb97fa640 10 client.5727 handle_command_reply: tid=0 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 client.5727 shutdown 2024-03-04T10:05:40.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 msgr2=0x55fdb56ad5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fefc8011010 tx=0x7fefc8014000 comp rx=0 tx=0).stop 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 shutdown_connections 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdb56bddf0 0x55fdb56adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdb56a7c40 0x55fdb56ad5f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55fdb56a5a10 0x55fdb572f6c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 --2- 172.21.15.99:0/2897866058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdb56a7860 0x55fdb57188a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefb97fa640 0 client.5727 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 >> 172.21.15.99:0/2897866058 conn(0x55fdb567ad40 msgr2=0x55fdb56809f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 shutdown_connections 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 1 -- 172.21.15.99:0/2897866058 wait complete. 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.163+0000 7fefe3051740 20 client.5727 trim_cache size 0 max 16384 2024-03-04T10:05:40.171 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:05:40.172 INFO:teuthology.orchestra.run.smithi099.stdout: "status": "no active scrubs running", 2024-03-04T10:05:40.172 INFO:teuthology.orchestra.run.smithi099.stdout: "scrubs": {} 2024-03-04T10:05:40.172 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:05:40.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefbc06b500 msgr2=0x7fefbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefbc06b500 0x7fefbc06d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fefd815bf30 tx=0x7fefc8009040 comp rx=0 tx=0).stop 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 msgr2=0x7fefd8150800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefd8150800 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fefd400f990 tx=0x7fefd400fe50 comp rx=0 tx=0).stop 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 shutdown_connections 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefbc06b500 0x7fefbc06d9c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fefd80b3cc0 0x7fefd80b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fefd80c4330 0x7fefd8150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 --2- 172.21.15.99:0/3191452385 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fefc40051c0 0x7fefd8150800 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 >> 172.21.15.99:0/3191452385 conn(0x7fefd80b22e0 msgr2=0x7fefd80cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 shutdown_connections 2024-03-04T10:05:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.164+0000 7fefe0473640 1 -- 172.21.15.99:0/3191452385 wait complete. 2024-03-04T10:05:40.176 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T10:05:40.176 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:05:40.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.277+0000 7fe14815b640 1 Processor -- start 2024-03-04T10:05:40.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.277+0000 7fe14815b640 1 -- start start 2024-03-04T10:05:40.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe14815b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe1400cf650 0x7fe1400cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe14815b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe140070de0 0x7fe140071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe14815b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe14015f020 con 0x7fe140154a80 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe14815b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1400cd010 con 0x7fe140070de0 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe14815b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1400cdaa0 con 0x7fe1400cf650 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe145ef7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe140154a80 0x7fe14015e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46724/0 (socket says 172.21.15.99:46724) 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe145ef7640 1 -- 172.21.15.99:0/4098858641 learned_addr learned my addr 172.21.15.99:0/4098858641 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe1466f8640 1 --2- 172.21.15.99:0/4098858641 >> v2:172.21.15.99:3300/0 conn(0x7fe140070de0 0x7fe140071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe144ef5640 1 -- 172.21.15.99:0/4098858641 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346359696 0 0) 0x7fe13c001110 con 0x7fe140154a80 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe1456f6640 1 --2- 172.21.15.99:0/4098858641 >> v2:172.21.15.162:3300/0 conn(0x7fe1400cf650 0x7fe1400cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe144ef5640 1 -- 172.21.15.99:0/4098858641 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe12c003520 con 0x7fe140154a80 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe1466f8640 1 -- 172.21.15.99:0/4098858641 >> v2:172.21.15.162:3300/0 conn(0x7fe1400cf650 msgr2=0x7fe1400cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:40.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe1466f8640 1 --2- 172.21.15.99:0/4098858641 >> v2:172.21.15.162:3300/0 conn(0x7fe1400cf650 0x7fe1400cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe1466f8640 1 -- 172.21.15.99:0/4098858641 >> v1:172.21.15.99:6789/0 conn(0x7fe140154a80 legacy=0x7fe14015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.278+0000 7fe1466f8640 1 -- 172.21.15.99:0/4098858641 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1400cdd80 con 0x7fe140070de0 2024-03-04T10:05:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.279+0000 7fe1466f8640 1 --2- 172.21.15.99:0/4098858641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140070de0 0x7fe140071270 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe140067960 tx=0x7fe13802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6160478c6a9abe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.279+0000 7fe144ef5640 1 -- 172.21.15.99:0/4098858641 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138039070 con 0x7fe140070de0 2024-03-04T10:05:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.279+0000 7fe144ef5640 1 -- 172.21.15.99:0/4098858641 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe13802cc80 con 0x7fe140070de0 2024-03-04T10:05:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.279+0000 7fe144ef5640 1 -- 172.21.15.99:0/4098858641 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138034660 con 0x7fe140070de0 2024-03-04T10:05:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 -- 172.21.15.99:0/4098858641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140070de0 msgr2=0x7fe140071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 --2- 172.21.15.99:0/4098858641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140070de0 0x7fe140071270 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe140067960 tx=0x7fe13802bed0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 -- 172.21.15.99:0/4098858641 shutdown_connections 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 --2- 172.21.15.99:0/4098858641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140070de0 0x7fe140071270 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 --2- 172.21.15.99:0/4098858641 >> v2:172.21.15.162:3300/0 conn(0x7fe1400cf650 0x7fe1400cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 -- 172.21.15.99:0/4098858641 >> 172.21.15.99:0/4098858641 conn(0x7fe14006f5f0 msgr2=0x7fe14006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 -- 172.21.15.99:0/4098858641 shutdown_connections 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.280+0000 7fe14815b640 1 -- 172.21.15.99:0/4098858641 wait complete. 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.281+0000 7fe14815b640 1 Processor -- start 2024-03-04T10:05:40.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.281+0000 7fe14815b640 1 -- start start 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe140070de0 0x7fe1400c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1400cf650 0x7fe140149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 0x7fe1400b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400ce620 con 0x7fe140154a80 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1400ce340 con 0x7fe1400cf650 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe140154e60 con 0x7fe140070de0 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 0x7fe1400b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 0x7fe1400b6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54726/0 (socket says 172.21.15.99:54726) 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 -- 172.21.15.99:0/3716480733 learned_addr learned my addr 172.21.15.99:0/3716480733 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1456f6640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1400cf650 0x7fe140149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe145ef7640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe140070de0 0x7fe1400c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 -- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe140070de0 msgr2=0x7fe1400c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe140070de0 0x7fe1400c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 -- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1400cf650 msgr2=0x7fe140149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1400cf650 0x7fe140149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 -- 172.21.15.99:0/3716480733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1400b6b80 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe145ef7640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe140070de0 0x7fe1400c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1466f8640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 0x7fe1400b6640 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe13802c4e0 tx=0x7fe138004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138039050 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe138045070 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1400c8f20 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe138009450 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.282+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1400c94e0 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.283+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe140074f60 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.284+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe13803e850 con 0x7fe140154a80 2024-03-04T10:05:40.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.284+0000 7fe1367fc640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe12406b4d0 0x7fe12406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:40.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.284+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fe1380ad560 con 0x7fe140154a80 2024-03-04T10:05:40.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.284+0000 7fe145ef7640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe12406b4d0 0x7fe12406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:40.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.285+0000 7fe145ef7640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe12406b4d0 0x7fe12406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe13c0012b0 tx=0x7fe13c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:40.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.289+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe138032e20 con 0x7fe140154a80 2024-03-04T10:05:40.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.494+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe1400b0d70 con 0x7fe140154a80 2024-03-04T10:05:40.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.494+0000 7fe1367fc640 1 -- 172.21.15.99:0/3716480733 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe13807c270 con 0x7fe140154a80 2024-03-04T10:05:40.495 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:05:40.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe12406b4d0 msgr2=0x7fe12406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe12406b4d0 0x7fe12406d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe13c0012b0 tx=0x7fe13c009040 comp rx=0 tx=0).stop 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 msgr2=0x7fe1400b6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 0x7fe1400b6640 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe13802c4e0 tx=0x7fe138004a40 comp rx=0 tx=0).stop 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 shutdown_connections 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe12406b4d0 0x7fe12406d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe140154a80 0x7fe1400b6640 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1400cf650 0x7fe140149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 --2- 172.21.15.99:0/3716480733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe140070de0 0x7fe1400c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 >> 172.21.15.99:0/3716480733 conn(0x7fe14006f5f0 msgr2=0x7fe14006c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 shutdown_connections 2024-03-04T10:05:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:40.496+0000 7fe14815b640 1 -- 172.21.15.99:0/3716480733 wait complete. 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:05:40.508 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:05:40.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.508 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:05:40.524 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42645sec preferred_lft 42645sec 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.525 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:05:40.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.526 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:05:40.589 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:05:40.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.591 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:05:40.647 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:05:40.647 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:05:40.647 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:05:40.647 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:05:40.647 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:05:40.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.648 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:05:40.648 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:05:40.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:05:40.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:05:40.648 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:05:40.648 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:05:40.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.743 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:05:40.758 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:05:40.758 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:05:40.758 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:05:40.758 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.758 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:05:40.825 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:05:40.825 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:40.880 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:05:40.880 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:05:40.880 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:40.934 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:05:40.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:40.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:05:41.019 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:05:41.019 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:05:41.026 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:05:41.026 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:05:41.026 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:05:41.026 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:05:41.027 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:05:41.027 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:05:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:05:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:05:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:05:41.027 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:05:41.029 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:05:41.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:41.029 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:41.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:41.055 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:05:41.112 INFO:teuthology.orchestra.run.smithi099.stdout:client5117 2024-03-04T10:05:41.112 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:41.112 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:41.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:41.167 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-03-04T10:05:41.222 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:05:41.223 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-03-04T10:05:41.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.304+0000 7f478180bf40 1 Processor -- start 2024-03-04T10:05:41.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.304+0000 7f478180bf40 1 -- start start 2024-03-04T10:05:41.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f478180bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c80e9b60 0x55b4c81b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f478180bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 0x55b4c81bd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f478180bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 0x55b4c81c0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f478180bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4c81bc320 con 0x55b4c812e470 2024-03-04T10:05:41.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f478180bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4c812ee30 con 0x55b4c80e9b60 2024-03-04T10:05:41.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f478180bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4c812eb90 con 0x55b4c81bbf40 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 0x55b4c81bd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 0x55b4c81bd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54752/0 (socket says 172.21.15.99:54752) 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 -- 172.21.15.99:0/3063763688 learned_addr learned my addr 172.21.15.99:0/3063763688 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 -- 172.21.15.99:0/3063763688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 msgr2=0x55b4c81c0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 0x55b4c81c0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 -- 172.21.15.99:0/3063763688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c80e9b60 msgr2=0x55b4c81b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c80e9b60 0x55b4c81b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 -- 172.21.15.99:0/3063763688 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4c81b9620 con 0x55b4c812e470 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477de4b640 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 0x55b4c81bd200 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4774009730 tx=0x7f4774009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b7905337ec829e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477ce49640 1 -- 172.21.15.99:0/3063763688 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774017070 con 0x55b4c812e470 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477ce49640 1 -- 172.21.15.99:0/3063763688 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47740090d0 con 0x55b4c812e470 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.305+0000 7f477ce49640 1 -- 172.21.15.99:0/3063763688 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774006b60 con 0x55b4c812e470 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 -- 172.21.15.99:0/3063763688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 msgr2=0x55b4c81bd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 0x55b4c81bd200 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4774009730 tx=0x7f4774009bf0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 -- 172.21.15.99:0/3063763688 shutdown_connections 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 0x55b4c81c0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c812e470 0x55b4c81bd200 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 --2- 172.21.15.99:0/3063763688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c80e9b60 0x55b4c81b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 -- 172.21.15.99:0/3063763688 >> 172.21.15.99:0/3063763688 conn(0x55b4c80e73d0 msgr2=0x55b4c812f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 -- 172.21.15.99:0/3063763688 shutdown_connections 2024-03-04T10:05:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 -- 172.21.15.99:0/3063763688 wait complete. 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 Processor -- start 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.306+0000 7f478180bf40 1 -- start start 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f478180bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c80e9b60 0x55b4c81ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f478180bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c812e470 0x55b4c81bb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f478180bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 0x55b4c812d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f478180bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4c81c2380 con 0x55b4c80e9b60 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f478180bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4c81b98c0 con 0x55b4c812e470 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f478180bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4c81c2660 con 0x55b4c81bbf40 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c80e9b60 0x55b4c81ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c80e9b60 0x55b4c81ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54764/0 (socket says 172.21.15.99:54764) 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 -- 172.21.15.99:0/1743341784 learned_addr learned my addr 172.21.15.99:0/1743341784 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477de4b640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c812e470 0x55b4c81bb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477ee4d640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 0x55b4c812d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 -- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 msgr2=0x55b4c812d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4c81bbf40 0x55b4c812d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 -- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c812e470 msgr2=0x55b4c81bb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4c812e470 0x55b4c81bb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.307+0000 7f477e64c640 1 -- 172.21.15.99:0/1743341784 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4c81b2650 con 0x55b4c80e9b60 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.308+0000 7f477e64c640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c80e9b60 0x55b4c81ad770 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f477001b7a0 tx=0x7f4770053b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.308+0000 7f476f7fe640 1 -- 172.21.15.99:0/1743341784 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770061020 con 0x55b4c80e9b60 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.308+0000 7f476f7fe640 1 -- 172.21.15.99:0/1743341784 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4770053ca0 con 0x55b4c80e9b60 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.308+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b4c812dba0 con 0x55b4c80e9b60 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.308+0000 7f476f7fe640 1 -- 172.21.15.99:0/1743341784 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770005b40 con 0x55b4c80e9b60 2024-03-04T10:05:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.308+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4c81b2090 con 0x55b4c80e9b60 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.309+0000 7f476f7fe640 1 -- 172.21.15.99:0/1743341784 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4770064020 con 0x55b4c80e9b60 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.309+0000 7f476f7fe640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4760069ee0 0x7f476006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f476f7fe640 1 -- 172.21.15.99:0/1743341784 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f47700cad00 con 0x55b4c80e9b60 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f478180bf40 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55b4c81c3400 0x55b4c81c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41) v9 -- 0x55b4c81c3da0 con 0x55b4c81c3400 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f477de4b640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4760069ee0 0x7f476006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f477ee4d640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55b4c81c3400 0x55b4c81c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f477ee4d640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55b4c81c3400 0x55b4c81c3860 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.310+0000 7f477de4b640 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4760069ee0 0x7f476006c3a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55b4c81172a0 tx=0x7f477400f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f477ee4d640 1 -- 172.21.15.99:0/1743341784 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v41'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4778007650 con 0x55b4c81c3400 2024-03-04T10:05:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55b4c81c3400 msgr2=0x55b4c81c3860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55b4c81c3400 0x55b4c81c3860 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4760069ee0 msgr2=0x7f476006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4760069ee0 0x7f476006c3a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55b4c81172a0 tx=0x7f477400f000 comp rx=0 tx=0).stop 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c80e9b60 msgr2=0x55b4c81ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 --2- 172.21.15.99:0/1743341784 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b4c80e9b60 0x55b4c81ad770 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f477001b7a0 tx=0x7f4770053b30 comp rx=0 tx=0).stop 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f477ee4d640 1 -- 172.21.15.99:0/1743341784 reap_dead start 2024-03-04T10:05:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 shutdown_connections 2024-03-04T10:05:41.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 >> 172.21.15.99:0/1743341784 conn(0x55b4c80e73d0 msgr2=0x55b4c81bd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 shutdown_connections 2024-03-04T10:05:41.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.312+0000 7f478180bf40 1 -- 172.21.15.99:0/1743341784 wait complete. 2024-03-04T10:05:41.320 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:41.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.458+0000 7fb056548640 1 Processor -- start 2024-03-04T10:05:41.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.458+0000 7fb056548640 1 -- start start 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb056548640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0500cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb056548640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb05006d3b0 0x7fb0500cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb056548640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500d0140 con 0x7fb0500ce9c0 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb056548640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05015cbb0 con 0x7fb05006d3b0 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb056548640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015c280 con 0x7fb0500cfd60 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb054ae5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb0500ce9c0 0x7fb0500cee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46738/0 (socket says 172.21.15.99:46738) 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb054ae5640 1 -- 172.21.15.99:0/805095816 learned_addr learned my addr 172.21.15.99:0/805095816 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/805095816 >> v2:172.21.15.162:3300/0 conn(0x7fb05006d3b0 0x7fb0500cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594118544 0 0) 0x7fb0500d0140 con 0x7fb0500ce9c0 2024-03-04T10:05:41.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb04ffff640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0500cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.459+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb038003520 con 0x7fb0500ce9c0 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3732314870 0 0) 0x7fb038003520 con 0x7fb0500ce9c0 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 msgr2=0x7fb0500cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0500cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 >> v2:172.21.15.162:3300/0 conn(0x7fb05006d3b0 msgr2=0x7fb0500cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 --2- 172.21.15.99:0/805095816 >> v2:172.21.15.162:3300/0 conn(0x7fb05006d3b0 0x7fb0500cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05015c560 con 0x7fb0500ce9c0 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fb0480019f0 con 0x7fb0500ce9c0 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 >> v1:172.21.15.99:6789/0 conn(0x7fb0500ce9c0 legacy=0x7fb0500cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0380039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb038003f50 0x7fb038004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038004d40 0x7fb0380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015cbb0 con 0x7fb038003f50 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015c280 con 0x7fb038004d40 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03800b9c0 con 0x7fb0500cfd60 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb038003f50 0x7fb038004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb054ae5640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038004d40 0x7fb0380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04ffff640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0380039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04f7fe640 1 -- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 msgr2=0x7fb0380039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0380039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04f7fe640 1 -- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038004d40 msgr2=0x7fb0380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038004d40 0x7fb0380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04f7fe640 1 -- 172.21.15.99:0/805095816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038003520 con 0x7fb038003f50 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb04ffff640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500cfd60 0x7fb0380039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:41.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.460+0000 7fb054ae5640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038004d40 0x7fb0380051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb038003f50 0x7fb038004800 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb044009ac0 tx=0x7fb038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752a78d81a1f36dd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044039070 con 0x7fb038003f50 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04402c6b0 con 0x7fb038003f50 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb04effd640 1 -- 172.21.15.99:0/805095816 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044009690 con 0x7fb038003f50 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb056548640 1 -- 172.21.15.99:0/805095816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb038003f50 msgr2=0x7fb038004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb056548640 1 --2- 172.21.15.99:0/805095816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb038003f50 0x7fb038004800 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb044009ac0 tx=0x7fb038005830 comp rx=0 tx=0).stop 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb054ae5640 1 -- 172.21.15.99:0/805095816 reap_dead start 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb056548640 1 -- 172.21.15.99:0/805095816 shutdown_connections 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.461+0000 7fb056548640 1 -- 172.21.15.99:0/805095816 >> 172.21.15.99:0/805095816 conn(0x7fb05006acd0 msgr2=0x7fb05006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.462+0000 7fb056548640 1 -- 172.21.15.99:0/805095816 shutdown_connections 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.462+0000 7fb056548640 1 -- 172.21.15.99:0/805095816 wait complete. 2024-03-04T10:05:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.463+0000 7fb056548640 1 Processor -- start 2024-03-04T10:05:41.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.463+0000 7fb056548640 1 -- start start 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.463+0000 7fb056548640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038003f50 0x7fb05014d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.463+0000 7fb056548640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 0x7fb0500a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.463+0000 7fb056548640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500acfd0 0x7fb0500ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.463+0000 7fb056548640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500cbdf0 con 0x7fb05014c6c0 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb056548640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015bf30 con 0x7fb038003f50 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb056548640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500cbf70 con 0x7fb0500acfd0 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 0x7fb0500a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 0x7fb0500a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54780/0 (socket says 172.21.15.99:54780) 2024-03-04T10:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 -- 172.21.15.99:0/4011668425 learned_addr learned my addr 172.21.15.99:0/4011668425 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04ffff640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038003f50 0x7fb05014d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb054ae5640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500acfd0 0x7fb0500ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 -- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500acfd0 msgr2=0x7fb0500ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500acfd0 0x7fb0500ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 -- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038003f50 msgr2=0x7fb05014d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038003f50 0x7fb05014d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 -- 172.21.15.99:0/4011668425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05014cb40 con 0x7fb05014c6c0 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04f7fe640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 0x7fb0500a8e70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb044037040 tx=0x7fb044004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044042020 con 0x7fb05014c6c0 2024-03-04T10:05:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04403a040 con 0x7fb05014c6c0 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0500a93b0 con 0x7fb05014c6c0 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044039070 con 0x7fb05014c6c0 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.464+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0500a9930 con 0x7fb05014c6c0 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.465+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb050074f60 con 0x7fb05014c6c0 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.466+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb044045020 con 0x7fb05014c6c0 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.466+0000 7fb04cff9640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb03c06b4d0 0x7fb03c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.466+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fb0440abc00 con 0x7fb05014c6c0 2024-03-04T10:05:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.466+0000 7fb04ffff640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb03c06b4d0 0x7fb03c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.467+0000 7fb04ffff640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb03c06b4d0 0x7fb03c06d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb0400099b0 tx=0x7fb040008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.470+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb0440b2080 con 0x7fb05014c6c0 2024-03-04T10:05:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.676+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb05015db60 con 0x7fb05014c6c0 2024-03-04T10:05:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.677+0000 7fb04cff9640 1 -- 172.21.15.99:0/4011668425 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7fb04404c030 con 0x7fb05014c6c0 2024-03-04T10:05:41.679 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:41.679 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:41.679 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb03c06b4d0 msgr2=0x7fb03c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb03c06b4d0 0x7fb03c06d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb0400099b0 tx=0x7fb040008040 comp rx=0 tx=0).stop 2024-03-04T10:05:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 msgr2=0x7fb0500a8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 0x7fb0500a8e70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fb044037040 tx=0x7fb044004730 comp rx=0 tx=0).stop 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 shutdown_connections 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb03c06b4d0 0x7fb03c06d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb0500acfd0 0x7fb0500ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb05014c6c0 0x7fb0500a8e70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 --2- 172.21.15.99:0/4011668425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb038003f50 0x7fb05014d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 >> 172.21.15.99:0/4011668425 conn(0x7fb05006acd0 msgr2=0x7fb0500b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 shutdown_connections 2024-03-04T10:05:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.679+0000 7fb056548640 1 -- 172.21.15.99:0/4011668425 wait complete. 2024-03-04T10:05:41.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 3253863887 2024-03-04T10:05:41.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.794+0000 7f3838e52640 1 Processor -- start 2024-03-04T10:05:41.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.794+0000 7f3838e52640 1 -- start start 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3838e52640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f38340cf660 0x7f38340cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3838e52640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f38341616e0 0x7f3834161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3838e52640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3834162450 con 0x7f383415da20 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3838e52640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38340d0000 con 0x7f38341616e0 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3838e52640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38340cc920 con 0x7f38340cf660 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f38341616e0 0x7f3834161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f38341616e0 0x7f3834161b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54788/0 (socket says 172.21.15.99:54788) 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 -- 172.21.15.99:0/1810297466 learned_addr learned my addr 172.21.15.99:0/1810297466 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 -- 172.21.15.99:0/1810297466 >> v2:172.21.15.162:3300/0 conn(0x7f38340cf660 msgr2=0x7f38340cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1810297466 >> v2:172.21.15.162:3300/0 conn(0x7f38340cf660 0x7f38340cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 -- 172.21.15.99:0/1810297466 >> v1:172.21.15.162:6789/0 conn(0x7f383415da20 legacy=0x7f38340ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.795+0000 7f3832d9d640 1 -- 172.21.15.99:0/1810297466 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340cc2d0 con 0x7f38341616e0 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.796+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1810297466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38341616e0 0x7f3834161b70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f38341615d0 tx=0x7f382c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e1bc2f768f6b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.796+0000 7f383159a640 1 -- 172.21.15.99:0/1810297466 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c02c9f0 con 0x7f38341616e0 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.796+0000 7f383159a640 1 -- 172.21.15.99:0/1810297466 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c02cb90 con 0x7f38341616e0 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.796+0000 7f383159a640 1 -- 172.21.15.99:0/1810297466 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c0343f0 con 0x7f38341616e0 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 -- 172.21.15.99:0/1810297466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38341616e0 msgr2=0x7f3834161b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 --2- 172.21.15.99:0/1810297466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38341616e0 0x7f3834161b70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f38341615d0 tx=0x7f382c02be50 comp rx=0 tx=0).stop 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 -- 172.21.15.99:0/1810297466 shutdown_connections 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 --2- 172.21.15.99:0/1810297466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38341616e0 0x7f3834161b70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 --2- 172.21.15.99:0/1810297466 >> v2:172.21.15.162:3300/0 conn(0x7f38340cf660 0x7f38340cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 -- 172.21.15.99:0/1810297466 >> 172.21.15.99:0/1810297466 conn(0x7f38340b33a0 msgr2=0x7f38340b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 -- 172.21.15.99:0/1810297466 shutdown_connections 2024-03-04T10:05:41.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.797+0000 7f3838e52640 1 -- 172.21.15.99:0/1810297466 wait complete. 2024-03-04T10:05:41.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.798+0000 7f3838e52640 1 Processor -- start 2024-03-04T10:05:41.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.798+0000 7f3838e52640 1 -- start start 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3838e52640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38340cf660 0x7f3834152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3838e52640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 0x7f38340caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3838e52640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38341616e0 0x7f38340cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3838e52640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383415de00 con 0x7f383415da20 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3838e52640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38340cbf80 con 0x7f38340cf660 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3838e52640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38341620b0 con 0x7f38341616e0 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 0x7f38340caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 0x7f38340caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54794/0 (socket says 172.21.15.99:54794) 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 -- 172.21.15.99:0/1715766947 learned_addr learned my addr 172.21.15.99:0/1715766947 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f383259c640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38340cf660 0x7f3834152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 -- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38341616e0 msgr2=0x7f38340cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38341616e0 0x7f38340cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38341616e0 0x7f38340cafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 -- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38340cf660 msgr2=0x7f3834152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38340cf660 0x7f3834152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.799+0000 7f3831d9b640 1 -- 172.21.15.99:0/1715766947 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38341531e0 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.800+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 0x7f38340caa90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f382800b4d0 tx=0x7f382800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.800+0000 7f3822ffd640 1 -- 172.21.15.99:0/1715766947 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828014070 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.800+0000 7f3822ffd640 1 -- 172.21.15.99:0/1715766947 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828010870 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.800+0000 7f3822ffd640 1 -- 172.21.15.99:0/1715766947 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828010c60 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.800+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38340afe30 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.800+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38340b03f0 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.801+0000 7f3822ffd640 1 -- 172.21.15.99:0/1715766947 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3828017070 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.801+0000 7f3822ffd640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.802+0000 7f3822ffd640 1 -- 172.21.15.99:0/1715766947 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f382808a180 con 0x7f383415da20 2024-03-04T10:05:41.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.802+0000 7f383259c640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.802+0000 7f383259c640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f381406b4d0 0x7f381406d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3824007fd0 tx=0x7f3824009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 Processor -- start 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 -- start start 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a363e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a366e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a363ecd0 0x55f6a366eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a35a7a70 con 0x55f6a366bbd0 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a3550c40 con 0x55f6a366b7f0 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.803+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6a35a3b10 con 0x55f6a363ecd0 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a366e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a366e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54804/0 (socket says 172.21.15.99:54804) 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 -- 172.21.15.99:0/1507542848 learned_addr learned my addr 172.21.15.99:0/1507542848 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f383259c640 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a363e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a363ecd0 0x55f6a366eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 -- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a363ecd0 msgr2=0x55f6a366eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a363ecd0 0x55f6a366eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 -- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 msgr2=0x55f6a363e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a363e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 -- 172.21.15.99:0/1507542848 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6a3388710 con 0x55f6a366bbd0 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a363ecd0 0x55f6a366eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a366e580 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f38280884e0 tx=0x7f3828058200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e86a1d497bbdc12 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3813fff640 1 -- 172.21.15.99:0/1507542848 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382801e070 con 0x55f6a366bbd0 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3813fff640 1 -- 172.21.15.99:0/1507542848 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828056a90 con 0x55f6a366bbd0 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.804+0000 7f3813fff640 1 -- 172.21.15.99:0/1507542848 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828070170 con 0x55f6a366bbd0 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 -- 172.21.15.99:0/1507542848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 msgr2=0x55f6a366e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a366e580 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f38280884e0 tx=0x7f3828058200 comp rx=0 tx=0).stop 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 -- 172.21.15.99:0/1507542848 shutdown_connections 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a363ecd0 0x55f6a366eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a366e580 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 --2- 172.21.15.99:0/1507542848 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a363e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 -- 172.21.15.99:0/1507542848 >> 172.21.15.99:0/1507542848 conn(0x55f6a3640a00 msgr2=0x55f6a3689580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 -- 172.21.15.99:0/1507542848 shutdown_connections 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 -- 172.21.15.99:0/1507542848 wait complete. 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 -- start start 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.805+0000 7f383ba32740 1 Processor -- start 2024-03-04T10:05:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a366ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366bbd0 0x55f6a3682030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a3671720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a3642fa0 con 0x55f6a36712a0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a35640a0 con 0x55f6a366bbd0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6a3642390 con 0x55f6a366b7f0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f383ba32740 10 client.0 _ll_register_callbacks cb 0x55f6a35a2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f3832d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a3671720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.806+0000 7f3832d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a3671720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54806/0 (socket says 172.21.15.99:54806) 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 -- 172.21.15.99:0/1209009279 learned_addr learned my addr 172.21.15.99:0/1209009279 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366bbd0 0x55f6a3682030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f383259c640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a366ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 -- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 msgr2=0x55f6a366ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a366ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 -- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366bbd0 msgr2=0x55f6a3682030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366bbd0 0x55f6a3682030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 -- 172.21.15.99:0/1209009279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f6a33e8ea0 con 0x55f6a36712a0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f383259c640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a366ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366bbd0 0x55f6a3682030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a3671720 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f382c009f40 tx=0x7f382c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8226267fa4bcb066 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f38117fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f38117fa640 1 -- 172.21.15.99:0/1209009279 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c009510 con 0x55f6a36712a0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f38117fa640 1 -- 172.21.15.99:0/1209009279 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f382c034bb0 con 0x55f6a36712a0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3828056a90 con 0x55f6a36712a0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f38117fa640 1 -- 172.21.15.99:0/1209009279 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f382c034db0 con 0x55f6a36712a0 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f383ba32740 10 client.5763 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55f6a327c000 con 0x55f6a36712a0 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f38117fa640 1 -- 172.21.15.99:0/1209009279 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828056a90 con 0x55f6a36712a0 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.807+0000 7f38117fa640 1 -- 172.21.15.99:0/1209009279 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f382c008040 con 0x55f6a36712a0 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 10 client.5763 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 10 client.5763 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 20 client.5763 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a369a210 0x55f6a369a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 4 client.5763 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6a35b4fb0 con 0x55f6a369a210 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383259c640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a369a210 0x55f6a369a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f38117fa640 10 client.5763 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383259c640 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a369a210 0x55f6a369a5f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7906ba496d473a22 server_cookie=4e87b104a2cfcff3 in_seq=0 out_seq=0 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f38117fa640 1 -- 172.21.15.99:0/1209009279 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f6a35b4fb0 con 0x55f6a369a210 2024-03-04T10:05:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f38117fa640 10 client.5763 handle_command_reply: tid=0 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.808+0000 7f383ba32740 1 client.5763 shutdown 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 msgr2=0x55f6a3671720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a3671720 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f382c009f40 tx=0x7f382c004ab0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 shutdown_connections 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a3671720 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a369a210 0x55f6a369a5f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366bbd0 0x55f6a3682030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 --2- 172.21.15.99:0/1209009279 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a366ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f38117fa640 0 client.5763 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 >> 172.21.15.99:0/1209009279 conn(0x55f6a363ecd0 msgr2=0x55f6a368ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 shutdown_connections 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 1 -- 172.21.15.99:0/1209009279 wait complete. 2024-03-04T10:05:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.809+0000 7f383ba32740 20 client.5763 trim_cache size 0 max 16384 2024-03-04T10:05:41.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.820+0000 7f383ba32740 1 Processor -- start 2024-03-04T10:05:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 -- start start 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a368a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366bbd0 0x55f6a3685410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a36a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38341ff900 con 0x55f6a36712a0 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38341605e0 con 0x55f6a366b7f0 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38341ff300 con 0x55f6a366bbd0 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f3832d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a36a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f3832d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a36a3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54814/0 (socket says 172.21.15.99:54814) 2024-03-04T10:05:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.821+0000 7f3832d9d640 1 -- 172.21.15.99:0/1087362386 learned_addr learned my addr 172.21.15.99:0/1087362386 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3831d9b640 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366bbd0 0x55f6a3685410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383259c640 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a368a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3832d9d640 1 -- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366bbd0 msgr2=0x55f6a3685410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366bbd0 0x55f6a3685410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3832d9d640 1 -- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 msgr2=0x55f6a368a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a368a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3832d9d640 1 -- 172.21.15.99:0/1087362386 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6a366b110 con 0x55f6a36712a0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383259c640 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a368a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f3832d9d640 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a36a3490 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f382c0094c0 tx=0x7f382c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422af5009bcb5e1d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f38137fe640 1 -- 172.21.15.99:0/1087362386 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c009510 con 0x55f6a36712a0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f38137fe640 1 -- 172.21.15.99:0/1087362386 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c009af0 con 0x55f6a36712a0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f38137fe640 1 -- 172.21.15.99:0/1087362386 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c034600 con 0x55f6a36712a0 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 -- 172.21.15.99:0/1087362386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 msgr2=0x55f6a36a3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a36a3490 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f382c0094c0 tx=0x7f382c03f000 comp rx=0 tx=0).stop 2024-03-04T10:05:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 -- 172.21.15.99:0/1087362386 shutdown_connections 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a36712a0 0x55f6a36a3490 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366bbd0 0x55f6a3685410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 --2- 172.21.15.99:0/1087362386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a366b7f0 0x55f6a368a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 -- 172.21.15.99:0/1087362386 >> 172.21.15.99:0/1087362386 conn(0x55f6a363ecd0 msgr2=0x55f6a366e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.822+0000 7f383ba32740 1 -- 172.21.15.99:0/1087362386 shutdown_connections 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.823+0000 7f383ba32740 1 -- 172.21.15.99:0/1087362386 wait complete. 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.823+0000 7f383ba32740 1 -- start start 2024-03-04T10:05:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.823+0000 7f383ba32740 1 Processor -- start 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a36ef420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a3643860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a36712a0 0x55f6a3643da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a369b210 con 0x55f6a366bbd0 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a36836c0 con 0x55f6a36712a0 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f6a3680b00 con 0x55f6a366b7f0 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383ba32740 10 client.0 _ll_register_callbacks cb 0x7f3828058560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a3643860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a3643860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54816/0 (socket says 172.21.15.99:54816) 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 -- 172.21.15.99:0/3135834801 learned_addr learned my addr 172.21.15.99:0/3135834801 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f383259c640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a36ef420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3832d9d640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a36712a0 0x55f6a3643da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 -- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 msgr2=0x55f6a36ef420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a36ef420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 -- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a36712a0 msgr2=0x55f6a3643da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a36712a0 0x55f6a3643da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 -- 172.21.15.99:0/3135834801 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f6a368a490 con 0x55f6a366bbd0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f3831d9b640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a3643860 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f382801c040 tx=0x7f382808a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae74950915f96fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.824+0000 7f38117fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f38117fa640 1 -- 172.21.15.99:0/3135834801 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3828058e90 con 0x55f6a366bbd0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f38117fa640 1 -- 172.21.15.99:0/3135834801 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f3828071dd0 con 0x55f6a366bbd0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f382c009af0 con 0x55f6a366bbd0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f38117fa640 1 -- 172.21.15.99:0/3135834801 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3828077300 con 0x55f6a366bbd0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 10 client.5769 fetch_fsmap learned FSMap version 21 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55f6a367efd0 con 0x55f6a366bbd0 2024-03-04T10:05:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f38117fa640 1 -- 172.21.15.99:0/3135834801 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c009af0 con 0x55f6a366bbd0 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f38117fa640 1 -- 172.21.15.99:0/3135834801 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 21) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3828070170 con 0x55f6a366bbd0 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 10 client.5769 fetch_fsmap finished waiting for FSMap version 21 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 10 client.5769 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 20 client.5769 populate_metadata read hostname 'smithi099' 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a36f5c60 0x55f6a36f6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 4 client.5769 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 3253863887}] 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 --> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] -- command(tid 0: {"prefix": "damage rm", "damage_id": 3253863887}) v1 -- 0x55f6a3642700 con 0x55f6a36f5c60 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383259c640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a36f5c60 0x55f6a36f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f38117fa640 10 client.5769 ms_handle_connect on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.825+0000 7f383259c640 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a36f5c60 0x55f6a36f6040 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=902e374953a8f72f server_cookie=b2f1614cbe934c00 in_seq=0 out_seq=0 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f38117fa640 1 -- 172.21.15.99:0/3135834801 <== mds.0 v2:172.21.15.99:6833/2085408316 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x55f6a3642700 con 0x55f6a36f5c60 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f38117fa640 10 client.5769 handle_command_reply: tid=0 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 client.5769 shutdown 2024-03-04T10:05:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 msgr2=0x55f6a3643860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a3643860 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f382801c040 tx=0x7f382808a040 comp rx=0 tx=0).stop 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 shutdown_connections 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:6833/2085408316,v1:172.21.15.99:6835/2085408316] conn(0x55f6a36f5c60 0x55f6a36f6040 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f6a36712a0 0x55f6a3643da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f6a366bbd0 0x55f6a3643860 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 --2- 172.21.15.99:0/3135834801 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f6a366b7f0 0x55f6a36ef420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f38117fa640 0 client.5769 ms_handle_reset on v2:172.21.15.99:6833/2085408316 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 >> 172.21.15.99:0/3135834801 conn(0x55f6a363ecd0 msgr2=0x55f6a3681f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 shutdown_connections 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 1 -- 172.21.15.99:0/3135834801 wait complete. 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f383ba32740 20 client.5769 trim_cache size 0 max 16384 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f381406b4d0 msgr2=0x7f381406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f381406b4d0 0x7f381406d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3824007fd0 tx=0x7f3824009040 comp rx=0 tx=0).stop 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 msgr2=0x7f38340caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 0x7f38340caa90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f382800b4d0 tx=0x7f382800b990 comp rx=0 tx=0).stop 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 shutdown_connections 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f381406b4d0 0x7f381406d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38341616e0 0x7f38340cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f383415da20 0x7f38340caa90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 --2- 172.21.15.99:0/1715766947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38340cf660 0x7f3834152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 >> 172.21.15.99:0/1715766947 conn(0x7f38340b33a0 msgr2=0x7f38340b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 shutdown_connections 2024-03-04T10:05:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41.826+0000 7f3838e52640 1 -- 172.21.15.99:0/1715766947 wait complete. 2024-03-04T10:05:41.838 DEBUG:teuthology.orchestra.run.smithi099:> stdin-killer -- python3 -c ' 2024-03-04T10:05:41.838 DEBUG:teuthology.orchestra.run.smithi099:> import sys 2024-03-04T10:05:41.838 DEBUG:teuthology.orchestra.run.smithi099:> import errno 2024-03-04T10:05:41.838 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:05:41.839 DEBUG:teuthology.orchestra.run.smithi099:> try: 2024-03-04T10:05:41.839 DEBUG:teuthology.orchestra.run.smithi099:> f = open("/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged", "w") 2024-03-04T10:05:41.839 DEBUG:teuthology.orchestra.run.smithi099:> f.close() 2024-03-04T10:05:41.839 DEBUG:teuthology.orchestra.run.smithi099:> except IOError as e: 2024-03-04T10:05:41.839 DEBUG:teuthology.orchestra.run.smithi099:> sys.exit(errno.EIO) 2024-03-04T10:05:41.839 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:05:41.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T10:05:41.916 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:05:41.916 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:05:41.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:05:42.053 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:05:42.053 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:42.053 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:42.068 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:42.069 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:42.124 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:42.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.262+0000 7fc7e5fc4640 1 Processor -- start 2024-03-04T10:05:42.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.262+0000 7fc7e5fc4640 1 -- start start 2024-03-04T10:05:42.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7e5fc4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc7e0071a60 0x7fc7e0150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7e5fc4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e015da40 con 0x7fc7e015d660 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7e5fc4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e00af090 con 0x7fc7e00aeb20 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7e5fc4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e015cbc0 con 0x7fc7e0071a60 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7dffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc7e0071a60 0x7fc7e0150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7dffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc7e0071a60 0x7fc7e0150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54830/0 (socket says 172.21.15.99:54830) 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7dffff640 1 -- 172.21.15.99:0/1831382848 learned_addr learned my addr 172.21.15.99:0/1831382848 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7dffff640 1 -- 172.21.15.99:0/1831382848 >> v1:172.21.15.162:6789/0 conn(0x7fc7e00aeb20 legacy=0x7fc7e0071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7dffff640 1 -- 172.21.15.99:0/1831382848 >> v1:172.21.15.99:6789/0 conn(0x7fc7e015d660 legacy=0x7fc7e015e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.263+0000 7fc7dffff640 1 -- 172.21.15.99:0/1831382848 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e015a9f0 con 0x7fc7e0071a60 2024-03-04T10:05:42.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.264+0000 7fc7dffff640 1 --2- 172.21.15.99:0/1831382848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e0150960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009e50 tx=0x7fc7d802cff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7bfcbb7a2e60e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.264+0000 7fc7deffd640 1 -- 172.21.15.99:0/1831382848 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d802db90 con 0x7fc7e0071a60 2024-03-04T10:05:42.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.264+0000 7fc7deffd640 1 -- 172.21.15.99:0/1831382848 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d802dd30 con 0x7fc7e0071a60 2024-03-04T10:05:42.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.264+0000 7fc7deffd640 1 -- 172.21.15.99:0/1831382848 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8035610 con 0x7fc7e0071a60 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/1831382848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 msgr2=0x7fc7e0150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/1831382848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e0150960 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8009e50 tx=0x7fc7d802cff0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/1831382848 shutdown_connections 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/1831382848 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e0150960 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/1831382848 >> 172.21.15.99:0/1831382848 conn(0x7fc7e006acd0 msgr2=0x7fc7e006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/1831382848 shutdown_connections 2024-03-04T10:05:42.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.265+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/1831382848 wait complete. 2024-03-04T10:05:42.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.266+0000 7fc7e5fc4640 1 Processor -- start 2024-03-04T10:05:42.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.266+0000 7fc7e5fc4640 1 -- start start 2024-03-04T10:05:42.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7e5fc4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e01483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7e5fc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7e00aeb20 0x7fc7e01497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7e5fc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc7e015d660 0x7fc7e00b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7e5fc4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e015a660 con 0x7fc7e0071a60 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7e5fc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e015afb0 con 0x7fc7e00aeb20 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7e5fc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e015b4d0 con 0x7fc7e015d660 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e01483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e01483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54832/0 (socket says 172.21.15.99:54832) 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 -- 172.21.15.99:0/3465692906 learned_addr learned my addr 172.21.15.99:0/3465692906 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7d7fff640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7e00aeb20 0x7fc7e01497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7dffff640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc7e015d660 0x7fc7e00b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 -- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc7e015d660 msgr2=0x7fc7e00b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc7e015d660 0x7fc7e00b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 -- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7e00aeb20 msgr2=0x7fc7e01497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7e00aeb20 0x7fc7e01497a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 -- 172.21.15.99:0/3465692906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00cf140 con 0x7fc7e0071a60 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7df7fe640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e01483e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c430 tx=0x7fc7d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.267+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0017070 con 0x7fc7e0071a60 2024-03-04T10:05:42.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.268+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d00052d0 con 0x7fc7e0071a60 2024-03-04T10:05:42.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.268+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e00cf420 con 0x7fc7e0071a60 2024-03-04T10:05:42.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.268+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0003850 con 0x7fc7e0071a60 2024-03-04T10:05:42.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.268+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e014a340 con 0x7fc7e0071a60 2024-03-04T10:05:42.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.269+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e00cf5f0 con 0x7fc7e0071a60 2024-03-04T10:05:42.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.269+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc7d001a070 con 0x7fc7e0071a60 2024-03-04T10:05:42.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.269+0000 7fc7dcff9640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc7c006b4d0 0x7fc7c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.270+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fc7d00897d0 con 0x7fc7e0071a60 2024-03-04T10:05:42.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.270+0000 7fc7d7fff640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc7c006b4d0 0x7fc7c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.270+0000 7fc7d7fff640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc7c006b4d0 0x7fc7c006d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00afc40 tx=0x7fc7c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.274+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc7d0057d70 con 0x7fc7e0071a60 2024-03-04T10:05:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.444+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7e0074f60 con 0x7fc7e0071a60 2024-03-04T10:05:42.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.445+0000 7fc7dcff9640 1 -- 172.21.15.99:0/3465692906 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17082 (secure 0 0 0) 0x7fc7d005be10 con 0x7fc7e0071a60 2024-03-04T10:05:42.446 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:42.446 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":41,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:21.677615+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/434536114":"2024-03-05T10:05:00.614507+0000","172.21.15.99:6833/434536114":"2024-03-05T10:05:00.614507+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc7c006b4d0 msgr2=0x7fc7c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc7c006b4d0 0x7fc7c006d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00afc40 tx=0x7fc7c8009040 comp rx=0 tx=0).stop 2024-03-04T10:05:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 msgr2=0x7fc7e01483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e01483e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c430 tx=0x7fc7d000c8f0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 shutdown_connections 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc7c006b4d0 0x7fc7c006d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc7e015d660 0x7fc7e00b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc7e00aeb20 0x7fc7e01497a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 --2- 172.21.15.99:0/3465692906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc7e0071a60 0x7fc7e01483e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 >> 172.21.15.99:0/3465692906 conn(0x7fc7e006acd0 msgr2=0x7fc7e006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 shutdown_connections 2024-03-04T10:05:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.447+0000 7fc7e5fc4640 1 -- 172.21.15.99:0/3465692906 wait complete. 2024-03-04T10:05:42.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:04:49.926068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:04:50.486347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:05:42.459 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:05:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.561+0000 7fb3587f1640 1 Processor -- start 2024-03-04T10:05:42.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.561+0000 7fb3587f1640 1 -- start start 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb3587f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb3501622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb3587f1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3500d2400 con 0x7fb3500d34a0 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb3587f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3500d26e0 con 0x7fb3500b5700 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb3587f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500d29c0 con 0x7fb3500b9410 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb355d8c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb3500d34a0 0x7fb350161be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46762/0 (socket says 172.21.15.99:46762) 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb355d8c640 1 -- 172.21.15.99:0/138416615 learned_addr learned my addr 172.21.15.99:0/138416615 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.562+0000 7fb356d8e640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb3501622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366470635 0 0) 0x7fb3500d2400 con 0x7fb3500d34a0 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33c003520 con 0x7fb3500d34a0 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941828918 0 0) 0x7fb3500d26e0 con 0x7fb3500b5700 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3500d2400 con 0x7fb3500b5700 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3808855874 0 0) 0x7fb33c003520 con 0x7fb3500d34a0 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 msgr2=0x7fb3501622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb3501622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 >> v1:172.21.15.162:6789/0 conn(0x7fb3500b5700 legacy=0x7fb3500d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500d2ca0 con 0x7fb3500d34a0 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb3501622f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fb340001670 con 0x7fb3500d34a0 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 >> v1:172.21.15.99:6789/0 conn(0x7fb3500d34a0 legacy=0x7fb350161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb33c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb33c0040b0 0x7fb33c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb33c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500d29c0 con 0x7fb33c004d40 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c00bb00 con 0x7fb33c0040b0 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c00bd00 con 0x7fb3500b9410 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb33c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb355d8c640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb33c0040b0 0x7fb33c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb35658d640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb33c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 -- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 msgr2=0x7fb33c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:42.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500b9410 0x7fb33c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 -- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb33c0040b0 msgr2=0x7fb33c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb33c0040b0 0x7fb33c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb356d8e640 1 -- 172.21.15.99:0/138416615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c003520 con 0x7fb33c004d40 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.563+0000 7fb355d8c640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb33c0040b0 0x7fb33c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.564+0000 7fb356d8e640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb33c0051c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb348009c70 tx=0x7fb33c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168a6e7f5a8f8821 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.564+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348039070 con 0x7fb33c004d40 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.564+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34802c910 con 0x7fb33c004d40 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.564+0000 7fb35558b640 1 -- 172.21.15.99:0/138416615 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348009840 con 0x7fb33c004d40 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb3587f1640 1 -- 172.21.15.99:0/138416615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 msgr2=0x7fb33c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb3587f1640 1 --2- 172.21.15.99:0/138416615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb33c0051c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb348009c70 tx=0x7fb33c005830 comp rx=0 tx=0).stop 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb356d8e640 1 -- 172.21.15.99:0/138416615 reap_dead start 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb3587f1640 1 -- 172.21.15.99:0/138416615 shutdown_connections 2024-03-04T10:05:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb3587f1640 1 -- 172.21.15.99:0/138416615 >> 172.21.15.99:0/138416615 conn(0x7fb3500d0490 msgr2=0x7fb3500d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:42.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb3587f1640 1 -- 172.21.15.99:0/138416615 shutdown_connections 2024-03-04T10:05:42.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.565+0000 7fb3587f1640 1 -- 172.21.15.99:0/138416615 wait complete. 2024-03-04T10:05:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.566+0000 7fb3587f1640 1 Processor -- start 2024-03-04T10:05:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.566+0000 7fb3587f1640 1 -- start start 2024-03-04T10:05:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb3500cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500c8840 0x7fb350152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3500c90f0 0x7fb3500c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501622f0 con 0x7fb33c004d40 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3500b5d30 con 0x7fb3500c90f0 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb350162470 con 0x7fb3500c8840 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb3500cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb3500cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54846/0 (socket says 172.21.15.99:54846) 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 -- 172.21.15.99:0/3141790541 learned_addr learned my addr 172.21.15.99:0/3141790541 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 -- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500c8840 msgr2=0x7fb350152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500c8840 0x7fb350152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb356d8e640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3500c90f0 0x7fb3500c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 -- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3500c90f0 msgr2=0x7fb3500c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3500c90f0 0x7fb3500c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 -- 172.21.15.99:0/3141790541 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3501526d0 con 0x7fb33c004d40 2024-03-04T10:05:42.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb35658d640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb3500cb4f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fb40 tx=0x7fb34c00d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c003b80 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34c018070 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3500cbbd0 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c00d920 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.567+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3500cc190 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.568+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb350074f60 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.569+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb34c00dac0 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.569+0000 7fb346ffd640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb33406b500 0x7fb33406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.569+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fb34c08a710 con 0x7fb33c004d40 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.570+0000 7fb355d8c640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb33406b500 0x7fb33406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.570+0000 7fb355d8c640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb33406b500 0x7fb33406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb340001200 tx=0x7fb340001bb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.574+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb34c059300 con 0x7fb33c004d40 2024-03-04T10:05:42.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.746+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb35006d1d0 con 0x7fb33c004d40 2024-03-04T10:05:42.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.747+0000 7fb346ffd640 1 -- 172.21.15.99:0/3141790541 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb34c058c00 con 0x7fb33c004d40 2024-03-04T10:05:42.748 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:05:42.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.748+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb33406b500 msgr2=0x7fb33406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.748+0000 7fb3587f1640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb33406b500 0x7fb33406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb340001200 tx=0x7fb340001bb0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.748+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 msgr2=0x7fb3500cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.748+0000 7fb3587f1640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb3500cb4f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00fb40 tx=0x7fb34c00d150 comp rx=0 tx=0).stop 2024-03-04T10:05:42.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 shutdown_connections 2024-03-04T10:05:42.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb33406b500 0x7fb33406d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb3500c90f0 0x7fb3500c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb3500c8840 0x7fb350152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 --2- 172.21.15.99:0/3141790541 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb33c004d40 0x7fb3500cb4f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 >> 172.21.15.99:0/3141790541 conn(0x7fb3500d0490 msgr2=0x7fb3500b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:42.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 shutdown_connections 2024-03-04T10:05:42.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.749+0000 7fb3587f1640 1 -- 172.21.15.99:0/3141790541 wait complete. 2024-03-04T10:05:42.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:05:42.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.863+0000 7f9855f3b640 1 Processor -- start 2024-03-04T10:05:42.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.863+0000 7f9855f3b640 1 -- start start 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f9855f3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f9855f3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9850161df0 0x7f98500cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f9855f3b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9850162450 con 0x7f98500cf660 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f9855f3b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98500d0000 con 0x7f9850161df0 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f9855f3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98500cc920 con 0x7f985015da20 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9850161df0 0x7f98500cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f98500cf660 0x7f98501616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46778/0 (socket says 172.21.15.99:46778) 2024-03-04T10:05:42.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984effd640 1 -- 172.21.15.99:0/1180705601 learned_addr learned my addr 172.21.15.99:0/1180705601 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984f7fe640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984ffff640 1 -- 172.21.15.99:0/1180705601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 msgr2=0x7f98500ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984ffff640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984ffff640 1 -- 172.21.15.99:0/1180705601 >> v1:172.21.15.99:6789/0 conn(0x7f98500cf660 legacy=0x7f98501616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984ffff640 1 -- 172.21.15.99:0/1180705601 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500cc2d0 con 0x7f9850161df0 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984e7fc640 1 -- 172.21.15.99:0/1180705601 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109659874 0 0) 0x7f9850162450 con 0x7f98500cf660 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.864+0000 7f984f7fe640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f984ffff640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500cfac0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9848009d90 tx=0x7f984802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bdba15b3172b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f984e7fc640 1 -- 172.21.15.99:0/1180705601 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848039070 con 0x7f9850161df0 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f984e7fc640 1 -- 172.21.15.99:0/1180705601 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984802cc50 con 0x7f9850161df0 2024-03-04T10:05:42.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f984e7fc640 1 -- 172.21.15.99:0/1180705601 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848034440 con 0x7f9850161df0 2024-03-04T10:05:42.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 -- 172.21.15.99:0/1180705601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 msgr2=0x7f98500cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500cfac0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9848009d90 tx=0x7f984802bea0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 -- 172.21.15.99:0/1180705601 shutdown_connections 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500cfac0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 --2- 172.21.15.99:0/1180705601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 -- 172.21.15.99:0/1180705601 >> 172.21.15.99:0/1180705601 conn(0x7f98500b33a0 msgr2=0x7f98500b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 -- 172.21.15.99:0/1180705601 shutdown_connections 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.865+0000 7f9855f3b640 1 -- 172.21.15.99:0/1180705601 wait complete. 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 Processor -- start 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 -- start start 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98500cf660 0x7f98500b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500b4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500cbf80 con 0x7f9850161df0 2024-03-04T10:05:42.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98500cd2f0 con 0x7f98500cf660 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.867+0000 7f9855f3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f985015de00 con 0x7f985015da20 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54858/0 (socket says 172.21.15.99:54858) 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 -- 172.21.15.99:0/3328330591 learned_addr learned my addr 172.21.15.99:0/3328330591 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984f7fe640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98500cf660 0x7f98500b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984effd640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500b4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 -- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 msgr2=0x7f98500b4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500b4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 -- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98500cf660 msgr2=0x7f98500b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98500cf660 0x7f98500b4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:42.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 -- 172.21.15.99:0/3328330591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98500ca570 con 0x7f9850161df0 2024-03-04T10:05:42.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984effd640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:42.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f984ffff640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500b4f80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9848009d90 tx=0x7f9848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848044020 con 0x7f9850161df0 2024-03-04T10:05:42.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984803a040 con 0x7f9850161df0 2024-03-04T10:05:42.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98500ca850 con 0x7f9850161df0 2024-03-04T10:05:42.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848039050 con 0x7f9850161df0 2024-03-04T10:05:42.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.868+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98500cae10 con 0x7f9850161df0 2024-03-04T10:05:42.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.869+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98501624f0 con 0x7f9850161df0 2024-03-04T10:05:42.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.870+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9848003990 con 0x7f9850161df0 2024-03-04T10:05:42.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.870+0000 7f9837fff640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f983c06b4d0 0x7f983c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:42.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.870+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f98480ac920 con 0x7f9850161df0 2024-03-04T10:05:42.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.870+0000 7f984f7fe640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f983c06b4d0 0x7f983c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:42.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.871+0000 7f984f7fe640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f983c06b4d0 0x7f983c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f985015c7f0 tx=0x7f9840008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:42.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:42.874+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f98501624f0 con 0x7f9850161df0 2024-03-04T10:05:43.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.043+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9850074f60 con 0x7f9850161df0 2024-03-04T10:05:43.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.043+0000 7f9837fff640 1 -- 172.21.15.99:0/3328330591 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7f984804e060 con 0x7f9850161df0 2024-03-04T10:05:43.044 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:05:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f983c06b4d0 msgr2=0x7f983c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f983c06b4d0 0x7f983c06d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f985015c7f0 tx=0x7f9840008040 comp rx=0 tx=0).stop 2024-03-04T10:05:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 msgr2=0x7f98500b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500b4f80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9848009d90 tx=0x7f9848004a40 comp rx=0 tx=0).stop 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 shutdown_connections 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f983c06b4d0 0x7f983c06d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9850161df0 0x7f98500b4f80 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f985015da20 0x7f98500b4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 --2- 172.21.15.99:0/3328330591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98500cf660 0x7f98500b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 >> 172.21.15.99:0/3328330591 conn(0x7f98500b33a0 msgr2=0x7f985015bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 shutdown_connections 2024-03-04T10:05:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.045+0000 7f9855f3b640 1 -- 172.21.15.99:0/3328330591 wait complete. 2024-03-04T10:05:43.057 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:05:43.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.158+0000 7f87f1b16640 1 Processor -- start 2024-03-04T10:05:43.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.159+0000 7f87f1b16640 1 -- start start 2024-03-04T10:05:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.159+0000 7f87f1b16640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f87ec158230 0x7f87ec0cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.159+0000 7f87f1b16640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0cc350 con 0x7f87ec0cd110 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.159+0000 7f87f1b16640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0cc630 con 0x7f87ec0d1e80 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.159+0000 7f87f1b16640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec0cc910 con 0x7f87ec158230 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eaffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f87ec0cd110 0x7f87ec0d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46792/0 (socket says 172.21.15.99:46792) 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eaffd640 1 -- 172.21.15.99:0/1319470842 learned_addr learned my addr 172.21.15.99:0/1319470842 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/1319470842 >> v2:172.21.15.162:3300/0 conn(0x7f87ec158230 0x7f87ec0cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014325531 0 0) 0x7f87ec0cc350 con 0x7f87ec0cd110 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d4003520 con 0x7f87ec0cd110 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541870741 0 0) 0x7f87d4003520 con 0x7f87ec0cd110 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 >> v2:172.21.15.162:3300/0 conn(0x7f87ec158230 msgr2=0x7f87ec0cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 --2- 172.21.15.99:0/1319470842 >> v2:172.21.15.162:3300/0 conn(0x7f87ec158230 0x7f87ec0cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 >> v1:172.21.15.162:6789/0 conn(0x7f87ec0d1e80 legacy=0x7f87ec0cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec0ccbf0 con 0x7f87ec0cd110 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f87e0001670 con 0x7f87ec0cd110 2024-03-04T10:05:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/1319470842 >> v2:172.21.15.162:3300/0 conn(0x7f87ec158230 0x7f87ec0cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 >> v1:172.21.15.99:6789/0 conn(0x7f87ec0cd110 legacy=0x7f87ec0d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87d40039b0 0x7f87d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87d4004800 0x7f87d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40051c0 0x7f87d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0cc910 con 0x7f87d40039b0 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d400c060 con 0x7f87d40051c0 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d400c260 con 0x7f87d4004800 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87d40039b0 0x7f87d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87ebfff640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40051c0 0x7f87d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eaffd640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87d4004800 0x7f87d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eb7fe640 1 -- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87d4004800 msgr2=0x7f87d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87d4004800 0x7f87d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.160+0000 7f87eb7fe640 1 -- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40051c0 msgr2=0x7f87d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40051c0 0x7f87d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87eb7fe640 1 -- 172.21.15.99:0/1319470842 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4003520 con 0x7f87d40039b0 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87ebfff640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40051c0 0x7f87d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87eaffd640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87d4004800 0x7f87d4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87ebfff640 1 -- 172.21.15.99:0/1319470842 reap_dead start 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87d40039b0 0x7f87d4003dd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009c90 tx=0x7f87d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d821ad55b58cb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc039070 con 0x7f87d40039b0 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc02c8f0 con 0x7f87d40039b0 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.161+0000 7f87ea7fc640 1 -- 172.21.15.99:0/1319470842 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc009860 con 0x7f87d40039b0 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 -- 172.21.15.99:0/1319470842 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87d40039b0 msgr2=0x7f87d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87d40039b0 0x7f87d4003dd0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009c90 tx=0x7f87d4005830 comp rx=0 tx=0).stop 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 -- 172.21.15.99:0/1319470842 shutdown_connections 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 --2- 172.21.15.99:0/1319470842 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87d40039b0 0x7f87d4003dd0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 -- 172.21.15.99:0/1319470842 >> 172.21.15.99:0/1319470842 conn(0x7f87ec0b3330 msgr2=0x7f87ec0b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 -- 172.21.15.99:0/1319470842 shutdown_connections 2024-03-04T10:05:43.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.162+0000 7f87f1b16640 1 -- 172.21.15.99:0/1319470842 wait complete. 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.163+0000 7f87f1b16640 1 Processor -- start 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.163+0000 7f87f1b16640 1 -- start start 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87f1b16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40039b0 0x7f87ec153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87f1b16640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87ec06ca00 0x7f87ec0c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87f1b16640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 0x7f87ec0c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87f1b16640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0cc0f0 con 0x7f87ec06ce80 2024-03-04T10:05:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87f1b16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0cbf70 con 0x7f87d40039b0 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87f1b16640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0d1770 con 0x7f87ec06ca00 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 0x7f87ec0c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 0x7f87ec0c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54874/0 (socket says 172.21.15.99:54874) 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 -- 172.21.15.99:0/4075169551 learned_addr learned my addr 172.21.15.99:0/4075169551 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40039b0 0x7f87ec153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87eaffd640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87ec06ca00 0x7f87ec0c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 -- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87ec06ca00 msgr2=0x7f87ec0c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87ec06ca00 0x7f87ec0c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 -- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40039b0 msgr2=0x7f87ec153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40039b0 0x7f87ec153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.164+0000 7f87ebfff640 1 -- 172.21.15.99:0/4075169551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec154020 con 0x7f87ec06ce80 2024-03-04T10:05:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.165+0000 7f87ebfff640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 0x7f87ec0c5f40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b0dd0 tx=0x7f87e400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.165+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4003b80 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.165+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e4018070 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.165+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec06d410 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.165+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e40050d0 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.165+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec06d9d0 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.166+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec074f60 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.166+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f87e4005270 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.166+0000 7f87d3fff640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.166+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f87e4089f70 con 0x7f87ec06ce80 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.167+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.167+0000 7f87eb7fe640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87d806b4d0 0x7f87d806d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f87dc036000 tx=0x7f87dc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.171+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f87e4058bf0 con 0x7f87ec06ce80 2024-03-04T10:05:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.340+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87ec0b7760 con 0x7f87ec06ce80 2024-03-04T10:05:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.340+0000 7f87d3fff640 1 -- 172.21.15.99:0/4075169551 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87e40584f0 con 0x7f87ec06ce80 2024-03-04T10:05:43.341 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:05:43.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87d806b4d0 msgr2=0x7f87d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87d806b4d0 0x7f87d806d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f87dc036000 tx=0x7f87dc030000 comp rx=0 tx=0).stop 2024-03-04T10:05:43.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 msgr2=0x7f87ec0c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 0x7f87ec0c5f40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b0dd0 tx=0x7f87e400e960 comp rx=0 tx=0).stop 2024-03-04T10:05:43.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 shutdown_connections 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87d806b4d0 0x7f87d806d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87ec06ce80 0x7f87ec0c5f40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87ec06ca00 0x7f87ec0c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 --2- 172.21.15.99:0/4075169551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87d40039b0 0x7f87ec153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 >> 172.21.15.99:0/4075169551 conn(0x7f87ec0b3330 msgr2=0x7f87ec0b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 shutdown_connections 2024-03-04T10:05:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.342+0000 7f87f1b16640 1 -- 172.21.15.99:0/4075169551 wait complete. 2024-03-04T10:05:43.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:05:43.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.453+0000 7f7326c9a640 1 Processor -- start 2024-03-04T10:05:43.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.453+0000 7f7326c9a640 1 -- start start 2024-03-04T10:05:43.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7326c9a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f732015de00 0x7f73200b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7326c9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73200b9380 0x7f73200b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7326c9a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732015e1e0 con 0x7f73200b70d0 2024-03-04T10:05:43.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7326c9a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73200d2820 con 0x7f732015de00 2024-03-04T10:05:43.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7326c9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200d1c10 con 0x7f73200b9380 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7324a36640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f732015de00 0x7f73200b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7325237640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f73200b70d0 0x7f73200b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46808/0 (socket says 172.21.15.99:46808) 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.454+0000 7f7325237640 1 -- 172.21.15.99:0/39291815 learned_addr learned my addr 172.21.15.99:0/39291815 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f731f7fe640 1 -- 172.21.15.99:0/39291815 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574350414 0 0) 0x7f732015e1e0 con 0x7f73200b70d0 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f731f7fe640 1 -- 172.21.15.99:0/39291815 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7308003520 con 0x7f73200b70d0 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f731ffff640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73200b9380 0x7f73200b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f7324a36640 1 -- 172.21.15.99:0/39291815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73200b9380 msgr2=0x7f73200b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f7324a36640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73200b9380 0x7f73200b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f7324a36640 1 -- 172.21.15.99:0/39291815 >> v1:172.21.15.99:6789/0 conn(0x7f73200b70d0 legacy=0x7f73200b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f7324a36640 1 -- 172.21.15.99:0/39291815 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200d1ef0 con 0x7f732015de00 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f731ffff640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73200b9380 0x7f73200b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f7324a36640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f732015de00 0x7f73200b2e60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f73100097a0 tx=0x7f731002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa91144e8b56c657 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.455+0000 7f731f7fe640 1 -- 172.21.15.99:0/39291815 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310035020 con 0x7f732015de00 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f731f7fe640 1 -- 172.21.15.99:0/39291815 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310031830 con 0x7f732015de00 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f731f7fe640 1 -- 172.21.15.99:0/39291815 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310031c00 con 0x7f732015de00 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 -- 172.21.15.99:0/39291815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f732015de00 msgr2=0x7f73200b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f732015de00 0x7f73200b2e60 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f73100097a0 tx=0x7f731002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 -- 172.21.15.99:0/39291815 shutdown_connections 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73200b9380 0x7f73200b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 --2- 172.21.15.99:0/39291815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f732015de00 0x7f73200b2e60 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 -- 172.21.15.99:0/39291815 >> 172.21.15.99:0/39291815 conn(0x7f73200d0470 msgr2=0x7f73200d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 -- 172.21.15.99:0/39291815 shutdown_connections 2024-03-04T10:05:43.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.456+0000 7f7326c9a640 1 -- 172.21.15.99:0/39291815 wait complete. 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.457+0000 7f7326c9a640 1 Processor -- start 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.457+0000 7f7326c9a640 1 -- start start 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7326c9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73200b70d0 0x7f73200b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7326c9a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 0x7f7320159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7326c9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f732015de00 0x7f73200b8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7326c9a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200b97e0 con 0x7f73200b9380 2024-03-04T10:05:43.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7326c9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200d24b0 con 0x7f73200b70d0 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7326c9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200b5b80 con 0x7f732015de00 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 0x7f7320159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 0x7f7320159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54888/0 (socket says 172.21.15.99:54888) 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 -- 172.21.15.99:0/2572025696 learned_addr learned my addr 172.21.15.99:0/2572025696 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7324a36640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73200b70d0 0x7f73200b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7325237640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f732015de00 0x7f73200b8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 -- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f732015de00 msgr2=0x7f73200b8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f732015de00 0x7f73200b8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 -- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73200b70d0 msgr2=0x7f73200b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73200b70d0 0x7f73200b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f731ffff640 1 -- 172.21.15.99:0/2572025696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200ccfe0 con 0x7f73200b9380 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.458+0000 7f7324a36640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73200b70d0 0x7f73200b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f7325237640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f732015de00 0x7f73200b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f731ffff640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 0x7f7320159cc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f731400c7f0 tx=0x7f731400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314015070 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7314007a30 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314007e00 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73200cd2c0 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.459+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73200cd880 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.460+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7320074f60 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.460+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7314018070 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.461+0000 7f731d7fa640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f730c06b500 0x7f730c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.461+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f731408a320 con 0x7f73200b9380 2024-03-04T10:05:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.461+0000 7f7324a36640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f730c06b500 0x7f730c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.462+0000 7f7324a36640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f730c06b500 0x7f730c06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7310036000 tx=0x7f7310008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.465+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7314059030 con 0x7f73200b9380 2024-03-04T10:05:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.648+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f73200b6bf0 con 0x7f73200b9380 2024-03-04T10:05:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.648+0000 7f731d7fa640 1 -- 172.21.15.99:0/2572025696 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v41) v1 ==== 80+0+120 (secure 0 0 0) 0x7f7314058930 con 0x7f73200b9380 2024-03-04T10:05:43.650 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6835/434536114 2024-03-05T10:05:00.614507+0000 2024-03-04T10:05:43.650 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6833/434536114 2024-03-05T10:05:00.614507+0000 2024-03-04T10:05:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:listed 2 entries 2024-03-04T10:05:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f730c06b500 msgr2=0x7f730c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f730c06b500 0x7f730c06d9c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7310036000 tx=0x7f7310008030 comp rx=0 tx=0).stop 2024-03-04T10:05:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 msgr2=0x7f7320159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 0x7f7320159cc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f731400c7f0 tx=0x7f731400cbf0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 shutdown_connections 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f730c06b500 0x7f730c06d9c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f732015de00 0x7f73200b8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73200b9380 0x7f7320159cc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 --2- 172.21.15.99:0/2572025696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73200b70d0 0x7f73200b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 >> 172.21.15.99:0/2572025696 conn(0x7f73200d0470 msgr2=0x7f73200d2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 shutdown_connections 2024-03-04T10:05:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.650+0000 7f7326c9a640 1 -- 172.21.15.99:0/2572025696 wait complete. 2024-03-04T10:05:43.662 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:05:43.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:43.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:05:43.692 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:05:43.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:05:43.693 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:05:43.748 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:05:43.748 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:43.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:05:43.768 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:43.768 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:43.803 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:05:43.803 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:05:43.803 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:43.825 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:05:43.825 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:43.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.942+0000 7f41f0537640 1 Processor -- start 2024-03-04T10:05:43.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.942+0000 7f41f0537640 1 -- start start 2024-03-04T10:05:43.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.943+0000 7f41f0537640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e80d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.943+0000 7f41f0537640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f41e80cc690 0x7f41e80d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.943+0000 7f41f0537640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e80cfb40 con 0x7f41e80d0e40 2024-03-04T10:05:43.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.943+0000 7f41f0537640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e80d2880 con 0x7f41e80cc690 2024-03-04T10:05:43.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.943+0000 7f41f0537640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80cfe20 con 0x7f41e815bbb0 2024-03-04T10:05:43.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.943+0000 7f41eead4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f41e80cc690 0x7f41e80d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41eead4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f41e80cc690 0x7f41e80d1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:46944/0 (socket says 172.21.15.99:46944) 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41eead4640 1 -- 172.21.15.99:0/1365107137 learned_addr learned my addr 172.21.15.99:0/1365107137 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41eead4640 1 -- 172.21.15.99:0/1365107137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 msgr2=0x7f41e80d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41ee2d3640 1 --2- 172.21.15.99:0/1365107137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e80d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41eead4640 1 --2- 172.21.15.99:0/1365107137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e80d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41eead4640 1 -- 172.21.15.99:0/1365107137 >> v1:172.21.15.162:6789/0 conn(0x7f41e80d0e40 legacy=0x7f41e80cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:43.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.944+0000 7f41eead4640 1 -- 172.21.15.99:0/1365107137 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80d0100 con 0x7f41e80cc690 2024-03-04T10:05:43.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41eead4640 1 --2- 172.21.15.99:0/1365107137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80cc690 0x7f41e80d1ca0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009870 tx=0x7f41e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670c7b502cb74712 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41ed2d1640 1 -- 172.21.15.99:0/1365107137 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0035020 con 0x7f41e80cc690 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41ed2d1640 1 -- 172.21.15.99:0/1365107137 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e0031640 con 0x7f41e80cc690 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41ed2d1640 1 -- 172.21.15.99:0/1365107137 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e002d040 con 0x7f41e80cc690 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 -- 172.21.15.99:0/1365107137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80cc690 msgr2=0x7f41e80d1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 --2- 172.21.15.99:0/1365107137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80cc690 0x7f41e80d1ca0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009870 tx=0x7f41e002b8f0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 -- 172.21.15.99:0/1365107137 shutdown_connections 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 --2- 172.21.15.99:0/1365107137 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80cc690 0x7f41e80d1ca0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 --2- 172.21.15.99:0/1365107137 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e80d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 -- 172.21.15.99:0/1365107137 >> 172.21.15.99:0/1365107137 conn(0x7f41e80b3480 msgr2=0x7f41e80b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 -- 172.21.15.99:0/1365107137 shutdown_connections 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.945+0000 7f41f0537640 1 -- 172.21.15.99:0/1365107137 wait complete. 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 Processor -- start 2024-03-04T10:05:43.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 -- start start 2024-03-04T10:05:43.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e80cc690 0x7f41e80b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 0x7f41e80cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e8158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e815bf90 con 0x7f41e80d0e40 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e80d21e0 con 0x7f41e80cc690 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41f0537640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e8155d10 con 0x7f41e815bbb0 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41edad2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 0x7f41e80cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41edad2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 0x7f41e80cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:46946/0 (socket says 172.21.15.99:46946) 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.947+0000 7f41edad2640 1 -- 172.21.15.99:0/3232284701 learned_addr learned my addr 172.21.15.99:0/3232284701 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41edad2640 1 -- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 msgr2=0x7f41e8158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41edad2640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e8158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41edad2640 1 -- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e80cc690 msgr2=0x7f41e80b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41ee2d3640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e80cc690 0x7f41e80b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41edad2640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e80cc690 0x7f41e80b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:43.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41edad2640 1 -- 172.21.15.99:0/3232284701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e80ca090 con 0x7f41e80d0e40 2024-03-04T10:05:43.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41ee2d3640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e80cc690 0x7f41e80b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:43.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41edad2640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 0x7f41e80cb130 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f41d800c400 tx=0x7f41d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8011790 con 0x7f41e80d0e40 2024-03-04T10:05:43.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d8017070 con 0x7f41e80d0e40 2024-03-04T10:05:43.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e80ca370 con 0x7f41e80d0e40 2024-03-04T10:05:43.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d8011bb0 con 0x7f41e80d0e40 2024-03-04T10:05:43.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.948+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e80cac10 con 0x7f41e80d0e40 2024-03-04T10:05:43.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.949+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e8074f60 con 0x7f41e80d0e40 2024-03-04T10:05:43.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.949+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f41d801c460 con 0x7f41e80d0e40 2024-03-04T10:05:43.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.950+0000 7f41deffd640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f41cc06b4d0 0x7f41cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:43.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.950+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f41d808a2e0 con 0x7f41e80d0e40 2024-03-04T10:05:43.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.950+0000 7f41ee2d3640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f41cc06b4d0 0x7f41cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:43.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.951+0000 7f41ee2d3640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f41cc06b4d0 0x7f41cc06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f41e4007eb0 tx=0x7f41e4007400 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:43.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:43.954+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f41d8058ed0 con 0x7f41e80d0e40 2024-03-04T10:05:44.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.159+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41e80ca540 con 0x7f41e80d0e40 2024-03-04T10:05:44.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.159+0000 7f41deffd640 1 -- 172.21.15.99:0/3232284701 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f41d80587d0 con 0x7f41e80d0e40 2024-03-04T10:05:44.161 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:44.161 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:44.161 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f41cc06b4d0 msgr2=0x7f41cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f41cc06b4d0 0x7f41cc06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f41e4007eb0 tx=0x7f41e4007400 comp rx=0 tx=0).stop 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 msgr2=0x7f41e80cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 0x7f41e80cb130 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f41d800c400 tx=0x7f41d800c8c0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 shutdown_connections 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f41cc06b4d0 0x7f41cc06d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41e815bbb0 0x7f41e8158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41e80d0e40 0x7f41e80cb130 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 --2- 172.21.15.99:0/3232284701 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41e80cc690 0x7f41e80b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.161+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 >> 172.21.15.99:0/3232284701 conn(0x7f41e80b3480 msgr2=0x7f41e80b5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:44.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.162+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 shutdown_connections 2024-03-04T10:05:44.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.162+0000 7f41f0537640 1 -- 172.21.15.99:0/3232284701 wait complete. 2024-03-04T10:05:44.174 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:44.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.274+0000 7f8d0328c640 1 Processor -- start 2024-03-04T10:05:44.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.274+0000 7f8d0328c640 1 -- start start 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d0328c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 0x7f8cfc15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d0328c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8cfc0cf650 0x7f8cfc0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d0328c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc0cdaa0 con 0x7f8cfc1616c0 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d0328c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cfc0cfff0 con 0x7f8cfc0cf650 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d0328c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc071df0 con 0x7f8cfc154a80 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d01829640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8cfc1616c0 0x7f8cfc161b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:43992/0 (socket says 172.21.15.99:43992) 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d01829640 1 -- 172.21.15.99:0/570853079 learned_addr learned my addr 172.21.15.99:0/570853079 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d00827640 1 --2- 172.21.15.99:0/570853079 >> v2:172.21.15.162:3300/0 conn(0x7f8cfc0cf650 0x7f8cfc0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d01028640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 0x7f8cfc15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454635279 0 0) 0x7f8cfc0cdaa0 con 0x7f8cfc1616c0 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce8003520 con 0x7f8cfc1616c0 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241192008 0 0) 0x7f8ce8003520 con 0x7f8cfc1616c0 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 msgr2=0x7f8cfc15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 0x7f8cfc15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 >> v2:172.21.15.162:3300/0 conn(0x7f8cfc0cf650 msgr2=0x7f8cfc0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 --2- 172.21.15.99:0/570853079 >> v2:172.21.15.162:3300/0 conn(0x7f8cfc0cf650 0x7f8cfc0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc0cd010 con 0x7f8cfc1616c0 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8cf40019f0 con 0x7f8cfc1616c0 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8d00827640 1 --2- 172.21.15.99:0/570853079 >> v2:172.21.15.162:3300/0 conn(0x7f8cfc0cf650 0x7f8cfc0cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 >> v1:172.21.15.99:6789/0 conn(0x7f8cfc1616c0 legacy=0x7f8cfc161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 0x7f8ce80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ce8003f60 0x7f8ce8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce8004d40 0x7f8ce80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc071df0 con 0x7f8ce8003f60 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce800b9c0 con 0x7f8ce8004d40 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.275+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce800bb40 con 0x7f8cfc154a80 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d00827640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ce8003f60 0x7f8ce8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d01028640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 0x7f8ce80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d01829640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce8004d40 0x7f8ce80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d00827640 1 -- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 msgr2=0x7f8ce80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d00827640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cfc154a80 0x7f8ce80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d00827640 1 -- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce8004d40 msgr2=0x7f8ce80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d00827640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ce8004d40 0x7f8ce80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.276+0000 7f8d00827640 1 -- 172.21.15.99:0/570853079 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8003520 con 0x7f8ce8003f60 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d00827640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ce8003f60 0x7f8ce8004800 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009c90 tx=0x7f8ce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e542d3d977d97c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec039070 con 0x7f8ce8003f60 2024-03-04T10:05:44.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec02c980 con 0x7f8ce8003f60 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8cf3fff640 1 -- 172.21.15.99:0/570853079 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec009860 con 0x7f8ce8003f60 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d0328c640 1 -- 172.21.15.99:0/570853079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ce8003f60 msgr2=0x7f8ce8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d0328c640 1 --2- 172.21.15.99:0/570853079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ce8003f60 0x7f8ce8004800 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009c90 tx=0x7f8ce8005830 comp rx=0 tx=0).stop 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d01829640 1 -- 172.21.15.99:0/570853079 reap_dead start 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d0328c640 1 -- 172.21.15.99:0/570853079 shutdown_connections 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d0328c640 1 -- 172.21.15.99:0/570853079 >> 172.21.15.99:0/570853079 conn(0x7f8cfc06f5f0 msgr2=0x7f8cfc06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d0328c640 1 -- 172.21.15.99:0/570853079 shutdown_connections 2024-03-04T10:05:44.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.277+0000 7f8d0328c640 1 -- 172.21.15.99:0/570853079 wait complete. 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 Processor -- start 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 -- start start 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ce8003f60 0x7f8cfc0b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 0x7f8cfc14d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cfc0b4490 0x7f8cfc0b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0ce340 con 0x7f8cfc14da40 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0ce060 con 0x7f8cfc0b4490 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d0328c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0ce620 con 0x7f8ce8003f60 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d00827640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 0x7f8cfc14d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d00827640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 0x7f8cfc14d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:46968/0 (socket says 172.21.15.99:46968) 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.279+0000 7f8d00827640 1 -- 172.21.15.99:0/322530199 learned_addr learned my addr 172.21.15.99:0/322530199 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d01829640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cfc0b4490 0x7f8cfc0b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d01028640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ce8003f60 0x7f8cfc0b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d00827640 1 -- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ce8003f60 msgr2=0x7f8cfc0b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d00827640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ce8003f60 0x7f8cfc0b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d00827640 1 -- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cfc0b4490 msgr2=0x7f8cfc0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d00827640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cfc0b4490 0x7f8cfc0b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d00827640 1 -- 172.21.15.99:0/322530199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc1550b0 con 0x7f8cfc14da40 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d01028640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ce8003f60 0x7f8cfc0b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d00827640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 0x7f8cfc14d1f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009db0 tx=0x7f8cec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec042020 con 0x7f8cfc14da40 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cec03a040 con 0x7f8cfc14da40 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cfc155390 con 0x7f8cfc14da40 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cec039070 con 0x7f8cfc14da40 2024-03-04T10:05:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.280+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cfc0b1df0 con 0x7f8cfc14da40 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.281+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cfc074f60 con 0x7f8cfc14da40 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.282+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8cec045020 con 0x7f8cfc14da40 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.282+0000 7f8cf1ffb640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ce006b4d0 0x7f8ce006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.282+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f8cec0abd30 con 0x7f8cfc14da40 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.282+0000 7f8d01028640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ce006b4d0 0x7f8ce006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.283+0000 7f8d01028640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ce006b4d0 0x7f8ce006d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009ae0 tx=0x7f8cf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.286+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cec04c030 con 0x7f8cfc14da40 2024-03-04T10:05:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.459+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cfc0b0f60 con 0x7f8cfc14da40 2024-03-04T10:05:44.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.460+0000 7f8cf1ffb640 1 -- 172.21.15.99:0/322530199 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17082 (secure 0 0 0) 0x7f8cec07a360 con 0x7f8cfc14da40 2024-03-04T10:05:44.462 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:44.462 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":41,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:21.677615+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/434536114":"2024-03-05T10:05:00.614507+0000","172.21.15.99:6833/434536114":"2024-03-05T10:05:00.614507+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.462+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ce006b4d0 msgr2=0x7f8ce006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.462+0000 7f8d0328c640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ce006b4d0 0x7f8ce006d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009ae0 tx=0x7f8cf8008040 comp rx=0 tx=0).stop 2024-03-04T10:05:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.462+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 msgr2=0x7f8cfc14d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.462+0000 7f8d0328c640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 0x7f8cfc14d1f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8cec009db0 tx=0x7f8cec004730 comp rx=0 tx=0).stop 2024-03-04T10:05:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 shutdown_connections 2024-03-04T10:05:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ce006b4d0 0x7f8ce006d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cfc0b4490 0x7f8cfc0b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cfc14da40 0x7f8cfc14d1f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 --2- 172.21.15.99:0/322530199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ce8003f60 0x7f8cfc0b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 >> 172.21.15.99:0/322530199 conn(0x7f8cfc06f5f0 msgr2=0x7f8cfc0737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:44.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 shutdown_connections 2024-03-04T10:05:44.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.463+0000 7f8d0328c640 1 -- 172.21.15.99:0/322530199 wait complete. 2024-03-04T10:05:44.475 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:04:49.926068+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:04:50.486347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:05:44.475 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:05:44.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.577+0000 7fbaccc2b640 1 Processor -- start 2024-03-04T10:05:44.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.577+0000 7fbaccc2b640 1 -- start start 2024-03-04T10:05:44.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbaccc2b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbac815da20 0x7fbac80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbaccc2b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbac8161df0 0x7fbac80cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbaccc2b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac8162450 con 0x7fbac80cf660 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbaccc2b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac80d0000 con 0x7fbac815da20 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbaccc2b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac80cc920 con 0x7fbac8161df0 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbac815da20 0x7fbac80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac5d9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbac80cf660 0x7fbac81616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44008/0 (socket says 172.21.15.99:44008) 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac5d9b640 1 -- 172.21.15.99:0/4024852469 learned_addr learned my addr 172.21.15.99:0/4024852469 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac6d9d640 1 --2- 172.21.15.99:0/4024852469 >> v2:172.21.15.162:3300/0 conn(0x7fbac8161df0 0x7fbac80cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac659c640 1 -- 172.21.15.99:0/4024852469 >> v2:172.21.15.162:3300/0 conn(0x7fbac8161df0 msgr2=0x7fbac80cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:44.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac659c640 1 --2- 172.21.15.99:0/4024852469 >> v2:172.21.15.162:3300/0 conn(0x7fbac8161df0 0x7fbac80cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac659c640 1 -- 172.21.15.99:0/4024852469 >> v1:172.21.15.99:6789/0 conn(0x7fbac80cf660 legacy=0x7fbac81616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac659c640 1 -- 172.21.15.99:0/4024852469 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac80cc2d0 con 0x7fbac815da20 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac559a640 1 -- 172.21.15.99:0/4024852469 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149411170 0 0) 0x7fbac8162450 con 0x7fbac80cf660 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.578+0000 7fbac6d9d640 1 --2- 172.21.15.99:0/4024852469 >> v2:172.21.15.162:3300/0 conn(0x7fbac8161df0 0x7fbac80cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.579+0000 7fbac659c640 1 --2- 172.21.15.99:0/4024852469 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac815da20 0x7fbac80ccdb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbac80d2890 tx=0x7fbab802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a023378fcf9f2f6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.579+0000 7fbac559a640 1 -- 172.21.15.99:0/4024852469 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8035020 con 0x7fbac815da20 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.579+0000 7fbac559a640 1 -- 172.21.15.99:0/4024852469 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8031830 con 0x7fbac815da20 2024-03-04T10:05:44.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.579+0000 7fbac559a640 1 -- 172.21.15.99:0/4024852469 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8031c00 con 0x7fbac815da20 2024-03-04T10:05:44.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 -- 172.21.15.99:0/4024852469 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac815da20 msgr2=0x7fbac80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/4024852469 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac815da20 0x7fbac80ccdb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fbac80d2890 tx=0x7fbab802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 -- 172.21.15.99:0/4024852469 shutdown_connections 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/4024852469 >> v2:172.21.15.162:3300/0 conn(0x7fbac8161df0 0x7fbac80cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/4024852469 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac815da20 0x7fbac80ccdb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 -- 172.21.15.99:0/4024852469 >> 172.21.15.99:0/4024852469 conn(0x7fbac80b33a0 msgr2=0x7fbac80b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 -- 172.21.15.99:0/4024852469 shutdown_connections 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.580+0000 7fbaccc2b640 1 -- 172.21.15.99:0/4024852469 wait complete. 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.581+0000 7fbaccc2b640 1 Processor -- start 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.581+0000 7fbaccc2b640 1 -- start start 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbaccc2b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 0x7fbac806d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbaccc2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbac815da20 0x7fbac8160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbaccc2b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbac8161df0 0x7fbac81558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbaccc2b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80cbf80 con 0x7fbac80cf660 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbaccc2b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80cd2f0 con 0x7fbac8161df0 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbaccc2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac815de00 con 0x7fbac815da20 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 0x7fbac806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 0x7fbac806d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:46982/0 (socket says 172.21.15.99:46982) 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 -- 172.21.15.99:0/3378004852 learned_addr learned my addr 172.21.15.99:0/3378004852 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac6d9d640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbac8161df0 0x7fbac81558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac5d9b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbac815da20 0x7fbac8160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 -- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbac815da20 msgr2=0x7fbac8160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbac815da20 0x7fbac8160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 -- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbac8161df0 msgr2=0x7fbac81558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbac8161df0 0x7fbac81558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac659c640 1 -- 172.21.15.99:0/3378004852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac81f6d20 con 0x7fbac80cf660 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.582+0000 7fbac5d9b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbac815da20 0x7fbac8160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.583+0000 7fbac659c640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 0x7fbac806d210 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009890 tx=0x7fbab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.583+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8041070 con 0x7fbac80cf660 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.583+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab803aa30 con 0x7fbac80cf660 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.583+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac81f7000 con 0x7fbac80cf660 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.583+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8035040 con 0x7fbac80cf660 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.583+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac81f75c0 con 0x7fbac80cf660 2024-03-04T10:05:44.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.584+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac8074f60 con 0x7fbac80cf660 2024-03-04T10:05:44.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.584+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbab8038020 con 0x7fbac80cf660 2024-03-04T10:05:44.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.584+0000 7fbab6ffd640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbaa806b500 0x7fbaa806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.585+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fbab80abc70 con 0x7fbac80cf660 2024-03-04T10:05:44.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.585+0000 7fbac5d9b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbaa806b500 0x7fbaa806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.585+0000 7fbac5d9b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbaa806b500 0x7fbaa806d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbac80b19f0 tx=0x7fbabc007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.589+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbab807a980 con 0x7fbac80cf660 2024-03-04T10:05:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.761+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbac80cdbf0 con 0x7fbac80cf660 2024-03-04T10:05:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.761+0000 7fbab6ffd640 1 -- 172.21.15.99:0/3378004852 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbab807a280 con 0x7fbac80cf660 2024-03-04T10:05:44.763 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:05:44.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbaa806b500 msgr2=0x7fbaa806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbaa806b500 0x7fbaa806d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbac80b19f0 tx=0x7fbabc007c10 comp rx=0 tx=0).stop 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 msgr2=0x7fbac806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 0x7fbac806d210 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009890 tx=0x7fbab80046c0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 shutdown_connections 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbaa806b500 0x7fbaa806d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbac8161df0 0x7fbac81558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbac815da20 0x7fbac8160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 --2- 172.21.15.99:0/3378004852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbac80cf660 0x7fbac806d210 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 >> 172.21.15.99:0/3378004852 conn(0x7fbac80b33a0 msgr2=0x7fbac80ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 shutdown_connections 2024-03-04T10:05:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.763+0000 7fbaccc2b640 1 -- 172.21.15.99:0/3378004852 wait complete. 2024-03-04T10:05:44.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:05:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.877+0000 7fcb18f3b640 1 Processor -- start 2024-03-04T10:05:44.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.878+0000 7fcb18f3b640 1 -- start start 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.878+0000 7fcb18f3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcb140b3340 0x7fcb1406d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.878+0000 7fcb18f3b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb14147890 con 0x7fcb140717a0 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.878+0000 7fcb18f3b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb1406df00 con 0x7fcb1406efa0 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.878+0000 7fcb18f3b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb140b2c70 con 0x7fcb140b3340 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb0bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcb140b3340 0x7fcb1406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb1259c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fcb140717a0 0x7fcb14071b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44024/0 (socket says 172.21.15.99:44024) 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb1259c640 1 -- 172.21.15.99:0/966831992 learned_addr learned my addr 172.21.15.99:0/966831992 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb0bfff640 1 -- 172.21.15.99:0/966831992 >> v1:172.21.15.162:6789/0 conn(0x7fcb1406efa0 legacy=0x7fcb140ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:44.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb0bfff640 1 -- 172.21.15.99:0/966831992 >> v1:172.21.15.99:6789/0 conn(0x7fcb140717a0 legacy=0x7fcb14071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb0bfff640 1 -- 172.21.15.99:0/966831992 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1406e1e0 con 0x7fcb140b3340 2024-03-04T10:05:44.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.879+0000 7fcb11d9b640 1 -- 172.21.15.99:0/966831992 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72650585 0 0) 0x7fcb14147890 con 0x7fcb140717a0 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb0bfff640 1 --2- 172.21.15.99:0/966831992 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb140b3340 0x7fcb1406d800 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0097a0 tx=0x7fcafc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab6214aabb2d3fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb11d9b640 1 -- 172.21.15.99:0/966831992 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc0315a0 con 0x7fcb140b3340 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb11d9b640 1 -- 172.21.15.99:0/966831992 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcafc037070 con 0x7fcb140b3340 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb11d9b640 1 -- 172.21.15.99:0/966831992 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcafc031960 con 0x7fcb140b3340 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 -- 172.21.15.99:0/966831992 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb140b3340 msgr2=0x7fcb1406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/966831992 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb140b3340 0x7fcb1406d800 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0097a0 tx=0x7fcafc02b840 comp rx=0 tx=0).stop 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 -- 172.21.15.99:0/966831992 shutdown_connections 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/966831992 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb140b3340 0x7fcb1406d800 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 -- 172.21.15.99:0/966831992 >> 172.21.15.99:0/966831992 conn(0x7fcb140cfad0 msgr2=0x7fcb140cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:44.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 -- 172.21.15.99:0/966831992 shutdown_connections 2024-03-04T10:05:44.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.880+0000 7fcb18f3b640 1 -- 172.21.15.99:0/966831992 wait complete. 2024-03-04T10:05:44.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 Processor -- start 2024-03-04T10:05:44.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 -- start start 2024-03-04T10:05:44.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 0x7fcb141f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb140717a0 0x7fcb1415d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb140b3340 0x7fcb14142130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1406ea80 con 0x7fcb1406efa0 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1406e7a0 con 0x7fcb140b3340 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb18f3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1406f430 con 0x7fcb140717a0 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 0x7fcb141f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb1259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 0x7fcb141f79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47008/0 (socket says 172.21.15.99:47008) 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.882+0000 7fcb1259c640 1 -- 172.21.15.99:0/1624785 learned_addr learned my addr 172.21.15.99:0/1624785 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb12d9d640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb140b3340 0x7fcb14142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb0bfff640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb140717a0 0x7fcb1415d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb1259c640 1 -- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb140717a0 msgr2=0x7fcb1415d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb1259c640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb140717a0 0x7fcb1415d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb1259c640 1 -- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb140b3340 msgr2=0x7fcb14142130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb1259c640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb140b3340 0x7fcb14142130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb1259c640 1 -- 172.21.15.99:0/1624785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb141f82b0 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb0bfff640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb140717a0 0x7fcb1415d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb1259c640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 0x7fcb141f79d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c500 tx=0x7fcb0400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04018070 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb04007580 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb141f8590 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04004a20 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.883+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1415fb00 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.884+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb141f8760 con 0x7fcb1406efa0 2024-03-04T10:05:44.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.884+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcb0401b070 con 0x7fcb1406efa0 2024-03-04T10:05:44.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.885+0000 7fcb0affd640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcaf406b4d0 0x7fcaf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:44.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.885+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fcb04089680 con 0x7fcb1406efa0 2024-03-04T10:05:44.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.885+0000 7fcb0bfff640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcaf406b4d0 0x7fcaf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:44.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.885+0000 7fcb0bfff640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcaf406b4d0 0x7fcaf406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcafc036000 tx=0x7fcafc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:44.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:44.889+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcb04058390 con 0x7fcb1406efa0 2024-03-04T10:05:45.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.059+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb140674d0 con 0x7fcb1406efa0 2024-03-04T10:05:45.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.059+0000 7fcb0affd640 1 -- 172.21.15.99:0/1624785 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb140674d0 con 0x7fcb1406efa0 2024-03-04T10:05:45.061 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcaf406b4d0 msgr2=0x7fcaf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcaf406b4d0 0x7fcaf406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcafc036000 tx=0x7fcafc008030 comp rx=0 tx=0).stop 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 msgr2=0x7fcb141f79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 0x7fcb141f79d0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcb0400c500 tx=0x7fcb0400c9c0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 shutdown_connections 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcaf406b4d0 0x7fcaf406d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb140b3340 0x7fcb14142130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcb140717a0 0x7fcb1415d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 --2- 172.21.15.99:0/1624785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcb1406efa0 0x7fcb141f79d0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.061+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 >> 172.21.15.99:0/1624785 conn(0x7fcb140cfad0 msgr2=0x7fcb14140840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.062+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 shutdown_connections 2024-03-04T10:05:45.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.062+0000 7fcb18f3b640 1 -- 172.21.15.99:0/1624785 wait complete. 2024-03-04T10:05:45.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:05:45.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.175+0000 7f4e25892640 1 Processor -- start 2024-03-04T10:05:45.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.175+0000 7f4e25892640 1 -- start start 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e25892640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4e200ceec0 0x7f4e200b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e25892640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2015ca50 con 0x7f4e200ccbd0 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e25892640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2015cd30 con 0x7f4e2015c2e0 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e25892640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2015d2f0 con 0x7f4e200ceec0 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e1e7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4e200ceec0 0x7f4e200b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e1effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f4e200ccbd0 0x7f4e200ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44028/0 (socket says 172.21.15.99:44028) 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e1effd640 1 -- 172.21.15.99:0/1575117147 learned_addr learned my addr 172.21.15.99:0/1575117147 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e1e7fc640 1 -- 172.21.15.99:0/1575117147 >> v1:172.21.15.162:6789/0 conn(0x7f4e2015c2e0 legacy=0x7f4e200b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e1e7fc640 1 -- 172.21.15.99:0/1575117147 >> v1:172.21.15.99:6789/0 conn(0x7f4e200ccbd0 legacy=0x7f4e200ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.176+0000 7f4e1e7fc640 1 -- 172.21.15.99:0/1575117147 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2015d010 con 0x7f4e200ceec0 2024-03-04T10:05:45.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.177+0000 7f4e1e7fc640 1 --2- 172.21.15.99:0/1575117147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e200ceec0 0x7f4e200b7320 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4e140097a0 tx=0x7f4e1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c385e7c70a532d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.177+0000 7f4e1dffb640 1 -- 172.21.15.99:0/1575117147 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14035020 con 0x7f4e200ceec0 2024-03-04T10:05:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.177+0000 7f4e1dffb640 1 -- 172.21.15.99:0/1575117147 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e14031830 con 0x7f4e200ceec0 2024-03-04T10:05:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.177+0000 7f4e1dffb640 1 -- 172.21.15.99:0/1575117147 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e14031c00 con 0x7f4e200ceec0 2024-03-04T10:05:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 -- 172.21.15.99:0/1575117147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e200ceec0 msgr2=0x7f4e200b7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 --2- 172.21.15.99:0/1575117147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e200ceec0 0x7f4e200b7320 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f4e140097a0 tx=0x7f4e1402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 -- 172.21.15.99:0/1575117147 shutdown_connections 2024-03-04T10:05:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 --2- 172.21.15.99:0/1575117147 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e200ceec0 0x7f4e200b7320 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 -- 172.21.15.99:0/1575117147 >> 172.21.15.99:0/1575117147 conn(0x7f4e20066dc0 msgr2=0x7f4e200671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 -- 172.21.15.99:0/1575117147 shutdown_connections 2024-03-04T10:05:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.178+0000 7f4e25892640 1 -- 172.21.15.99:0/1575117147 wait complete. 2024-03-04T10:05:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.179+0000 7f4e25892640 1 Processor -- start 2024-03-04T10:05:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.179+0000 7f4e25892640 1 -- start start 2024-03-04T10:05:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e25892640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e200ccbd0 0x7f4e20151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e25892640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e200ceec0 0x7f4e200c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e25892640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 0x7f4e200c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e25892640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e200ccfb0 con 0x7f4e2015c2e0 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e25892640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2015d6d0 con 0x7f4e200ceec0 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e25892640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2015eb70 con 0x7f4e200ccbd0 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 0x7f4e200c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 0x7f4e200c5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47038/0 (socket says 172.21.15.99:47038) 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 -- 172.21.15.99:0/186918167 learned_addr learned my addr 172.21.15.99:0/186918167 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1e7fc640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e200ceec0 0x7f4e200c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1effd640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e200ccbd0 0x7f4e20151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 -- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e200ccbd0 msgr2=0x7f4e20151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e200ccbd0 0x7f4e20151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 -- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e200ceec0 msgr2=0x7f4e200c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e200ceec0 0x7f4e200c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.180+0000 7f4e1f7fe640 1 -- 172.21.15.99:0/186918167 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e200c7df0 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.181+0000 7f4e1f7fe640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 0x7f4e200c5d80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800b440 tx=0x7f4e1800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.181+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18015020 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.181+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e18011830 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.181+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e200c80d0 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.181+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18011c20 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.181+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2006cdb0 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.182+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e20074f60 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.182+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4e18018020 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.183+0000 7f4e07fff640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.183+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f4e18089e90 con 0x7f4e2015c2e0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.183+0000 7f4e1effd640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.183+0000 7f4e1effd640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e100012e0 tx=0x7f4e10001a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.187+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4e1801f030 con 0x7f4e2015c2e0 2024-03-04T10:05:45.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.356+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4e200cd1c0 con 0x7f4e2015c2e0 2024-03-04T10:05:45.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.356+0000 7f4e07fff640 1 -- 172.21.15.99:0/186918167 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4e180584c0 con 0x7f4e2015c2e0 2024-03-04T10:05:45.358 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e0c06b4d0 msgr2=0x7f4e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e100012e0 tx=0x7f4e10001a30 comp rx=0 tx=0).stop 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 msgr2=0x7f4e200c5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 0x7f4e200c5d80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800b440 tx=0x7f4e1800b900 comp rx=0 tx=0).stop 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 shutdown_connections 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e0c06b4d0 0x7f4e0c06d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e2015c2e0 0x7f4e200c5d80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e200ceec0 0x7f4e200c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 --2- 172.21.15.99:0/186918167 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e200ccbd0 0x7f4e20151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 >> 172.21.15.99:0/186918167 conn(0x7f4e20066dc0 msgr2=0x7f4e200d1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 shutdown_connections 2024-03-04T10:05:45.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.358+0000 7f4e25892640 1 -- 172.21.15.99:0/186918167 wait complete. 2024-03-04T10:05:45.371 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:45.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.470+0000 7f330f1df640 1 Processor -- start 2024-03-04T10:05:45.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.470+0000 7f330f1df640 1 -- start start 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330f1df640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f33081597f0 0x7f330815c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330f1df640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f330806acd0 0x7f330806b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330f1df640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330815cfe0 con 0x7f330806b690 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330f1df640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33080d0240 con 0x7f330806acd0 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330f1df640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33080cf630 con 0x7f33081597f0 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f3307fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f330806acd0 0x7f330806b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330d77c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f330806b690 0x7f33080ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44038/0 (socket says 172.21.15.99:44038) 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f330d77c640 1 -- 172.21.15.99:0/3372880957 learned_addr learned my addr 172.21.15.99:0/3372880957 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f3307fff640 1 -- 172.21.15.99:0/3372880957 >> v2:172.21.15.162:3300/0 conn(0x7f33081597f0 msgr2=0x7f330815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f3307fff640 1 --2- 172.21.15.99:0/3372880957 >> v2:172.21.15.162:3300/0 conn(0x7f33081597f0 0x7f330815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f3307fff640 1 -- 172.21.15.99:0/3372880957 >> v1:172.21.15.99:6789/0 conn(0x7f330806b690 legacy=0x7f33080ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f3307fff640 1 -- 172.21.15.99:0/3372880957 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33080cf910 con 0x7f330806acd0 2024-03-04T10:05:45.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.471+0000 7f33077fe640 1 -- 172.21.15.99:0/3372880957 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148420057 0 0) 0x7f330815cfe0 con 0x7f330806b690 2024-03-04T10:05:45.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f3307fff640 1 --2- 172.21.15.99:0/3372880957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f330806b150 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f33080ce230 tx=0x7f32f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbc9d1212e6a073 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f33077fe640 1 -- 172.21.15.99:0/3372880957 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f40315a0 con 0x7f330806acd0 2024-03-04T10:05:45.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f33077fe640 1 -- 172.21.15.99:0/3372880957 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f4037070 con 0x7f330806acd0 2024-03-04T10:05:45.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f33077fe640 1 -- 172.21.15.99:0/3372880957 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4031960 con 0x7f330806acd0 2024-03-04T10:05:45.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 -- 172.21.15.99:0/3372880957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 msgr2=0x7f330806b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 --2- 172.21.15.99:0/3372880957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f330806b150 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f33080ce230 tx=0x7f32f402b840 comp rx=0 tx=0).stop 2024-03-04T10:05:45.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 -- 172.21.15.99:0/3372880957 shutdown_connections 2024-03-04T10:05:45.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 --2- 172.21.15.99:0/3372880957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f330806b150 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 --2- 172.21.15.99:0/3372880957 >> v2:172.21.15.162:3300/0 conn(0x7f33081597f0 0x7f330815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 -- 172.21.15.99:0/3372880957 >> 172.21.15.99:0/3372880957 conn(0x7f33080d0710 msgr2=0x7f33080d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 -- 172.21.15.99:0/3372880957 shutdown_connections 2024-03-04T10:05:45.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.472+0000 7f330f1df640 1 -- 172.21.15.99:0/3372880957 wait complete. 2024-03-04T10:05:45.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 Processor -- start 2024-03-04T10:05:45.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 -- start start 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f3308157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f330806b690 0x7f33080c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33081597f0 0x7f33080c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080d1350 con 0x7f330806acd0 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080cfed0 con 0x7f33081597f0 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.474+0000 7f330f1df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33080cd270 con 0x7f330806b690 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f3308157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f3308157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47052/0 (socket says 172.21.15.99:47052) 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 -- 172.21.15.99:0/2613607266 learned_addr learned my addr 172.21.15.99:0/2613607266 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330d77c640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33081597f0 0x7f33080c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f3307fff640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f330806b690 0x7f33080c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 -- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f330806b690 msgr2=0x7f33080c5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f330806b690 0x7f33080c5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 -- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33081597f0 msgr2=0x7f33080c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33081597f0 0x7f33080c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 -- 172.21.15.99:0/2613607266 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33080b51b0 con 0x7f330806acd0 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330d77c640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33081597f0 0x7f33080c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f3307fff640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f330806b690 0x7f33080c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330cf7b640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f3308157db0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00b410 tx=0x7f32fc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc015020 con 0x7f330806acd0 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32fc011850 con 0x7f330806acd0 2024-03-04T10:05:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33080b5490 con 0x7f330806acd0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32fc011c40 con 0x7f330806acd0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.475+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33080b5a50 con 0x7f330806acd0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.476+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330806cd70 con 0x7f330806acd0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.476+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f32fc018020 con 0x7f330806acd0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.477+0000 7f33057fa640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32ec06b4d0 0x7f32ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.477+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f32fc08a3a0 con 0x7f330806acd0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.477+0000 7f3307fff640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32ec06b4d0 0x7f32ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.478+0000 7f3307fff640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32ec06b4d0 0x7f32ec06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f32f4036000 tx=0x7f32f4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.481+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f32fc01f020 con 0x7f330806acd0 2024-03-04T10:05:45.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.688+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3308074f60 con 0x7f330806acd0 2024-03-04T10:05:45.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.688+0000 7f33057fa640 1 -- 172.21.15.99:0/2613607266 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f32fc0172c0 con 0x7f330806acd0 2024-03-04T10:05:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:45.691 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:45.693 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:45.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32ec06b4d0 msgr2=0x7f32ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32ec06b4d0 0x7f32ec06d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f32f4036000 tx=0x7f32f4008030 comp rx=0 tx=0).stop 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 msgr2=0x7f3308157db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f3308157db0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f32fc00b410 tx=0x7f32fc00b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 shutdown_connections 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32ec06b4d0 0x7f32ec06d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33081597f0 0x7f33080c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f330806b690 0x7f33080c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 --2- 172.21.15.99:0/2613607266 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f330806acd0 0x7f3308157db0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 >> 172.21.15.99:0/2613607266 conn(0x7f33080d0710 msgr2=0x7f330806f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 shutdown_connections 2024-03-04T10:05:45.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.690+0000 7f330f1df640 1 -- 172.21.15.99:0/2613607266 wait complete. 2024-03-04T10:05:45.705 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:45.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.805+0000 7f288e5c4640 1 Processor -- start 2024-03-04T10:05:45.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.805+0000 7f288e5c4640 1 -- start start 2024-03-04T10:05:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f288e5c4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2888154a80 0x7f288815e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f288e5c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f28880cfae0 0x7f2888161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f288e5c4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28880cff70 con 0x7f28880cf650 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f288e5c4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2888071df0 con 0x7f2888154a80 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f288e5c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28880cd010 con 0x7f28880cfae0 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f2887fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2888154a80 0x7f288815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f2887fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2888154a80 0x7f288815e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47056/0 (socket says 172.21.15.99:47056) 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f2887fff640 1 -- 172.21.15.99:0/1970083859 learned_addr learned my addr 172.21.15.99:0/1970083859 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.806+0000 7f288cb61640 1 --2- 172.21.15.99:0/1970083859 >> v2:172.21.15.162:3300/0 conn(0x7f28880cfae0 0x7f2888161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2887fff640 1 -- 172.21.15.99:0/1970083859 >> v2:172.21.15.162:3300/0 conn(0x7f28880cfae0 msgr2=0x7f2888161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2887fff640 1 --2- 172.21.15.99:0/1970083859 >> v2:172.21.15.162:3300/0 conn(0x7f28880cfae0 0x7f2888161dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2887fff640 1 -- 172.21.15.99:0/1970083859 >> v1:172.21.15.162:6789/0 conn(0x7f28880cf650 legacy=0x7f28881616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2887fff640 1 -- 172.21.15.99:0/1970083859 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28880cdaa0 con 0x7f2888154a80 2024-03-04T10:05:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f288cb61640 1 --2- 172.21.15.99:0/1970083859 >> v2:172.21.15.162:3300/0 conn(0x7f28880cfae0 0x7f2888161dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:45.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2887fff640 1 --2- 172.21.15.99:0/1970083859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f288815e910 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2888158690 tx=0x7f287802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74945c909e427318 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2886ffd640 1 -- 172.21.15.99:0/1970083859 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878035020 con 0x7f2888154a80 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2886ffd640 1 -- 172.21.15.99:0/1970083859 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2878031870 con 0x7f2888154a80 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.807+0000 7f2886ffd640 1 -- 172.21.15.99:0/1970083859 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878031c40 con 0x7f2888154a80 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 -- 172.21.15.99:0/1970083859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 msgr2=0x7f288815e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 --2- 172.21.15.99:0/1970083859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f288815e910 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2888158690 tx=0x7f287802b910 comp rx=0 tx=0).stop 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 -- 172.21.15.99:0/1970083859 shutdown_connections 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 --2- 172.21.15.99:0/1970083859 >> v2:172.21.15.162:3300/0 conn(0x7f28880cfae0 0x7f2888161dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 --2- 172.21.15.99:0/1970083859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f288815e910 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 -- 172.21.15.99:0/1970083859 >> 172.21.15.99:0/1970083859 conn(0x7f288806f5f0 msgr2=0x7f288806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 -- 172.21.15.99:0/1970083859 shutdown_connections 2024-03-04T10:05:45.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.808+0000 7f288e5c4640 1 -- 172.21.15.99:0/1970083859 wait complete. 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.809+0000 7f288e5c4640 1 Processor -- start 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.809+0000 7f288e5c4640 1 -- start start 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288e5c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28880cf650 0x7f28880c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288e5c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28880cfae0 0x7f28880c7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288e5c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f2888141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288e5c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880ce340 con 0x7f2888154a80 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288e5c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880ce060 con 0x7f28880cfae0 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288e5c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28880ce620 con 0x7f28880cf650 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f2888141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f2888141a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47060/0 (socket says 172.21.15.99:47060) 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 -- 172.21.15.99:0/3263571201 learned_addr learned my addr 172.21.15.99:0/3263571201 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f2887fff640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28880cf650 0x7f28880c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f28877fe640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28880cfae0 0x7f28880c7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 -- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28880cf650 msgr2=0x7f28880c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28880cf650 0x7f28880c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 -- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28880cfae0 msgr2=0x7f28880c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28880cfae0 0x7f28880c7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f288cb61640 1 -- 172.21.15.99:0/3263571201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2888142430 con 0x7f2888154a80 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f28877fe640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28880cfae0 0x7f28880c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.810+0000 7f2887fff640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28880cf650 0x7f28880c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.811+0000 7f288cb61640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f2888141a30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f288000cef0 tx=0x7f288000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.811+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288001a070 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.811+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288000d860 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.811+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2888142710 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.811+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288000dc50 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.811+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28880c8040 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.812+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2888074f60 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.812+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f288001d070 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.813+0000 7f2884ff9640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f287406b4d0 0x7f287406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.813+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f288008ac80 con 0x7f2888154a80 2024-03-04T10:05:45.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.813+0000 7f2887fff640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f287406b4d0 0x7f287406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:45.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.813+0000 7f2887fff640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f287406b4d0 0x7f287406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2878036000 tx=0x7f287803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:45.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.817+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f28800599c0 con 0x7f2888154a80 2024-03-04T10:05:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.988+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28880b4e00 con 0x7f2888154a80 2024-03-04T10:05:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.989+0000 7f2884ff9640 1 -- 172.21.15.99:0/3263571201 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17082 (secure 0 0 0) 0x7f28800592c0 con 0x7f2888154a80 2024-03-04T10:05:45.991 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:45.991 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":41,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:21.677615+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/434536114":"2024-03-05T10:05:00.614507+0000","172.21.15.99:6833/434536114":"2024-03-05T10:05:00.614507+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f287406b4d0 msgr2=0x7f287406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f287406b4d0 0x7f287406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2878036000 tx=0x7f287803c040 comp rx=0 tx=0).stop 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 msgr2=0x7f2888141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f2888141a30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f288000cef0 tx=0x7f288000a710 comp rx=0 tx=0).stop 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 shutdown_connections 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f287406b4d0 0x7f287406d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2888154a80 0x7f2888141a30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28880cfae0 0x7f28880c7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 --2- 172.21.15.99:0/3263571201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28880cf650 0x7f28880c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 >> 172.21.15.99:0/3263571201 conn(0x7f288806f5f0 msgr2=0x7f28881530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 shutdown_connections 2024-03-04T10:05:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:45.991+0000 7f288e5c4640 1 -- 172.21.15.99:0/3263571201 wait complete. 2024-03-04T10:05:46.004 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:05:46.004 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.103+0000 7fa8dd2d7640 1 Processor -- start 2024-03-04T10:05:46.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.103+0000 7fa8dd2d7640 1 -- start start 2024-03-04T10:05:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8dd2d7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa8d80cd270 0x7fa8d8166660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8dd2d7640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d815cfe0 con 0x7fa8d81597f0 2024-03-04T10:05:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8dd2d7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80cf630 con 0x7fa8d815c030 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8dd2d7640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d806baf0 con 0x7fa8d80cd270 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d6ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa8d81597f0 0x7fa8d806acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44042/0 (socket says 172.21.15.99:44042) 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d6ffd640 1 -- 172.21.15.99:0/2717441182 learned_addr learned my addr 172.21.15.99:0/2717441182 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d77fe640 1 --2- 172.21.15.99:0/2717441182 >> v2:172.21.15.99:3300/0 conn(0x7fa8d80cd270 0x7fa8d8166660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d5ffb640 1 -- 172.21.15.99:0/2717441182 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782784610 0 0) 0x7fa8d815cfe0 con 0x7fa8d81597f0 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d5ffb640 1 -- 172.21.15.99:0/2717441182 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c0003520 con 0x7fa8d81597f0 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d77fe640 1 -- 172.21.15.99:0/2717441182 >> v1:172.21.15.162:6789/0 conn(0x7fa8d815c030 legacy=0x7fa8d806b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d77fe640 1 -- 172.21.15.99:0/2717441182 >> v1:172.21.15.99:6789/0 conn(0x7fa8d81597f0 legacy=0x7fa8d806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.104+0000 7fa8d77fe640 1 -- 172.21.15.99:0/2717441182 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80cf910 con 0x7fa8d80cd270 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.105+0000 7fa8d77fe640 1 --2- 172.21.15.99:0/2717441182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d80cd270 0x7fa8d8166660 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009d90 tx=0x7fa8d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b8188531c6d1c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.105+0000 7fa8d5ffb640 1 -- 172.21.15.99:0/2717441182 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0039070 con 0x7fa8d80cd270 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.105+0000 7fa8d5ffb640 1 -- 172.21.15.99:0/2717441182 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d002cc60 con 0x7fa8d80cd270 2024-03-04T10:05:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.105+0000 7fa8d5ffb640 1 -- 172.21.15.99:0/2717441182 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0034660 con 0x7fa8d80cd270 2024-03-04T10:05:46.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/2717441182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d80cd270 msgr2=0x7fa8d8166660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/2717441182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d80cd270 0x7fa8d8166660 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009d90 tx=0x7fa8d002beb0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/2717441182 shutdown_connections 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/2717441182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d80cd270 0x7fa8d8166660 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/2717441182 >> 172.21.15.99:0/2717441182 conn(0x7fa8d80d0710 msgr2=0x7fa8d80d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/2717441182 shutdown_connections 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.106+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/2717441182 wait complete. 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.107+0000 7fa8dd2d7640 1 Processor -- start 2024-03-04T10:05:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.107+0000 7fa8dd2d7640 1 -- start start 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8dd2d7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8d80cd270 0x7fa8d80c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8dd2d7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 0x7fa8d8070070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8dd2d7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8d815c030 0x7fa8d80c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8dd2d7640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80cfed0 con 0x7fa8d81597f0 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8dd2d7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80d0240 con 0x7fa8d80cd270 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8dd2d7640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80d1350 con 0x7fa8d815c030 2024-03-04T10:05:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 0x7fa8d8070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 0x7fa8d8070070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47076/0 (socket says 172.21.15.99:47076) 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 -- 172.21.15.99:0/1201524375 learned_addr learned my addr 172.21.15.99:0/1201524375 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d6ffd640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8d80cd270 0x7fa8d80c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d77fe640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8d815c030 0x7fa8d80c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 -- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8d815c030 msgr2=0x7fa8d80c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8d815c030 0x7fa8d80c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 -- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8d80cd270 msgr2=0x7fa8d80c5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8d80cd270 0x7fa8d80c5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.108+0000 7fa8d67fc640 1 -- 172.21.15.99:0/1201524375 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80c49b0 con 0x7fa8d81597f0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.109+0000 7fa8d67fc640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 0x7fa8d8070070 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00c400 tx=0x7fa8cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.109+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0052d0 con 0x7fa8d81597f0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.109+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc005490 con 0x7fa8d81597f0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.109+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc007260 con 0x7fa8d81597f0 2024-03-04T10:05:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.109+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d80c4c90 con 0x7fa8d81597f0 2024-03-04T10:05:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.109+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d80c5250 con 0x7fa8d81597f0 2024-03-04T10:05:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.110+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d806cd70 con 0x7fa8d81597f0 2024-03-04T10:05:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.110+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa8cc007400 con 0x7fa8d81597f0 2024-03-04T10:05:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.111+0000 7fa8bf7fe640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8c406b4d0 0x7fa8c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.111+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fa8cc089810 con 0x7fa8d81597f0 2024-03-04T10:05:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.111+0000 7fa8d6ffd640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8c406b4d0 0x7fa8c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.111+0000 7fa8d6ffd640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8c406b4d0 0x7fa8c406d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80011e0 tx=0x7fa8c8007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.115+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa8cc057ed0 con 0x7fa8d81597f0 2024-03-04T10:05:46.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.323+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d81591f0 con 0x7fa8d81597f0 2024-03-04T10:05:46.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.324+0000 7fa8bf7fe640 1 -- 172.21.15.99:0/1201524375 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7fa8cc05bf70 con 0x7fa8d81597f0 2024-03-04T10:05:46.325 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:46.325 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:46.325 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:46.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8c406b4d0 msgr2=0x7fa8c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8c406b4d0 0x7fa8c406d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80011e0 tx=0x7fa8c8007d30 comp rx=0 tx=0).stop 2024-03-04T10:05:46.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 msgr2=0x7fa8d8070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 0x7fa8d8070070 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00c400 tx=0x7fa8cc00c8c0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 shutdown_connections 2024-03-04T10:05:46.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa8c406b4d0 0x7fa8c406d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8d815c030 0x7fa8d80c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa8d81597f0 0x7fa8d8070070 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 --2- 172.21.15.99:0/1201524375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8d80cd270 0x7fa8d80c5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 >> 172.21.15.99:0/1201524375 conn(0x7fa8d80d0710 msgr2=0x7fa8d815b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:46.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 shutdown_connections 2024-03-04T10:05:46.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.326+0000 7fa8dd2d7640 1 -- 172.21.15.99:0/1201524375 wait complete. 2024-03-04T10:05:46.338 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:46.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.436+0000 7f566a668640 1 Processor -- start 2024-03-04T10:05:46.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.437+0000 7f566a668640 1 -- start start 2024-03-04T10:05:46.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.437+0000 7f566a668640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f56640b0700 0x7f56640b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.437+0000 7f566a668640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 0x7f56640b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.437+0000 7f566a668640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640ccad0 con 0x7f566415b960 2024-03-04T10:05:46.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.437+0000 7f566a668640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56640b5190 con 0x7f56640b0700 2024-03-04T10:05:46.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.437+0000 7f566a668640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56640cf860 con 0x7f56640b47c0 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 0x7f56640b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 0x7f56640b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:56400/0 (socket says 172.21.15.99:56400) 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5663fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f566415b960 0x7f56640d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:36898/0 (socket says 172.21.15.99:36898) 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 -- 172.21.15.99:0/361592453 learned_addr learned my addr 172.21.15.99:0/361592453 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f56637fe640 1 --2- 172.21.15.99:0/361592453 >> v2:172.21.15.162:3300/0 conn(0x7f56640b0700 0x7f56640b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5662ffd640 1 -- 172.21.15.99:0/361592453 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112137614 0 0) 0x7f56640ccad0 con 0x7f566415b960 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 -- 172.21.15.99:0/361592453 >> v2:172.21.15.162:3300/0 conn(0x7f56640b0700 msgr2=0x7f56640b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 --2- 172.21.15.99:0/361592453 >> v2:172.21.15.162:3300/0 conn(0x7f56640b0700 0x7f56640b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 -- 172.21.15.99:0/361592453 >> v1:172.21.15.162:6789/0 conn(0x7f566415b960 legacy=0x7f56640d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.438+0000 7f5668c05640 1 -- 172.21.15.99:0/361592453 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56640cfab0 con 0x7f56640b47c0 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.439+0000 7f5668c05640 1 --2- 172.21.15.99:0/361592453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 0x7f56640b4c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f565c009e50 tx=0x7f565c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5860747ce77f0462 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.439+0000 7f5662ffd640 1 -- 172.21.15.99:0/361592453 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c02ca70 con 0x7f56640b47c0 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.439+0000 7f5662ffd640 1 -- 172.21.15.99:0/361592453 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c02cc10 con 0x7f56640b47c0 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.439+0000 7f5662ffd640 1 -- 172.21.15.99:0/361592453 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c0343f0 con 0x7f56640b47c0 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 -- 172.21.15.99:0/361592453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 msgr2=0x7f56640b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 --2- 172.21.15.99:0/361592453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 0x7f56640b4c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f565c009e50 tx=0x7f565c02bed0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 -- 172.21.15.99:0/361592453 shutdown_connections 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 --2- 172.21.15.99:0/361592453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b47c0 0x7f56640b4c50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 --2- 172.21.15.99:0/361592453 >> v2:172.21.15.162:3300/0 conn(0x7f56640b0700 0x7f56640b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 -- 172.21.15.99:0/361592453 >> 172.21.15.99:0/361592453 conn(0x7f566406f5f0 msgr2=0x7f566406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 -- 172.21.15.99:0/361592453 shutdown_connections 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.440+0000 7f566a668640 1 -- 172.21.15.99:0/361592453 wait complete. 2024-03-04T10:05:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.441+0000 7f566a668640 1 Processor -- start 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.441+0000 7f566a668640 1 -- start start 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b0700 0x7f5664159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 0x7f56640ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f566415b960 0x7f566415e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566415bd40 con 0x7f56640b47c0 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56640d0100 con 0x7f566415b960 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566415bec0 con 0x7f56640b0700 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 0x7f56640ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 0x7f56640ace70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47088/0 (socket says 172.21.15.99:47088) 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 -- 172.21.15.99:0/278073849 learned_addr learned my addr 172.21.15.99:0/278073849 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f5663fff640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b0700 0x7f5664159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f5668c05640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f566415b960 0x7f566415e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 -- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b0700 msgr2=0x7f5664159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b0700 0x7f5664159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 -- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f566415b960 msgr2=0x7f566415e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f566415b960 0x7f566415e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 -- 172.21.15.99:0/278073849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56640a8680 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f5668c05640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f566415b960 0x7f566415e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f56637fe640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 0x7f56640ace70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f565800e9b0 tx=0x7f565800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658017070 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565800c9a0 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56640a8960 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565800cd90 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.442+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56640a8f20 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.443+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5664074f60 con 0x7f56640b47c0 2024-03-04T10:05:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.444+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f565801a070 con 0x7f56640b47c0 2024-03-04T10:05:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.444+0000 7f5660ff9640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f565006b500 0x7f565006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.444+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f565808a2d0 con 0x7f56640b47c0 2024-03-04T10:05:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.445+0000 7f5663fff640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f565006b500 0x7f565006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.445+0000 7f5663fff640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f565006b500 0x7f565006d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5654007fd0 tx=0x7f5654007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.449+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5658058910 con 0x7f56640b47c0 2024-03-04T10:05:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.655+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56640b0d30 con 0x7f56640b47c0 2024-03-04T10:05:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.656+0000 7f5660ff9640 1 -- 172.21.15.99:0/278073849 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4945 (secure 0 0 0) 0x7f565805c9b0 con 0x7f56640b47c0 2024-03-04T10:05:46.657 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:46.657 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:04:51.516792+0000","modified":"2024-03-04T10:05:03.628528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6835/2085408316","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2085408316},{"type":"v1","addr":"172.21.15.99:6835","nonce":2085408316}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-04T10:05:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 21 2024-03-04T10:05:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f565006b500 msgr2=0x7f565006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f565006b500 0x7f565006d9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5654007fd0 tx=0x7f5654007de0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 msgr2=0x7f56640ace70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 0x7f56640ace70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f565800e9b0 tx=0x7f565800ee70 comp rx=0 tx=0).stop 2024-03-04T10:05:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 shutdown_connections 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f565006b500 0x7f565006d9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f566415b960 0x7f566415e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56640b47c0 0x7f56640ace70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 --2- 172.21.15.99:0/278073849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56640b0700 0x7f5664159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 >> 172.21.15.99:0/278073849 conn(0x7f566406f5f0 msgr2=0x7f56640b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 shutdown_connections 2024-03-04T10:05:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.658+0000 7f566a668640 1 -- 172.21.15.99:0/278073849 wait complete. 2024-03-04T10:05:46.670 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:46.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.770+0000 7f65ea7b2640 1 Processor -- start 2024-03-04T10:05:46.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.770+0000 7f65ea7b2640 1 -- start start 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65ea7b2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f65e40cf650 0x7f65e40cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65ea7b2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e415f020 con 0x7f65e4070de0 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65ea7b2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e40cd010 con 0x7f65e4154a80 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65ea7b2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e40cdaa0 con 0x7f65e40cf650 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65e8d4f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f65e4070de0 0x7f65e4071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44046/0 (socket says 172.21.15.99:44046) 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65e8d4f640 1 -- 172.21.15.99:0/4097231423 learned_addr learned my addr 172.21.15.99:0/4097231423 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737571320 0 0) 0x7f65dc001490 con 0x7f65e4070de0 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65cc003520 con 0x7f65e4070de0 2024-03-04T10:05:46.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.771+0000 7f65e37fe640 1 --2- 172.21.15.99:0/4097231423 >> v2:172.21.15.162:3300/0 conn(0x7f65e40cf650 0x7f65e40cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217796750 0 0) 0x7f65cc003520 con 0x7f65e4070de0 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 >> v2:172.21.15.162:3300/0 conn(0x7f65e40cf650 msgr2=0x7f65e40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 --2- 172.21.15.99:0/4097231423 >> v2:172.21.15.162:3300/0 conn(0x7f65e40cf650 0x7f65e40cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 >> v1:172.21.15.162:6789/0 conn(0x7f65e4154a80 legacy=0x7f65e415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e40cdd80 con 0x7f65e4070de0 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f65dc001bc0 con 0x7f65e4070de0 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 >> v1:172.21.15.99:6789/0 conn(0x7f65e4070de0 legacy=0x7f65e4071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc0039b0 0x7f65cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0051c0 0x7f65cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40cdaa0 con 0x7f65cc004800 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc00c110 con 0x7f65cc0051c0 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc00c310 con 0x7f65cc0039b0 2024-03-04T10:05:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e37fe640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e8d4f640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0051c0 0x7f65cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e3fff640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc0039b0 0x7f65cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e37fe640 1 -- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc0039b0 msgr2=0x7f65cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e37fe640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65cc0039b0 0x7f65cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e37fe640 1 -- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0051c0 msgr2=0x7f65cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e37fe640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc0051c0 0x7f65cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e8d4f640 1 -- 172.21.15.99:0/4097231423 reap_dead start 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.772+0000 7f65e37fe640 1 -- 172.21.15.99:0/4097231423 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc003520 con 0x7f65cc004800 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65e37fe640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65d800ba30 tx=0x7f65cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1336c8f396ab063f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8036020 con 0x7f65cc004800 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d8032640 con 0x7f65cc004800 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65e2ffd640 1 -- 172.21.15.99:0/4097231423 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d800b600 con 0x7f65cc004800 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65ea7b2640 1 -- 172.21.15.99:0/4097231423 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc004800 msgr2=0x7f65cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f65d800ba30 tx=0x7f65cc005830 comp rx=0 tx=0).stop 2024-03-04T10:05:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65ea7b2640 1 -- 172.21.15.99:0/4097231423 shutdown_connections 2024-03-04T10:05:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.773+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/4097231423 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65cc004800 0x7f65cc004c80 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.774+0000 7f65ea7b2640 1 -- 172.21.15.99:0/4097231423 >> 172.21.15.99:0/4097231423 conn(0x7f65e406f5f0 msgr2=0x7f65e406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.774+0000 7f65ea7b2640 1 -- 172.21.15.99:0/4097231423 shutdown_connections 2024-03-04T10:05:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.774+0000 7f65ea7b2640 1 -- 172.21.15.99:0/4097231423 wait complete. 2024-03-04T10:05:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.775+0000 7f65ea7b2640 1 Processor -- start 2024-03-04T10:05:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.775+0000 7f65ea7b2640 1 -- start start 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65ea7b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc004800 0x7f65e40c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65ea7b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 0x7f65e40b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65ea7b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65e40b29f0 0x7f65e4161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65ea7b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40ce620 con 0x7f65e41f7d00 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65ea7b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40ce340 con 0x7f65cc004800 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65ea7b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e4154e60 con 0x7f65e40b29f0 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 0x7f65e40b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 0x7f65e40b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47114/0 (socket says 172.21.15.99:47114) 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 -- 172.21.15.99:0/3641775829 learned_addr learned my addr 172.21.15.99:0/3641775829 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e3fff640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc004800 0x7f65e40c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e8d4f640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65e40b29f0 0x7f65e4161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 -- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65e40b29f0 msgr2=0x7f65e4161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65e40b29f0 0x7f65e4161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 -- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc004800 msgr2=0x7f65e40c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc004800 0x7f65e40c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 -- 172.21.15.99:0/3641775829 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e40c8200 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e37fe640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 0x7f65e40b24b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f65d8035040 tx=0x7f65d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8041070 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.776+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d8030a50 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.777+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e40ca910 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.777+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8036040 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.777+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e40caed0 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.777+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e4074f60 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.778+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f65d804c060 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.778+0000 7f65e0ff9640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65d006b4d0 0x7f65d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.778+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f65d80ac0d0 con 0x7f65e41f7d00 2024-03-04T10:05:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.779+0000 7f65e3fff640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65d006b4d0 0x7f65d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.779+0000 7f65e3fff640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65d006b4d0 0x7f65d006d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f65d4000c00 tx=0x7f65d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.782+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f65d807a700 con 0x7f65e41f7d00 2024-03-04T10:05:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.955+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65e40cd450 con 0x7f65e41f7d00 2024-03-04T10:05:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.956+0000 7f65e0ff9640 1 -- 172.21.15.99:0/3641775829 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17082 (secure 0 0 0) 0x7f65d807e7a0 con 0x7f65e41f7d00 2024-03-04T10:05:46.957 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:46.958 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":41,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:21.677615+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:04:49.926068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-04T10:04:50.486347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/434536114":"2024-03-05T10:05:00.614507+0000","172.21.15.99:6833/434536114":"2024-03-05T10:05:00.614507+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65d006b4d0 msgr2=0x7f65d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65d006b4d0 0x7f65d006d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f65d4000c00 tx=0x7f65d4009040 comp rx=0 tx=0).stop 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 msgr2=0x7f65e40b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 0x7f65e40b24b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f65d8035040 tx=0x7f65d8004730 comp rx=0 tx=0).stop 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 shutdown_connections 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f65d006b4d0 0x7f65d006d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f65e40b29f0 0x7f65e4161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f65e41f7d00 0x7f65e40b24b0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 --2- 172.21.15.99:0/3641775829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65cc004800 0x7f65e40c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 >> 172.21.15.99:0/3641775829 conn(0x7f65e406f5f0 msgr2=0x7f65e4153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 shutdown_connections 2024-03-04T10:05:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:46.958+0000 7f65ea7b2640 1 -- 172.21.15.99:0/3641775829 wait complete. 2024-03-04T10:05:46.971 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:05:47.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.070+0000 7f8751a44640 1 Processor -- start 2024-03-04T10:05:47.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.070+0000 7f8751a44640 1 -- start start 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f8751a44640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f874c15de00 0x7f874c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f8751a44640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874c15e1e0 con 0x7f874c0b5470 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f8751a44640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874c0d1c10 con 0x7f874c0b9380 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f8751a44640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874c0d2820 con 0x7f874c15de00 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f874b7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f874c0b5470 0x7f874c0d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44060/0 (socket says 172.21.15.99:44060) 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f874b7fe640 1 -- 172.21.15.99:0/149614682 learned_addr learned my addr 172.21.15.99:0/149614682 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289722291 0 0) 0x7f874c15e1e0 con 0x7f874c0b5470 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8738003520 con 0x7f874c0b5470 2024-03-04T10:05:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.071+0000 7f874affd640 1 --2- 172.21.15.99:0/149614682 >> v2:172.21.15.162:3300/0 conn(0x7f874c15de00 0x7f874c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201906222 0 0) 0x7f8738003520 con 0x7f874c0b5470 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 >> v2:172.21.15.162:3300/0 conn(0x7f874c15de00 msgr2=0x7f874c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 --2- 172.21.15.99:0/149614682 >> v2:172.21.15.162:3300/0 conn(0x7f874c15de00 0x7f874c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 >> v1:172.21.15.162:6789/0 conn(0x7f874c0b9380 legacy=0x7f874c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c0d1ef0 con 0x7f874c0b5470 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f87440019f0 con 0x7f874c0b5470 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 >> v1:172.21.15.99:6789/0 conn(0x7f874c0b5470 legacy=0x7f874c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87380051c0 0x7f87380141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0d2820 con 0x7f87380039b0 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8738014870 con 0x7f87380051c0 2024-03-04T10:05:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8738014a70 con 0x7f8738004800 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874affd640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874b7fe640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87380051c0 0x7f87380141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f8743fff640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874affd640 1 -- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8738004800 msgr2=0x7f8738004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874affd640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8738004800 0x7f8738004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874affd640 1 -- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87380051c0 msgr2=0x7f87380141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874affd640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87380051c0 0x7f87380141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874b7fe640 1 -- 172.21.15.99:0/149614682 reap_dead start 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.072+0000 7f874affd640 1 -- 172.21.15.99:0/149614682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8738003520 con 0x7f87380039b0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f874affd640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f873c009ac0 tx=0x7f8738005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ec31c447be4a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c039070 con 0x7f87380039b0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c02c6b0 con 0x7f87380039b0 2024-03-04T10:05:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f874a7fc640 1 -- 172.21.15.99:0/149614682 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c009690 con 0x7f87380039b0 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f8751a44640 1 -- 172.21.15.99:0/149614682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87380039b0 msgr2=0x7f8738003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f8751a44640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f873c009ac0 tx=0x7f8738005830 comp rx=0 tx=0).stop 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f8751a44640 1 -- 172.21.15.99:0/149614682 shutdown_connections 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f8751a44640 1 --2- 172.21.15.99:0/149614682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87380039b0 0x7f8738003dd0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.073+0000 7f8751a44640 1 -- 172.21.15.99:0/149614682 >> 172.21.15.99:0/149614682 conn(0x7f874c0d0470 msgr2=0x7f874c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.074+0000 7f8751a44640 1 -- 172.21.15.99:0/149614682 shutdown_connections 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.074+0000 7f8751a44640 1 -- 172.21.15.99:0/149614682 wait complete. 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 Processor -- start 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 -- start start 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87380039b0 0x7f874c160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 0x7f874c0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f874c0c9160 0x7f874c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0b97e0 con 0x7f874c14ede0 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0d24b0 con 0x7f874c0c9160 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8751a44640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f874c0b2e60 con 0x7f87380039b0 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8743fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 0x7f874c0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8743fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 0x7f874c0b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47124/0 (socket says 172.21.15.99:47124) 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.075+0000 7f8743fff640 1 -- 172.21.15.99:0/1025998609 learned_addr learned my addr 172.21.15.99:0/1025998609 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f874b7fe640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f874c0c9160 0x7f874c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8743fff640 1 -- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87380039b0 msgr2=0x7f874c160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f874affd640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87380039b0 0x7f874c160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8743fff640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87380039b0 0x7f874c160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8743fff640 1 -- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f874c0c9160 msgr2=0x7f874c0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8743fff640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f874c0c9160 0x7f874c0c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8743fff640 1 -- 172.21.15.99:0/1025998609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874c1550e0 con 0x7f874c14ede0 2024-03-04T10:05:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f874affd640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87380039b0 0x7f874c160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f874b7fe640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f874c0c9160 0x7f874c0c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8743fff640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 0x7f874c0b3460 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f873400e530 tx=0x7f873400e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340053a0 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734018070 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874c1553c0 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8734003810 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.076+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874c161040 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.077+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874c074f60 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.077+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f87340039b0 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.078+0000 7f8748ff9640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.078+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f873408a040 con 0x7f874c14ede0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.078+0000 7f874affd640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.079+0000 7f874affd640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f873c036000 tx=0x7f873c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:47.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.082+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8734058cd0 con 0x7f874c14ede0 2024-03-04T10:05:47.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.291+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f874c0ce7a0 con 0x7f874c14ede0 2024-03-04T10:05:47.292 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:05:47.291+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:05:47.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.293+0000 7f8748ff9640 1 -- 172.21.15.99:0/1025998609 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) v1 ==== 161+0+0 (secure 0 0 0) 0x7f87340585d0 con 0x7f874c14ede0 2024-03-04T10:05:47.294 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:05:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872c06b4d0 msgr2=0x7f872c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872c06b4d0 0x7f872c06d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f873c036000 tx=0x7f873c030000 comp rx=0 tx=0).stop 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 msgr2=0x7f874c0b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 0x7f874c0b3460 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f873400e530 tx=0x7f873400e9f0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 shutdown_connections 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872c06b4d0 0x7f872c06d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f874c0c9160 0x7f874c0c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874c14ede0 0x7f874c0b3460 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 --2- 172.21.15.99:0/1025998609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87380039b0 0x7f874c160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 >> 172.21.15.99:0/1025998609 conn(0x7f874c0d0470 msgr2=0x7f874c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 shutdown_connections 2024-03-04T10:05:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.295+0000 7f8751a44640 1 -- 172.21.15.99:0/1025998609 wait complete. 2024-03-04T10:05:47.299 INFO:tasks.ceph.mds.c.smithi099.stderr: -3355> 2024-03-04T10:05:06.624+0000 7f8c8a11f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:05:47.309 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T10:05:47.328 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:05:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.410+0000 7ff4a4b69640 1 Processor -- start 2024-03-04T10:05:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.410+0000 7ff4a4b69640 1 -- start start 2024-03-04T10:05:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff4a4b69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 0x7ff4a006acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff4a4b69640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff4a00b0150 0x7ff4a006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff4a4b69640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a015cf50 con 0x7ff4a00c3e20 2024-03-04T10:05:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff4a4b69640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a00cc390 con 0x7ff4a00b0150 2024-03-04T10:05:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff4a4b69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00cc670 con 0x7ff4a015c7f0 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 0x7ff4a006acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49dd9b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff4a00b0150 0x7ff4a006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 0x7ff4a006acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:56480/0 (socket says 172.21.15.99:56480) 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49ed9d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff4a00c3e20 0x7ff4a015f750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:36932/0 (socket says 172.21.15.99:36932) 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 -- 172.21.15.99:0/2093133845 learned_addr learned my addr 172.21.15.99:0/2093133845 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 -- 172.21.15.99:0/2093133845 >> v2:172.21.15.162:3300/0 conn(0x7ff4a00b0150 msgr2=0x7ff4a006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 --2- 172.21.15.99:0/2093133845 >> v2:172.21.15.162:3300/0 conn(0x7ff4a00b0150 0x7ff4a006b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 -- 172.21.15.99:0/2093133845 >> v1:172.21.15.162:6789/0 conn(0x7ff4a00c3e20 legacy=0x7ff4a015f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49e59c640 1 -- 172.21.15.99:0/2093133845 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00cc950 con 0x7ff4a015c7f0 2024-03-04T10:05:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.411+0000 7ff49dd9b640 1 --2- 172.21.15.99:0/2093133845 >> v2:172.21.15.162:3300/0 conn(0x7ff4a00b0150 0x7ff4a006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.412+0000 7ff49e59c640 1 --2- 172.21.15.99:0/2093133845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 0x7ff4a006acd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff490009870 tx=0x7ff49002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a48e08ee3fc8023 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.412+0000 7ff49d59a640 1 -- 172.21.15.99:0/2093133845 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4900317c0 con 0x7ff4a015c7f0 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.412+0000 7ff49d59a640 1 -- 172.21.15.99:0/2093133845 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff490037070 con 0x7ff4a015c7f0 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.412+0000 7ff49d59a640 1 -- 172.21.15.99:0/2093133845 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff490031b80 con 0x7ff4a015c7f0 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 -- 172.21.15.99:0/2093133845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 msgr2=0x7ff4a006acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/2093133845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 0x7ff4a006acd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff490009870 tx=0x7ff49002b860 comp rx=0 tx=0).stop 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 -- 172.21.15.99:0/2093133845 shutdown_connections 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/2093133845 >> v2:172.21.15.162:3300/0 conn(0x7ff4a00b0150 0x7ff4a006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/2093133845 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a015c7f0 0x7ff4a006acd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 -- 172.21.15.99:0/2093133845 >> 172.21.15.99:0/2093133845 conn(0x7ff4a00b22e0 msgr2=0x7ff4a00b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 -- 172.21.15.99:0/2093133845 shutdown_connections 2024-03-04T10:05:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.413+0000 7ff4a4b69640 1 -- 172.21.15.99:0/2093133845 wait complete. 2024-03-04T10:05:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.414+0000 7ff4a4b69640 1 Processor -- start 2024-03-04T10:05:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.414+0000 7ff4a4b69640 1 -- start start 2024-03-04T10:05:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff4a4b69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4a00b0150 0x7ff4a01f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff4a4b69640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 0x7ff4a00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff4a4b69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a00ca050 0x7ff4a01588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff4a4b69640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00b2e00 con 0x7ff4a00c3e20 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff4a4b69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a015cbe0 con 0x7ff4a00b0150 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff4a4b69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00b2f80 con 0x7ff4a00ca050 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff49dd9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 0x7ff4a00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff49dd9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 0x7ff4a00c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47156/0 (socket says 172.21.15.99:47156) 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff49dd9b640 1 -- 172.21.15.99:0/3897514902 learned_addr learned my addr 172.21.15.99:0/3897514902 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff49e59c640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4a00b0150 0x7ff4a01f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.415+0000 7ff49ed9d640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a00ca050 0x7ff4a01588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff49dd9b640 1 -- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a00ca050 msgr2=0x7ff4a01588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff49dd9b640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a00ca050 0x7ff4a01588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff49dd9b640 1 -- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4a00b0150 msgr2=0x7ff4a01f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff49dd9b640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4a00b0150 0x7ff4a01f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff49dd9b640 1 -- 172.21.15.99:0/3897514902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00c9150 con 0x7ff4a00c3e20 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff49dd9b640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 0x7ff4a00c8c10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff49400e900 tx=0x7ff49400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494017070 con 0x7ff4a00c3e20 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49400ca00 con 0x7ff4a00c3e20 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49400cdf0 con 0x7ff4a00c3e20 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a014efc0 con 0x7ff4a00c3e20 2024-03-04T10:05:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.416+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a00c65b0 con 0x7ff4a00c3e20 2024-03-04T10:05:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.417+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a0074f60 con 0x7ff4a00c3e20 2024-03-04T10:05:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.417+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff49401a070 con 0x7ff4a00c3e20 2024-03-04T10:05:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.418+0000 7ff48effd640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.418+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6302+0+0 (secure 0 0 0) 0x7ff49408a240 con 0x7ff4a00c3e20 2024-03-04T10:05:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.418+0000 7ff49e59c640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.418+0000 7ff49e59c640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff48006b4d0 0x7ff48006d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff490036000 tx=0x7ff49003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.422+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff4940588b0 con 0x7ff4a00c3e20 2024-03-04T10:05:47.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:47.630+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff4a0072f90 con 0x7ff4a00c3e20 2024-03-04T10:05:48.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.297+0000 7ff48effd640 1 -- 172.21.15.99:0/3897514902 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff49405c950 con 0x7ff4a00c3e20 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff48006b4d0 msgr2=0x7ff48006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff48006b4d0 0x7ff48006d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff490036000 tx=0x7ff49003d040 comp rx=0 tx=0).stop 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 msgr2=0x7ff4a00c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 0x7ff4a00c8c10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff49400e900 tx=0x7ff49400edc0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 shutdown_connections 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff48006b4d0 0x7ff48006d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff4a00ca050 0x7ff4a01588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff4a00c3e20 0x7ff4a00c8c10 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 --2- 172.21.15.99:0/3897514902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff4a00b0150 0x7ff4a01f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.299+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 >> 172.21.15.99:0/3897514902 conn(0x7ff4a00b22e0 msgr2=0x7ff4a015d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:48.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.300+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 shutdown_connections 2024-03-04T10:05:48.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.300+0000 7ff4a4b69640 1 -- 172.21.15.99:0/3897514902 wait complete. 2024-03-04T10:05:48.313 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T10:05:48.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.459+0000 7f546e716640 1 Processor -- start 2024-03-04T10:05:48.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.459+0000 7f546e716640 1 -- start start 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f546e716640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5468072300 0x7f546815de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f546e716640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54680cf090 con 0x7f54680cc9b0 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f546e716640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54680cb910 con 0x7f546813cf30 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f546e716640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54680cc520 con 0x7f5468072300 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f546ccb3640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f54680cc9b0 0x7f546815e380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44072/0 (socket says 172.21.15.99:44072) 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f546ccb3640 1 -- 172.21.15.99:0/1773561843 learned_addr learned my addr 172.21.15.99:0/1773561843 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:48.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f54677fe640 1 --2- 172.21.15.99:0/1773561843 >> v2:172.21.15.99:3300/0 conn(0x7f5468072300 0x7f546815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:48.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f5466ffd640 1 -- 172.21.15.99:0/1773561843 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044189142 0 0) 0x7f54680cf090 con 0x7f54680cc9b0 2024-03-04T10:05:48.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f5466ffd640 1 -- 172.21.15.99:0/1773561843 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450003520 con 0x7f54680cc9b0 2024-03-04T10:05:48.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.460+0000 7f54677fe640 1 -- 172.21.15.99:0/1773561843 >> v1:172.21.15.162:6789/0 conn(0x7f546813cf30 legacy=0x7f5468071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:48.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.461+0000 7f54677fe640 1 -- 172.21.15.99:0/1773561843 >> v1:172.21.15.99:6789/0 conn(0x7f54680cc9b0 legacy=0x7f546815e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:48.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.461+0000 7f54677fe640 1 -- 172.21.15.99:0/1773561843 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54680cbbf0 con 0x7f5468072300 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.461+0000 7f54677fe640 1 --2- 172.21.15.99:0/1773561843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5468072300 0x7f546815de40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3778214885433a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.461+0000 7f5466ffd640 1 -- 172.21.15.99:0/1773561843 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c030800 con 0x7f5468072300 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.461+0000 7f5466ffd640 1 -- 172.21.15.99:0/1773561843 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c0309a0 con 0x7f5468072300 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.461+0000 7f5466ffd640 1 -- 172.21.15.99:0/1773561843 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c030d70 con 0x7f5468072300 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 -- 172.21.15.99:0/1773561843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5468072300 msgr2=0x7f546815de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 --2- 172.21.15.99:0/1773561843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5468072300 0x7f546815de40 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f545c0097a0 tx=0x7f545c02c9e0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 -- 172.21.15.99:0/1773561843 shutdown_connections 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 --2- 172.21.15.99:0/1773561843 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5468072300 0x7f546815de40 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 -- 172.21.15.99:0/1773561843 >> 172.21.15.99:0/1773561843 conn(0x7f54680b1600 msgr2=0x7f54680b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 -- 172.21.15.99:0/1773561843 shutdown_connections 2024-03-04T10:05:48.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.462+0000 7f546e716640 1 -- 172.21.15.99:0/1773561843 wait complete. 2024-03-04T10:05:48.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.463+0000 7f546e716640 1 Processor -- start 2024-03-04T10:05:48.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.463+0000 7f546e716640 1 -- start start 2024-03-04T10:05:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546e716640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5468072300 0x7f546815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546e716640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 0x7f546813ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546e716640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546813cf30 0x7f546813bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546e716640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f546813d310 con 0x7f54680cc9b0 2024-03-04T10:05:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546e716640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54680cc1b0 con 0x7f5468072300 2024-03-04T10:05:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546e716640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f546813d490 con 0x7f546813cf30 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 0x7f546813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 0x7f546813ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47170/0 (socket says 172.21.15.99:47170) 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 -- 172.21.15.99:0/647726793 learned_addr learned my addr 172.21.15.99:0/647726793 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f5467fff640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5468072300 0x7f546815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546ccb3640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546813cf30 0x7f546813bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 -- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546813cf30 msgr2=0x7f546813bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546813cf30 0x7f546813bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 -- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5468072300 msgr2=0x7f546815ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5468072300 0x7f546815ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f54677fe640 1 -- 172.21.15.99:0/647726793 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546814cc60 con 0x7f54680cc9b0 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f5467fff640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5468072300 0x7f546815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:48.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.464+0000 7f546ccb3640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546813cf30 0x7f546813bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.465+0000 7f54677fe640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 0x7f546813ba80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f545c0098c0 tx=0x7f545c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.465+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c0389c0 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.465+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545c038b80 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.465+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545c0413f0 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.465+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546814cf40 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.465+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f546814d500 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.466+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5468074f60 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.466+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f545c041590 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.467+0000 7f5464ff9640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f545406b4d0 0x7f545406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.467+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f545c0ac630 con 0x7f54680cc9b0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.467+0000 7f5467fff640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f545406b4d0 0x7f545406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.467+0000 7f5467fff640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f545406b4d0 0x7f545406d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5458006fd0 tx=0x7f5458008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:48.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.471+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f545c034030 con 0x7f54680cc9b0 2024-03-04T10:05:48.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:48.641+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54680aef60 con 0x7f54680cc9b0 2024-03-04T10:05:49.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.298+0000 7f5464ff9640 1 -- 172.21.15.99:0/647726793 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v43) v1 ==== 181+0+0 (secure 0 0 0) 0x7f545c07b3b0 con 0x7f54680cc9b0 2024-03-04T10:05:49.300 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' removed 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.301+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f545406b4d0 msgr2=0x7f545406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.301+0000 7f546e716640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f545406b4d0 0x7f545406d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5458006fd0 tx=0x7f5458008040 comp rx=0 tx=0).stop 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.301+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 msgr2=0x7f546813ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.301+0000 7f546e716640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 0x7f546813ba80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f545c0098c0 tx=0x7f545c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 shutdown_connections 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f545406b4d0 0x7f545406d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f546813cf30 0x7f546813bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54680cc9b0 0x7f546813ba80 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 --2- 172.21.15.99:0/647726793 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5468072300 0x7f546815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 >> 172.21.15.99:0/647726793 conn(0x7f54680b1600 msgr2=0x7f546815e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 shutdown_connections 2024-03-04T10:05:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.302+0000 7f546e716640 1 -- 172.21.15.99:0/647726793 wait complete. 2024-03-04T10:05:49.317 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T10:05:49.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.423+0000 7fa3b85df640 1 Processor -- start 2024-03-04T10:05:49.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.423+0000 7fa3b85df640 1 -- start start 2024-03-04T10:05:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b85df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b01438e0 0x7fa3b00cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b85df640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa3b0143cc0 0x7fa3b00ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b85df640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3b00cedc0 0x7fa3b0071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b85df640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b00b2aa0 con 0x7fa3b00cedc0 2024-03-04T10:05:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b85df640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b015e5d0 con 0x7fa3b0143cc0 2024-03-04T10:05:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b85df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b015e8b0 con 0x7fa3b01438e0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3b00cedc0 0x7fa3b0071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3b00cedc0 0x7fa3b0071360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47184/0 (socket says 172.21.15.99:47184) 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 -- 172.21.15.99:0/2733698891 learned_addr learned my addr 172.21.15.99:0/2733698891 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b5b7a640 1 --2- 172.21.15.99:0/2733698891 >> v2:172.21.15.162:3300/0 conn(0x7fa3b0143cc0 0x7fa3b00ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b637b640 1 --2- 172.21.15.99:0/2733698891 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b01438e0 0x7fa3b00cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 -- 172.21.15.99:0/2733698891 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b01438e0 msgr2=0x7fa3b00cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 --2- 172.21.15.99:0/2733698891 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b01438e0 0x7fa3b00cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 -- 172.21.15.99:0/2733698891 >> v2:172.21.15.162:3300/0 conn(0x7fa3b0143cc0 msgr2=0x7fa3b00ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 --2- 172.21.15.99:0/2733698891 >> v2:172.21.15.162:3300/0 conn(0x7fa3b0143cc0 0x7fa3b00ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.424+0000 7fa3b6b7c640 1 -- 172.21.15.99:0/2733698891 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b015eb90 con 0x7fa3b00cedc0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.425+0000 7fa3b6b7c640 1 --2- 172.21.15.99:0/2733698891 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b00cedc0 0x7fa3b0071360 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa3b006f620 tx=0x7fa3a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3bdfd190d71a960 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.425+0000 7fa3b5379640 1 -- 172.21.15.99:0/2733698891 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a802ca70 con 0x7fa3b00cedc0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.425+0000 7fa3b5379640 1 -- 172.21.15.99:0/2733698891 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a802cc10 con 0x7fa3b00cedc0 2024-03-04T10:05:49.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.425+0000 7fa3b5379640 1 -- 172.21.15.99:0/2733698891 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80343f0 con 0x7fa3b00cedc0 2024-03-04T10:05:49.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 -- 172.21.15.99:0/2733698891 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b00cedc0 msgr2=0x7fa3b0071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2733698891 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b00cedc0 0x7fa3b0071360 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa3b006f620 tx=0x7fa3a802bed0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 -- 172.21.15.99:0/2733698891 shutdown_connections 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2733698891 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b00cedc0 0x7fa3b0071360 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2733698891 >> v2:172.21.15.162:3300/0 conn(0x7fa3b0143cc0 0x7fa3b00ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2733698891 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b01438e0 0x7fa3b00cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 -- 172.21.15.99:0/2733698891 >> 172.21.15.99:0/2733698891 conn(0x7fa3b00b1600 msgr2=0x7fa3b00b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:49.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 -- 172.21.15.99:0/2733698891 shutdown_connections 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.426+0000 7fa3b85df640 1 -- 172.21.15.99:0/2733698891 wait complete. 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.427+0000 7fa3b85df640 1 Processor -- start 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.427+0000 7fa3b85df640 1 -- start start 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b00cedc0 0x7fa3b013af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 0x7fa3b015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b0141f50 0x7fa3b01423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00cf250 con 0x7fa3b01438e0 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b015f150 con 0x7fa3b0141f50 2024-03-04T10:05:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00cbe50 con 0x7fa3b00cedc0 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 0x7fa3b015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 0x7fa3b015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47192/0 (socket says 172.21.15.99:47192) 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 -- 172.21.15.99:0/2257958918 learned_addr learned my addr 172.21.15.99:0/2257958918 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 -- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b00cedc0 msgr2=0x7fa3b013af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b00cedc0 0x7fa3b013af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 -- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b0141f50 msgr2=0x7fa3b01423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b0141f50 0x7fa3b01423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 -- 172.21.15.99:0/2257958918 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0159e80 con 0x7fa3b01438e0 2024-03-04T10:05:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b5b7a640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 0x7fa3b015c550 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000aee0 tx=0x7fa3a000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0004220 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a00043c0 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b015b900 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0003830 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.428+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b013a750 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.429+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b00b0e20 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.430+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa3a0003be0 con 0x7fa3b01438e0 2024-03-04T10:05:49.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.430+0000 7fa3a6ffd640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa39406b500 0x7fa39406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:49.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.430+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5864+0+0 (secure 0 0 0) 0x7fa3a008c420 con 0x7fa3b01438e0 2024-03-04T10:05:49.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.430+0000 7fa3b637b640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa39406b500 0x7fa39406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:49.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.431+0000 7fa3b637b640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa39406b500 0x7fa39406d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00a810 tx=0x7fa3ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:49.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.434+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa3b00b0e20 con 0x7fa3b01438e0 2024-03-04T10:05:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:49.606+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3b0074f60 con 0x7fa3b01438e0 2024-03-04T10:05:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.303+0000 7fa3a6ffd640 1 -- 172.21.15.99:0/2257958918 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v44) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa3b0159e80 con 0x7fa3b01438e0 2024-03-04T10:05:50.305 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' removed 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa39406b500 msgr2=0x7fa39406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa39406b500 0x7fa39406d9c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00a810 tx=0x7fa3ac008040 comp rx=0 tx=0).stop 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 msgr2=0x7fa3b015c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 0x7fa3b015c550 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000aee0 tx=0x7fa3a000cb90 comp rx=0 tx=0).stop 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 shutdown_connections 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa39406b500 0x7fa39406d9c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b0141f50 0x7fa3b01423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b01438e0 0x7fa3b015c550 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 --2- 172.21.15.99:0/2257958918 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3b00cedc0 0x7fa3b013af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 >> 172.21.15.99:0/2257958918 conn(0x7fa3b00b1600 msgr2=0x7fa3b006acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 shutdown_connections 2024-03-04T10:05:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.306+0000 7fa3b85df640 1 -- 172.21.15.99:0/2257958918 wait complete. 2024-03-04T10:05:50.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T10:05:50.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.423+0000 7fc6ad429640 1 Processor -- start 2024-03-04T10:05:50.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.423+0000 7fc6ad429640 1 -- start start 2024-03-04T10:05:50.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6ad429640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a806b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6ad429640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc6a806fa80 0x7fc6a80cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6ad429640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a80cb290 con 0x7fc6a80cfa40 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6ad429640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a80701d0 con 0x7fc6a806fa80 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6ad429640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80704b0 con 0x7fc6a815df00 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6a67fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc6a806fa80 0x7fc6a80cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6a67fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc6a806fa80 0x7fc6a80cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47200/0 (socket says 172.21.15.99:47200) 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.424+0000 7fc6a67fc640 1 -- 172.21.15.99:0/2377991983 learned_addr learned my addr 172.21.15.99:0/2377991983 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.425+0000 7fc6a67fc640 1 -- 172.21.15.99:0/2377991983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 msgr2=0x7fc6a806b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.425+0000 7fc6a67fc640 1 --2- 172.21.15.99:0/2377991983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a806b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.425+0000 7fc6a67fc640 1 -- 172.21.15.99:0/2377991983 >> v1:172.21.15.162:6789/0 conn(0x7fc6a80cfa40 legacy=0x7fc6a80cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.425+0000 7fc6a67fc640 1 -- 172.21.15.99:0/2377991983 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a80cc8e0 con 0x7fc6a806fa80 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.425+0000 7fc6a67fc640 1 --2- 172.21.15.99:0/2377991983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a80cad50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009870 tx=0x7fc69c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9d0c158e94dbaa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.425+0000 7fc6a5ffb640 1 -- 172.21.15.99:0/2377991983 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0317c0 con 0x7fc6a806fa80 2024-03-04T10:05:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6a5ffb640 1 -- 172.21.15.99:0/2377991983 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c037070 con 0x7fc6a806fa80 2024-03-04T10:05:50.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6a5ffb640 1 -- 172.21.15.99:0/2377991983 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c031b80 con 0x7fc6a806fa80 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 -- 172.21.15.99:0/2377991983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 msgr2=0x7fc6a80cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 --2- 172.21.15.99:0/2377991983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a80cad50 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009870 tx=0x7fc69c02b860 comp rx=0 tx=0).stop 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 -- 172.21.15.99:0/2377991983 shutdown_connections 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 --2- 172.21.15.99:0/2377991983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a80cad50 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 --2- 172.21.15.99:0/2377991983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a806b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 -- 172.21.15.99:0/2377991983 >> 172.21.15.99:0/2377991983 conn(0x7fc6a80b2250 msgr2=0x7fc6a80b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 -- 172.21.15.99:0/2377991983 shutdown_connections 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.426+0000 7fc6ad429640 1 -- 172.21.15.99:0/2377991983 wait complete. 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.427+0000 7fc6ad429640 1 Processor -- start 2024-03-04T10:05:50.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.427+0000 7fc6ad429640 1 -- start start 2024-03-04T10:05:50.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6ad429640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a814dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6ad429640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6a80cfa40 0x7fc6a80c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6ad429640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6ad429640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a806fee0 con 0x7fc6a806fa80 2024-03-04T10:05:50.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6ad429640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80ccea0 con 0x7fc6a80cfa40 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6ad429640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80cd3c0 con 0x7fc6a815df00 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6a6ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a814dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6a6ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a814dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47210/0 (socket says 172.21.15.99:47210) 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.428+0000 7fc6a6ffd640 1 -- 172.21.15.99:0/387110632 learned_addr learned my addr 172.21.15.99:0/387110632 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a77fe640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a6ffd640 1 -- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 msgr2=0x7fc6a80b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a67fc640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6a80cfa40 0x7fc6a80c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a6ffd640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a80b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a6ffd640 1 -- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6a80cfa40 msgr2=0x7fc6a80c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a6ffd640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6a80cfa40 0x7fc6a80c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a6ffd640 1 -- 172.21.15.99:0/387110632 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a80c68a0 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a67fc640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6a80cfa40 0x7fc6a80c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6a6ffd640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a814dd90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc69800ba50 tx=0x7fc69800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698019070 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69800cba0 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698014440 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a80c5050 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.429+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a80c5610 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.430+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a8074f60 con 0x7fc6a806fa80 2024-03-04T10:05:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.430+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc69801c070 con 0x7fc6a806fa80 2024-03-04T10:05:50.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.431+0000 7fc68f7fe640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc69406b500 0x7fc69406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.431+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc698089d90 con 0x7fc6a806fa80 2024-03-04T10:05:50.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.431+0000 7fc6a67fc640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc69406b500 0x7fc69406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.432+0000 7fc6a67fc640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc69406b500 0x7fc69406d9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc69c036000 tx=0x7fc69c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:50.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.435+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc698058dc0 con 0x7fc6a806fa80 2024-03-04T10:05:50.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.609+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc6a806b4b0 con 0x7fc6a806fa80 2024-03-04T10:05:50.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.612+0000 7fc68f7fe640 1 -- 172.21.15.99:0/387110632 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc6980586c0 con 0x7fc6a806fa80 2024-03-04T10:05:50.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc69406b500 msgr2=0x7fc69406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc69406b500 0x7fc69406d9c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc69c036000 tx=0x7fc69c008030 comp rx=0 tx=0).stop 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 msgr2=0x7fc6a814dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a814dd90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc69800ba50 tx=0x7fc69800bf10 comp rx=0 tx=0).stop 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 shutdown_connections 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc69406b500 0x7fc69406d9c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6a815df00 0x7fc6a80b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6a80cfa40 0x7fc6a80c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 --2- 172.21.15.99:0/387110632 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6a806fa80 0x7fc6a814dd90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 >> 172.21.15.99:0/387110632 conn(0x7fc6a80b2250 msgr2=0x7fc6a80b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 shutdown_connections 2024-03-04T10:05:50.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.614+0000 7fc6ad429640 1 -- 172.21.15.99:0/387110632 wait complete. 2024-03-04T10:05:50.626 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry' 2024-03-04T10:05:50.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.725+0000 7f30ab297640 1 Processor -- start 2024-03-04T10:05:50.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.726+0000 7f30ab297640 1 -- start start 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.726+0000 7f30ab297640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a415b960 0x7f30a40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.726+0000 7f30ab297640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f30a40d0ad0 0x7f30a40b0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.726+0000 7f30ab297640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40ccad0 con 0x7f30a40b0c40 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.726+0000 7f30ab297640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a40cf860 con 0x7f30a40d0ad0 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.726+0000 7f30ab297640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40d0100 con 0x7f30a415b960 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a8832640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f30a40d0ad0 0x7f30a40b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a9834640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f30a40b0c40 0x7f30a40b47c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44078/0 (socket says 172.21.15.99:44078) 2024-03-04T10:05:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a9834640 1 -- 172.21.15.99:0/576969818 learned_addr learned my addr 172.21.15.99:0/576969818 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a8832640 1 -- 172.21.15.99:0/576969818 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a415b960 msgr2=0x7f30a40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a8832640 1 --2- 172.21.15.99:0/576969818 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a415b960 0x7f30a40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a8832640 1 -- 172.21.15.99:0/576969818 >> v1:172.21.15.99:6789/0 conn(0x7f30a40b0c40 legacy=0x7f30a40b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f309bfff640 1 -- 172.21.15.99:0/576969818 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700035340 0 0) 0x7f30a40ccad0 con 0x7f30a40b0c40 2024-03-04T10:05:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a8832640 1 -- 172.21.15.99:0/576969818 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a40cfab0 con 0x7f30a40d0ad0 2024-03-04T10:05:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.727+0000 7f30a8832640 1 --2- 172.21.15.99:0/576969818 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a40d0ad0 0x7f30a40b0700 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f30940097a0 tx=0x7f309402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86708cf2f4ab6cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f309bfff640 1 -- 172.21.15.99:0/576969818 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30940315a0 con 0x7f30a40d0ad0 2024-03-04T10:05:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f309bfff640 1 -- 172.21.15.99:0/576969818 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3094037070 con 0x7f30a40d0ad0 2024-03-04T10:05:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f309bfff640 1 -- 172.21.15.99:0/576969818 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094031960 con 0x7f30a40d0ad0 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 -- 172.21.15.99:0/576969818 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a40d0ad0 msgr2=0x7f30a40b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 --2- 172.21.15.99:0/576969818 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a40d0ad0 0x7f30a40b0700 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f30940097a0 tx=0x7f309402b840 comp rx=0 tx=0).stop 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 -- 172.21.15.99:0/576969818 shutdown_connections 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 --2- 172.21.15.99:0/576969818 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a40d0ad0 0x7f30a40b0700 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 --2- 172.21.15.99:0/576969818 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a415b960 0x7f30a40d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 -- 172.21.15.99:0/576969818 >> 172.21.15.99:0/576969818 conn(0x7f30a406f5f0 msgr2=0x7f30a406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 -- 172.21.15.99:0/576969818 shutdown_connections 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.728+0000 7f30ab297640 1 -- 172.21.15.99:0/576969818 wait complete. 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.729+0000 7f30ab297640 1 Processor -- start 2024-03-04T10:05:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.729+0000 7f30ab297640 1 -- start start 2024-03-04T10:05:50.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30ab297640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a40b0c40 0x7f30a4154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30ab297640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a40d0ad0 0x7f30a4157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30ab297640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 0x7f30a40b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30ab297640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a415bec0 con 0x7f30a415b960 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30ab297640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a415bd40 con 0x7f30a40d0ad0 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30ab297640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40b4ed0 con 0x7f30a40b0c40 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30a9834640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 0x7f30a40b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30a9834640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 0x7f30a40b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47222/0 (socket says 172.21.15.99:47222) 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.730+0000 7f30a9834640 1 -- 172.21.15.99:0/4064370189 learned_addr learned my addr 172.21.15.99:0/4064370189 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30a9834640 1 -- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a40b0c40 msgr2=0x7f30a4154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30a9834640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a40b0c40 0x7f30a4154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30a9834640 1 -- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a40d0ad0 msgr2=0x7f30a4157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30a9834640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a40d0ad0 0x7f30a4157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30a9834640 1 -- 172.21.15.99:0/4064370189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4158870 con 0x7f30a415b960 2024-03-04T10:05:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30a9834640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 0x7f30a40b17c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c410 tx=0x7f309c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c0074e0 con 0x7f30a415b960 2024-03-04T10:05:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c0047e0 con 0x7f30a415b960 2024-03-04T10:05:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a40ca490 con 0x7f30a415b960 2024-03-04T10:05:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309c004bd0 con 0x7f30a415b960 2024-03-04T10:05:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.731+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a40caf70 con 0x7f30a415b960 2024-03-04T10:05:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.732+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4074f60 con 0x7f30a415b960 2024-03-04T10:05:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.732+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f309c011070 con 0x7f30a415b960 2024-03-04T10:05:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.733+0000 7f3099ffb640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308806b4d0 0x7f308806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.733+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f309c089c80 con 0x7f30a415b960 2024-03-04T10:05:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.733+0000 7f30a9033640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308806b4d0 0x7f308806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.733+0000 7f30a9033640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308806b4d0 0x7f308806d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f30a000ab90 tx=0x7f30a0009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:50.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.737+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f309c01a2b0 con 0x7f30a415b960 2024-03-04T10:05:50.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:50.972+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]} v 0) v1 -- 0x7f30a415b360 con 0x7f30a415b960 2024-03-04T10:05:51.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.304+0000 7f3099ffb640 1 -- 172.21.15.99:0/4064370189 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_damaged_dentry"]}]=0 v137) v1 ==== 134+0+0 (secure 0 0 0) 0x7f309c058c20 con 0x7f30a415b960 2024-03-04T10:05:51.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308806b4d0 msgr2=0x7f308806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308806b4d0 0x7f308806d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f30a000ab90 tx=0x7f30a0009250 comp rx=0 tx=0).stop 2024-03-04T10:05:51.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 msgr2=0x7f30a40b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 0x7f30a40b17c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f309c00c410 tx=0x7f309c00c8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 shutdown_connections 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308806b4d0 0x7f308806d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f30a415b960 0x7f30a40b17c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30a40d0ad0 0x7f30a4157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 --2- 172.21.15.99:0/4064370189 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f30a40b0c40 0x7f30a4154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 >> 172.21.15.99:0/4064370189 conn(0x7f30a406f5f0 msgr2=0x7f30a40b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 shutdown_connections 2024-03-04T10:05:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.307+0000 7f30ab297640 1 -- 172.21.15.99:0/4064370189 wait complete. 2024-03-04T10:05:51.319 INFO:tasks.cephfs_test_runner:test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) ... ok 2024-03-04T10:05:51.319 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) That the MDS won't abort when the dentry is already known to be damaged. 2024-03-04T10:05:51.319 INFO:tasks.cephfs_test_runner:test_dentry_first_existing (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:05:51.320 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.423+0000 7f52f136f640 1 Processor -- start 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.423+0000 7f52f136f640 1 -- start start 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.423+0000 7f52f136f640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f52ec15de00 0x7f52ec0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.423+0000 7f52f136f640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.423+0000 7f52f136f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec15e1e0 con 0x7f52ec0b9380 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52f136f640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52ec0d2820 con 0x7f52ec15de00 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52f136f640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0d1c10 con 0x7f52ec0b5470 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eaffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f52ec15de00 0x7f52ec0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52e3fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f52ec0b9380 0x7f52ec0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:36964/0 (socket says 172.21.15.99:36964) 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52e3fff640 1 -- 172.21.15.99:0/1811967737 learned_addr learned my addr 172.21.15.99:0/1811967737 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eb7fe640 1 -- 172.21.15.99:0/1811967737 >> v2:172.21.15.162:3300/0 conn(0x7f52ec15de00 msgr2=0x7f52ec0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eb7fe640 1 --2- 172.21.15.99:0/1811967737 >> v2:172.21.15.162:3300/0 conn(0x7f52ec15de00 0x7f52ec0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eb7fe640 1 -- 172.21.15.99:0/1811967737 >> v1:172.21.15.162:6789/0 conn(0x7f52ec0b9380 legacy=0x7f52ec0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eb7fe640 1 -- 172.21.15.99:0/1811967737 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0d1ef0 con 0x7f52ec0b5470 2024-03-04T10:05:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.424+0000 7f52eaffd640 1 --2- 172.21.15.99:0/1811967737 >> v2:172.21.15.162:3300/0 conn(0x7f52ec15de00 0x7f52ec0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.425+0000 7f52eb7fe640 1 --2- 172.21.15.99:0/1811967737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec0b5900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009d90 tx=0x7f52e402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=176c64e83cf87e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.425+0000 7f52ea7fc640 1 -- 172.21.15.99:0/1811967737 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4039070 con 0x7f52ec0b5470 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.425+0000 7f52ea7fc640 1 -- 172.21.15.99:0/1811967737 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e402cc50 con 0x7f52ec0b5470 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.425+0000 7f52ea7fc640 1 -- 172.21.15.99:0/1811967737 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4034440 con 0x7f52ec0b5470 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 -- 172.21.15.99:0/1811967737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 msgr2=0x7f52ec0b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 --2- 172.21.15.99:0/1811967737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec0b5900 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009d90 tx=0x7f52e402bea0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 -- 172.21.15.99:0/1811967737 shutdown_connections 2024-03-04T10:05:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 --2- 172.21.15.99:0/1811967737 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec0b5900 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 --2- 172.21.15.99:0/1811967737 >> v2:172.21.15.162:3300/0 conn(0x7f52ec15de00 0x7f52ec0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 -- 172.21.15.99:0/1811967737 >> 172.21.15.99:0/1811967737 conn(0x7f52ec0d0470 msgr2=0x7f52ec0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 -- 172.21.15.99:0/1811967737 shutdown_connections 2024-03-04T10:05:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.426+0000 7f52f136f640 1 -- 172.21.15.99:0/1811967737 wait complete. 2024-03-04T10:05:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.427+0000 7f52f136f640 1 Processor -- start 2024-03-04T10:05:51.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.427+0000 7f52f136f640 1 -- start start 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52f136f640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52f136f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52ec0b9380 0x7f52ec14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52f136f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 0x7f52ec1f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52f136f640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0b97e0 con 0x7f52ec15de00 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52f136f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0d24b0 con 0x7f52ec0b9380 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52f136f640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0b72a0 con 0x7f52ec0b5470 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 0x7f52ec1f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 0x7f52ec1f3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47226/0 (socket says 172.21.15.99:47226) 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 -- 172.21.15.99:0/2602062097 learned_addr learned my addr 172.21.15.99:0/2602062097 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52e3fff640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52ec0b9380 0x7f52ec14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eaffd640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 -- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 msgr2=0x7f52ec157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 -- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52ec0b9380 msgr2=0x7f52ec14de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52ec0b9380 0x7f52ec14de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.428+0000 7f52eb7fe640 1 -- 172.21.15.99:0/2602062097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0c8c80 con 0x7f52ec15de00 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.429+0000 7f52eb7fe640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 0x7f52ec1f3a00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009eb0 tx=0x7f52e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.429+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4039050 con 0x7f52ec15de00 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.429+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e4045070 con 0x7f52ec15de00 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.429+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e403e610 con 0x7f52ec15de00 2024-03-04T10:05:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.429+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec0c8f60 con 0x7f52ec15de00 2024-03-04T10:05:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.429+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec0c9520 con 0x7f52ec15de00 2024-03-04T10:05:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.430+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec074f60 con 0x7f52ec15de00 2024-03-04T10:05:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.430+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f52e403e7b0 con 0x7f52ec15de00 2024-03-04T10:05:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.431+0000 7f52e37fe640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52cc06b4d0 0x7f52cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.431+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f52e40aca60 con 0x7f52ec15de00 2024-03-04T10:05:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.431+0000 7f52eaffd640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52cc06b4d0 0x7f52cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.431+0000 7f52eaffd640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52cc06b4d0 0x7f52cc06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009560 tx=0x7f52dc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.435+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f52e407b440 con 0x7f52ec15de00 2024-03-04T10:05:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.606+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52ec0d0a10 con 0x7f52ec15de00 2024-03-04T10:05:51.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.607+0000 7f52e37fe640 1 -- 172.21.15.99:0/2602062097 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+13385 (secure 0 0 0) 0x7f52e407f4e0 con 0x7f52ec15de00 2024-03-04T10:05:51.608 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:51.608 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":44,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:50.299550+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/2085408316":"2024-03-05T10:05:47.291956+0000","172.21.15.99:6835/434536114":"2024-03-05T10:05:00.614507+0000","172.21.15.99:6833/2085408316":"2024-03-05T10:05:47.291956+0000","172.21.15.99:6833/434536114":"2024-03-05T10:05:00.614507+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:51.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52cc06b4d0 msgr2=0x7f52cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52cc06b4d0 0x7f52cc06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009560 tx=0x7f52dc009190 comp rx=0 tx=0).stop 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 msgr2=0x7f52ec1f3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 0x7f52ec1f3a00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009eb0 tx=0x7f52e4004a40 comp rx=0 tx=0).stop 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 shutdown_connections 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52cc06b4d0 0x7f52cc06d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f52ec15de00 0x7f52ec1f3a00 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f52ec0b9380 0x7f52ec14de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 --2- 172.21.15.99:0/2602062097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f52ec0b5470 0x7f52ec157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 >> 172.21.15.99:0/2602062097 conn(0x7f52ec0d0470 msgr2=0x7f52ec0b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 shutdown_connections 2024-03-04T10:05:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.609+0000 7f52f136f640 1 -- 172.21.15.99:0/2602062097 wait complete. 2024-03-04T10:05:51.621 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:05:51.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:05:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.723+0000 7fdcad3c4640 1 Processor -- start 2024-03-04T10:05:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.723+0000 7fdcad3c4640 1 -- start start 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdcad3c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca80ccbd0 0x7fdca80b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdcad3c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fdca80ceec0 0x7fdca80b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdcad3c4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca815ca50 con 0x7fdca815c2e0 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdcad3c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdca815d2f0 con 0x7fdca80ceec0 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdcad3c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca815d6d0 con 0x7fdca80ccbd0 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdca67fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdca815c2e0 0x7fdca80ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44082/0 (socket says 172.21.15.99:44082) 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdca67fc640 1 -- 172.21.15.99:0/2847866083 learned_addr learned my addr 172.21.15.99:0/2847866083 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdca77fe640 1 --2- 172.21.15.99:0/2847866083 >> v2:172.21.15.162:3300/0 conn(0x7fdca80ceec0 0x7fdca80b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.724+0000 7fdca6ffd640 1 --2- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca80ccbd0 0x7fdca80b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.725+0000 7fdca5ffb640 1 -- 172.21.15.99:0/2847866083 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746080470 0 0) 0x7fdca815ca50 con 0x7fdca815c2e0 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.725+0000 7fdca5ffb640 1 -- 172.21.15.99:0/2847866083 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc94003520 con 0x7fdca815c2e0 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.725+0000 7fdca77fe640 1 -- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca80ccbd0 msgr2=0x7fdca80b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.725+0000 7fdca77fe640 1 --2- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca80ccbd0 0x7fdca80b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.725+0000 7fdca77fe640 1 -- 172.21.15.99:0/2847866083 >> v1:172.21.15.99:6789/0 conn(0x7fdca815c2e0 legacy=0x7fdca80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.725+0000 7fdca77fe640 1 -- 172.21.15.99:0/2847866083 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca815cd30 con 0x7fdca80ceec0 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdca77fe640 1 --2- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ceec0 0x7fdca80b0560 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdca00097a0 tx=0x7fdca002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974fc87f90ab9a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdca5ffb640 1 -- 172.21.15.99:0/2847866083 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0035020 con 0x7fdca80ceec0 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdca5ffb640 1 -- 172.21.15.99:0/2847866083 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca0031830 con 0x7fdca80ceec0 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdca5ffb640 1 -- 172.21.15.99:0/2847866083 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0031be0 con 0x7fdca80ceec0 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 -- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ceec0 msgr2=0x7fdca80b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ceec0 0x7fdca80b0560 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdca00097a0 tx=0x7fdca002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 -- 172.21.15.99:0/2847866083 shutdown_connections 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ceec0 0x7fdca80b0560 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/2847866083 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca80ccbd0 0x7fdca80b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 -- 172.21.15.99:0/2847866083 >> 172.21.15.99:0/2847866083 conn(0x7fdca8066dc0 msgr2=0x7fdca80671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 -- 172.21.15.99:0/2847866083 shutdown_connections 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.726+0000 7fdcad3c4640 1 -- 172.21.15.99:0/2847866083 wait complete. 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 Processor -- start 2024-03-04T10:05:51.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 -- start start 2024-03-04T10:05:51.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ccbd0 0x7fdca8158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 0x7fdca80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca815c2e0 0x7fdca80cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80ccfb0 con 0x7fdca80ceec0 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca815c760 con 0x7fdca80ccbd0 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.728+0000 7fdcad3c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca80b0aa0 con 0x7fdca815c2e0 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 0x7fdca80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 0x7fdca80cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47242/0 (socket says 172.21.15.99:47242) 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 -- 172.21.15.99:0/1363754626 learned_addr learned my addr 172.21.15.99:0/1363754626 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca6ffd640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ccbd0 0x7fdca8158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca77fe640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca815c2e0 0x7fdca80cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 -- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca815c2e0 msgr2=0x7fdca80cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca815c2e0 0x7fdca80cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 -- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ccbd0 msgr2=0x7fdca8158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ccbd0 0x7fdca8158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 -- 172.21.15.99:0/1363754626 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80d1da0 con 0x7fdca80ceec0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca6ffd640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ccbd0 0x7fdca8158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdca67fc640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 0x7fdca80cae80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000c390 tx=0x7fdc9000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc90017070 con 0x7fdca80ceec0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc90005ce0 con 0x7fdca80ceec0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc900072f0 con 0x7fdca80ceec0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca80d1220 con 0x7fdca80ceec0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.729+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80d1ac0 con 0x7fdca80ceec0 2024-03-04T10:05:51.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.730+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca8074f60 con 0x7fdca80ceec0 2024-03-04T10:05:51.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.731+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdc9001a070 con 0x7fdca80ceec0 2024-03-04T10:05:51.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.731+0000 7fdc9f7fe640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:51.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.731+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fdc90089460 con 0x7fdca80ceec0 2024-03-04T10:05:51.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.731+0000 7fdca6ffd640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:51.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.732+0000 7fdca6ffd640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc8806b4d0 0x7fdc8806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdc980074c0 tx=0x7fdc9800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:51.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.735+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdc90058490 con 0x7fdca80ceec0 2024-03-04T10:05:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.905+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdca80c5f40 con 0x7fdca80ceec0 2024-03-04T10:05:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.905+0000 7fdc9f7fe640 1 -- 172.21.15.99:0/1363754626 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc90057d90 con 0x7fdca80ceec0 2024-03-04T10:05:51.906 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:05:51.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc8806b4d0 msgr2=0x7fdc8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc8806b4d0 0x7fdc8806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdc980074c0 tx=0x7fdc9800b040 comp rx=0 tx=0).stop 2024-03-04T10:05:51.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 msgr2=0x7fdca80cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:51.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 0x7fdca80cae80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000c390 tx=0x7fdc9000c850 comp rx=0 tx=0).stop 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 shutdown_connections 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc8806b4d0 0x7fdc8806d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdca815c2e0 0x7fdca80cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdca80ceec0 0x7fdca80cae80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 --2- 172.21.15.99:0/1363754626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdca80ccbd0 0x7fdca8158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 >> 172.21.15.99:0/1363754626 conn(0x7fdca8066dc0 msgr2=0x7fdca815b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 shutdown_connections 2024-03-04T10:05:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:51.907+0000 7fdcad3c4640 1 -- 172.21.15.99:0/1363754626 wait complete. 2024-03-04T10:05:51.919 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-03-04T10:05:52.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.018+0000 7fec3b73b640 1 Processor -- start 2024-03-04T10:05:52.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.018+0000 7fec3b73b640 1 -- start start 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec3b73b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fec3415b030 0x7fec3415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec3b73b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fec3415b410 0x7fec3415b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec3b73b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3406d8b0 con 0x7fec3406d3b0 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec3b73b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec340b2720 con 0x7fec3415b410 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec3b73b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3406db90 con 0x7fec3415b030 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fec3415b410 0x7fec3415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fec3415b410 0x7fec3415b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47250/0 (socket says 172.21.15.99:47250) 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 -- 172.21.15.99:0/40816068 learned_addr learned my addr 172.21.15.99:0/40816068 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec394d7640 1 --2- 172.21.15.99:0/40816068 >> v2:172.21.15.162:3300/0 conn(0x7fec3415b030 0x7fec3415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 -- 172.21.15.99:0/40816068 >> v2:172.21.15.162:3300/0 conn(0x7fec3415b030 msgr2=0x7fec3415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 --2- 172.21.15.99:0/40816068 >> v2:172.21.15.162:3300/0 conn(0x7fec3415b030 0x7fec3415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 -- 172.21.15.99:0/40816068 >> v1:172.21.15.162:6789/0 conn(0x7fec3406d3b0 legacy=0x7fec3415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec39cd8640 1 -- 172.21.15.99:0/40816068 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3406de70 con 0x7fec3415b410 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.019+0000 7fec394d7640 1 --2- 172.21.15.99:0/40816068 >> v2:172.21.15.162:3300/0 conn(0x7fec3415b030 0x7fec3415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec39cd8640 1 --2- 172.21.15.99:0/40816068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3415b410 0x7fec3415b880 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fec340b28a0 tx=0x7fec2c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab89cc84997fa8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec2bfff640 1 -- 172.21.15.99:0/40816068 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c02c9f0 con 0x7fec3415b410 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec2bfff640 1 -- 172.21.15.99:0/40816068 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c02cb90 con 0x7fec3415b410 2024-03-04T10:05:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec2bfff640 1 -- 172.21.15.99:0/40816068 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0343f0 con 0x7fec3415b410 2024-03-04T10:05:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec3b73b640 1 -- 172.21.15.99:0/40816068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3415b410 msgr2=0x7fec3415b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec3b73b640 1 --2- 172.21.15.99:0/40816068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3415b410 0x7fec3415b880 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fec340b28a0 tx=0x7fec2c02be50 comp rx=0 tx=0).stop 2024-03-04T10:05:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec3b73b640 1 -- 172.21.15.99:0/40816068 shutdown_connections 2024-03-04T10:05:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec3b73b640 1 --2- 172.21.15.99:0/40816068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3415b410 0x7fec3415b880 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec3b73b640 1 --2- 172.21.15.99:0/40816068 >> v2:172.21.15.162:3300/0 conn(0x7fec3415b030 0x7fec3415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.020+0000 7fec3b73b640 1 -- 172.21.15.99:0/40816068 >> 172.21.15.99:0/40816068 conn(0x7fec3406acd0 msgr2=0x7fec3406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.021+0000 7fec3b73b640 1 -- 172.21.15.99:0/40816068 shutdown_connections 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.021+0000 7fec3b73b640 1 -- 172.21.15.99:0/40816068 wait complete. 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 Processor -- start 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 -- start start 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 0x7fec3414f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec3415b030 0x7fec340c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec3415b410 0x7fec340c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3406e690 con 0x7fec3406d3b0 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3415bdc0 con 0x7fec3415b410 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec3b73b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3406e810 con 0x7fec3415b030 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.022+0000 7fec394d7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 0x7fec3414f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 0x7fec3414f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47252/0 (socket says 172.21.15.99:47252) 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 -- 172.21.15.99:0/712204193 learned_addr learned my addr 172.21.15.99:0/712204193 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec39cd8640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec3415b410 0x7fec340c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec38cd6640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec3415b030 0x7fec340c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 -- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec3415b030 msgr2=0x7fec340c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec3415b030 0x7fec340c4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 -- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec3415b410 msgr2=0x7fec340c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec3415b410 0x7fec340c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 -- 172.21.15.99:0/712204193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec340aa290 con 0x7fec3406d3b0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec39cd8640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec3415b410 0x7fec340c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec394d7640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 0x7fec3414f1a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fec3000eb90 tx=0x7fec30007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30007d20 con 0x7fec3406d3b0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec30010440 con 0x7fec3406d3b0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec340aa570 con 0x7fec3406d3b0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30010810 con 0x7fec3406d3b0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.023+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec340aab30 con 0x7fec3406d3b0 2024-03-04T10:05:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.024+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec34074f60 con 0x7fec3406d3b0 2024-03-04T10:05:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.025+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fec3000b030 con 0x7fec3406d3b0 2024-03-04T10:05:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.025+0000 7fec29ffb640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec1806b4d0 0x7fec1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.025+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fec3008c4a0 con 0x7fec3406d3b0 2024-03-04T10:05:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.026+0000 7fec38cd6640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec1806b4d0 0x7fec1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.026+0000 7fec38cd6640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec1806b4d0 0x7fec1806d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fec3415e080 tx=0x7fec24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.030+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fec3005b440 con 0x7fec3406d3b0 2024-03-04T10:05:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.266+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) v1 -- 0x7fec3414f6e0 con 0x7fec3406d3b0 2024-03-04T10:05:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.310+0000 7fec29ffb640 1 -- 172.21.15.99:0/712204193 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v138) v1 ==== 144+0+0 (secure 0 0 0) 0x7fec3005ad40 con 0x7fec3406d3b0 2024-03-04T10:05:52.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec1806b4d0 msgr2=0x7fec1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec1806b4d0 0x7fec1806d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fec3415e080 tx=0x7fec24009040 comp rx=0 tx=0).stop 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 msgr2=0x7fec3414f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 0x7fec3414f1a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fec3000eb90 tx=0x7fec30007410 comp rx=0 tx=0).stop 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 shutdown_connections 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fec1806b4d0 0x7fec1806d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fec3415b410 0x7fec340c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fec3415b030 0x7fec340c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 --2- 172.21.15.99:0/712204193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fec3406d3b0 0x7fec3414f1a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 >> 172.21.15.99:0/712204193 conn(0x7fec3406acd0 msgr2=0x7fec340ae6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 shutdown_connections 2024-03-04T10:05:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.312+0000 7fec3b73b640 1 -- 172.21.15.99:0/712204193 wait complete. 2024-03-04T10:05:52.324 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T10:05:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.425+0000 7f1374692640 1 Processor -- start 2024-03-04T10:05:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.425+0000 7f1374692640 1 -- start start 2024-03-04T10:05:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.426+0000 7f1374692640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f136c0b5700 0x7f136c0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.426+0000 7f1374692640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f136c0d34a0 0x7f136c06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.426+0000 7f1374692640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136c0d2400 con 0x7f136c0b9410 2024-03-04T10:05:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.426+0000 7f1374692640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136c0d26e0 con 0x7f136c0b5700 2024-03-04T10:05:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.426+0000 7f1374692640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136c0d29c0 con 0x7f136c0d34a0 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f136c0b5700 0x7f136c0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f136c0b5700 0x7f136c0d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47254/0 (socket says 172.21.15.99:47254) 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 -- 172.21.15.99:0/3588400888 learned_addr learned my addr 172.21.15.99:0/3588400888 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f1371c2d640 1 --2- 172.21.15.99:0/3588400888 >> v2:172.21.15.162:3300/0 conn(0x7f136c0d34a0 0x7f136c06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 -- 172.21.15.99:0/3588400888 >> v2:172.21.15.162:3300/0 conn(0x7f136c0d34a0 msgr2=0x7f136c06c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 --2- 172.21.15.99:0/3588400888 >> v2:172.21.15.162:3300/0 conn(0x7f136c0d34a0 0x7f136c06c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 -- 172.21.15.99:0/3588400888 >> v1:172.21.15.162:6789/0 conn(0x7f136c0b9410 legacy=0x7f136c06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.427+0000 7f137242e640 1 -- 172.21.15.99:0/3588400888 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0d2ca0 con 0x7f136c0b5700 2024-03-04T10:05:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f137242e640 1 --2- 172.21.15.99:0/3588400888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b5700 0x7f136c0d1b00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f136c0c7470 tx=0x7f136402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70a7c4746a452b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f137142c640 1 -- 172.21.15.99:0/3588400888 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136402c9f0 con 0x7f136c0b5700 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f137142c640 1 -- 172.21.15.99:0/3588400888 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136402cb90 con 0x7f136c0b5700 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f137142c640 1 -- 172.21.15.99:0/3588400888 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13640343f0 con 0x7f136c0b5700 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 -- 172.21.15.99:0/3588400888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b5700 msgr2=0x7f136c0d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 --2- 172.21.15.99:0/3588400888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b5700 0x7f136c0d1b00 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f136c0c7470 tx=0x7f136402be50 comp rx=0 tx=0).stop 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 -- 172.21.15.99:0/3588400888 shutdown_connections 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 --2- 172.21.15.99:0/3588400888 >> v2:172.21.15.162:3300/0 conn(0x7f136c0d34a0 0x7f136c06c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 --2- 172.21.15.99:0/3588400888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b5700 0x7f136c0d1b00 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 -- 172.21.15.99:0/3588400888 >> 172.21.15.99:0/3588400888 conn(0x7f136c0d0490 msgr2=0x7f136c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 -- 172.21.15.99:0/3588400888 shutdown_connections 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.428+0000 7f1374692640 1 -- 172.21.15.99:0/3588400888 wait complete. 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.429+0000 7f1374692640 1 Processor -- start 2024-03-04T10:05:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 -- start start 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f136c0b5700 0x7f136c156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 0x7f136c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f136c0d34a0 0x7f136c0ce970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c06ceb0 con 0x7f136c0b9410 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136c06cd30 con 0x7f136c0b5700 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1374692640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136c0b5ae0 con 0x7f136c0d34a0 2024-03-04T10:05:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1371c2d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 0x7f136c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1371c2d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 0x7f136c159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47256/0 (socket says 172.21.15.99:47256) 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.430+0000 7f1371c2d640 1 -- 172.21.15.99:0/1354055934 learned_addr learned my addr 172.21.15.99:0/1354055934 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f137242e640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f136c0b5700 0x7f136c156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1371c2d640 1 -- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f136c0d34a0 msgr2=0x7f136c0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1371c2d640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f136c0d34a0 0x7f136c0ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1371c2d640 1 -- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f136c0b5700 msgr2=0x7f136c156f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1371c2d640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f136c0b5700 0x7f136c156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1371c2d640 1 -- 172.21.15.99:0/1354055934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136c0cb3b0 con 0x7f136c0b9410 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1371c2d640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 0x7f136c159eb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c7f0 tx=0x7f135c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c015070 con 0x7f136c0b9410 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c007820 con 0x7f136c0b9410 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136c0cb690 con 0x7f136c0b9410 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c007bd0 con 0x7f136c0b9410 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.431+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136c0cbc50 con 0x7f136c0b9410 2024-03-04T10:05:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.432+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136c074f60 con 0x7f136c0b9410 2024-03-04T10:05:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.432+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f135c018070 con 0x7f136c0b9410 2024-03-04T10:05:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.433+0000 7f1362ffd640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f135006b500 0x7f135006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.433+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f135c089e30 con 0x7f136c0b9410 2024-03-04T10:05:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.433+0000 7f137242e640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f135006b500 0x7f135006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.434+0000 7f137242e640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f135006b500 0x7f135006d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1364035000 tx=0x7f1364030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.437+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f135c058d40 con 0x7f136c0b9410 2024-03-04T10:05:52.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.607+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f136c159260 con 0x7f136c0b9410 2024-03-04T10:05:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.610+0000 7f1362ffd640 1 -- 172.21.15.99:0/1354055934 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f135c058640 con 0x7f136c0b9410 2024-03-04T10:05:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.612+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f135006b500 msgr2=0x7f135006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.612+0000 7f1374692640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f135006b500 0x7f135006d9c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1364035000 tx=0x7f1364030000 comp rx=0 tx=0).stop 2024-03-04T10:05:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.612+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 msgr2=0x7f136c159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.612+0000 7f1374692640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 0x7f136c159eb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c7f0 tx=0x7f135c00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 shutdown_connections 2024-03-04T10:05:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f135006b500 0x7f135006d9c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f136c0d34a0 0x7f136c0ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f136c0b9410 0x7f136c159eb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 --2- 172.21.15.99:0/1354055934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f136c0b5700 0x7f136c156f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 >> 172.21.15.99:0/1354055934 conn(0x7f136c0d0490 msgr2=0x7f136c0b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 shutdown_connections 2024-03-04T10:05:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.613+0000 7f1374692640 1 -- 172.21.15.99:0/1354055934 wait complete. 2024-03-04T10:05:52.624 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:05:52.625 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:05:52.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:05:52.727 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:05:52.727 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:52.728 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:52.743 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:05:52.743 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:05:52.743 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:52.761 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:05:52.761 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:05:52.761 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:52.798 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:05:52.798 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:05:52.798 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:05:52.821 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:05:52.821 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T10:05:52.821 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:05:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.937+0000 7f3c3df74640 1 Processor -- start 2024-03-04T10:05:52.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.937+0000 7f3c3df74640 1 -- start start 2024-03-04T10:05:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.937+0000 7f3c3df74640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 0x7f3c38166660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.937+0000 7f3c3df74640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3815cfe0 con 0x7f3c3815c030 2024-03-04T10:05:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.937+0000 7f3c3df74640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c380cf630 con 0x7f3c381597f0 2024-03-04T10:05:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.937+0000 7f3c3df74640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3806baf0 con 0x7f3c380cd270 2024-03-04T10:05:52.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c36ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3c3815c030 0x7f3c3806b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44084/0 (socket says 172.21.15.99:44084) 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c36ffd640 1 -- 172.21.15.99:0/1602439786 learned_addr learned my addr 172.21.15.99:0/1602439786 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161006188 0 0) 0x7f3c3815cfe0 con 0x7f3c3815c030 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c37fff640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 0x7f3c38166660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c20003520 con 0x7f3c3815c030 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3656209288 0 0) 0x7f3c20003520 con 0x7f3c3815c030 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 msgr2=0x7f3c38166660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 0x7f3c38166660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 >> v1:172.21.15.162:6789/0 conn(0x7f3c381597f0 legacy=0x7f3c3806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380cf910 con 0x7f3c3815c030 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3c2c001670 con 0x7f3c3815c030 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 >> v1:172.21.15.99:6789/0 conn(0x7f3c3815c030 legacy=0x7f3c3806b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c200039e0 0x7f3c20003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 0x7f3c20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3806baf0 con 0x7f3c20004d40 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2000bba0 con 0x7f3c200039e0 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2000bda0 con 0x7f3c380cd270 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c37fff640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c377fe640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c200039e0 0x7f3c20003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.938+0000 7f3c36ffd640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 0x7f3c20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c37fff640 1 -- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 msgr2=0x7f3c20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c37fff640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380cd270 0x7f3c20004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c37fff640 1 -- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c200039e0 msgr2=0x7f3c20003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c37fff640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c200039e0 0x7f3c20003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c37fff640 1 -- 172.21.15.99:0/1602439786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c20003520 con 0x7f3c20004d40 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c37fff640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3c30007360 tx=0x7f3c20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96342399994c3c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30039070 con 0x7f3c20004d40 2024-03-04T10:05:52.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3002ccd0 con 0x7f3c20004d40 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.939+0000 7f3c367fc640 1 -- 172.21.15.99:0/1602439786 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30009be0 con 0x7f3c20004d40 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c3df74640 1 -- 172.21.15.99:0/1602439786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c20004d40 msgr2=0x7f3c200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c3df74640 1 --2- 172.21.15.99:0/1602439786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c20004d40 0x7f3c200051c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f3c30007360 tx=0x7f3c20005830 comp rx=0 tx=0).stop 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c37fff640 1 -- 172.21.15.99:0/1602439786 reap_dead start 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c3df74640 1 -- 172.21.15.99:0/1602439786 shutdown_connections 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c3df74640 1 -- 172.21.15.99:0/1602439786 >> 172.21.15.99:0/1602439786 conn(0x7f3c380d0710 msgr2=0x7f3c380d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c3df74640 1 -- 172.21.15.99:0/1602439786 shutdown_connections 2024-03-04T10:05:52.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.940+0000 7f3c3df74640 1 -- 172.21.15.99:0/1602439786 wait complete. 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.941+0000 7f3c3df74640 1 Processor -- start 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 -- start start 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c380b5e40 0x7f3c380b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380c6cb0 0x7f3c381f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 0x7f3c380c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380cfed0 con 0x7f3c381f4ee0 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380d0240 con 0x7f3c380b5e40 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c3df74640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c380d1350 con 0x7f3c380c6cb0 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c37fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 0x7f3c380c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c37fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 0x7f3c380c4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47274/0 (socket says 172.21.15.99:47274) 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c37fff640 1 -- 172.21.15.99:0/2668499416 learned_addr learned my addr 172.21.15.99:0/2668499416 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:52.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c36ffd640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380c6cb0 0x7f3c381f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.942+0000 7f3c377fe640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c380b5e40 0x7f3c380b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c37fff640 1 -- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380c6cb0 msgr2=0x7f3c381f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c37fff640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380c6cb0 0x7f3c381f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c37fff640 1 -- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c380b5e40 msgr2=0x7f3c380b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c37fff640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c380b5e40 0x7f3c380b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c37fff640 1 -- 172.21.15.99:0/2668499416 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c380b2ca0 con 0x7f3c381f4ee0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c37fff640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 0x7f3c380c4fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3c30007480 tx=0x7f3c30004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30039070 con 0x7f3c381f4ee0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3003a040 con 0x7f3c381f4ee0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c30009be0 con 0x7f3c381f4ee0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c380b2f80 con 0x7f3c381f4ee0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.943+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c380c6350 con 0x7f3c381f4ee0 2024-03-04T10:05:52.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.944+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3806cd70 con 0x7f3c381f4ee0 2024-03-04T10:05:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.944+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3c30046030 con 0x7f3c381f4ee0 2024-03-04T10:05:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.945+0000 7f3c1ffff640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2406b4d0 0x7f3c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.945+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3c300ab580 con 0x7f3c381f4ee0 2024-03-04T10:05:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.945+0000 7f3c377fe640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2406b4d0 0x7f3c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:52.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.945+0000 7f3c377fe640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2406b4d0 0x7f3c2406d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c280017e0 tx=0x7f3c28009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:52.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:52.949+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3c3004d060 con 0x7f3c381f4ee0 2024-03-04T10:05:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.154+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c380af5e0 con 0x7f3c381f4ee0 2024-03-04T10:05:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.155+0000 7f3c1ffff640 1 -- 172.21.15.99:0/2668499416 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f3c30079f90 con 0x7f3c381f4ee0 2024-03-04T10:05:53.157 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:53.157 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4880,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/2608019617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2608019617},{"type":"v1","addr":"172.21.15.162:6836","nonce":2608019617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5053,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/3428516778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":3428516778},{"type":"v1","addr":"172.21.15.162:6837","nonce":3428516778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/1652277159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":1652277159},{"type":"v1","addr":"172.21.15.99:6834","nonce":1652277159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5892,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/211510885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":211510885},{"type":"v1","addr":"172.21.15.99:6835","nonce":211510885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[]} 2024-03-04T10:05:53.159 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 23 2024-03-04T10:05:53.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2406b4d0 msgr2=0x7f3c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:53.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2406b4d0 0x7f3c2406d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c280017e0 tx=0x7f3c28009040 comp rx=0 tx=0).stop 2024-03-04T10:05:53.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 msgr2=0x7f3c380c4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 0x7f3c380c4fc0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3c30007480 tx=0x7f3c30004ab0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 shutdown_connections 2024-03-04T10:05:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2406b4d0 0x7f3c2406d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c381f4ee0 0x7f3c380c4fc0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c380c6cb0 0x7f3c381f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 --2- 172.21.15.99:0/2668499416 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c380b5e40 0x7f3c380b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.156+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 >> 172.21.15.99:0/2668499416 conn(0x7f3c380d0710 msgr2=0x7f3c380ad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.157+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 shutdown_connections 2024-03-04T10:05:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.157+0000 7f3c3df74640 1 -- 172.21.15.99:0/2668499416 wait complete. 2024-03-04T10:05:53.170 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:05:53.170 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T10:05:53.171 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:05:53.171 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:05:53.171 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:05:53.172 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T10:05:53.172 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:05:53.172 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:05:53.173 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T10:05:53.173 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T10:05:53.173 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T10:05:53.173 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:05:53.174 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T10:05:53.174 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T10:05:53.174 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T10:05:53.175 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:05:53.180 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:05:53.181 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:05:53.182 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:05:53.182 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:05:53.185 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T10:05:53.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T10:05:53.186 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T10:05:53.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T10:05:53.224 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:05:53.225 DEBUG:teuthology.parallel:result is None 2024-03-04T10:05:53.226 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:05:53.226 DEBUG:teuthology.parallel:result is None 2024-03-04T10:05:53.228 INFO:tasks.ceph.mds.b:Started 2024-03-04T10:05:53.228 DEBUG:teuthology.parallel:result is None 2024-03-04T10:05:53.229 INFO:tasks.ceph.mds.d:Started 2024-03-04T10:05:53.229 DEBUG:teuthology.parallel:result is None 2024-03-04T10:05:53.229 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T10:05:53.299 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:05:53.300 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:05:53.321 INFO:tasks.ceph.mds.b.smithi162.stdout:starting mds.b at 2024-03-04T10:05:53.321 INFO:tasks.ceph.mds.d.smithi162.stdout:starting mds.d at 2024-03-04T10:05:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 Processor -- start 2024-03-04T10:05:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 -- start start 2024-03-04T10:05:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd815be90 0x7f8cd80cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8cd80cfcd0 0x7f8cd80cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd815f8c0 con 0x7f8cd815bab0 2024-03-04T10:05:53.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd80b7970 con 0x7f8cd80cfcd0 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.334+0000 7f8cdf0f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd815c7f0 con 0x7f8cd815be90 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd815be90 0x7f8cd80cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdd68e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8cd80cfcd0 0x7f8cd80cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdce8d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8cd815bab0 0x7f8cd80b7260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:37006/0 (socket says 172.21.15.99:37006) 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd815be90 0x7f8cd80cf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:56660/0 (socket says 172.21.15.99:56660) 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdce8d640 1 -- 172.21.15.99:0/4066693349 learned_addr learned my addr 172.21.15.99:0/4066693349 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdd68e640 1 -- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd815be90 msgr2=0x7f8cd80cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdd68e640 1 --2- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd815be90 0x7f8cd80cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdd68e640 1 -- 172.21.15.99:0/4066693349 >> v1:172.21.15.162:6789/0 conn(0x7f8cd815bab0 legacy=0x7f8cd80b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:53.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cdd68e640 1 -- 172.21.15.99:0/4066693349 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd815cad0 con 0x7f8cd80cfcd0 2024-03-04T10:05:53.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.335+0000 7f8cd77fe640 1 -- 172.21.15.99:0/4066693349 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779444289 0 0) 0x7f8cd815f8c0 con 0x7f8cd815bab0 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdd68e640 1 --2- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd80cfcd0 0x7f8cd80cc1b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009d90 tx=0x7f8cd002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f63ee243dbfa0261 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cd77fe640 1 -- 172.21.15.99:0/4066693349 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0039070 con 0x7f8cd80cfcd0 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cd77fe640 1 -- 172.21.15.99:0/4066693349 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd002cc50 con 0x7f8cd80cfcd0 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cd77fe640 1 -- 172.21.15.99:0/4066693349 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0034440 con 0x7f8cd80cfcd0 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd80cfcd0 msgr2=0x7f8cd80cc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd80cfcd0 0x7f8cd80cc1b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009d90 tx=0x7f8cd002bea0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/4066693349 shutdown_connections 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd815be90 0x7f8cd80cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/4066693349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd80cfcd0 0x7f8cd80cc1b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/4066693349 >> 172.21.15.99:0/4066693349 conn(0x7f8cd806f6e0 msgr2=0x7f8cd806faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/4066693349 shutdown_connections 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.336+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/4066693349 wait complete. 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.337+0000 7f8cdf0f1640 1 Processor -- start 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.337+0000 7f8cdf0f1640 1 -- start start 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cdf0f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd80cfcd0 0x7f8cd80b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cdf0f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 0x7f8cd80713a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cdf0f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd815be90 0x7f8cd814fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cdf0f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd815c320 con 0x7f8cd815bab0 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cdf0f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd815d090 con 0x7f8cd815be90 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cdf0f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd815c4a0 con 0x7f8cd80cfcd0 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 0x7f8cd80713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cd7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 0x7f8cd80713a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47302/0 (socket says 172.21.15.99:47302) 2024-03-04T10:05:53.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.338+0000 7f8cd7fff640 1 -- 172.21.15.99:0/2040767572 learned_addr learned my addr 172.21.15.99:0/2040767572 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cdce8d640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd80cfcd0 0x7f8cd80b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cdd68e640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd815be90 0x7f8cd814fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd7fff640 1 -- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd80cfcd0 msgr2=0x7f8cd80b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd7fff640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd80cfcd0 0x7f8cd80b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd7fff640 1 -- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd815be90 msgr2=0x7f8cd814fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd7fff640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd815be90 0x7f8cd814fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd7fff640 1 -- 172.21.15.99:0/2040767572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8159410 con 0x7f8cd815bab0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cdd68e640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd815be90 0x7f8cd814fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd7fff640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 0x7f8cd80713a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400e9b0 tx=0x7f8cc400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4017070 con 0x7f8cd815bab0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc400c9a0 con 0x7f8cd815bab0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400cd90 con 0x7f8cd815bab0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd81596f0 con 0x7f8cd815bab0 2024-03-04T10:05:53.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.339+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd80c8a20 con 0x7f8cd815bab0 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.340+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd8159ae0 con 0x7f8cd815bab0 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.340+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8cc401a070 con 0x7f8cd815bab0 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.341+0000 7f8cd57fa640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cbc06b4d0 0x7f8cbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.341+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f8cc408a170 con 0x7f8cd815bab0 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.341+0000 7f8cdce8d640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cbc06b4d0 0x7f8cbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.341+0000 7f8cdce8d640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cbc06b4d0 0x7f8cbc06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc000f60 tx=0x7f8ccc007b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.345+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cc4059070 con 0x7f8cd815bab0 2024-03-04T10:05:53.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:53.528+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8cd81598c0 con 0x7f8cd815bab0 2024-03-04T10:05:54.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.317+0000 7f8cd57fa640 1 -- 172.21.15.99:0/2040767572 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v47) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8cc4058970 con 0x7f8cd815bab0 2024-03-04T10:05:54.319 INFO:teuthology.orchestra.run.smithi099.stderr: removed all blocklist entries 2024-03-04T10:05:54.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cbc06b4d0 msgr2=0x7f8cbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cbc06b4d0 0x7f8cbc06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc000f60 tx=0x7f8ccc007b50 comp rx=0 tx=0).stop 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 msgr2=0x7f8cd80713a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 0x7f8cd80713a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400e9b0 tx=0x7f8cc400ee70 comp rx=0 tx=0).stop 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 shutdown_connections 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cbc06b4d0 0x7f8cbc06d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cd815be90 0x7f8cd814fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cd815bab0 0x7f8cd80713a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 --2- 172.21.15.99:0/2040767572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cd80cfcd0 0x7f8cd80b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 >> 172.21.15.99:0/2040767572 conn(0x7f8cd806f6e0 msgr2=0x7f8cd80d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 shutdown_connections 2024-03-04T10:05:54.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.319+0000 7f8cdf0f1640 1 -- 172.21.15.99:0/2040767572 wait complete. 2024-03-04T10:05:54.335 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T10:05:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 Processor -- start 2024-03-04T10:05:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 -- start start 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb0c15de00 0x7ffb0c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b5470 0x7ffb0c0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb0c15e1e0 con 0x7ffb0c0b9380 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0c0d2820 con 0x7ffb0c15de00 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.436+0000 7ffb120ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0d1c10 con 0x7ffb0c0b5470 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb0c15de00 0x7ffb0c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0affd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ffb0c0b9380 0x7ffb0c0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53198/0 (socket says 172.21.15.99:53198) 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0affd640 1 -- 172.21.15.99:0/782367858 learned_addr learned my addr 172.21.15.99:0/782367858 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0b7fe640 1 -- 172.21.15.99:0/782367858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b5470 msgr2=0x7ffb0c0b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0bfff640 1 --2- 172.21.15.99:0/782367858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b5470 0x7ffb0c0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0b7fe640 1 --2- 172.21.15.99:0/782367858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b5470 0x7ffb0c0b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0a7fc640 1 -- 172.21.15.99:0/782367858 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371540518 0 0) 0x7ffb0c15e1e0 con 0x7ffb0c0b9380 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0b7fe640 1 -- 172.21.15.99:0/782367858 >> v1:172.21.15.99:6789/0 conn(0x7ffb0c0b9380 legacy=0x7ffb0c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.437+0000 7ffb0b7fe640 1 -- 172.21.15.99:0/782367858 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0d1ef0 con 0x7ffb0c15de00 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb0b7fe640 1 --2- 172.21.15.99:0/782367858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c0b2e60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0097a0 tx=0x7ffafc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a59c0d36bd6f04 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb0a7fc640 1 -- 172.21.15.99:0/782367858 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc035020 con 0x7ffb0c15de00 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb0a7fc640 1 -- 172.21.15.99:0/782367858 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc031830 con 0x7ffb0c15de00 2024-03-04T10:05:54.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb0a7fc640 1 -- 172.21.15.99:0/782367858 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc031c00 con 0x7ffb0c15de00 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 -- 172.21.15.99:0/782367858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 msgr2=0x7ffb0c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 --2- 172.21.15.99:0/782367858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c0b2e60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0097a0 tx=0x7ffafc02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 -- 172.21.15.99:0/782367858 shutdown_connections 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 --2- 172.21.15.99:0/782367858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b5470 0x7ffb0c0b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 --2- 172.21.15.99:0/782367858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c0b2e60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 -- 172.21.15.99:0/782367858 >> 172.21.15.99:0/782367858 conn(0x7ffb0c0d0470 msgr2=0x7ffb0c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:54.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 -- 172.21.15.99:0/782367858 shutdown_connections 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.438+0000 7ffb120ab640 1 -- 172.21.15.99:0/782367858 wait complete. 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.439+0000 7ffb120ab640 1 Processor -- start 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 -- start start 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb0c0b5470 0x7ffb0c15e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b9380 0x7ffb0c0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c1e7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0b97e0 con 0x7ffb0c15de00 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0d24b0 con 0x7ffb0c0b5470 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb120ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0c0b72a0 con 0x7ffb0c0b9380 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb0bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c1e7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb0bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c1e7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35506/0 (socket says 172.21.15.99:35506) 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.440+0000 7ffb0bfff640 1 -- 172.21.15.99:0/4049071983 learned_addr learned my addr 172.21.15.99:0/4049071983 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0affd640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b9380 0x7ffb0c0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0b7fe640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb0c0b5470 0x7ffb0c15e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0bfff640 1 -- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b9380 msgr2=0x7ffb0c0cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0bfff640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b9380 0x7ffb0c0cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0bfff640 1 -- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb0c0b5470 msgr2=0x7ffb0c15e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0bfff640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb0c0b5470 0x7ffb0c15e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0bfff640 1 -- 172.21.15.99:0/4049071983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0c0b34f0 con 0x7ffb0c15de00 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb0bfff640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c1e7ae0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400cd70 tx=0x7ffb0400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003b90 con 0x7ffb0c15de00 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04003d30 con 0x7ffb0c15de00 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb0c1e8860 con 0x7ffb0c15de00 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04004240 con 0x7ffb0c15de00 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.441+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0c0b3a90 con 0x7ffb0c15de00 2024-03-04T10:05:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.442+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0c074f60 con 0x7ffb0c15de00 2024-03-04T10:05:54.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.442+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffb040043e0 con 0x7ffb0c15de00 2024-03-04T10:05:54.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.443+0000 7ffaf3fff640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaf806b4d0 0x7ffaf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.443+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ffb0408ab40 con 0x7ffb0c15de00 2024-03-04T10:05:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.443+0000 7ffb0b7fe640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaf806b4d0 0x7ffaf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.444+0000 7ffb0b7fe640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaf806b4d0 0x7ffaf806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffafc036000 tx=0x7ffafc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.447+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffb04059b80 con 0x7ffb0c15de00 2024-03-04T10:05:54.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.684+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ffb0c0b3f30 con 0x7ffb0c15de00 2024-03-04T10:05:54.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.684+0000 7ffaf3fff640 1 -- 172.21.15.99:0/4049071983 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ffb04059480 con 0x7ffb0c15de00 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_dump": [ 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.a", 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg==", 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:05:54.687 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.b", 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw==", 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.688 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.c", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg==", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.d", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw==", 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.689 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.0", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg==", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.690 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.1", 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ==", 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.2", 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw==", 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.691 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.3", 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA==", 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.692 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.4", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw==", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.5", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg==", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.693 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.6", 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg==", 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.7", 2024-03-04T10:05:54.694 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA==", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.0", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ==", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow r", 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T10:05:54.695 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.1", 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA==", 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.2", 2024-03-04T10:05:54.696 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg==", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.3", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ==", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:05:54.697 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.admin", 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBAnOVlV1MhNhAAJTJsIc9qGmeQBnHKjrkPvw==", 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.698 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.x", 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw==", 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.y", 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ==", 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:05:54.699 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:05:54.700 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:05:54.700 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:05:54.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.700 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:54.700 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-03-04T10:05:54.700 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:05:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaf806b4d0 msgr2=0x7ffaf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaf806b4d0 0x7ffaf806d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffafc036000 tx=0x7ffafc03c040 comp rx=0 tx=0).stop 2024-03-04T10:05:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 msgr2=0x7ffb0c1e7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c1e7ae0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400cd70 tx=0x7ffb0400a660 comp rx=0 tx=0).stop 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 shutdown_connections 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffaf806b4d0 0x7ffaf806d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb0c15de00 0x7ffb0c1e7ae0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb0c0b9380 0x7ffb0c0cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 --2- 172.21.15.99:0/4049071983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb0c0b5470 0x7ffb0c15e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 >> 172.21.15.99:0/4049071983 conn(0x7ffb0c0d0470 msgr2=0x7ffb0c1e9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 shutdown_connections 2024-03-04T10:05:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.686+0000 7ffb120ab640 1 -- 172.21.15.99:0/4049071983 wait complete. 2024-03-04T10:05:54.703 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:54.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.800+0000 7f1906b97640 1 Processor -- start 2024-03-04T10:05:54.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.800+0000 7f1906b97640 1 -- start start 2024-03-04T10:05:54.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1906b97640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f19000cc8d0 0x7f19000cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1906b97640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190006fd20 con 0x7f190015dea0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1906b97640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1900070000 con 0x7f190006b3e0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1906b97640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190006fad0 con 0x7f19000cc8d0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1905134640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f19000cc8d0 0x7f19000cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1904933640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f190015dea0 0x7f190006acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53214/0 (socket says 172.21.15.99:53214) 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1905134640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f19000cc8d0 0x7f19000cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35516/0 (socket says 172.21.15.99:35516) 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1904933640 1 -- 172.21.15.99:0/484446390 learned_addr learned my addr 172.21.15.99:0/484446390 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.801+0000 7f1905134640 1 -- 172.21.15.99:0/484446390 >> v1:172.21.15.162:6789/0 conn(0x7f190006b3e0 legacy=0x7f19000cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f18ff7fe640 1 -- 172.21.15.99:0/484446390 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816881111 0 0) 0x7f190006fd20 con 0x7f190015dea0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f1905134640 1 -- 172.21.15.99:0/484446390 >> v1:172.21.15.99:6789/0 conn(0x7f190015dea0 legacy=0x7f190006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f1905134640 1 -- 172.21.15.99:0/484446390 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19000702e0 con 0x7f19000cc8d0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f1905134640 1 --2- 172.21.15.99:0/484446390 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19000cc8d0 0x7f19000cad60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009d90 tx=0x7f18f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9643be3319eb796b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:54.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f18ff7fe640 1 -- 172.21.15.99:0/484446390 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8039070 con 0x7f19000cc8d0 2024-03-04T10:05:54.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f18ff7fe640 1 -- 172.21.15.99:0/484446390 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f802cc60 con 0x7f19000cc8d0 2024-03-04T10:05:54.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.802+0000 7f18ff7fe640 1 -- 172.21.15.99:0/484446390 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8034660 con 0x7f19000cc8d0 2024-03-04T10:05:54.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 -- 172.21.15.99:0/484446390 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19000cc8d0 msgr2=0x7f19000cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 --2- 172.21.15.99:0/484446390 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19000cc8d0 0x7f19000cad60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009d90 tx=0x7f18f802beb0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 -- 172.21.15.99:0/484446390 shutdown_connections 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 --2- 172.21.15.99:0/484446390 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f19000cc8d0 0x7f19000cad60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 -- 172.21.15.99:0/484446390 >> 172.21.15.99:0/484446390 conn(0x7f19000d0710 msgr2=0x7f19000d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 -- 172.21.15.99:0/484446390 shutdown_connections 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.803+0000 7f1906b97640 1 -- 172.21.15.99:0/484446390 wait complete. 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.804+0000 7f1906b97640 1 Processor -- start 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.804+0000 7f1906b97640 1 -- start start 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1906b97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f190006b3e0 0x7f19000c4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1906b97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19000cc8d0 0x7f190014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1906b97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 0x7f19000b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1906b97640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d1380 con 0x7f190015dea0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1906b97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d1200 con 0x7f19000cc8d0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1906b97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19000d1500 con 0x7f190006b3e0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 0x7f19000b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 0x7f19000b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35528/0 (socket says 172.21.15.99:35528) 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 -- 172.21.15.99:0/3917319657 learned_addr learned my addr 172.21.15.99:0/3917319657 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1904933640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f190006b3e0 0x7f19000c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f18fffff640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19000cc8d0 0x7f190014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 -- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f190006b3e0 msgr2=0x7f19000c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f190006b3e0 0x7f19000c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 -- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19000cc8d0 msgr2=0x7f190014e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19000cc8d0 0x7f190014e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.805+0000 7f1905134640 1 -- 172.21.15.99:0/3917319657 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900151ba0 con 0x7f190015dea0 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f1904933640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f190006b3e0 0x7f19000c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f18fffff640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19000cc8d0 0x7f190014e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f1905134640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 0x7f19000b45b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f18f8037040 tx=0x7f18f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8044030 con 0x7f190015dea0 2024-03-04T10:05:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f802ca60 con 0x7f190015dea0 2024-03-04T10:05:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19000c5e20 con 0x7f190015dea0 2024-03-04T10:05:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8039040 con 0x7f190015dea0 2024-03-04T10:05:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.806+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190015af80 con 0x7f190015dea0 2024-03-04T10:05:54.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.807+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190006ba10 con 0x7f190015dea0 2024-03-04T10:05:54.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.807+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f18f8047070 con 0x7f190015dea0 2024-03-04T10:05:54.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.808+0000 7f18fd7fa640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f18ec06b500 0x7f18ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:54.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.808+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f18f80ac890 con 0x7f190015dea0 2024-03-04T10:05:54.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.808+0000 7f1904933640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f18ec06b500 0x7f18ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:54.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.808+0000 7f1904933640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f18ec06b500 0x7f18ec06d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f18f0000f60 tx=0x7f18f0007ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:54.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.812+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f190006ba10 con 0x7f190015dea0 2024-03-04T10:05:54.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.984+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1900074f60 con 0x7f190015dea0 2024-03-04T10:05:54.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.985+0000 7f18fd7fa640 1 -- 172.21.15.99:0/3917319657 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13128 (secure 0 0 0) 0x7f18f804e070 con 0x7f190015dea0 2024-03-04T10:05:54.987 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:54.987 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":48,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:54.325414+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f18ec06b500 msgr2=0x7f18ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f18ec06b500 0x7f18ec06d9c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f18f0000f60 tx=0x7f18f0007ca0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 msgr2=0x7f19000b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 0x7f19000b45b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f18f8037040 tx=0x7f18f8004a40 comp rx=0 tx=0).stop 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 shutdown_connections 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f18ec06b500 0x7f18ec06d9c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f190015dea0 0x7f19000b45b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f19000cc8d0 0x7f190014e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 --2- 172.21.15.99:0/3917319657 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f190006b3e0 0x7f19000c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 >> 172.21.15.99:0/3917319657 conn(0x7f19000d0710 msgr2=0x7f190006acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:54.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 shutdown_connections 2024-03-04T10:05:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:54.987+0000 7f1906b97640 1 -- 172.21.15.99:0/3917319657 wait complete. 2024-03-04T10:05:55.001 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:05:55.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:05:55.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.102+0000 7f4e4480b640 1 Processor -- start 2024-03-04T10:05:55.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.102+0000 7f4e4480b640 1 -- start start 2024-03-04T10:05:55.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e4480b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4e4015da20 0x7f4e400ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e4480b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f4e400cf660 0x7f4e400cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e4480b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e401616e0 0x7f4e40161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e4480b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e400d0000 con 0x7f4e4015da20 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e4480b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e400cc920 con 0x7f4e400cf660 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e4480b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e40162450 con 0x7f4e401616e0 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4e4015da20 0x7f4e400ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4e4015da20 0x7f4e400ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35534/0 (socket says 172.21.15.99:35534) 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 -- 172.21.15.99:0/948425319 learned_addr learned my addr 172.21.15.99:0/948425319 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3ed9d640 1 --2- 172.21.15.99:0/948425319 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e401616e0 0x7f4e40161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3dd9b640 1 --2- 172.21.15.99:0/948425319 >> v2:172.21.15.162:3300/0 conn(0x7f4e400cf660 0x7f4e400cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 -- 172.21.15.99:0/948425319 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e401616e0 msgr2=0x7f4e40161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 --2- 172.21.15.99:0/948425319 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e401616e0 0x7f4e40161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 -- 172.21.15.99:0/948425319 >> v2:172.21.15.162:3300/0 conn(0x7f4e400cf660 msgr2=0x7f4e400cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 --2- 172.21.15.99:0/948425319 >> v2:172.21.15.162:3300/0 conn(0x7f4e400cf660 0x7f4e400cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.103+0000 7f4e3e59c640 1 -- 172.21.15.99:0/948425319 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e400cc2d0 con 0x7f4e4015da20 2024-03-04T10:05:55.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.104+0000 7f4e3e59c640 1 --2- 172.21.15.99:0/948425319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e4015da20 0x7f4e400ccdb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e401615d0 tx=0x7f4e3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9b10dd3275a02b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.104+0000 7f4e3d59a640 1 -- 172.21.15.99:0/948425319 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30035020 con 0x7f4e4015da20 2024-03-04T10:05:55.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.104+0000 7f4e3d59a640 1 -- 172.21.15.99:0/948425319 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30031870 con 0x7f4e4015da20 2024-03-04T10:05:55.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.104+0000 7f4e3d59a640 1 -- 172.21.15.99:0/948425319 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30031c40 con 0x7f4e4015da20 2024-03-04T10:05:55.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 -- 172.21.15.99:0/948425319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e4015da20 msgr2=0x7f4e400ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 --2- 172.21.15.99:0/948425319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e4015da20 0x7f4e400ccdb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4e401615d0 tx=0x7f4e3002b910 comp rx=0 tx=0).stop 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 -- 172.21.15.99:0/948425319 shutdown_connections 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 --2- 172.21.15.99:0/948425319 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e401616e0 0x7f4e40161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 --2- 172.21.15.99:0/948425319 >> v2:172.21.15.162:3300/0 conn(0x7f4e400cf660 0x7f4e400cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 --2- 172.21.15.99:0/948425319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e4015da20 0x7f4e400ccdb0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 -- 172.21.15.99:0/948425319 >> 172.21.15.99:0/948425319 conn(0x7f4e400b33a0 msgr2=0x7f4e400b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 -- 172.21.15.99:0/948425319 shutdown_connections 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.105+0000 7f4e4480b640 1 -- 172.21.15.99:0/948425319 wait complete. 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.106+0000 7f4e4480b640 1 Processor -- start 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.106+0000 7f4e4480b640 1 -- start start 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e4480b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 0x7f4e40152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e4480b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e4015da20 0x7f4e40154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e4480b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e401616e0 0x7f4e40154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e4480b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400cbf80 con 0x7f4e400cf660 2024-03-04T10:05:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e4480b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400cd2f0 con 0x7f4e401616e0 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e4480b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4015de00 con 0x7f4e4015da20 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e3e59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 0x7f4e40152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e3e59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 0x7f4e40152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35540/0 (socket says 172.21.15.99:35540) 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e3e59c640 1 -- 172.21.15.99:0/2640785392 learned_addr learned my addr 172.21.15.99:0/2640785392 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e3dd9b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e4015da20 0x7f4e40154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.107+0000 7f4e3ed9d640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e401616e0 0x7f4e40154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e3e59c640 1 -- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e4015da20 msgr2=0x7f4e40154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e3e59c640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e4015da20 0x7f4e40154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e3e59c640 1 -- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e401616e0 msgr2=0x7f4e40154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e3e59c640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e401616e0 0x7f4e40154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e3e59c640 1 -- 172.21.15.99:0/2640785392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e400c9d00 con 0x7f4e400cf660 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e3e59c640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 0x7f4e40152900 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4e30000c00 tx=0x7f4e300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30035040 con 0x7f4e400cf660 2024-03-04T10:05:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30030d00 con 0x7f4e400cf660 2024-03-04T10:05:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e400c9fe0 con 0x7f4e400cf660 2024-03-04T10:05:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e300456a0 con 0x7f4e400cf660 2024-03-04T10:05:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.108+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e400ca880 con 0x7f4e400cf660 2024-03-04T10:05:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.109+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e40074f60 con 0x7f4e400cf660 2024-03-04T10:05:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.109+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4e30038020 con 0x7f4e400cf660 2024-03-04T10:05:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.110+0000 7f4e2effd640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e2006b4d0 0x7f4e2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.110+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4e300ac2c0 con 0x7f4e400cf660 2024-03-04T10:05:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.110+0000 7f4e3dd9b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e2006b4d0 0x7f4e2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.110+0000 7f4e3dd9b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e2006b4d0 0x7f4e2006d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e400c9c30 tx=0x7f4e34008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.114+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4e3007ad40 con 0x7f4e400cf660 2024-03-04T10:05:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.285+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4e4015d000 con 0x7f4e400cf660 2024-03-04T10:05:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.285+0000 7f4e2effd640 1 -- 172.21.15.99:0/2640785392 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e3007ede0 con 0x7f4e400cf660 2024-03-04T10:05:55.287 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:05:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e2006b4d0 msgr2=0x7f4e2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e2006b4d0 0x7f4e2006d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e400c9c30 tx=0x7f4e34008040 comp rx=0 tx=0).stop 2024-03-04T10:05:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 msgr2=0x7f4e40152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 0x7f4e40152900 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4e30000c00 tx=0x7f4e300046c0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 shutdown_connections 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4e2006b4d0 0x7f4e2006d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4e401616e0 0x7f4e40154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4e4015da20 0x7f4e40154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 --2- 172.21.15.99:0/2640785392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4e400cf660 0x7f4e40152900 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 >> 172.21.15.99:0/2640785392 conn(0x7f4e400b33a0 msgr2=0x7f4e4006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 shutdown_connections 2024-03-04T10:05:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.287+0000 7f4e4480b640 1 -- 172.21.15.99:0/2640785392 wait complete. 2024-03-04T10:05:55.300 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T10:05:55.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.399+0000 7f752def1640 1 Processor -- start 2024-03-04T10:05:55.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.399+0000 7f752def1640 1 -- start start 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.399+0000 7f752def1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f75280af7b0 0x7f75280d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.399+0000 7f752def1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75280cc760 con 0x7f75280cf080 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.399+0000 7f752def1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75280ce720 con 0x7f752815ac40 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.399+0000 7f752def1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752815b020 con 0x7f75280af7b0 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f751ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f75280af7b0 0x7f75280d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f7527fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f75280cf080 0x7f75280ad7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53226/0 (socket says 172.21.15.99:53226) 2024-03-04T10:05:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f7527fff640 1 -- 172.21.15.99:0/366542702 learned_addr learned my addr 172.21.15.99:0/366542702 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f751ffff640 1 -- 172.21.15.99:0/366542702 >> v1:172.21.15.162:6789/0 conn(0x7f752815ac40 legacy=0x7f75280ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f751ffff640 1 -- 172.21.15.99:0/366542702 >> v1:172.21.15.99:6789/0 conn(0x7f75280cf080 legacy=0x7f75280ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f751ffff640 1 -- 172.21.15.99:0/366542702 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75280cbe30 con 0x7f75280af7b0 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.400+0000 7f7526ffd640 1 -- 172.21.15.99:0/366542702 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070544442 0 0) 0x7f75280cc760 con 0x7f75280cf080 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f751ffff640 1 --2- 172.21.15.99:0/366542702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75280af7b0 0x7f75280d1290 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f75100097a0 tx=0x7f751002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7870011520f01480 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f7526ffd640 1 -- 172.21.15.99:0/366542702 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510034070 con 0x7f75280af7b0 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f7526ffd640 1 -- 172.21.15.99:0/366542702 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510030610 con 0x7f75280af7b0 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f7526ffd640 1 -- 172.21.15.99:0/366542702 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100309e0 con 0x7f75280af7b0 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 -- 172.21.15.99:0/366542702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75280af7b0 msgr2=0x7f75280d1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 --2- 172.21.15.99:0/366542702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75280af7b0 0x7f75280d1290 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f75100097a0 tx=0x7f751002b960 comp rx=0 tx=0).stop 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 -- 172.21.15.99:0/366542702 shutdown_connections 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 --2- 172.21.15.99:0/366542702 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f75280af7b0 0x7f75280d1290 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 -- 172.21.15.99:0/366542702 >> 172.21.15.99:0/366542702 conn(0x7f75280d0770 msgr2=0x7f75280d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:55.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 -- 172.21.15.99:0/366542702 shutdown_connections 2024-03-04T10:05:55.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.401+0000 7f752def1640 1 -- 172.21.15.99:0/366542702 wait complete. 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.402+0000 7f752def1640 1 Processor -- start 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.402+0000 7f752def1640 1 -- start start 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f752def1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75280cf080 0x7f75281f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f752def1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f752815ac40 0x7f75280ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f752def1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 0x7f752814db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f752def1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75280ce340 con 0x7f752814d6d0 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f752def1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75280cc3f0 con 0x7f752815ac40 2024-03-04T10:05:55.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f752def1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75280ce4c0 con 0x7f75280cf080 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f7527fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 0x7f752814db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f7527fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 0x7f752814db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35560/0 (socket says 172.21.15.99:35560) 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f7527fff640 1 -- 172.21.15.99:0/3310407409 learned_addr learned my addr 172.21.15.99:0/3310407409 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f75277fe640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75280cf080 0x7f75281f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.403+0000 7f751ffff640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f752815ac40 0x7f75280ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7527fff640 1 -- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75280cf080 msgr2=0x7f75281f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7527fff640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75280cf080 0x7f75281f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7527fff640 1 -- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f752815ac40 msgr2=0x7f75280ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7527fff640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f752815ac40 0x7f75280ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7527fff640 1 -- 172.21.15.99:0/3310407409 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75281f7b40 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f75277fe640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75280cf080 0x7f75281f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f751ffff640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f752815ac40 0x7f75280ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7527fff640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 0x7f752814db80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f752000c7d0 tx=0x7f752000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520018070 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752000a640 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75280c73b0 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752000aa70 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.404+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75280c6df0 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.405+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752806cd70 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.405+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f752001b070 con 0x7f752814d6d0 2024-03-04T10:05:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.406+0000 7f7524ff9640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f750806b4d0 0x7f750806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.406+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7520089620 con 0x7f752814d6d0 2024-03-04T10:05:55.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.406+0000 7f75277fe640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f750806b4d0 0x7f750806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.407+0000 7f75277fe640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f750806b4d0 0x7f750806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7518006fd0 tx=0x7f7518008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.410+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7520058660 con 0x7f752814d6d0 2024-03-04T10:05:55.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.582+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f75280af450 con 0x7f752814d6d0 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.583+0000 7f7524ff9640 1 -- 172.21.15.99:0/3310407409 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7520057f60 con 0x7f752814d6d0 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:05:55.585 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:05:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f750806b4d0 msgr2=0x7f750806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f750806b4d0 0x7f750806d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7518006fd0 tx=0x7f7518008040 comp rx=0 tx=0).stop 2024-03-04T10:05:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 msgr2=0x7f752814db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 0x7f752814db80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f752000c7d0 tx=0x7f752000cc90 comp rx=0 tx=0).stop 2024-03-04T10:05:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 shutdown_connections 2024-03-04T10:05:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f750806b4d0 0x7f750806d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f752814d6d0 0x7f752814db80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f752815ac40 0x7f75280ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 --2- 172.21.15.99:0/3310407409 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75280cf080 0x7f75281f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.584+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 >> 172.21.15.99:0/3310407409 conn(0x7f75280d0770 msgr2=0x7f75280ae630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.585+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 shutdown_connections 2024-03-04T10:05:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.585+0000 7f752def1640 1 -- 172.21.15.99:0/3310407409 wait complete. 2024-03-04T10:05:55.597 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T10:05:55.598 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T10:05:55.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.699+0000 7ff7a3d69640 1 Processor -- start 2024-03-04T10:05:55.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.699+0000 7ff7a3d69640 1 -- start start 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a3d69640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff79c0700e0 0x7ff79c06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a3d69640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff79c06f1b0 0x7ff79c0cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a3d69640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79c06b790 con 0x7ff79c06b210 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a3d69640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c06e6d0 con 0x7ff79c06f1b0 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a3d69640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79c06e9b0 con 0x7ff79c0700e0 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a1304640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff79c06f1b0 0x7ff79c0cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a2306640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff79c06b210 0x7ff79c15d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53236/0 (socket says 172.21.15.99:53236) 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a2306640 1 -- 172.21.15.99:0/551583015 learned_addr learned my addr 172.21.15.99:0/551583015 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a1b05640 1 --2- 172.21.15.99:0/551583015 >> v2:172.21.15.162:3300/0 conn(0x7ff79c0700e0 0x7ff79c06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a1304640 1 -- 172.21.15.99:0/551583015 >> v2:172.21.15.162:3300/0 conn(0x7ff79c0700e0 msgr2=0x7ff79c06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a1304640 1 --2- 172.21.15.99:0/551583015 >> v2:172.21.15.162:3300/0 conn(0x7ff79c0700e0 0x7ff79c06acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a1304640 1 -- 172.21.15.99:0/551583015 >> v1:172.21.15.99:6789/0 conn(0x7ff79c06b210 legacy=0x7ff79c15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a1304640 1 -- 172.21.15.99:0/551583015 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c06ec90 con 0x7ff79c06f1b0 2024-03-04T10:05:55.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.700+0000 7ff7a0b03640 1 -- 172.21.15.99:0/551583015 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089267392 0 0) 0x7ff79c06b790 con 0x7ff79c06b210 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.701+0000 7ff7a1304640 1 --2- 172.21.15.99:0/551583015 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0cf9b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff79c072440 tx=0x7ff78c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7348e7d3a1ee66d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.701+0000 7ff7a0b03640 1 -- 172.21.15.99:0/551583015 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c035020 con 0x7ff79c06f1b0 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.701+0000 7ff7a0b03640 1 -- 172.21.15.99:0/551583015 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c031830 con 0x7ff79c06f1b0 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.701+0000 7ff7a0b03640 1 -- 172.21.15.99:0/551583015 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c031c00 con 0x7ff79c06f1b0 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 -- 172.21.15.99:0/551583015 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 msgr2=0x7ff79c0cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/551583015 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0cf9b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff79c072440 tx=0x7ff78c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 -- 172.21.15.99:0/551583015 shutdown_connections 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/551583015 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0cf9b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/551583015 >> v2:172.21.15.162:3300/0 conn(0x7ff79c0700e0 0x7ff79c06acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 -- 172.21.15.99:0/551583015 >> 172.21.15.99:0/551583015 conn(0x7ff79c099a00 msgr2=0x7ff79c0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:55.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 -- 172.21.15.99:0/551583015 shutdown_connections 2024-03-04T10:05:55.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.702+0000 7ff7a3d69640 1 -- 172.21.15.99:0/551583015 wait complete. 2024-03-04T10:05:55.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.703+0000 7ff7a3d69640 1 Processor -- start 2024-03-04T10:05:55.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.703+0000 7ff7a3d69640 1 -- start start 2024-03-04T10:05:55.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a3d69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff79c06b210 0x7ff79c1e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a3d69640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a3d69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff79c0700e0 0x7ff79c0b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a3d69640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0cfef0 con 0x7ff79c06f1b0 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a3d69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0704c0 con 0x7ff79c06b210 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a3d69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c0d0070 con 0x7ff79c0700e0 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35566/0 (socket says 172.21.15.99:35566) 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 -- 172.21.15.99:0/2232289117 learned_addr learned my addr 172.21.15.99:0/2232289117 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1b05640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff79c06b210 0x7ff79c1e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a2306640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff79c0700e0 0x7ff79c0b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 -- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff79c0700e0 msgr2=0x7ff79c0b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff79c0700e0 0x7ff79c0b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 -- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff79c06b210 msgr2=0x7ff79c1e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff79c06b210 0x7ff79c1e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:55.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 -- 172.21.15.99:0/2232289117 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c0c8ea0 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1b05640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff79c06b210 0x7ff79c1e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a2306640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff79c0700e0 0x7ff79c0b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.704+0000 7ff7a1304640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0b54b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff78c000c00 tx=0x7ff78c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.705+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c035070 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.705+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78c004250 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.705+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c03a6f0 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.705+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c0b3470 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.705+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c1e4fc0 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.706+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c074f60 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.706+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff78c038020 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.706+0000 7ff7927fc640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff78006b500 0x7ff78006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.706+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff78c0abd10 con 0x7ff79c06f1b0 2024-03-04T10:05:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.707+0000 7ff7a1b05640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff78006b500 0x7ff78006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:55.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.707+0000 7ff7a1b05640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff78006b500 0x7ff78006d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff798006fd0 tx=0x7ff798008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:55.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.711+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff78c040070 con 0x7ff79c06f1b0 2024-03-04T10:05:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:55.879+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff79c0b66a0 con 0x7ff79c06f1b0 2024-03-04T10:05:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.324+0000 7ff7927fc640 1 -- 172.21.15.99:0/2232289117 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v49) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff78c07ade0 con 0x7ff79c06f1b0 2024-03-04T10:05:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff78006b500 msgr2=0x7ff78006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff78006b500 0x7ff78006d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff798006fd0 tx=0x7ff798008040 comp rx=0 tx=0).stop 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 msgr2=0x7ff79c0b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0b54b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff78c000c00 tx=0x7ff78c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 shutdown_connections 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff78006b500 0x7ff78006d9c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff79c0700e0 0x7ff79c0b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff79c06f1b0 0x7ff79c0b54b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 --2- 172.21.15.99:0/2232289117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff79c06b210 0x7ff79c1e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 >> 172.21.15.99:0/2232289117 conn(0x7ff79c099a00 msgr2=0x7ff79c0d0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 shutdown_connections 2024-03-04T10:05:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.327+0000 7ff7a3d69640 1 -- 172.21.15.99:0/2232289117 wait complete. 2024-03-04T10:05:56.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T10:05:56.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.450+0000 7f9fe94b2640 1 Processor -- start 2024-03-04T10:05:56.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.450+0000 7f9fe94b2640 1 -- start start 2024-03-04T10:05:56.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe94b2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9fe40d34a0 0x7f9fe4161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe94b2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe40d2400 con 0x7f9fe40b5700 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe94b2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe40d26e0 con 0x7f9fe40b9410 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe94b2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe40d29c0 con 0x7f9fe40d34a0 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe2ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9fe40b5700 0x7f9fe40d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53246/0 (socket says 172.21.15.99:53246) 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe2ffd640 1 -- 172.21.15.99:0/1043217896 learned_addr learned my addr 172.21.15.99:0/1043217896 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1043217896 >> v2:172.21.15.162:3300/0 conn(0x7f9fe40d34a0 0x7f9fe4161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676586007 0 0) 0x7f9fe40d2400 con 0x7f9fe40b5700 2024-03-04T10:05:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd0003520 con 0x7f9fe40b5700 2024-03-04T10:05:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593457256 0 0) 0x7f9fd0003520 con 0x7f9fe40b5700 2024-03-04T10:05:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 >> v2:172.21.15.162:3300/0 conn(0x7f9fe40d34a0 msgr2=0x7f9fe4161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 --2- 172.21.15.99:0/1043217896 >> v2:172.21.15.162:3300/0 conn(0x7f9fe40d34a0 0x7f9fe4161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 >> v1:172.21.15.162:6789/0 conn(0x7f9fe40b9410 legacy=0x7f9fe4162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe40d2ca0 con 0x7f9fe40b5700 2024-03-04T10:05:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f9fd4001670 con 0x7f9fe40b5700 2024-03-04T10:05:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1043217896 >> v2:172.21.15.162:3300/0 conn(0x7f9fe40d34a0 0x7f9fe4161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:05:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 >> v1:172.21.15.99:6789/0 conn(0x7f9fe40b5700 legacy=0x7f9fe40d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd00039b0 0x7f9fd0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.451+0000 7f9fe27fc640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe27fc640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fd00051c0 0x7f9fd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40d29c0 con 0x7f9fd0004800 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd000c160 con 0x7f9fd00039b0 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd000c360 con 0x7f9fd00051c0 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe2ffd640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd00039b0 0x7f9fd0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe37fe640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fd00051c0 0x7f9fd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fdbfff640 1 -- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fd00051c0 msgr2=0x7f9fd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fd00051c0 0x7f9fd000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fdbfff640 1 -- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd00039b0 msgr2=0x7f9fd0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd00039b0 0x7f9fd0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe37fe640 1 -- 172.21.15.99:0/1043217896 reap_dead start 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fdbfff640 1 -- 172.21.15.99:0/1043217896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd0003520 con 0x7f9fd0004800 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.452+0000 7f9fe2ffd640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd00039b0 0x7f9fd0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc009c60 tx=0x7f9fd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fa38e165702de1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc039070 con 0x7f9fd0004800 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fcc02c8f0 con 0x7f9fd0004800 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe27fc640 1 -- 172.21.15.99:0/1043217896 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc009860 con 0x7f9fd0004800 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1043217896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 msgr2=0x7f9fd0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc009c60 tx=0x7f9fd0005830 comp rx=0 tx=0).stop 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1043217896 shutdown_connections 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1043217896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fd0004c80 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1043217896 >> 172.21.15.99:0/1043217896 conn(0x7f9fe40d0490 msgr2=0x7f9fe40d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1043217896 shutdown_connections 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.453+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1043217896 wait complete. 2024-03-04T10:05:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 Processor -- start 2024-03-04T10:05:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 -- start start 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fe40cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe40b4590 0x7f9fe4158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fe40b49f0 0x7f9fe4158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40d1b00 con 0x7f9fd0004800 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40b5cf0 con 0x7f9fe40b4590 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe94b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40d1c80 con 0x7f9fe40b49f0 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.455+0000 7f9fe2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fe40cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fe40cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35570/0 (socket says 172.21.15.99:35570) 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 -- 172.21.15.99:0/1821560396 learned_addr learned my addr 172.21.15.99:0/1821560396 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 -- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fe40b49f0 msgr2=0x7f9fe4158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fe40b49f0 0x7f9fe4158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 -- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe40b4590 msgr2=0x7f9fe4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe40b4590 0x7f9fe4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 -- 172.21.15.99:0/1821560396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe40cc010 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe2ffd640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fe40cd0b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400e730 tx=0x7f9fd400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400c660 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd4018070 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400ca00 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe40cc2f0 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.456+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe40cc8b0 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.457+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4074f60 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.458+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9fd400cba0 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.458+0000 7f9fdb7fe640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9fc406b500 0x7f9fc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.458+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f9fd4089ff0 con 0x7f9fd0004800 2024-03-04T10:05:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.458+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9fc406b500 0x7f9fc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:56.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.459+0000 7f9fdbfff640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9fc406b500 0x7f9fc406d9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc036000 tx=0x7f9fcc02e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:56.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.463+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9fd4058e30 con 0x7f9fd0004800 2024-03-04T10:05:56.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:56.633+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9fe40b3b20 con 0x7f9fd0004800 2024-03-04T10:05:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.330+0000 7f9fdb7fe640 1 -- 172.21.15.99:0/1821560396 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v50) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9fd4058730 con 0x7f9fd0004800 2024-03-04T10:05:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2024-03-04T10:05:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9fc406b500 msgr2=0x7f9fc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9fc406b500 0x7f9fc406d9c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc036000 tx=0x7f9fcc02e000 comp rx=0 tx=0).stop 2024-03-04T10:05:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 msgr2=0x7f9fe40cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fe40cd0b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400e730 tx=0x7f9fd400ebf0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 shutdown_connections 2024-03-04T10:05:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9fc406b500 0x7f9fc406d9c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9fe40b49f0 0x7f9fe4158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fe40b4590 0x7f9fe4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 --2- 172.21.15.99:0/1821560396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9fd0004800 0x7f9fe40cd0b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 >> 172.21.15.99:0/1821560396 conn(0x7f9fe40d0490 msgr2=0x7f9fe40c98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 shutdown_connections 2024-03-04T10:05:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.334+0000 7f9fe94b2640 1 -- 172.21.15.99:0/1821560396 wait complete. 2024-03-04T10:05:57.358 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T10:05:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.461+0000 7f42e6709640 1 Processor -- start 2024-03-04T10:05:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.461+0000 7f42e6709640 1 -- start start 2024-03-04T10:05:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e6709640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f42e015b700 0x7f42e00ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e6709640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f42e015e810 0x7f42e006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e6709640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e015bae0 con 0x7f42e006bce0 2024-03-04T10:05:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e6709640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e006c680 con 0x7f42e015e810 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e6709640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e00adb90 con 0x7f42e015b700 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f42e015e810 0x7f42e006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f42e015e810 0x7f42e006c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35572/0 (socket says 172.21.15.99:35572) 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 -- 172.21.15.99:0/251574462 learned_addr learned my addr 172.21.15.99:0/251574462 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42dffff640 1 --2- 172.21.15.99:0/251574462 >> v2:172.21.15.162:3300/0 conn(0x7f42e015b700 0x7f42e00ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 -- 172.21.15.99:0/251574462 >> v2:172.21.15.162:3300/0 conn(0x7f42e015b700 msgr2=0x7f42e00ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 --2- 172.21.15.99:0/251574462 >> v2:172.21.15.162:3300/0 conn(0x7f42e015b700 0x7f42e00ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 -- 172.21.15.99:0/251574462 >> v1:172.21.15.162:6789/0 conn(0x7f42e006bce0 legacy=0x7f42e015e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42e4ca6640 1 -- 172.21.15.99:0/251574462 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e015ee60 con 0x7f42e015e810 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.462+0000 7f42dffff640 1 --2- 172.21.15.99:0/251574462 >> v2:172.21.15.162:3300/0 conn(0x7f42e015b700 0x7f42e00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.463+0000 7f42e4ca6640 1 --2- 172.21.15.99:0/251574462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015e810 0x7f42e006c140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f42e0071410 tx=0x7f42d802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7297d030d15db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.463+0000 7f42deffd640 1 -- 172.21.15.99:0/251574462 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d802cb20 con 0x7f42e015e810 2024-03-04T10:05:57.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.463+0000 7f42deffd640 1 -- 172.21.15.99:0/251574462 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d802ccc0 con 0x7f42e015e810 2024-03-04T10:05:57.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.463+0000 7f42deffd640 1 -- 172.21.15.99:0/251574462 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8034610 con 0x7f42e015e810 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 -- 172.21.15.99:0/251574462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015e810 msgr2=0x7f42e006c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 --2- 172.21.15.99:0/251574462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015e810 0x7f42e006c140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f42e0071410 tx=0x7f42d802bf80 comp rx=0 tx=0).stop 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 -- 172.21.15.99:0/251574462 shutdown_connections 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 --2- 172.21.15.99:0/251574462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015e810 0x7f42e006c140 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 --2- 172.21.15.99:0/251574462 >> v2:172.21.15.162:3300/0 conn(0x7f42e015b700 0x7f42e00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 -- 172.21.15.99:0/251574462 >> 172.21.15.99:0/251574462 conn(0x7f42e0071bc0 msgr2=0x7f42e0071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 -- 172.21.15.99:0/251574462 shutdown_connections 2024-03-04T10:05:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.464+0000 7f42e6709640 1 -- 172.21.15.99:0/251574462 wait complete. 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.465+0000 7f42e6709640 1 Processor -- start 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.465+0000 7f42e6709640 1 -- start start 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e6709640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42e006bce0 0x7f42e014b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e6709640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 0x7f42e0157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e6709640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42e015e810 0x7f42e006f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e6709640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00ae560 con 0x7f42e015b700 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e6709640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00ad820 con 0x7f42e006bce0 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e6709640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00ae6e0 con 0x7f42e015e810 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 0x7f42e0157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 0x7f42e0157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35586/0 (socket says 172.21.15.99:35586) 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 -- 172.21.15.99:0/3516559316 learned_addr learned my addr 172.21.15.99:0/3516559316 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42e4ca6640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42e015e810 0x7f42e006f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42dffff640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42e006bce0 0x7f42e014b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 -- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42e015e810 msgr2=0x7f42e006f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42e015e810 0x7f42e006f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 -- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42e006bce0 msgr2=0x7f42e014b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42e006bce0 0x7f42e014b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.466+0000 7f42df7fe640 1 -- 172.21.15.99:0/3516559316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e0154870 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.467+0000 7f42df7fe640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 0x7f42e0157340 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c400 tx=0x7f42d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.467+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00052d0 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.467+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d0005490 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.467+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e0157880 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.467+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0007260 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.467+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e00cd0a0 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.468+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e0074f60 con 0x7f42e015b700 2024-03-04T10:05:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.468+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f42d0007400 con 0x7f42e015b700 2024-03-04T10:05:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.469+0000 7f42dcff9640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f42cc06b4d0 0x7f42cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.469+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f42d00897d0 con 0x7f42e015b700 2024-03-04T10:05:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.469+0000 7f42dffff640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f42cc06b4d0 0x7f42cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.469+0000 7f42dffff640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f42cc06b4d0 0x7f42cc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f42d4006e80 tx=0x7f42d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:57.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.473+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f42e0154870 con 0x7f42e015b700 2024-03-04T10:05:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:57.678+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f42e006cba0 con 0x7f42e015b700 2024-03-04T10:05:58.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.342+0000 7f42dcff9640 1 -- 172.21.15.99:0/3516559316 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v26) v1 ==== 172+0+0 (secure 0 0 0) 0x7f42d0057ed0 con 0x7f42e015b700 2024-03-04T10:05:58.348 INFO:teuthology.orchestra.run.smithi099.stderr:new fs with metadata pool 7 and data pool 8 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.351+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f42cc06b4d0 msgr2=0x7f42cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.351+0000 7f42e6709640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f42cc06b4d0 0x7f42cc06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f42d4006e80 tx=0x7f42d4008040 comp rx=0 tx=0).stop 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.351+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 msgr2=0x7f42e0157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.351+0000 7f42e6709640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 0x7f42e0157340 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c400 tx=0x7f42d000c8c0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.352+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 shutdown_connections 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.352+0000 7f42e6709640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f42cc06b4d0 0x7f42cc06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.352+0000 7f42e6709640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f42e015e810 0x7f42e006f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.352+0000 7f42e6709640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f42e015b700 0x7f42e0157340 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.352+0000 7f42e6709640 1 --2- 172.21.15.99:0/3516559316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42e006bce0 0x7f42e014b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.352+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 >> 172.21.15.99:0/3516559316 conn(0x7f42e0071bc0 msgr2=0x7f42e00b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:58.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.354+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 shutdown_connections 2024-03-04T10:05:58.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.354+0000 7f42e6709640 1 -- 172.21.15.99:0/3516559316 wait complete. 2024-03-04T10:05:58.375 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:58.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.473+0000 7f56bb59d640 1 Processor -- start 2024-03-04T10:05:58.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.473+0000 7f56bb59d640 1 -- start start 2024-03-04T10:05:58.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56bb59d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f56b415dea0 0x7f56b40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56bb59d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f56b406b3e0 0x7f56b40cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56bb59d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56b406fd20 con 0x7f56b40cc8d0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56bb59d640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b406fad0 con 0x7f56b415dea0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56bb59d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56b4070000 con 0x7f56b406b3e0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9339640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f56b415dea0 0x7f56b40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b8b38640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f56b40cc8d0 0x7f56b406acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53252/0 (socket says 172.21.15.99:53252) 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b8b38640 1 -- 172.21.15.99:0/3974505331 learned_addr learned my addr 172.21.15.99:0/3974505331 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9339640 1 -- 172.21.15.99:0/3974505331 >> v2:172.21.15.162:3300/0 conn(0x7f56b406b3e0 msgr2=0x7f56b40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9b3a640 1 --2- 172.21.15.99:0/3974505331 >> v2:172.21.15.162:3300/0 conn(0x7f56b406b3e0 0x7f56b40cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56abfff640 1 -- 172.21.15.99:0/3974505331 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589422553 0 0) 0x7f56b406fd20 con 0x7f56b40cc8d0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9339640 1 --2- 172.21.15.99:0/3974505331 >> v2:172.21.15.162:3300/0 conn(0x7f56b406b3e0 0x7f56b40cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9339640 1 -- 172.21.15.99:0/3974505331 >> v1:172.21.15.99:6789/0 conn(0x7f56b40cc8d0 legacy=0x7f56b406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9339640 1 -- 172.21.15.99:0/3974505331 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40702e0 con 0x7f56b415dea0 2024-03-04T10:05:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.474+0000 7f56b9b3a640 1 --2- 172.21.15.99:0/3974505331 >> v2:172.21.15.162:3300/0 conn(0x7f56b406b3e0 0x7f56b40cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:05:58.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.475+0000 7f56b9339640 1 --2- 172.21.15.99:0/3974505331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b40cad60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f56b40cceb0 tx=0x7f56b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31936f3f468c5d02 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:58.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.475+0000 7f56abfff640 1 -- 172.21.15.99:0/3974505331 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0035020 con 0x7f56b415dea0 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.475+0000 7f56abfff640 1 -- 172.21.15.99:0/3974505331 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b0031830 con 0x7f56b415dea0 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.475+0000 7f56abfff640 1 -- 172.21.15.99:0/3974505331 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0031c00 con 0x7f56b415dea0 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 -- 172.21.15.99:0/3974505331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 msgr2=0x7f56b40cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 --2- 172.21.15.99:0/3974505331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b40cad60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f56b40cceb0 tx=0x7f56b002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 -- 172.21.15.99:0/3974505331 shutdown_connections 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 --2- 172.21.15.99:0/3974505331 >> v2:172.21.15.162:3300/0 conn(0x7f56b406b3e0 0x7f56b40cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 --2- 172.21.15.99:0/3974505331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b40cad60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 -- 172.21.15.99:0/3974505331 >> 172.21.15.99:0/3974505331 conn(0x7f56b40d0710 msgr2=0x7f56b40d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 -- 172.21.15.99:0/3974505331 shutdown_connections 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.476+0000 7f56bb59d640 1 -- 172.21.15.99:0/3974505331 wait complete. 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.477+0000 7f56bb59d640 1 Processor -- start 2024-03-04T10:05:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.477+0000 7f56bb59d640 1 -- start start 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b406b3e0 0x7f56b41455d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56b40cc8d0 0x7f56b41f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b414b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40d1380 con 0x7f56b415dea0 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40d1200 con 0x7f56b406b3e0 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56b40d1500 con 0x7f56b40cc8d0 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b414b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b414b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35604/0 (socket says 172.21.15.99:35604) 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 -- 172.21.15.99:0/1851243327 learned_addr learned my addr 172.21.15.99:0/1851243327 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b8b38640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56b40cc8d0 0x7f56b41f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9339640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b406b3e0 0x7f56b41455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 -- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56b40cc8d0 msgr2=0x7f56b41f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56b40cc8d0 0x7f56b41f3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 -- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b406b3e0 msgr2=0x7f56b41455d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b406b3e0 0x7f56b41455d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 -- 172.21.15.99:0/1851243327 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b41481a0 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56b9b3a640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b414b670 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00cea0 tx=0x7f56ac00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac003b90 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac003d50 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b4148480 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.478+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac004260 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.479+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b414b3d0 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.480+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b4074f60 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.480+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f56ac004400 con 0x7f56b415dea0 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.480+0000 7f56a9ffb640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.480+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f56ac08ada0 con 0x7f56b415dea0 2024-03-04T10:05:58.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.480+0000 7f56b9339640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.481+0000 7f56b9339640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f569806b4d0 0x7f569806d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f56b0036000 tx=0x7f56b0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:58.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.485+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f56ac059a90 con 0x7f56b415dea0 2024-03-04T10:05:58.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.655+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56b406f160 con 0x7f56b415dea0 2024-03-04T10:05:58.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.656+0000 7f56a9ffb640 1 -- 172.21.15.99:0/1851243327 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f56ac059390 con 0x7f56b415dea0 2024-03-04T10:05:58.659 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:58.659 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":51,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:57.680794+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.658+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f569806b4d0 msgr2=0x7f569806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.658+0000 7f56bb59d640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f569806b4d0 0x7f569806d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f56b0036000 tx=0x7f56b0008030 comp rx=0 tx=0).stop 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.658+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 msgr2=0x7f56b414b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.658+0000 7f56bb59d640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b414b670 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00cea0 tx=0x7f56ac00cf90 comp rx=0 tx=0).stop 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.658+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 shutdown_connections 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f569806b4d0 0x7f569806d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f56b415dea0 0x7f56b414b670 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f56b40cc8d0 0x7f56b41f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 --2- 172.21.15.99:0/1851243327 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f56b406b3e0 0x7f56b41455d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 >> 172.21.15.99:0/1851243327 conn(0x7f56b40d0710 msgr2=0x7f56b406acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:58.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 shutdown_connections 2024-03-04T10:05:58.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.659+0000 7f56bb59d640 1 -- 172.21.15.99:0/1851243327 wait complete. 2024-03-04T10:05:58.672 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:05:58.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.770+0000 7f3400937640 1 Processor -- start 2024-03-04T10:05:58.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.770+0000 7f3400937640 1 -- start start 2024-03-04T10:05:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f3400937640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f33fc06c090 0x7f33fc06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f3400937640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc06c470 0x7f33fc0af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f3400937640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc0ad6c0 con 0x7f33fc066de0 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f3400937640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc0ad960 con 0x7f33fc06c090 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f3400937640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc06ebc0 con 0x7f33fc06c470 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f33fa59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f33fc06c090 0x7f33fc06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f33fa59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f33fc06c090 0x7f33fc06f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35608/0 (socket says 172.21.15.99:35608) 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.771+0000 7f33fa59c640 1 -- 172.21.15.99:0/2508955895 learned_addr learned my addr 172.21.15.99:0/2508955895 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33f9d9b640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc06c470 0x7f33fc0af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33fa59c640 1 -- 172.21.15.99:0/2508955895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc06c470 msgr2=0x7f33fc0af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33fa59c640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc06c470 0x7f33fc0af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33fa59c640 1 -- 172.21.15.99:0/2508955895 >> v1:172.21.15.162:6789/0 conn(0x7f33fc066de0 legacy=0x7f33fc15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33fa59c640 1 -- 172.21.15.99:0/2508955895 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc06eea0 con 0x7f33fc06c090 2024-03-04T10:05:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33f9d9b640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc06c470 0x7f33fc0af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:05:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.772+0000 7f33fa59c640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc06f680 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009e50 tx=0x7f33ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d77fb4caa48d434 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f33f959a640 1 -- 172.21.15.99:0/2508955895 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec02ca70 con 0x7f33fc06c090 2024-03-04T10:05:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f33f959a640 1 -- 172.21.15.99:0/2508955895 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec02cc10 con 0x7f33fc06c090 2024-03-04T10:05:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f33f959a640 1 -- 172.21.15.99:0/2508955895 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0343f0 con 0x7f33fc06c090 2024-03-04T10:05:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 -- 172.21.15.99:0/2508955895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 msgr2=0x7f33fc06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc06f680 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009e50 tx=0x7f33ec02bed0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 -- 172.21.15.99:0/2508955895 shutdown_connections 2024-03-04T10:05:58.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc06c470 0x7f33fc0af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 --2- 172.21.15.99:0/2508955895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc06f680 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 -- 172.21.15.99:0/2508955895 >> 172.21.15.99:0/2508955895 conn(0x7f33fc0cfef0 msgr2=0x7f33fc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 -- 172.21.15.99:0/2508955895 shutdown_connections 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.773+0000 7f3400937640 1 -- 172.21.15.99:0/2508955895 wait complete. 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 Processor -- start 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 -- start start 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33fc066de0 0x7f33fc09b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc14ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc1531b0 0x7f33fc15aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0adc40 con 0x7f33fc06c090 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0af760 con 0x7f33fc066de0 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f3400937640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0addc0 con 0x7f33fc1531b0 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.775+0000 7f33f9d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc14ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc14ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35614/0 (socket says 172.21.15.99:35614) 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 -- 172.21.15.99:0/745905019 learned_addr learned my addr 172.21.15.99:0/745905019 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33fa59c640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33fc066de0 0x7f33fc09b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33fad9d640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc1531b0 0x7f33fc15aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 -- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc1531b0 msgr2=0x7f33fc15aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc1531b0 0x7f33fc15aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 -- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33fc066de0 msgr2=0x7f33fc09b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33fc066de0 0x7f33fc09b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 -- 172.21.15.99:0/745905019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc099aa0 con 0x7f33fc06c090 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33f9d9b640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc14ef40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f33f000e7e0 tx=0x7f33f000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f000f040 con 0x7f33fc06c090 2024-03-04T10:05:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f0004030 con 0x7f33fc06c090 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0004400 con 0x7f33fc06c090 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc099d80 con 0x7f33fc06c090 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.776+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc09a340 con 0x7f33fc06c090 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.777+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc074f60 con 0x7f33fc06c090 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.778+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f33f000c6d0 con 0x7f33fc06c090 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.778+0000 7f33eaffd640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33dc06b4d0 0x7f33dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.778+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f33f008a0c0 con 0x7f33fc06c090 2024-03-04T10:05:58.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.778+0000 7f33fa59c640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33dc06b4d0 0x7f33dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:58.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.779+0000 7f33fa59c640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33dc06b4d0 0x7f33dc06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f33ec035000 tx=0x7f33ec030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:58.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.782+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33f0017020 con 0x7f33fc06c090 2024-03-04T10:05:58.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.953+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33fc06def0 con 0x7f33fc06c090 2024-03-04T10:05:59.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.954+0000 7f33eaffd640 1 -- 172.21.15.99:0/745905019 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16919 (secure 0 0 0) 0x7f33f0058760 con 0x7f33fc06c090 2024-03-04T10:05:59.463 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:05:59.463 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":51,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:57.680794+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33dc06b4d0 msgr2=0x7f33dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33dc06b4d0 0x7f33dc06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f33ec035000 tx=0x7f33ec030000 comp rx=0 tx=0).stop 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 msgr2=0x7f33fc14ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc14ef40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f33f000e7e0 tx=0x7f33f000eca0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 shutdown_connections 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33dc06b4d0 0x7f33dc06d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33fc1531b0 0x7f33fc15aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33fc06c090 0x7f33fc14ef40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 --2- 172.21.15.99:0/745905019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33fc066de0 0x7f33fc09b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 >> 172.21.15.99:0/745905019 conn(0x7f33fc0cfef0 msgr2=0x7f33fc06acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 shutdown_connections 2024-03-04T10:05:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:58.956+0000 7f3400937640 1 -- 172.21.15.99:0/745905019 wait complete. 2024-03-04T10:05:59.466 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T10:05:59.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.573+0000 7f29c9a43640 1 Processor -- start 2024-03-04T10:05:59.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.573+0000 7f29c9a43640 1 -- start start 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c9a43640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f29c40cf020 0x7f29c40b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c9a43640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c40ca970 con 0x7f29c415de40 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c9a43640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c4161d90 con 0x7f29c40cec40 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c9a43640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c4162070 con 0x7f29c40cf020 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c27fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f29c415de40 0x7f29c40d2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53256/0 (socket says 172.21.15.99:53256) 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c27fc640 1 -- 172.21.15.99:0/564229193 learned_addr learned my addr 172.21.15.99:0/564229193 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427720909 0 0) 0x7f29c40ca970 con 0x7f29c415de40 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c37fe640 1 --2- 172.21.15.99:0/564229193 >> v2:172.21.15.162:3300/0 conn(0x7f29c40cf020 0x7f29c40b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.574+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ac003520 con 0x7f29c415de40 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2920088499 0 0) 0x7f29ac003520 con 0x7f29c415de40 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 >> v2:172.21.15.162:3300/0 conn(0x7f29c40cf020 msgr2=0x7f29c40b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 --2- 172.21.15.99:0/564229193 >> v2:172.21.15.162:3300/0 conn(0x7f29c40cf020 0x7f29c40b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 >> v1:172.21.15.162:6789/0 conn(0x7f29c40cec40 legacy=0x7f29c415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c4162350 con 0x7f29c415de40 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f29b8001670 con 0x7f29c415de40 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 >> v1:172.21.15.99:6789/0 conn(0x7f29c415de40 legacy=0x7f29c40d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac0039b0 0x7f29ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29ac004800 0x7f29ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29ac0051c0 0x7f29ac0141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4162070 con 0x7f29ac004800 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac014870 con 0x7f29ac0039b0 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29ac014a70 con 0x7f29ac0051c0 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c27fc640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29ac004800 0x7f29ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c2ffd640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac0039b0 0x7f29ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c37fe640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29ac0051c0 0x7f29ac0141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c27fc640 1 -- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29ac0051c0 msgr2=0x7f29ac0141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c27fc640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29ac0051c0 0x7f29ac0141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c27fc640 1 -- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac0039b0 msgr2=0x7f29ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c27fc640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac0039b0 0x7f29ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c27fc640 1 -- 172.21.15.99:0/564229193 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ac003520 con 0x7f29ac004800 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.575+0000 7f29c37fe640 1 -- 172.21.15.99:0/564229193 reap_dead start 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c27fc640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29ac004800 0x7f29ac004c80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f29b800c240 tx=0x7f29ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcd28be4d27f6e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8007cf0 con 0x7f29ac004800 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b8030070 con 0x7f29ac004800 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c1ffb640 1 -- 172.21.15.99:0/564229193 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8005d30 con 0x7f29ac004800 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c9a43640 1 -- 172.21.15.99:0/564229193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29ac004800 msgr2=0x7f29ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c9a43640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29ac004800 0x7f29ac004c80 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f29b800c240 tx=0x7f29ac005830 comp rx=0 tx=0).stop 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c9a43640 1 -- 172.21.15.99:0/564229193 shutdown_connections 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c9a43640 1 --2- 172.21.15.99:0/564229193 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29ac004800 0x7f29ac004c80 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c9a43640 1 -- 172.21.15.99:0/564229193 >> 172.21.15.99:0/564229193 conn(0x7f29c40d0580 msgr2=0x7f29c40d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.576+0000 7f29c9a43640 1 -- 172.21.15.99:0/564229193 shutdown_connections 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.577+0000 7f29c9a43640 1 -- 172.21.15.99:0/564229193 wait complete. 2024-03-04T10:05:59.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 Processor -- start 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 -- start start 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac004800 0x7f29c4150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29c406fe40 0x7f29c40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 0x7f29c4160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40d3300 con 0x7f29c4160830 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40d1bf0 con 0x7f29ac004800 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.578+0000 7f29c9a43640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40d3480 con 0x7f29c406fe40 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 0x7f29c4160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 0x7f29c4160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35632/0 (socket says 172.21.15.99:35632) 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 -- 172.21.15.99:0/2889953365 learned_addr learned my addr 172.21.15.99:0/2889953365 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c2ffd640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac004800 0x7f29c4150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c27fc640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29c406fe40 0x7f29c40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 -- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29c406fe40 msgr2=0x7f29c40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29c406fe40 0x7f29c40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 -- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac004800 msgr2=0x7f29c4150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac004800 0x7f29c4150680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 -- 172.21.15.99:0/2889953365 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c415eca0 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c37fe640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 0x7f29c4160cb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f29c40d18d0 tx=0x7f29bc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc011e10 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc01b070 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c415ef80 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc003990 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.579+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c40708e0 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.580+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c40d2f70 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.580+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f29bc003b50 con 0x7f29c4160830 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.581+0000 7f29abfff640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29b006b500 0x7f29b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:05:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.581+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29bc08aba0 con 0x7f29c4160830 2024-03-04T10:05:59.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.581+0000 7f29c2ffd640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29b006b500 0x7f29b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:05:59.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.582+0000 7f29c2ffd640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29b006b500 0x7f29b006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000c00 tx=0x7f29b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:05:59.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.585+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f29c40d2f70 con 0x7f29c4160830 2024-03-04T10:05:59.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:05:59.792+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f29c4074f60 con 0x7f29c4160830 2024-03-04T10:06:00.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.348+0000 7f29abfff640 1 -- 172.21.15.99:0/2889953365 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7f29bc059920 con 0x7f29c4160830 2024-03-04T10:06:00.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29b006b500 msgr2=0x7f29b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:00.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29b006b500 0x7f29b006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000c00 tx=0x7f29b4009040 comp rx=0 tx=0).stop 2024-03-04T10:06:00.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 msgr2=0x7f29c4160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:00.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 0x7f29c4160cb0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f29c40d18d0 tx=0x7f29bc0115a0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 shutdown_connections 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29b006b500 0x7f29b006d9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29c4160830 0x7f29c4160cb0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29c406fe40 0x7f29c40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 --2- 172.21.15.99:0/2889953365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f29ac004800 0x7f29c4150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 >> 172.21.15.99:0/2889953365 conn(0x7f29c40d0580 msgr2=0x7f29c40b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 shutdown_connections 2024-03-04T10:06:00.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.351+0000 7f29c9a43640 1 -- 172.21.15.99:0/2889953365 wait complete. 2024-03-04T10:06:00.366 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T10:06:00.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T10:06:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.474+0000 7fa3eefa2640 1 Processor -- start 2024-03-04T10:06:00.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.474+0000 7fa3eefa2640 1 -- start start 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3eefa2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3e815bac0 0x7fa3e809a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3eefa2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e815bea0 con 0x7fa3e80cf720 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3eefa2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e809ada0 con 0x7fa3e80cfc70 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3eefa2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e8099aa0 con 0x7fa3e815bac0 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3edfa0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa3e815bac0 0x7fa3e809a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3ed79f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa3e80cf720 0x7fa3e80ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53268/0 (socket says 172.21.15.99:53268) 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.475+0000 7fa3ed79f640 1 -- 172.21.15.99:0/3619039114 learned_addr learned my addr 172.21.15.99:0/3619039114 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.476+0000 7fa3edfa0640 1 -- 172.21.15.99:0/3619039114 >> v1:172.21.15.162:6789/0 conn(0x7fa3e80cfc70 legacy=0x7fa3e80c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.476+0000 7fa3edfa0640 1 -- 172.21.15.99:0/3619039114 >> v1:172.21.15.99:6789/0 conn(0x7fa3e80cf720 legacy=0x7fa3e80ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.476+0000 7fa3edfa0640 1 -- 172.21.15.99:0/3619039114 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8099d80 con 0x7fa3e815bac0 2024-03-04T10:06:00.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.476+0000 7fa3ecf9e640 1 -- 172.21.15.99:0/3619039114 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885473219 0 0) 0x7fa3e815bea0 con 0x7fa3e80cf720 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.476+0000 7fa3edfa0640 1 --2- 172.21.15.99:0/3619039114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e815bac0 0x7fa3e809a860 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009d90 tx=0x7fa3e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84178a31ba8799d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3ecf9e640 1 -- 172.21.15.99:0/3619039114 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4039070 con 0x7fa3e815bac0 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3ecf9e640 1 -- 172.21.15.99:0/3619039114 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e402cc60 con 0x7fa3e815bac0 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3ecf9e640 1 -- 172.21.15.99:0/3619039114 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4034660 con 0x7fa3e815bac0 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 -- 172.21.15.99:0/3619039114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e815bac0 msgr2=0x7fa3e809a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/3619039114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e815bac0 0x7fa3e809a860 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009d90 tx=0x7fa3e402beb0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 -- 172.21.15.99:0/3619039114 shutdown_connections 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/3619039114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e815bac0 0x7fa3e809a860 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 -- 172.21.15.99:0/3619039114 >> 172.21.15.99:0/3619039114 conn(0x7fa3e80b31a0 msgr2=0x7fa3e80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 -- 172.21.15.99:0/3619039114 shutdown_connections 2024-03-04T10:06:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.477+0000 7fa3eefa2640 1 -- 172.21.15.99:0/3619039114 wait complete. 2024-03-04T10:06:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.478+0000 7fa3eefa2640 1 Processor -- start 2024-03-04T10:06:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.478+0000 7fa3eefa2640 1 -- start start 2024-03-04T10:06:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3eefa2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3e80cf720 0x7fa3e8150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:00.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3eefa2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 0x7fa3e815d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3eefa2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3e815bac0 0x7fa3e815d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3eefa2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80c5910 con 0x7fa3e80cfc70 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3eefa2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80cd620 con 0x7fa3e80cf720 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3eefa2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e80c5a90 con 0x7fa3e815bac0 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 0x7fa3e815d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 0x7fa3e815d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35658/0 (socket says 172.21.15.99:35658) 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 -- 172.21.15.99:0/1606266872 learned_addr learned my addr 172.21.15.99:0/1606266872 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 -- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3e815bac0 msgr2=0x7fa3e815d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3e815bac0 0x7fa3e815d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 -- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3e80cf720 msgr2=0x7fa3e8150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3e80cf720 0x7fa3e8150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.479+0000 7fa3ed79f640 1 -- 172.21.15.99:0/1606266872 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8151080 con 0x7fa3e80cfc70 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.480+0000 7fa3ed79f640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 0x7fa3e815d370 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800c4a0 tx=0x7fa3d800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.480+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8018070 con 0x7fa3e80cfc70 2024-03-04T10:06:00.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.480+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d80074e0 con 0x7fa3e80cfc70 2024-03-04T10:06:00.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.480+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8004a20 con 0x7fa3e80cfc70 2024-03-04T10:06:00.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.480+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e815c1c0 con 0x7fa3e80cfc70 2024-03-04T10:06:00.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.480+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e815c780 con 0x7fa3e80cfc70 2024-03-04T10:06:00.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.481+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e804f090 con 0x7fa3e80cfc70 2024-03-04T10:06:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.481+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa3d801b070 con 0x7fa3e80cfc70 2024-03-04T10:06:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.481+0000 7fa3de7fc640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3cc06b500 0x7fa3cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.482+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa3d80895b0 con 0x7fa3e80cfc70 2024-03-04T10:06:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.482+0000 7fa3edfa0640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3cc06b500 0x7fa3cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.482+0000 7fa3edfa0640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3cc06b500 0x7fa3cc06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4036000 tx=0x7fa3e402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:00.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.486+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa3e8151080 con 0x7fa3e80cfc70 2024-03-04T10:06:00.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:00.691+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa3e8073ed0 con 0x7fa3e80cfc70 2024-03-04T10:06:01.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.353+0000 7fa3de7fc640 1 -- 172.21.15.99:0/1606266872 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa3d8058330 con 0x7fa3e80cfc70 2024-03-04T10:06:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3cc06b500 msgr2=0x7fa3cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3cc06b500 0x7fa3cc06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4036000 tx=0x7fa3e402e000 comp rx=0 tx=0).stop 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 msgr2=0x7fa3e815d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 0x7fa3e815d370 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800c4a0 tx=0x7fa3d800c960 comp rx=0 tx=0).stop 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 shutdown_connections 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3cc06b500 0x7fa3cc06d9c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3e815bac0 0x7fa3e815d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3e80cfc70 0x7fa3e815d370 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 --2- 172.21.15.99:0/1606266872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3e80cf720 0x7fa3e8150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 >> 172.21.15.99:0/1606266872 conn(0x7fa3e80b31a0 msgr2=0x7fa3e80c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 shutdown_connections 2024-03-04T10:06:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.356+0000 7fa3eefa2640 1 -- 172.21.15.99:0/1606266872 wait complete. 2024-03-04T10:06:01.371 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T10:06:01.372 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.476+0000 7feb4890a640 1 Processor -- start 2024-03-04T10:06:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.476+0000 7feb4890a640 1 -- start start 2024-03-04T10:06:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.477+0000 7feb4890a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb4415de00 0x7feb440b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.477+0000 7feb4890a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb440b5470 0x7feb440b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.477+0000 7feb4890a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4415e1e0 con 0x7feb440b9380 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.477+0000 7feb4890a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb440d2820 con 0x7feb440b5470 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.477+0000 7feb4890a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb440d1c10 con 0x7feb4415de00 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.477+0000 7feb42d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb440b5470 0x7feb440b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb41d9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7feb440b9380 0x7feb440b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53272/0 (socket says 172.21.15.99:53272) 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb42d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb440b5470 0x7feb440b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35670/0 (socket says 172.21.15.99:35670) 2024-03-04T10:06:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb41d9b640 1 -- 172.21.15.99:0/1853205696 learned_addr learned my addr 172.21.15.99:0/1853205696 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb4259c640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb4415de00 0x7feb440b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb42d9d640 1 -- 172.21.15.99:0/1853205696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb4415de00 msgr2=0x7feb440b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb42d9d640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb4415de00 0x7feb440b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb42d9d640 1 -- 172.21.15.99:0/1853205696 >> v1:172.21.15.99:6789/0 conn(0x7feb440b9380 legacy=0x7feb440b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb42d9d640 1 -- 172.21.15.99:0/1853205696 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb440d1ef0 con 0x7feb440b5470 2024-03-04T10:06:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb4159a640 1 -- 172.21.15.99:0/1853205696 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541631116 0 0) 0x7feb4415e1e0 con 0x7feb440b9380 2024-03-04T10:06:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.478+0000 7feb4259c640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb4415de00 0x7feb440b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb42d9d640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b5470 0x7feb440b5900 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009d90 tx=0x7feb3c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515746a9ac648387 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4159a640 1 -- 172.21.15.99:0/1853205696 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c039070 con 0x7feb440b5470 2024-03-04T10:06:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4159a640 1 -- 172.21.15.99:0/1853205696 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c02cc50 con 0x7feb440b5470 2024-03-04T10:06:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4159a640 1 -- 172.21.15.99:0/1853205696 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c034440 con 0x7feb440b5470 2024-03-04T10:06:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 -- 172.21.15.99:0/1853205696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b5470 msgr2=0x7feb440b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b5470 0x7feb440b5900 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009d90 tx=0x7feb3c02bea0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 -- 172.21.15.99:0/1853205696 shutdown_connections 2024-03-04T10:06:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b5470 0x7feb440b5900 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 --2- 172.21.15.99:0/1853205696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb4415de00 0x7feb440b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 -- 172.21.15.99:0/1853205696 >> 172.21.15.99:0/1853205696 conn(0x7feb440d0470 msgr2=0x7feb440d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 -- 172.21.15.99:0/1853205696 shutdown_connections 2024-03-04T10:06:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.479+0000 7feb4890a640 1 -- 172.21.15.99:0/1853205696 wait complete. 2024-03-04T10:06:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 Processor -- start 2024-03-04T10:06:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 -- start start 2024-03-04T10:06:01.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb440b5470 0x7feb441605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 0x7feb44152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb4415de00 0x7feb440d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440b97e0 con 0x7feb440b9380 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb440d24b0 con 0x7feb4415de00 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb4890a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb440b72a0 con 0x7feb440b5470 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb41d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 0x7feb44152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb41d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 0x7feb44152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35676/0 (socket says 172.21.15.99:35676) 2024-03-04T10:06:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.481+0000 7feb41d9b640 1 -- 172.21.15.99:0/2666582403 learned_addr learned my addr 172.21.15.99:0/2666582403 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb4259c640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb440b5470 0x7feb441605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb42d9d640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb4415de00 0x7feb440d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb41d9b640 1 -- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb440b5470 msgr2=0x7feb441605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb41d9b640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb440b5470 0x7feb441605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb41d9b640 1 -- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb4415de00 msgr2=0x7feb440d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb41d9b640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb4415de00 0x7feb440d2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb41d9b640 1 -- 172.21.15.99:0/2666582403 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb441534c0 con 0x7feb440b9380 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb4259c640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb440b5470 0x7feb441605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb41d9b640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 0x7feb44152f80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7feb3400c500 tx=0x7feb3400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34018070 con 0x7feb440b9380 2024-03-04T10:06:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34007580 con 0x7feb440b9380 2024-03-04T10:06:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34004a20 con 0x7feb440b9380 2024-03-04T10:06:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb44160d10 con 0x7feb440b9380 2024-03-04T10:06:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.482+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb441f3e20 con 0x7feb440b9380 2024-03-04T10:06:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.483+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb44074f60 con 0x7feb440b9380 2024-03-04T10:06:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.484+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7feb3401b070 con 0x7feb440b9380 2024-03-04T10:06:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.484+0000 7feb32ffd640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.484+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb340896e0 con 0x7feb440b9380 2024-03-04T10:06:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.484+0000 7feb4259c640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.485+0000 7feb4259c640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb2406b4d0 0x7feb2406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb38006fd0 tx=0x7feb38008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.488+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feb441534c0 con 0x7feb440b9380 2024-03-04T10:06:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.692+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb4406b1b0 con 0x7feb440b9380 2024-03-04T10:06:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.692+0000 7feb32ffd640 1 -- 172.21.15.99:0/2666582403 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4938 (secure 0 0 0) 0x7feb340583d0 con 0x7feb440b9380 2024-03-04T10:06:01.695 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:01.695 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:00.693772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5961},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5961":{"gid":5961,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6833/370337902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":370337902},{"type":"v1","addr":"172.21.15.99:6833","nonce":370337902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 30 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb2406b4d0 msgr2=0x7feb2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb2406b4d0 0x7feb2406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feb38006fd0 tx=0x7feb38008040 comp rx=0 tx=0).stop 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 msgr2=0x7feb44152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 0x7feb44152f80 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7feb3400c500 tx=0x7feb3400c9c0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 shutdown_connections 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb4415de00 0x7feb440d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb440b9380 0x7feb44152f80 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 --2- 172.21.15.99:0/2666582403 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb440b5470 0x7feb441605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 >> 172.21.15.99:0/2666582403 conn(0x7feb440d0470 msgr2=0x7feb440b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 shutdown_connections 2024-03-04T10:06:01.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.694+0000 7feb4890a640 1 -- 172.21.15.99:0/2666582403 wait complete. 2024-03-04T10:06:01.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:01.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.807+0000 7fa0707b9640 1 Processor -- start 2024-03-04T10:06:01.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.807+0000 7fa0707b9640 1 -- start start 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa0707b9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 0x7fa06815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa0707b9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06806d8b0 con 0x7fa06815b030 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa0707b9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06806db90 con 0x7fa06815b410 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa0707b9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680b2720 con 0x7fa06806d3b0 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06e555640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa06815b030 0x7fa06815dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53278/0 (socket says 172.21.15.99:53278) 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06e555640 1 -- 172.21.15.99:0/1324659709 learned_addr learned my addr 172.21.15.99:0/1324659709 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:01.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06dd54640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 0x7fa06815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721698640 0 0) 0x7fa06806d8b0 con 0x7fa06815b030 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa054005730 con 0x7fa06815b030 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908049302 0 0) 0x7fa054005730 con 0x7fa06815b030 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 msgr2=0x7fa06815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 0x7fa06815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 >> v1:172.21.15.162:6789/0 conn(0x7fa06815b410 legacy=0x7fa06815b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06806de70 con 0x7fa06815b030 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fa064001670 con 0x7fa06815b030 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 >> v1:172.21.15.99:6789/0 conn(0x7fa06815b030 legacy=0x7fa06815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa054005bf0 0x7fa054006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 0x7fa054006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa054006f50 0x7fa0540073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.808+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0680b2720 con 0x7fa054005bf0 2024-03-04T10:06:01.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05400dd00 con 0x7fa054006f50 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05400df00 con 0x7fa06806d3b0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06e555640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa054005bf0 0x7fa054006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06dd54640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 0x7fa054006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06ed56640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa054006f50 0x7fa0540073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06e555640 1 -- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 msgr2=0x7fa054006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06e555640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa06806d3b0 0x7fa054006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06e555640 1 -- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa054006f50 msgr2=0x7fa0540073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06e555640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa054006f50 0x7fa0540073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06e555640 1 -- 172.21.15.99:0/1324659709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa054005730 con 0x7fa054005bf0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.809+0000 7fa06ed56640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa054006f50 0x7fa0540073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa06e555640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa054005bf0 0x7fa054006010 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa06400c240 tx=0x7fa054007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a43fcf5e21a644c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064007d10 con 0x7fa054005bf0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa064030070 con 0x7fa054005bf0 2024-03-04T10:06:01.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa06d553640 1 -- 172.21.15.99:0/1324659709 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064005d50 con 0x7fa054005bf0 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa0707b9640 1 -- 172.21.15.99:0/1324659709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa054005bf0 msgr2=0x7fa054006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa0707b9640 1 --2- 172.21.15.99:0/1324659709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa054005bf0 0x7fa054006010 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fa06400c240 tx=0x7fa054007a40 comp rx=0 tx=0).stop 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa06ed56640 1 -- 172.21.15.99:0/1324659709 reap_dead start 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa0707b9640 1 -- 172.21.15.99:0/1324659709 shutdown_connections 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa0707b9640 1 -- 172.21.15.99:0/1324659709 >> 172.21.15.99:0/1324659709 conn(0x7fa06806acd0 msgr2=0x7fa06806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa0707b9640 1 -- 172.21.15.99:0/1324659709 shutdown_connections 2024-03-04T10:06:01.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.810+0000 7fa0707b9640 1 -- 172.21.15.99:0/1324659709 wait complete. 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.811+0000 7fa0707b9640 1 Processor -- start 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.811+0000 7fa0707b9640 1 -- start start 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa0707b9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa054005bf0 0x7fa06814c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa0707b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0680ab450 0x7fa0680ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa0707b9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 0x7fa068158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa0707b9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06806e810 con 0x7fa0681588b0 2024-03-04T10:06:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa0707b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06806e690 con 0x7fa0680ab450 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa0707b9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06815c120 con 0x7fa054005bf0 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa06ed56640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 0x7fa068158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa06ed56640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 0x7fa068158d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35698/0 (socket says 172.21.15.99:35698) 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa06ed56640 1 -- 172.21.15.99:0/4189543016 learned_addr learned my addr 172.21.15.99:0/4189543016 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa06e555640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa054005bf0 0x7fa06814c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.812+0000 7fa06dd54640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0680ab450 0x7fa0680ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06ed56640 1 -- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa054005bf0 msgr2=0x7fa06814c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06ed56640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa054005bf0 0x7fa06814c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06ed56640 1 -- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0680ab450 msgr2=0x7fa0680ab8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06ed56640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0680ab450 0x7fa0680ab8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06ed56640 1 -- 172.21.15.99:0/4189543016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068159670 con 0x7fa0681588b0 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06dd54640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0680ab450 0x7fa0680ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa06ed56640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 0x7fa068158d30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa060000f80 tx=0x7fa06000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060006b50 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa060006d10 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0680a8cd0 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa060003970 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.813+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06814c560 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.814+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa068074f60 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.814+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa06001a020 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.815+0000 7fa05effd640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa04c06b4d0 0x7fa04c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.815+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa0600899e0 con 0x7fa0681588b0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.815+0000 7fa06e555640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa04c06b4d0 0x7fa04c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:01.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.816+0000 7fa06e555640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa04c06b4d0 0x7fa04c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa064034000 tx=0x7fa064009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:01.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.819+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa060058760 con 0x7fa0681588b0 2024-03-04T10:06:01.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.990+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0680cc7c0 con 0x7fa0681588b0 2024-03-04T10:06:01.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.992+0000 7fa05effd640 1 -- 172.21.15.99:0/4189543016 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16906 (secure 0 0 0) 0x7fa060058060 con 0x7fa0681588b0 2024-03-04T10:06:01.994 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:01.994 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":52,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:59.341669+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.993+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa04c06b4d0 msgr2=0x7fa04c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.993+0000 7fa0707b9640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa04c06b4d0 0x7fa04c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa064034000 tx=0x7fa064009030 comp rx=0 tx=0).stop 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.993+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 msgr2=0x7fa068158d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.993+0000 7fa0707b9640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 0x7fa068158d30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa060000f80 tx=0x7fa06000eb90 comp rx=0 tx=0).stop 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 shutdown_connections 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa04c06b4d0 0x7fa04c06d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa0681588b0 0x7fa068158d30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa0680ab450 0x7fa0680ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 --2- 172.21.15.99:0/4189543016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa054005bf0 0x7fa06814c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 >> 172.21.15.99:0/4189543016 conn(0x7fa06806acd0 msgr2=0x7fa0680b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 shutdown_connections 2024-03-04T10:06:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:01.994+0000 7fa0707b9640 1 -- 172.21.15.99:0/4189543016 wait complete. 2024-03-04T10:06:02.007 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T10:06:02.007 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.106+0000 7fa3d32f8640 1 Processor -- start 2024-03-04T10:06:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.106+0000 7fa3d32f8640 1 -- start start 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d32f8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3cc15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d32f8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc15cfe0 con 0x7fa3cc06b860 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d32f8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc0cf630 con 0x7fa3cc06acd0 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d32f8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0d0240 con 0x7fa3cc1597f0 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d1895640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa3cc06b860 0x7fa3cc0cc260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53286/0 (socket says 172.21.15.99:53286) 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d1895640 1 -- 172.21.15.99:0/433046047 learned_addr learned my addr 172.21.15.99:0/433046047 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3d1094640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3cc15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884988158 0 0) 0x7fa3cc15cfe0 con 0x7fa3cc06b860 2024-03-04T10:06:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b8003520 con 0x7fa3cc06b860 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290582329 0 0) 0x7fa3b8003520 con 0x7fa3cc06b860 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 msgr2=0x7fa3cc15c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3cc15c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 >> v1:172.21.15.162:6789/0 conn(0x7fa3cc06acd0 legacy=0x7fa3cc06b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0cf910 con 0x7fa3cc06b860 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fa3c40019f0 con 0x7fa3cc06b860 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 >> v1:172.21.15.99:6789/0 conn(0x7fa3cc06b860 legacy=0x7fa3cc0cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.107+0000 7fa3c3fff640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3c3fff640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3c3fff640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0d0240 con 0x7fa3b80039e0 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b800ba70 con 0x7fa3b8004d40 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b800bc70 con 0x7fa3cc1597f0 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1094640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d0893640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1895640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1094640 1 -- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 msgr2=0x7fa3b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1094640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1094640 1 -- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b8004d40 msgr2=0x7fa3b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1094640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1094640 1 -- 172.21.15.99:0/433046047 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8003520 con 0x7fa3b80039e0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d0893640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1597f0 0x7fa3b8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.108+0000 7fa3d1895640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3b8004d40 0x7fa3b80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d1094640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3b8003dc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009ac0 tx=0x7fa3b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89653fdc873128c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8039070 con 0x7fa3b80039e0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c802c6b0 con 0x7fa3b80039e0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3c3fff640 1 -- 172.21.15.99:0/433046047 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8009690 con 0x7fa3b80039e0 2024-03-04T10:06:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d32f8640 1 -- 172.21.15.99:0/433046047 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 msgr2=0x7fa3b8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/433046047 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3b8003dc0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8009ac0 tx=0x7fa3b8005830 comp rx=0 tx=0).stop 2024-03-04T10:06:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d1895640 1 -- 172.21.15.99:0/433046047 reap_dead start 2024-03-04T10:06:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d32f8640 1 -- 172.21.15.99:0/433046047 shutdown_connections 2024-03-04T10:06:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d32f8640 1 -- 172.21.15.99:0/433046047 >> 172.21.15.99:0/433046047 conn(0x7fa3cc0d0710 msgr2=0x7fa3cc0d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d32f8640 1 -- 172.21.15.99:0/433046047 shutdown_connections 2024-03-04T10:06:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.109+0000 7fa3d32f8640 1 -- 172.21.15.99:0/433046047 wait complete. 2024-03-04T10:06:02.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.110+0000 7fa3d32f8640 1 Processor -- start 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.110+0000 7fa3d32f8640 1 -- start start 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d32f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3cc1446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d32f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc0c68c0 0x7fa3cc1f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d32f8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1f3600 0x7fa3cc141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d32f8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0d1350 con 0x7fa3b80039e0 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d32f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cfed0 con 0x7fa3cc0c68c0 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d32f8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc15c030 con 0x7fa3cc1f3600 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3cc1446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3cc1446c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35710/0 (socket says 172.21.15.99:35710) 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 -- 172.21.15.99:0/2294097946 learned_addr learned my addr 172.21.15.99:0/2294097946 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d0893640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc0c68c0 0x7fa3cc1f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1895640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1f3600 0x7fa3cc141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 -- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1f3600 msgr2=0x7fa3cc141c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1f3600 0x7fa3cc141c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 -- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc0c68c0 msgr2=0x7fa3cc1f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.111+0000 7fa3d1094640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc0c68c0 0x7fa3cc1f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3d1094640 1 -- 172.21.15.99:0/2294097946 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc144c00 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3d1895640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1f3600 0x7fa3cc141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3d1094640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3cc1446c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8000c00 tx=0x7fa3c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8042020 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c803a040 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc0b1910 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8039070 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.112+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc0b1bb0 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.113+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc06cd70 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.113+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa3c8045020 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.114+0000 7fa3c1ffb640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3b006b4d0 0x7fa3b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.114+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa3c80ab7d0 con 0x7fa3b80039e0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.114+0000 7fa3d0893640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3b006b4d0 0x7fa3b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.114+0000 7fa3d0893640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3b006b4d0 0x7fa3b006d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc14ec70 tx=0x7fa3bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.118+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa3c804c070 con 0x7fa3b80039e0 2024-03-04T10:06:02.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.285+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3cc15e050 con 0x7fa3b006b4d0 2024-03-04T10:06:02.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.292+0000 7fa3c1ffb640 1 -- 172.21.15.99:0/2294097946 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246637 (secure 0 0 0) 0x7fa3cc15e050 con 0x7fa3b006b4d0 2024-03-04T10:06:02.297 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:02.298 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3b006b4d0 msgr2=0x7fa3b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3b006b4d0 0x7fa3b006d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc14ec70 tx=0x7fa3bc009040 comp rx=0 tx=0).stop 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 msgr2=0x7fa3cc1446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3cc1446c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8000c00 tx=0x7fa3c8004730 comp rx=0 tx=0).stop 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 shutdown_connections 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa3b006b4d0 0x7fa3b006d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa3cc1f3600 0x7fa3cc141c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa3cc0c68c0 0x7fa3cc1f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 --2- 172.21.15.99:0/2294097946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa3b80039e0 0x7fa3cc1446c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 >> 172.21.15.99:0/2294097946 conn(0x7fa3cc0d0710 msgr2=0x7fa3cc15e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 shutdown_connections 2024-03-04T10:06:02.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.295+0000 7fa3d32f8640 1 -- 172.21.15.99:0/2294097946 wait complete. 2024-03-04T10:06:02.308 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-04T10:06:01.471627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221888,"kb_used_data":5704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752832,"statfs":{"total":773094113280,"available":772866899968,"internally_reserved":0,"allocated":5840896,"data_stored":3235896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132451"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342480+0000","last_change":"2024-03-04T10:05:57.342480+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342480+0000","last_undegraded":"2024-03-04T10:05:57.342480+0000","last_fullsized":"2024-03-04T10:05:57.342480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338459+0000","last_change":"2024-03-04T10:05:57.338459+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338459+0000","last_undegraded":"2024-03-04T10:05:57.338459+0000","last_fullsized":"2024-03-04T10:05:57.338459+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341510+0000","last_change":"2024-03-04T10:05:57.341510+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341510+0000","last_undegraded":"2024-03-04T10:05:57.341510+0000","last_fullsized":"2024-03-04T10:05:57.341510+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339130+0000","last_change":"2024-03-04T10:05:57.339130+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339130+0000","last_undegraded":"2024-03-04T10:05:57.339130+0000","last_fullsized":"2024-03-04T10:05:57.339130+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338288+0000","last_change":"2024-03-04T10:05:57.338288+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338288+0000","last_undegraded":"2024-03-04T10:05:57.338288+0000","last_fullsized":"2024-03-04T10:05:57.338288+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343828+0000","last_change":"2024-03-04T10:05:57.343828+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343828+0000","last_undegraded":"2024-03-04T10:05:57.343828+0000","last_fullsized":"2024-03-04T10:05:57.343828+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340823+0000","last_change":"2024-03-04T10:05:57.340823+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340823+0000","last_undegraded":"2024-03-04T10:05:57.340823+0000","last_fullsized":"2024-03-04T10:05:57.340823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343184+0000","last_change":"2024-03-04T10:05:57.343184+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343184+0000","last_undegraded":"2024-03-04T10:05:57.343184+0000","last_fullsized":"2024-03-04T10:05:57.343184+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338669+0000","last_change":"2024-03-04T10:05:57.338669+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338669+0000","last_undegraded":"2024-03-04T10:05:57.338669+0000","last_fullsized":"2024-03-04T10:05:57.338669+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342054+0000","last_change":"2024-03-04T10:05:57.342054+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342054+0000","last_undegraded":"2024-03-04T10:05:57.342054+0000","last_fullsized":"2024-03-04T10:05:57.342054+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340974+0000","last_change":"2024-03-04T10:05:57.340974+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340974+0000","last_undegraded":"2024-03-04T10:05:57.340974+0000","last_fullsized":"2024-03-04T10:05:57.340974+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339260+0000","last_change":"2024-03-04T10:05:57.339260+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339260+0000","last_undegraded":"2024-03-04T10:05:57.339260+0000","last_fullsized":"2024-03-04T10:05:57.339260+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341698+0000","last_change":"2024-03-04T10:05:57.341698+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341698+0000","last_undegraded":"2024-03-04T10:05:57.341698+0000","last_fullsized":"2024-03-04T10:05:57.341698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341083+0000","last_change":"2024-03-04T10:05:57.341083+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341083+0000","last_undegraded":"2024-03-04T10:05:57.341083+0000","last_fullsized":"2024-03-04T10:05:57.341083+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338937+0000","last_change":"2024-03-04T10:05:57.338937+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338937+0000","last_undegraded":"2024-03-04T10:05:57.338937+0000","last_fullsized":"2024-03-04T10:05:57.338937+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339257+0000","last_change":"2024-03-04T10:05:57.339257+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339257+0000","last_undegraded":"2024-03-04T10:05:57.339257+0000","last_fullsized":"2024-03-04T10:05:57.339257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342715+0000","last_change":"2024-03-04T10:05:57.342715+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342715+0000","last_undegraded":"2024-03-04T10:05:57.342715+0000","last_fullsized":"2024-03-04T10:05:57.342715+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337971+0000","last_change":"2024-03-04T10:05:57.337971+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337971+0000","last_undegraded":"2024-03-04T10:05:57.337971+0000","last_fullsized":"2024-03-04T10:05:57.337971+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341139+0000","last_change":"2024-03-04T10:05:57.341139+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341139+0000","last_undegraded":"2024-03-04T10:05:57.341139+0000","last_fullsized":"2024-03-04T10:05:57.341139+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341520+0000","last_change":"2024-03-04T10:05:57.341520+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341520+0000","last_undegraded":"2024-03-04T10:05:57.341520+0000","last_fullsized":"2024-03-04T10:05:57.341520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340984+0000","last_change":"2024-03-04T10:05:57.340984+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340984+0000","last_undegraded":"2024-03-04T10:05:57.340984+0000","last_fullsized":"2024-03-04T10:05:57.340984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338695+0000","last_change":"2024-03-04T10:05:57.338695+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338695+0000","last_undegraded":"2024-03-04T10:05:57.338695+0000","last_fullsized":"2024-03-04T10:05:57.338695+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342052+0000","last_change":"2024-03-04T10:05:57.342052+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342052+0000","last_undegraded":"2024-03-04T10:05:57.342052+0000","last_fullsized":"2024-03-04T10:05:57.342052+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339928+0000","last_change":"2024-03-04T10:05:57.339928+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339928+0000","last_undegraded":"2024-03-04T10:05:57.339928+0000","last_fullsized":"2024-03-04T10:05:57.339928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343322+0000","last_change":"2024-03-04T10:05:57.343322+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343322+0000","last_undegraded":"2024-03-04T10:05:57.343322+0000","last_fullsized":"2024-03-04T10:05:57.343322+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338555+0000","last_change":"2024-03-04T10:05:57.338555+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338555+0000","last_undegraded":"2024-03-04T10:05:57.338555+0000","last_fullsized":"2024-03-04T10:05:57.338555+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343243+0000","last_change":"2024-03-04T10:05:57.343243+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343243+0000","last_undegraded":"2024-03-04T10:05:57.343243+0000","last_fullsized":"2024-03-04T10:05:57.343243+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341030+0000","last_change":"2024-03-04T10:05:57.341030+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341030+0000","last_undegraded":"2024-03-04T10:05:57.341030+0000","last_fullsized":"2024-03-04T10:05:57.341030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342563+0000","last_change":"2024-03-04T10:05:57.342563+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342563+0000","last_undegraded":"2024-03-04T10:05:57.342563+0000","last_fullsized":"2024-03-04T10:05:57.342563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339783+0000","last_change":"2024-03-04T10:05:57.339783+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339783+0000","last_undegraded":"2024-03-04T10:05:57.339783+0000","last_fullsized":"2024-03-04T10:05:57.339783+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340387+0000","last_change":"2024-03-04T10:05:57.340387+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340387+0000","last_undegraded":"2024-03-04T10:05:57.340387+0000","last_fullsized":"2024-03-04T10:05:57.340387+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342493+0000","last_change":"2024-03-04T10:05:57.342493+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342493+0000","last_undegraded":"2024-03-04T10:05:57.342493+0000","last_fullsized":"2024-03-04T10:05:57.342493+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338611+0000","last_change":"2024-03-04T10:05:57.338611+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338611+0000","last_undegraded":"2024-03-04T10:05:57.338611+0000","last_fullsized":"2024-03-04T10:05:57.338611+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.350566+0000","last_change":"2024-03-04T10:05:57.347237+0000","last_active":"2024-03-04T10:05:57.350566+0000","last_peered":"2024-03-04T10:05:57.350566+0000","last_clean":"2024-03-04T10:05:57.350566+0000","last_became_active":"2024-03-04T10:05:57.346945+0000","last_became_peered":"2024-03-04T10:05:57.346945+0000","last_unstale":"2024-03-04T10:05:57.350566+0000","last_undegraded":"2024-03-04T10:05:57.350566+0000","last_fullsized":"2024-03-04T10:05:57.350566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:39:16.941315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339395+0000","last_change":"2024-03-04T10:05:57.339395+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339395+0000","last_undegraded":"2024-03-04T10:05:57.339395+0000","last_fullsized":"2024-03-04T10:05:57.339395+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356689+0000","last_change":"2024-03-04T10:05:57.354647+0000","last_active":"2024-03-04T10:05:57.356689+0000","last_peered":"2024-03-04T10:05:57.356689+0000","last_clean":"2024-03-04T10:05:57.356689+0000","last_became_active":"2024-03-04T10:05:57.354296+0000","last_became_peered":"2024-03-04T10:05:57.354296+0000","last_unstale":"2024-03-04T10:05:57.356689+0000","last_undegraded":"2024-03-04T10:05:57.356689+0000","last_fullsized":"2024-03-04T10:05:57.356689+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343540+0000","last_change":"2024-03-04T10:05:57.343540+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343540+0000","last_undegraded":"2024-03-04T10:05:57.343540+0000","last_fullsized":"2024-03-04T10:05:57.343540+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.349231+0000","last_change":"2024-03-04T10:05:57.348436+0000","last_active":"2024-03-04T10:05:57.349231+0000","last_peered":"2024-03-04T10:05:57.349231+0000","last_clean":"2024-03-04T10:05:57.349231+0000","last_became_active":"2024-03-04T10:05:57.348160+0000","last_became_peered":"2024-03-04T10:05:57.348160+0000","last_unstale":"2024-03-04T10:05:57.349231+0000","last_undegraded":"2024-03-04T10:05:57.349231+0000","last_fullsized":"2024-03-04T10:05:57.349231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339087+0000","last_change":"2024-03-04T10:05:57.339087+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339087+0000","last_undegraded":"2024-03-04T10:05:57.339087+0000","last_fullsized":"2024-03-04T10:05:57.339087+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353404+0000","last_change":"2024-03-04T10:05:57.348949+0000","last_active":"2024-03-04T10:05:57.353404+0000","last_peered":"2024-03-04T10:05:57.353404+0000","last_clean":"2024-03-04T10:05:57.353404+0000","last_became_active":"2024-03-04T10:05:57.348244+0000","last_became_peered":"2024-03-04T10:05:57.348244+0000","last_unstale":"2024-03-04T10:05:57.353404+0000","last_undegraded":"2024-03-04T10:05:57.353404+0000","last_fullsized":"2024-03-04T10:05:57.353404+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338012+0000","last_change":"2024-03-04T10:05:57.338012+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338012+0000","last_undegraded":"2024-03-04T10:05:57.338012+0000","last_fullsized":"2024-03-04T10:05:57.338012+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355522+0000","last_change":"2024-03-04T10:05:57.353664+0000","last_active":"2024-03-04T10:05:57.355522+0000","last_peered":"2024-03-04T10:05:57.355522+0000","last_clean":"2024-03-04T10:05:57.355522+0000","last_became_active":"2024-03-04T10:05:57.353393+0000","last_became_peered":"2024-03-04T10:05:57.353393+0000","last_unstale":"2024-03-04T10:05:57.355522+0000","last_undegraded":"2024-03-04T10:05:57.355522+0000","last_fullsized":"2024-03-04T10:05:57.355522+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:27.622975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340503+0000","last_change":"2024-03-04T10:05:57.340503+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340503+0000","last_undegraded":"2024-03-04T10:05:57.340503+0000","last_fullsized":"2024-03-04T10:05:57.340503+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354018+0000","last_change":"2024-03-04T10:05:57.349177+0000","last_active":"2024-03-04T10:05:57.354018+0000","last_peered":"2024-03-04T10:05:57.354018+0000","last_clean":"2024-03-04T10:05:57.354018+0000","last_became_active":"2024-03-04T10:05:57.347881+0000","last_became_peered":"2024-03-04T10:05:57.347881+0000","last_unstale":"2024-03-04T10:05:57.354018+0000","last_undegraded":"2024-03-04T10:05:57.354018+0000","last_fullsized":"2024-03-04T10:05:57.354018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337663+0000","last_change":"2024-03-04T10:05:57.337663+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337663+0000","last_undegraded":"2024-03-04T10:05:57.337663+0000","last_fullsized":"2024-03-04T10:05:57.337663+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353263+0000","last_change":"2024-03-04T10:05:57.348567+0000","last_active":"2024-03-04T10:05:57.353263+0000","last_peered":"2024-03-04T10:05:57.353263+0000","last_clean":"2024-03-04T10:05:57.353263+0000","last_became_active":"2024-03-04T10:05:57.348178+0000","last_became_peered":"2024-03-04T10:05:57.348178+0000","last_unstale":"2024-03-04T10:05:57.353263+0000","last_undegraded":"2024-03-04T10:05:57.353263+0000","last_fullsized":"2024-03-04T10:05:57.353263+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:12.651082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338926+0000","last_change":"2024-03-04T10:05:57.338926+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338926+0000","last_undegraded":"2024-03-04T10:05:57.338926+0000","last_fullsized":"2024-03-04T10:05:57.338926+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354413+0000","last_change":"2024-03-04T10:05:57.352999+0000","last_active":"2024-03-04T10:05:57.354413+0000","last_peered":"2024-03-04T10:05:57.354413+0000","last_clean":"2024-03-04T10:05:57.354413+0000","last_became_active":"2024-03-04T10:05:57.352754+0000","last_became_peered":"2024-03-04T10:05:57.352754+0000","last_unstale":"2024-03-04T10:05:57.354413+0000","last_undegraded":"2024-03-04T10:05:57.354413+0000","last_fullsized":"2024-03-04T10:05:57.354413+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:07.296828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.336768+0000","last_change":"2024-03-04T10:05:57.336768+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.336768+0000","last_undegraded":"2024-03-04T10:05:57.336768+0000","last_fullsized":"2024-03-04T10:05:57.336768+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353253+0000","last_change":"2024-03-04T10:05:57.349469+0000","last_active":"2024-03-04T10:05:57.353253+0000","last_peered":"2024-03-04T10:05:57.353253+0000","last_clean":"2024-03-04T10:05:57.353253+0000","last_became_active":"2024-03-04T10:05:57.349159+0000","last_became_peered":"2024-03-04T10:05:57.349159+0000","last_unstale":"2024-03-04T10:05:57.353253+0000","last_undegraded":"2024-03-04T10:05:57.353253+0000","last_fullsized":"2024-03-04T10:05:57.353253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:53.178998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339883+0000","last_change":"2024-03-04T10:05:57.339883+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339883+0000","last_undegraded":"2024-03-04T10:05:57.339883+0000","last_fullsized":"2024-03-04T10:05:57.339883+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355198+0000","last_change":"2024-03-04T10:05:57.354131+0000","last_active":"2024-03-04T10:05:57.355198+0000","last_peered":"2024-03-04T10:05:57.355198+0000","last_clean":"2024-03-04T10:05:57.355198+0000","last_became_active":"2024-03-04T10:05:57.353829+0000","last_became_peered":"2024-03-04T10:05:57.353829+0000","last_unstale":"2024-03-04T10:05:57.355198+0000","last_undegraded":"2024-03-04T10:05:57.355198+0000","last_fullsized":"2024-03-04T10:05:57.355198+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340720+0000","last_change":"2024-03-04T10:05:57.340720+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340720+0000","last_undegraded":"2024-03-04T10:05:57.340720+0000","last_fullsized":"2024-03-04T10:05:57.340720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353322+0000","last_change":"2024-03-04T10:05:57.349719+0000","last_active":"2024-03-04T10:05:57.353322+0000","last_peered":"2024-03-04T10:05:57.353322+0000","last_clean":"2024-03-04T10:05:57.353322+0000","last_became_active":"2024-03-04T10:05:57.349424+0000","last_became_peered":"2024-03-04T10:05:57.349424+0000","last_unstale":"2024-03-04T10:05:57.353322+0000","last_undegraded":"2024-03-04T10:05:57.353322+0000","last_fullsized":"2024-03-04T10:05:57.353322+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:22.446636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337789+0000","last_change":"2024-03-04T10:05:57.337789+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337789+0000","last_undegraded":"2024-03-04T10:05:57.337789+0000","last_fullsized":"2024-03-04T10:05:57.337789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357144+0000","last_change":"2024-03-04T10:05:57.353978+0000","last_active":"2024-03-04T10:05:57.357144+0000","last_peered":"2024-03-04T10:05:57.357144+0000","last_clean":"2024-03-04T10:05:57.357144+0000","last_became_active":"2024-03-04T10:05:57.347397+0000","last_became_peered":"2024-03-04T10:05:57.347397+0000","last_unstale":"2024-03-04T10:05:57.357144+0000","last_undegraded":"2024-03-04T10:05:57.357144+0000","last_fullsized":"2024-03-04T10:05:57.357144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:39.719928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342228+0000","last_change":"2024-03-04T10:05:57.342228+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342228+0000","last_undegraded":"2024-03-04T10:05:57.342228+0000","last_fullsized":"2024-03-04T10:05:57.342228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357253+0000","last_change":"2024-03-04T10:05:57.355370+0000","last_active":"2024-03-04T10:05:57.357253+0000","last_peered":"2024-03-04T10:05:57.357253+0000","last_clean":"2024-03-04T10:05:57.357253+0000","last_became_active":"2024-03-04T10:05:57.355114+0000","last_became_peered":"2024-03-04T10:05:57.355114+0000","last_unstale":"2024-03-04T10:05:57.357253+0000","last_undegraded":"2024-03-04T10:05:57.357253+0000","last_fullsized":"2024-03-04T10:05:57.357253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:08.382759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340369+0000","last_change":"2024-03-04T10:05:57.340369+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340369+0000","last_undegraded":"2024-03-04T10:05:57.340369+0000","last_fullsized":"2024-03-04T10:05:57.340369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355879+0000","last_change":"2024-03-04T10:05:57.354657+0000","last_active":"2024-03-04T10:05:57.355879+0000","last_peered":"2024-03-04T10:05:57.355879+0000","last_clean":"2024-03-04T10:05:57.355879+0000","last_became_active":"2024-03-04T10:05:57.354347+0000","last_became_peered":"2024-03-04T10:05:57.354347+0000","last_unstale":"2024-03-04T10:05:57.355879+0000","last_undegraded":"2024-03-04T10:05:57.355879+0000","last_fullsized":"2024-03-04T10:05:57.355879+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338070+0000","last_change":"2024-03-04T10:05:57.338070+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338070+0000","last_undegraded":"2024-03-04T10:05:57.338070+0000","last_fullsized":"2024-03-04T10:05:57.338070+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.349156+0000","last_change":"2024-03-04T10:05:57.346933+0000","last_active":"2024-03-04T10:05:57.349156+0000","last_peered":"2024-03-04T10:05:57.349156+0000","last_clean":"2024-03-04T10:05:57.349156+0000","last_became_active":"2024-03-04T10:05:57.346652+0000","last_became_peered":"2024-03-04T10:05:57.346652+0000","last_unstale":"2024-03-04T10:05:57.349156+0000","last_undegraded":"2024-03-04T10:05:57.349156+0000","last_fullsized":"2024-03-04T10:05:57.349156+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:07.296828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343697+0000","last_change":"2024-03-04T10:05:57.343697+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343697+0000","last_undegraded":"2024-03-04T10:05:57.343697+0000","last_fullsized":"2024-03-04T10:05:57.343697+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353593+0000","last_change":"2024-03-04T10:05:57.347278+0000","last_active":"2024-03-04T10:05:57.353593+0000","last_peered":"2024-03-04T10:05:57.353593+0000","last_clean":"2024-03-04T10:05:57.353593+0000","last_became_active":"2024-03-04T10:05:57.346983+0000","last_became_peered":"2024-03-04T10:05:57.346983+0000","last_unstale":"2024-03-04T10:05:57.353593+0000","last_undegraded":"2024-03-04T10:05:57.353593+0000","last_fullsized":"2024-03-04T10:05:57.353593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337987+0000","last_change":"2024-03-04T10:05:57.337987+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337987+0000","last_undegraded":"2024-03-04T10:05:57.337987+0000","last_fullsized":"2024-03-04T10:05:57.337987+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357529+0000","last_change":"2024-03-04T10:05:57.356865+0000","last_active":"2024-03-04T10:05:57.357529+0000","last_peered":"2024-03-04T10:05:57.357529+0000","last_clean":"2024-03-04T10:05:57.357529+0000","last_became_active":"2024-03-04T10:05:57.356604+0000","last_became_peered":"2024-03-04T10:05:57.356604+0000","last_unstale":"2024-03-04T10:05:57.357529+0000","last_undegraded":"2024-03-04T10:05:57.357529+0000","last_fullsized":"2024-03-04T10:05:57.357529+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338655+0000","last_change":"2024-03-04T10:05:57.338655+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338655+0000","last_undegraded":"2024-03-04T10:05:57.338655+0000","last_fullsized":"2024-03-04T10:05:57.338655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353131+0000","last_change":"2024-03-04T10:05:57.349592+0000","last_active":"2024-03-04T10:05:57.353131+0000","last_peered":"2024-03-04T10:05:57.353131+0000","last_clean":"2024-03-04T10:05:57.353131+0000","last_became_active":"2024-03-04T10:05:57.349329+0000","last_became_peered":"2024-03-04T10:05:57.349329+0000","last_unstale":"2024-03-04T10:05:57.353131+0000","last_undegraded":"2024-03-04T10:05:57.353131+0000","last_fullsized":"2024-03-04T10:05:57.353131+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:33:03.432962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337290+0000","last_change":"2024-03-04T10:05:57.337290+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337290+0000","last_undegraded":"2024-03-04T10:05:57.337290+0000","last_fullsized":"2024-03-04T10:05:57.337290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353786+0000","last_change":"2024-03-04T10:05:57.350127+0000","last_active":"2024-03-04T10:05:57.353786+0000","last_peered":"2024-03-04T10:05:57.353786+0000","last_clean":"2024-03-04T10:05:57.353786+0000","last_became_active":"2024-03-04T10:05:57.349795+0000","last_became_peered":"2024-03-04T10:05:57.349795+0000","last_unstale":"2024-03-04T10:05:57.353786+0000","last_undegraded":"2024-03-04T10:05:57.353786+0000","last_fullsized":"2024-03-04T10:05:57.353786+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:58.362487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339440+0000","last_change":"2024-03-04T10:05:57.339440+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339440+0000","last_undegraded":"2024-03-04T10:05:57.339440+0000","last_fullsized":"2024-03-04T10:05:57.339440+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354102+0000","last_change":"2024-03-04T10:05:57.352501+0000","last_active":"2024-03-04T10:05:57.354102+0000","last_peered":"2024-03-04T10:05:57.354102+0000","last_clean":"2024-03-04T10:05:57.354102+0000","last_became_active":"2024-03-04T10:05:57.352263+0000","last_became_peered":"2024-03-04T10:05:57.352263+0000","last_unstale":"2024-03-04T10:05:57.354102+0000","last_undegraded":"2024-03-04T10:05:57.354102+0000","last_fullsized":"2024-03-04T10:05:57.354102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:06.471395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340153+0000","last_change":"2024-03-04T10:05:57.340153+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340153+0000","last_undegraded":"2024-03-04T10:05:57.340153+0000","last_fullsized":"2024-03-04T10:05:57.340153+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.351766+0000","last_change":"2024-03-04T10:05:57.350188+0000","last_active":"2024-03-04T10:05:57.351766+0000","last_peered":"2024-03-04T10:05:57.351766+0000","last_clean":"2024-03-04T10:05:57.351766+0000","last_became_active":"2024-03-04T10:05:57.349869+0000","last_became_peered":"2024-03-04T10:05:57.349869+0000","last_unstale":"2024-03-04T10:05:57.351766+0000","last_undegraded":"2024-03-04T10:05:57.351766+0000","last_fullsized":"2024-03-04T10:05:57.351766+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341628+0000","last_change":"2024-03-04T10:05:57.341628+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341628+0000","last_undegraded":"2024-03-04T10:05:57.341628+0000","last_fullsized":"2024-03-04T10:05:57.341628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353669+0000","last_change":"2024-03-04T10:05:57.348152+0000","last_active":"2024-03-04T10:05:57.353669+0000","last_peered":"2024-03-04T10:05:57.353669+0000","last_clean":"2024-03-04T10:05:57.353669+0000","last_became_active":"2024-03-04T10:05:57.347666+0000","last_became_peered":"2024-03-04T10:05:57.347666+0000","last_unstale":"2024-03-04T10:05:57.353669+0000","last_undegraded":"2024-03-04T10:05:57.353669+0000","last_fullsized":"2024-03-04T10:05:57.353669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342010+0000","last_change":"2024-03-04T10:05:57.342010+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342010+0000","last_undegraded":"2024-03-04T10:05:57.342010+0000","last_fullsized":"2024-03-04T10:05:57.342010+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357245+0000","last_change":"2024-03-04T10:05:57.355286+0000","last_active":"2024-03-04T10:05:57.357245+0000","last_peered":"2024-03-04T10:05:57.357245+0000","last_clean":"2024-03-04T10:05:57.357245+0000","last_became_active":"2024-03-04T10:05:57.354851+0000","last_became_peered":"2024-03-04T10:05:57.354851+0000","last_unstale":"2024-03-04T10:05:57.357245+0000","last_undegraded":"2024-03-04T10:05:57.357245+0000","last_fullsized":"2024-03-04T10:05:57.357245+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:22.446636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340221+0000","last_change":"2024-03-04T10:05:57.340221+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340221+0000","last_undegraded":"2024-03-04T10:05:57.340221+0000","last_fullsized":"2024-03-04T10:05:57.340221+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353590+0000","last_change":"2024-03-04T10:05:57.350358+0000","last_active":"2024-03-04T10:05:57.353590+0000","last_peered":"2024-03-04T10:05:57.353590+0000","last_clean":"2024-03-04T10:05:57.353590+0000","last_became_active":"2024-03-04T10:05:57.350087+0000","last_became_peered":"2024-03-04T10:05:57.350087+0000","last_unstale":"2024-03-04T10:05:57.353590+0000","last_undegraded":"2024-03-04T10:05:57.353590+0000","last_fullsized":"2024-03-04T10:05:57.353590+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:42.909689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339252+0000","last_change":"2024-03-04T10:05:57.339252+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339252+0000","last_undegraded":"2024-03-04T10:05:57.339252+0000","last_fullsized":"2024-03-04T10:05:57.339252+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355278+0000","last_change":"2024-03-04T10:05:57.354106+0000","last_active":"2024-03-04T10:05:57.355278+0000","last_peered":"2024-03-04T10:05:57.355278+0000","last_clean":"2024-03-04T10:05:57.355278+0000","last_became_active":"2024-03-04T10:05:57.353627+0000","last_became_peered":"2024-03-04T10:05:57.353627+0000","last_unstale":"2024-03-04T10:05:57.355278+0000","last_undegraded":"2024-03-04T10:05:57.355278+0000","last_fullsized":"2024-03-04T10:05:57.355278+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:23.216649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339491+0000","last_change":"2024-03-04T10:05:57.339491+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339491+0000","last_undegraded":"2024-03-04T10:05:57.339491+0000","last_fullsized":"2024-03-04T10:05:57.339491+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.350556+0000","last_change":"2024-03-04T10:05:57.347633+0000","last_active":"2024-03-04T10:05:57.350556+0000","last_peered":"2024-03-04T10:05:57.350556+0000","last_clean":"2024-03-04T10:05:57.350556+0000","last_became_active":"2024-03-04T10:05:57.347321+0000","last_became_peered":"2024-03-04T10:05:57.347321+0000","last_unstale":"2024-03-04T10:05:57.350556+0000","last_undegraded":"2024-03-04T10:05:57.350556+0000","last_fullsized":"2024-03-04T10:05:57.350556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:27.622975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337788+0000","last_change":"2024-03-04T10:05:57.337788+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337788+0000","last_undegraded":"2024-03-04T10:05:57.337788+0000","last_fullsized":"2024-03-04T10:05:57.337788+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354379+0000","last_change":"2024-03-04T10:05:57.352257+0000","last_active":"2024-03-04T10:05:57.354379+0000","last_peered":"2024-03-04T10:05:57.354379+0000","last_clean":"2024-03-04T10:05:57.354379+0000","last_became_active":"2024-03-04T10:05:57.351978+0000","last_became_peered":"2024-03-04T10:05:57.351978+0000","last_unstale":"2024-03-04T10:05:57.354379+0000","last_undegraded":"2024-03-04T10:05:57.354379+0000","last_fullsized":"2024-03-04T10:05:57.354379+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:12.894467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340438+0000","last_change":"2024-03-04T10:05:57.340438+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340438+0000","last_undegraded":"2024-03-04T10:05:57.340438+0000","last_fullsized":"2024-03-04T10:05:57.340438+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353782+0000","last_change":"2024-03-04T10:05:57.347958+0000","last_active":"2024-03-04T10:05:57.353782+0000","last_peered":"2024-03-04T10:05:57.353782+0000","last_clean":"2024-03-04T10:05:57.353782+0000","last_became_active":"2024-03-04T10:05:57.347694+0000","last_became_peered":"2024-03-04T10:05:57.347694+0000","last_unstale":"2024-03-04T10:05:57.353782+0000","last_undegraded":"2024-03-04T10:05:57.353782+0000","last_fullsized":"2024-03-04T10:05:57.353782+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342652+0000","last_change":"2024-03-04T10:05:57.342652+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342652+0000","last_undegraded":"2024-03-04T10:05:57.342652+0000","last_fullsized":"2024-03-04T10:05:57.342652+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356093+0000","last_change":"2024-03-04T10:05:57.352564+0000","last_active":"2024-03-04T10:05:57.356093+0000","last_peered":"2024-03-04T10:05:57.356093+0000","last_clean":"2024-03-04T10:05:57.356093+0000","last_became_active":"2024-03-04T10:05:57.352324+0000","last_became_peered":"2024-03-04T10:05:57.352324+0000","last_unstale":"2024-03-04T10:05:57.356093+0000","last_undegraded":"2024-03-04T10:05:57.356093+0000","last_fullsized":"2024-03-04T10:05:57.356093+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342406+0000","last_change":"2024-03-04T10:05:57.342406+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342406+0000","last_undegraded":"2024-03-04T10:05:57.342406+0000","last_fullsized":"2024-03-04T10:05:57.342406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.345907+0000","last_change":"2024-03-04T10:05:57.341577+0000","last_active":"2024-03-04T10:05:57.345907+0000","last_peered":"2024-03-04T10:05:57.345907+0000","last_clean":"2024-03-04T10:05:57.345907+0000","last_became_active":"2024-03-04T10:05:57.341307+0000","last_became_peered":"2024-03-04T10:05:57.341307+0000","last_unstale":"2024-03-04T10:05:57.345907+0000","last_undegraded":"2024-03-04T10:05:57.345907+0000","last_fullsized":"2024-03-04T10:05:57.345907+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:23.216649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337281+0000","last_change":"2024-03-04T10:05:57.337281+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337281+0000","last_undegraded":"2024-03-04T10:05:57.337281+0000","last_fullsized":"2024-03-04T10:05:57.337281+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354473+0000","last_change":"2024-03-04T10:05:57.351387+0000","last_active":"2024-03-04T10:05:57.354473+0000","last_peered":"2024-03-04T10:05:57.354473+0000","last_clean":"2024-03-04T10:05:57.354473+0000","last_became_active":"2024-03-04T10:05:57.351012+0000","last_became_peered":"2024-03-04T10:05:57.351012+0000","last_unstale":"2024-03-04T10:05:57.354473+0000","last_undegraded":"2024-03-04T10:05:57.354473+0000","last_fullsized":"2024-03-04T10:05:57.354473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341292+0000","last_change":"2024-03-04T10:05:57.341292+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341292+0000","last_undegraded":"2024-03-04T10:05:57.341292+0000","last_fullsized":"2024-03-04T10:05:57.341292+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356535+0000","last_change":"2024-03-04T10:05:57.354681+0000","last_active":"2024-03-04T10:05:57.356535+0000","last_peered":"2024-03-04T10:05:57.356535+0000","last_clean":"2024-03-04T10:05:57.356535+0000","last_became_active":"2024-03-04T10:05:57.354415+0000","last_became_peered":"2024-03-04T10:05:57.354415+0000","last_unstale":"2024-03-04T10:05:57.356535+0000","last_undegraded":"2024-03-04T10:05:57.356535+0000","last_fullsized":"2024-03-04T10:05:57.356535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:53.178998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":231,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.338829+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:05:57.338829+0000","last_peered":"2024-03-04T10:05:57.338829+0000","last_clean":"2024-03-04T10:05:57.338829+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:05:57.338829+0000","last_undegraded":"2024-03-04T10:05:57.338829+0000","last_fullsized":"2024-03-04T10:05:57.338829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28176,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343664,"statfs":{"total":96636764160,"available":96607911936,"internally_reserved":0,"allocated":1175552,"data_stored":847263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":847263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:02.312 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:02.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.446+0000 7f9592e19640 1 Processor -- start 2024-03-04T10:06:02.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.446+0000 7f9592e19640 1 -- start start 2024-03-04T10:06:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f9592e19640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 0x7f958c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f9592e19640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c15e1e0 con 0x7f958c0b5470 2024-03-04T10:06:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f9592e19640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c0d1c10 con 0x7f958c15de00 2024-03-04T10:06:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f9592e19640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0b98c0 con 0x7f958c0b2e60 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f95913b6640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f958c0b5470 0x7f958c0d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53298/0 (socket says 172.21.15.99:53298) 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f95913b6640 1 -- 172.21.15.99:0/2100573054 learned_addr learned my addr 172.21.15.99:0/2100573054 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658585154 0 0) 0x7f958c15e1e0 con 0x7f958c0b5470 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c003520 con 0x7f958c0b5470 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958bfff640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 0x7f958c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402486093 0 0) 0x7f957c003520 con 0x7f958c0b5470 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 msgr2=0x7f958c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 0x7f958c0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 >> v1:172.21.15.162:6789/0 conn(0x7f958c15de00 legacy=0x7f958c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c0d1ef0 con 0x7f958c0b5470 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f95840019f0 con 0x7f958c0b5470 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 >> v1:172.21.15.99:6789/0 conn(0x7f958c0b5470 legacy=0x7f958c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 0x7f957c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f957c003f60 0x7f957c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f957c004d40 0x7f957c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0b98c0 con 0x7f957c003f60 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c00baa0 con 0x7f957c004d40 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.447+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957c00bca0 con 0x7f958c0b2e60 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f957c003f60 0x7f957c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f9590bb5640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 0x7f957c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f95913b6640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f957c004d40 0x7f957c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 -- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 msgr2=0x7f957c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f958c0b2e60 0x7f957c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 -- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f957c004d40 msgr2=0x7f957c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f957c004d40 0x7f957c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 -- 172.21.15.99:0/2100573054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c003520 con 0x7f957c003f60 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f95913b6640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f957c004d40 0x7f957c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.448+0000 7f958bfff640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f957c003f60 0x7f957c004800 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9578009ac0 tx=0x7f957c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a960fc49973d1e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578039070 con 0x7f957c003f60 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957802c6b0 con 0x7f957c003f60 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f958b7fe640 1 -- 172.21.15.99:0/2100573054 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578009690 con 0x7f957c003f60 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f9592e19640 1 -- 172.21.15.99:0/2100573054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f957c003f60 msgr2=0x7f957c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f9592e19640 1 --2- 172.21.15.99:0/2100573054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f957c003f60 0x7f957c004800 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9578009ac0 tx=0x7f957c005830 comp rx=0 tx=0).stop 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f95913b6640 1 -- 172.21.15.99:0/2100573054 reap_dead start 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f9592e19640 1 -- 172.21.15.99:0/2100573054 shutdown_connections 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f9592e19640 1 -- 172.21.15.99:0/2100573054 >> 172.21.15.99:0/2100573054 conn(0x7f958c0d0470 msgr2=0x7f958c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f9592e19640 1 -- 172.21.15.99:0/2100573054 shutdown_connections 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.449+0000 7f9592e19640 1 -- 172.21.15.99:0/2100573054 wait complete. 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.450+0000 7f9592e19640 1 Processor -- start 2024-03-04T10:06:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.450+0000 7f9592e19640 1 -- start start 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f9592e19640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f957c003f60 0x7f958c153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f9592e19640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f958c0c9ac0 0x7f958c155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f9592e19640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 0x7f958c156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f9592e19640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0d2820 con 0x7f958c155cb0 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f9592e19640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0d24b0 con 0x7f958c0c9ac0 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f9592e19640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0b72a0 con 0x7f957c003f60 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f95913b6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 0x7f958c156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f95913b6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 0x7f958c156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35728/0 (socket says 172.21.15.99:35728) 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.451+0000 7f95913b6640 1 -- 172.21.15.99:0/2040574163 learned_addr learned my addr 172.21.15.99:0/2040574163 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f9590bb5640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f957c003f60 0x7f958c153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f958bfff640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f958c0c9ac0 0x7f958c155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95913b6640 1 -- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f957c003f60 msgr2=0x7f958c153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95913b6640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f957c003f60 0x7f958c153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95913b6640 1 -- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f958c0c9ac0 msgr2=0x7f958c155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95913b6640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f958c0c9ac0 0x7f958c155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95913b6640 1 -- 172.21.15.99:0/2040574163 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c159bb0 con 0x7f958c155cb0 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f9590bb5640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f957c003f60 0x7f958c153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f958bfff640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f958c0c9ac0 0x7f958c155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95913b6640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 0x7f958c156130 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f958c0d4a50 tx=0x7f9584007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958400eb60 con 0x7f958c155cb0 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9584018070 con 0x7f958c155cb0 2024-03-04T10:06:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584013420 con 0x7f958c155cb0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c0cb600 con 0x7f958c155cb0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.452+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c0cb8a0 con 0x7f958c155cb0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.453+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c074f60 con 0x7f958c155cb0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.453+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f95840135c0 con 0x7f958c155cb0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.454+0000 7f95897fa640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f957006b500 0x7f957006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.454+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f958408a550 con 0x7f958c155cb0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.454+0000 7f9590bb5640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f957006b500 0x7f957006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.455+0000 7f9590bb5640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f957006b500 0x7f957006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f958c159fc0 tx=0x7f9580009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.458+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f95840592d0 con 0x7f958c155cb0 2024-03-04T10:06:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.626+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f958c06ccb0 con 0x7f957006b500 2024-03-04T10:06:02.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.633+0000 7f95897fa640 1 -- 172.21.15.99:0/2040574163 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246637 (secure 0 0 0) 0x7f958c06ccb0 con 0x7f957006b500 2024-03-04T10:06:02.637 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:02.639 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:02.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.635+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f957006b500 msgr2=0x7f957006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.635+0000 7f9592e19640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f957006b500 0x7f957006d9c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f958c159fc0 tx=0x7f9580009040 comp rx=0 tx=0).stop 2024-03-04T10:06:02.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.635+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 msgr2=0x7f958c156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.635+0000 7f9592e19640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 0x7f958c156130 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f958c0d4a50 tx=0x7f9584007bc0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 shutdown_connections 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f957006b500 0x7f957006d9c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f958c155cb0 0x7f958c156130 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f958c0c9ac0 0x7f958c155770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 --2- 172.21.15.99:0/2040574163 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f957c003f60 0x7f958c153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 >> 172.21.15.99:0/2040574163 conn(0x7f958c0d0470 msgr2=0x7f958c0b58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 shutdown_connections 2024-03-04T10:06:02.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.636+0000 7f9592e19640 1 -- 172.21.15.99:0/2040574163 wait complete. 2024-03-04T10:06:02.649 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-04T10:06:01.471627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221888,"kb_used_data":5704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752832,"statfs":{"total":773094113280,"available":772866899968,"internally_reserved":0,"allocated":5840896,"data_stored":3235896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132451"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342480+0000","last_change":"2024-03-04T10:05:57.342480+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342480+0000","last_undegraded":"2024-03-04T10:05:57.342480+0000","last_fullsized":"2024-03-04T10:05:57.342480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338459+0000","last_change":"2024-03-04T10:05:57.338459+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338459+0000","last_undegraded":"2024-03-04T10:05:57.338459+0000","last_fullsized":"2024-03-04T10:05:57.338459+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341510+0000","last_change":"2024-03-04T10:05:57.341510+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341510+0000","last_undegraded":"2024-03-04T10:05:57.341510+0000","last_fullsized":"2024-03-04T10:05:57.341510+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339130+0000","last_change":"2024-03-04T10:05:57.339130+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339130+0000","last_undegraded":"2024-03-04T10:05:57.339130+0000","last_fullsized":"2024-03-04T10:05:57.339130+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338288+0000","last_change":"2024-03-04T10:05:57.338288+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338288+0000","last_undegraded":"2024-03-04T10:05:57.338288+0000","last_fullsized":"2024-03-04T10:05:57.338288+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343828+0000","last_change":"2024-03-04T10:05:57.343828+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343828+0000","last_undegraded":"2024-03-04T10:05:57.343828+0000","last_fullsized":"2024-03-04T10:05:57.343828+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340823+0000","last_change":"2024-03-04T10:05:57.340823+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340823+0000","last_undegraded":"2024-03-04T10:05:57.340823+0000","last_fullsized":"2024-03-04T10:05:57.340823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343184+0000","last_change":"2024-03-04T10:05:57.343184+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343184+0000","last_undegraded":"2024-03-04T10:05:57.343184+0000","last_fullsized":"2024-03-04T10:05:57.343184+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338669+0000","last_change":"2024-03-04T10:05:57.338669+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338669+0000","last_undegraded":"2024-03-04T10:05:57.338669+0000","last_fullsized":"2024-03-04T10:05:57.338669+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342054+0000","last_change":"2024-03-04T10:05:57.342054+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342054+0000","last_undegraded":"2024-03-04T10:05:57.342054+0000","last_fullsized":"2024-03-04T10:05:57.342054+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340974+0000","last_change":"2024-03-04T10:05:57.340974+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340974+0000","last_undegraded":"2024-03-04T10:05:57.340974+0000","last_fullsized":"2024-03-04T10:05:57.340974+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339260+0000","last_change":"2024-03-04T10:05:57.339260+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339260+0000","last_undegraded":"2024-03-04T10:05:57.339260+0000","last_fullsized":"2024-03-04T10:05:57.339260+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341698+0000","last_change":"2024-03-04T10:05:57.341698+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341698+0000","last_undegraded":"2024-03-04T10:05:57.341698+0000","last_fullsized":"2024-03-04T10:05:57.341698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341083+0000","last_change":"2024-03-04T10:05:57.341083+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341083+0000","last_undegraded":"2024-03-04T10:05:57.341083+0000","last_fullsized":"2024-03-04T10:05:57.341083+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338937+0000","last_change":"2024-03-04T10:05:57.338937+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338937+0000","last_undegraded":"2024-03-04T10:05:57.338937+0000","last_fullsized":"2024-03-04T10:05:57.338937+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339257+0000","last_change":"2024-03-04T10:05:57.339257+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339257+0000","last_undegraded":"2024-03-04T10:05:57.339257+0000","last_fullsized":"2024-03-04T10:05:57.339257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342715+0000","last_change":"2024-03-04T10:05:57.342715+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342715+0000","last_undegraded":"2024-03-04T10:05:57.342715+0000","last_fullsized":"2024-03-04T10:05:57.342715+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337971+0000","last_change":"2024-03-04T10:05:57.337971+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337971+0000","last_undegraded":"2024-03-04T10:05:57.337971+0000","last_fullsized":"2024-03-04T10:05:57.337971+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341139+0000","last_change":"2024-03-04T10:05:57.341139+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341139+0000","last_undegraded":"2024-03-04T10:05:57.341139+0000","last_fullsized":"2024-03-04T10:05:57.341139+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341520+0000","last_change":"2024-03-04T10:05:57.341520+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341520+0000","last_undegraded":"2024-03-04T10:05:57.341520+0000","last_fullsized":"2024-03-04T10:05:57.341520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340984+0000","last_change":"2024-03-04T10:05:57.340984+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340984+0000","last_undegraded":"2024-03-04T10:05:57.340984+0000","last_fullsized":"2024-03-04T10:05:57.340984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338695+0000","last_change":"2024-03-04T10:05:57.338695+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338695+0000","last_undegraded":"2024-03-04T10:05:57.338695+0000","last_fullsized":"2024-03-04T10:05:57.338695+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342052+0000","last_change":"2024-03-04T10:05:57.342052+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342052+0000","last_undegraded":"2024-03-04T10:05:57.342052+0000","last_fullsized":"2024-03-04T10:05:57.342052+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339928+0000","last_change":"2024-03-04T10:05:57.339928+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339928+0000","last_undegraded":"2024-03-04T10:05:57.339928+0000","last_fullsized":"2024-03-04T10:05:57.339928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343322+0000","last_change":"2024-03-04T10:05:57.343322+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343322+0000","last_undegraded":"2024-03-04T10:05:57.343322+0000","last_fullsized":"2024-03-04T10:05:57.343322+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338555+0000","last_change":"2024-03-04T10:05:57.338555+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338555+0000","last_undegraded":"2024-03-04T10:05:57.338555+0000","last_fullsized":"2024-03-04T10:05:57.338555+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343243+0000","last_change":"2024-03-04T10:05:57.343243+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343243+0000","last_undegraded":"2024-03-04T10:05:57.343243+0000","last_fullsized":"2024-03-04T10:05:57.343243+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341030+0000","last_change":"2024-03-04T10:05:57.341030+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341030+0000","last_undegraded":"2024-03-04T10:05:57.341030+0000","last_fullsized":"2024-03-04T10:05:57.341030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342563+0000","last_change":"2024-03-04T10:05:57.342563+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342563+0000","last_undegraded":"2024-03-04T10:05:57.342563+0000","last_fullsized":"2024-03-04T10:05:57.342563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339783+0000","last_change":"2024-03-04T10:05:57.339783+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339783+0000","last_undegraded":"2024-03-04T10:05:57.339783+0000","last_fullsized":"2024-03-04T10:05:57.339783+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340387+0000","last_change":"2024-03-04T10:05:57.340387+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340387+0000","last_undegraded":"2024-03-04T10:05:57.340387+0000","last_fullsized":"2024-03-04T10:05:57.340387+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342493+0000","last_change":"2024-03-04T10:05:57.342493+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342493+0000","last_undegraded":"2024-03-04T10:05:57.342493+0000","last_fullsized":"2024-03-04T10:05:57.342493+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338611+0000","last_change":"2024-03-04T10:05:57.338611+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338611+0000","last_undegraded":"2024-03-04T10:05:57.338611+0000","last_fullsized":"2024-03-04T10:05:57.338611+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.350566+0000","last_change":"2024-03-04T10:05:57.347237+0000","last_active":"2024-03-04T10:05:57.350566+0000","last_peered":"2024-03-04T10:05:57.350566+0000","last_clean":"2024-03-04T10:05:57.350566+0000","last_became_active":"2024-03-04T10:05:57.346945+0000","last_became_peered":"2024-03-04T10:05:57.346945+0000","last_unstale":"2024-03-04T10:05:57.350566+0000","last_undegraded":"2024-03-04T10:05:57.350566+0000","last_fullsized":"2024-03-04T10:05:57.350566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:39:16.941315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339395+0000","last_change":"2024-03-04T10:05:57.339395+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339395+0000","last_undegraded":"2024-03-04T10:05:57.339395+0000","last_fullsized":"2024-03-04T10:05:57.339395+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356689+0000","last_change":"2024-03-04T10:05:57.354647+0000","last_active":"2024-03-04T10:05:57.356689+0000","last_peered":"2024-03-04T10:05:57.356689+0000","last_clean":"2024-03-04T10:05:57.356689+0000","last_became_active":"2024-03-04T10:05:57.354296+0000","last_became_peered":"2024-03-04T10:05:57.354296+0000","last_unstale":"2024-03-04T10:05:57.356689+0000","last_undegraded":"2024-03-04T10:05:57.356689+0000","last_fullsized":"2024-03-04T10:05:57.356689+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343540+0000","last_change":"2024-03-04T10:05:57.343540+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343540+0000","last_undegraded":"2024-03-04T10:05:57.343540+0000","last_fullsized":"2024-03-04T10:05:57.343540+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.349231+0000","last_change":"2024-03-04T10:05:57.348436+0000","last_active":"2024-03-04T10:05:57.349231+0000","last_peered":"2024-03-04T10:05:57.349231+0000","last_clean":"2024-03-04T10:05:57.349231+0000","last_became_active":"2024-03-04T10:05:57.348160+0000","last_became_peered":"2024-03-04T10:05:57.348160+0000","last_unstale":"2024-03-04T10:05:57.349231+0000","last_undegraded":"2024-03-04T10:05:57.349231+0000","last_fullsized":"2024-03-04T10:05:57.349231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339087+0000","last_change":"2024-03-04T10:05:57.339087+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339087+0000","last_undegraded":"2024-03-04T10:05:57.339087+0000","last_fullsized":"2024-03-04T10:05:57.339087+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353404+0000","last_change":"2024-03-04T10:05:57.348949+0000","last_active":"2024-03-04T10:05:57.353404+0000","last_peered":"2024-03-04T10:05:57.353404+0000","last_clean":"2024-03-04T10:05:57.353404+0000","last_became_active":"2024-03-04T10:05:57.348244+0000","last_became_peered":"2024-03-04T10:05:57.348244+0000","last_unstale":"2024-03-04T10:05:57.353404+0000","last_undegraded":"2024-03-04T10:05:57.353404+0000","last_fullsized":"2024-03-04T10:05:57.353404+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338012+0000","last_change":"2024-03-04T10:05:57.338012+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338012+0000","last_undegraded":"2024-03-04T10:05:57.338012+0000","last_fullsized":"2024-03-04T10:05:57.338012+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355522+0000","last_change":"2024-03-04T10:05:57.353664+0000","last_active":"2024-03-04T10:05:57.355522+0000","last_peered":"2024-03-04T10:05:57.355522+0000","last_clean":"2024-03-04T10:05:57.355522+0000","last_became_active":"2024-03-04T10:05:57.353393+0000","last_became_peered":"2024-03-04T10:05:57.353393+0000","last_unstale":"2024-03-04T10:05:57.355522+0000","last_undegraded":"2024-03-04T10:05:57.355522+0000","last_fullsized":"2024-03-04T10:05:57.355522+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:27.622975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340503+0000","last_change":"2024-03-04T10:05:57.340503+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340503+0000","last_undegraded":"2024-03-04T10:05:57.340503+0000","last_fullsized":"2024-03-04T10:05:57.340503+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354018+0000","last_change":"2024-03-04T10:05:57.349177+0000","last_active":"2024-03-04T10:05:57.354018+0000","last_peered":"2024-03-04T10:05:57.354018+0000","last_clean":"2024-03-04T10:05:57.354018+0000","last_became_active":"2024-03-04T10:05:57.347881+0000","last_became_peered":"2024-03-04T10:05:57.347881+0000","last_unstale":"2024-03-04T10:05:57.354018+0000","last_undegraded":"2024-03-04T10:05:57.354018+0000","last_fullsized":"2024-03-04T10:05:57.354018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337663+0000","last_change":"2024-03-04T10:05:57.337663+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337663+0000","last_undegraded":"2024-03-04T10:05:57.337663+0000","last_fullsized":"2024-03-04T10:05:57.337663+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353263+0000","last_change":"2024-03-04T10:05:57.348567+0000","last_active":"2024-03-04T10:05:57.353263+0000","last_peered":"2024-03-04T10:05:57.353263+0000","last_clean":"2024-03-04T10:05:57.353263+0000","last_became_active":"2024-03-04T10:05:57.348178+0000","last_became_peered":"2024-03-04T10:05:57.348178+0000","last_unstale":"2024-03-04T10:05:57.353263+0000","last_undegraded":"2024-03-04T10:05:57.353263+0000","last_fullsized":"2024-03-04T10:05:57.353263+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:12.651082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338926+0000","last_change":"2024-03-04T10:05:57.338926+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338926+0000","last_undegraded":"2024-03-04T10:05:57.338926+0000","last_fullsized":"2024-03-04T10:05:57.338926+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354413+0000","last_change":"2024-03-04T10:05:57.352999+0000","last_active":"2024-03-04T10:05:57.354413+0000","last_peered":"2024-03-04T10:05:57.354413+0000","last_clean":"2024-03-04T10:05:57.354413+0000","last_became_active":"2024-03-04T10:05:57.352754+0000","last_became_peered":"2024-03-04T10:05:57.352754+0000","last_unstale":"2024-03-04T10:05:57.354413+0000","last_undegraded":"2024-03-04T10:05:57.354413+0000","last_fullsized":"2024-03-04T10:05:57.354413+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:07.296828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.336768+0000","last_change":"2024-03-04T10:05:57.336768+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.336768+0000","last_undegraded":"2024-03-04T10:05:57.336768+0000","last_fullsized":"2024-03-04T10:05:57.336768+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353253+0000","last_change":"2024-03-04T10:05:57.349469+0000","last_active":"2024-03-04T10:05:57.353253+0000","last_peered":"2024-03-04T10:05:57.353253+0000","last_clean":"2024-03-04T10:05:57.353253+0000","last_became_active":"2024-03-04T10:05:57.349159+0000","last_became_peered":"2024-03-04T10:05:57.349159+0000","last_unstale":"2024-03-04T10:05:57.353253+0000","last_undegraded":"2024-03-04T10:05:57.353253+0000","last_fullsized":"2024-03-04T10:05:57.353253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:53.178998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339883+0000","last_change":"2024-03-04T10:05:57.339883+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339883+0000","last_undegraded":"2024-03-04T10:05:57.339883+0000","last_fullsized":"2024-03-04T10:05:57.339883+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355198+0000","last_change":"2024-03-04T10:05:57.354131+0000","last_active":"2024-03-04T10:05:57.355198+0000","last_peered":"2024-03-04T10:05:57.355198+0000","last_clean":"2024-03-04T10:05:57.355198+0000","last_became_active":"2024-03-04T10:05:57.353829+0000","last_became_peered":"2024-03-04T10:05:57.353829+0000","last_unstale":"2024-03-04T10:05:57.355198+0000","last_undegraded":"2024-03-04T10:05:57.355198+0000","last_fullsized":"2024-03-04T10:05:57.355198+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340720+0000","last_change":"2024-03-04T10:05:57.340720+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340720+0000","last_undegraded":"2024-03-04T10:05:57.340720+0000","last_fullsized":"2024-03-04T10:05:57.340720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353322+0000","last_change":"2024-03-04T10:05:57.349719+0000","last_active":"2024-03-04T10:05:57.353322+0000","last_peered":"2024-03-04T10:05:57.353322+0000","last_clean":"2024-03-04T10:05:57.353322+0000","last_became_active":"2024-03-04T10:05:57.349424+0000","last_became_peered":"2024-03-04T10:05:57.349424+0000","last_unstale":"2024-03-04T10:05:57.353322+0000","last_undegraded":"2024-03-04T10:05:57.353322+0000","last_fullsized":"2024-03-04T10:05:57.353322+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:22.446636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337789+0000","last_change":"2024-03-04T10:05:57.337789+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337789+0000","last_undegraded":"2024-03-04T10:05:57.337789+0000","last_fullsized":"2024-03-04T10:05:57.337789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357144+0000","last_change":"2024-03-04T10:05:57.353978+0000","last_active":"2024-03-04T10:05:57.357144+0000","last_peered":"2024-03-04T10:05:57.357144+0000","last_clean":"2024-03-04T10:05:57.357144+0000","last_became_active":"2024-03-04T10:05:57.347397+0000","last_became_peered":"2024-03-04T10:05:57.347397+0000","last_unstale":"2024-03-04T10:05:57.357144+0000","last_undegraded":"2024-03-04T10:05:57.357144+0000","last_fullsized":"2024-03-04T10:05:57.357144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:39.719928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342228+0000","last_change":"2024-03-04T10:05:57.342228+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342228+0000","last_undegraded":"2024-03-04T10:05:57.342228+0000","last_fullsized":"2024-03-04T10:05:57.342228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357253+0000","last_change":"2024-03-04T10:05:57.355370+0000","last_active":"2024-03-04T10:05:57.357253+0000","last_peered":"2024-03-04T10:05:57.357253+0000","last_clean":"2024-03-04T10:05:57.357253+0000","last_became_active":"2024-03-04T10:05:57.355114+0000","last_became_peered":"2024-03-04T10:05:57.355114+0000","last_unstale":"2024-03-04T10:05:57.357253+0000","last_undegraded":"2024-03-04T10:05:57.357253+0000","last_fullsized":"2024-03-04T10:05:57.357253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:08.382759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340369+0000","last_change":"2024-03-04T10:05:57.340369+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340369+0000","last_undegraded":"2024-03-04T10:05:57.340369+0000","last_fullsized":"2024-03-04T10:05:57.340369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355879+0000","last_change":"2024-03-04T10:05:57.354657+0000","last_active":"2024-03-04T10:05:57.355879+0000","last_peered":"2024-03-04T10:05:57.355879+0000","last_clean":"2024-03-04T10:05:57.355879+0000","last_became_active":"2024-03-04T10:05:57.354347+0000","last_became_peered":"2024-03-04T10:05:57.354347+0000","last_unstale":"2024-03-04T10:05:57.355879+0000","last_undegraded":"2024-03-04T10:05:57.355879+0000","last_fullsized":"2024-03-04T10:05:57.355879+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338070+0000","last_change":"2024-03-04T10:05:57.338070+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338070+0000","last_undegraded":"2024-03-04T10:05:57.338070+0000","last_fullsized":"2024-03-04T10:05:57.338070+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.349156+0000","last_change":"2024-03-04T10:05:57.346933+0000","last_active":"2024-03-04T10:05:57.349156+0000","last_peered":"2024-03-04T10:05:57.349156+0000","last_clean":"2024-03-04T10:05:57.349156+0000","last_became_active":"2024-03-04T10:05:57.346652+0000","last_became_peered":"2024-03-04T10:05:57.346652+0000","last_unstale":"2024-03-04T10:05:57.349156+0000","last_undegraded":"2024-03-04T10:05:57.349156+0000","last_fullsized":"2024-03-04T10:05:57.349156+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:07.296828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343697+0000","last_change":"2024-03-04T10:05:57.343697+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343697+0000","last_undegraded":"2024-03-04T10:05:57.343697+0000","last_fullsized":"2024-03-04T10:05:57.343697+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353593+0000","last_change":"2024-03-04T10:05:57.347278+0000","last_active":"2024-03-04T10:05:57.353593+0000","last_peered":"2024-03-04T10:05:57.353593+0000","last_clean":"2024-03-04T10:05:57.353593+0000","last_became_active":"2024-03-04T10:05:57.346983+0000","last_became_peered":"2024-03-04T10:05:57.346983+0000","last_unstale":"2024-03-04T10:05:57.353593+0000","last_undegraded":"2024-03-04T10:05:57.353593+0000","last_fullsized":"2024-03-04T10:05:57.353593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337987+0000","last_change":"2024-03-04T10:05:57.337987+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337987+0000","last_undegraded":"2024-03-04T10:05:57.337987+0000","last_fullsized":"2024-03-04T10:05:57.337987+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357529+0000","last_change":"2024-03-04T10:05:57.356865+0000","last_active":"2024-03-04T10:05:57.357529+0000","last_peered":"2024-03-04T10:05:57.357529+0000","last_clean":"2024-03-04T10:05:57.357529+0000","last_became_active":"2024-03-04T10:05:57.356604+0000","last_became_peered":"2024-03-04T10:05:57.356604+0000","last_unstale":"2024-03-04T10:05:57.357529+0000","last_undegraded":"2024-03-04T10:05:57.357529+0000","last_fullsized":"2024-03-04T10:05:57.357529+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338655+0000","last_change":"2024-03-04T10:05:57.338655+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338655+0000","last_undegraded":"2024-03-04T10:05:57.338655+0000","last_fullsized":"2024-03-04T10:05:57.338655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353131+0000","last_change":"2024-03-04T10:05:57.349592+0000","last_active":"2024-03-04T10:05:57.353131+0000","last_peered":"2024-03-04T10:05:57.353131+0000","last_clean":"2024-03-04T10:05:57.353131+0000","last_became_active":"2024-03-04T10:05:57.349329+0000","last_became_peered":"2024-03-04T10:05:57.349329+0000","last_unstale":"2024-03-04T10:05:57.353131+0000","last_undegraded":"2024-03-04T10:05:57.353131+0000","last_fullsized":"2024-03-04T10:05:57.353131+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:33:03.432962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337290+0000","last_change":"2024-03-04T10:05:57.337290+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337290+0000","last_undegraded":"2024-03-04T10:05:57.337290+0000","last_fullsized":"2024-03-04T10:05:57.337290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353786+0000","last_change":"2024-03-04T10:05:57.350127+0000","last_active":"2024-03-04T10:05:57.353786+0000","last_peered":"2024-03-04T10:05:57.353786+0000","last_clean":"2024-03-04T10:05:57.353786+0000","last_became_active":"2024-03-04T10:05:57.349795+0000","last_became_peered":"2024-03-04T10:05:57.349795+0000","last_unstale":"2024-03-04T10:05:57.353786+0000","last_undegraded":"2024-03-04T10:05:57.353786+0000","last_fullsized":"2024-03-04T10:05:57.353786+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:58.362487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339440+0000","last_change":"2024-03-04T10:05:57.339440+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339440+0000","last_undegraded":"2024-03-04T10:05:57.339440+0000","last_fullsized":"2024-03-04T10:05:57.339440+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354102+0000","last_change":"2024-03-04T10:05:57.352501+0000","last_active":"2024-03-04T10:05:57.354102+0000","last_peered":"2024-03-04T10:05:57.354102+0000","last_clean":"2024-03-04T10:05:57.354102+0000","last_became_active":"2024-03-04T10:05:57.352263+0000","last_became_peered":"2024-03-04T10:05:57.352263+0000","last_unstale":"2024-03-04T10:05:57.354102+0000","last_undegraded":"2024-03-04T10:05:57.354102+0000","last_fullsized":"2024-03-04T10:05:57.354102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:06.471395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340153+0000","last_change":"2024-03-04T10:05:57.340153+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340153+0000","last_undegraded":"2024-03-04T10:05:57.340153+0000","last_fullsized":"2024-03-04T10:05:57.340153+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.351766+0000","last_change":"2024-03-04T10:05:57.350188+0000","last_active":"2024-03-04T10:05:57.351766+0000","last_peered":"2024-03-04T10:05:57.351766+0000","last_clean":"2024-03-04T10:05:57.351766+0000","last_became_active":"2024-03-04T10:05:57.349869+0000","last_became_peered":"2024-03-04T10:05:57.349869+0000","last_unstale":"2024-03-04T10:05:57.351766+0000","last_undegraded":"2024-03-04T10:05:57.351766+0000","last_fullsized":"2024-03-04T10:05:57.351766+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341628+0000","last_change":"2024-03-04T10:05:57.341628+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341628+0000","last_undegraded":"2024-03-04T10:05:57.341628+0000","last_fullsized":"2024-03-04T10:05:57.341628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353669+0000","last_change":"2024-03-04T10:05:57.348152+0000","last_active":"2024-03-04T10:05:57.353669+0000","last_peered":"2024-03-04T10:05:57.353669+0000","last_clean":"2024-03-04T10:05:57.353669+0000","last_became_active":"2024-03-04T10:05:57.347666+0000","last_became_peered":"2024-03-04T10:05:57.347666+0000","last_unstale":"2024-03-04T10:05:57.353669+0000","last_undegraded":"2024-03-04T10:05:57.353669+0000","last_fullsized":"2024-03-04T10:05:57.353669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342010+0000","last_change":"2024-03-04T10:05:57.342010+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342010+0000","last_undegraded":"2024-03-04T10:05:57.342010+0000","last_fullsized":"2024-03-04T10:05:57.342010+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357245+0000","last_change":"2024-03-04T10:05:57.355286+0000","last_active":"2024-03-04T10:05:57.357245+0000","last_peered":"2024-03-04T10:05:57.357245+0000","last_clean":"2024-03-04T10:05:57.357245+0000","last_became_active":"2024-03-04T10:05:57.354851+0000","last_became_peered":"2024-03-04T10:05:57.354851+0000","last_unstale":"2024-03-04T10:05:57.357245+0000","last_undegraded":"2024-03-04T10:05:57.357245+0000","last_fullsized":"2024-03-04T10:05:57.357245+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:22.446636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340221+0000","last_change":"2024-03-04T10:05:57.340221+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340221+0000","last_undegraded":"2024-03-04T10:05:57.340221+0000","last_fullsized":"2024-03-04T10:05:57.340221+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353590+0000","last_change":"2024-03-04T10:05:57.350358+0000","last_active":"2024-03-04T10:05:57.353590+0000","last_peered":"2024-03-04T10:05:57.353590+0000","last_clean":"2024-03-04T10:05:57.353590+0000","last_became_active":"2024-03-04T10:05:57.350087+0000","last_became_peered":"2024-03-04T10:05:57.350087+0000","last_unstale":"2024-03-04T10:05:57.353590+0000","last_undegraded":"2024-03-04T10:05:57.353590+0000","last_fullsized":"2024-03-04T10:05:57.353590+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:42.909689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339252+0000","last_change":"2024-03-04T10:05:57.339252+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339252+0000","last_undegraded":"2024-03-04T10:05:57.339252+0000","last_fullsized":"2024-03-04T10:05:57.339252+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355278+0000","last_change":"2024-03-04T10:05:57.354106+0000","last_active":"2024-03-04T10:05:57.355278+0000","last_peered":"2024-03-04T10:05:57.355278+0000","last_clean":"2024-03-04T10:05:57.355278+0000","last_became_active":"2024-03-04T10:05:57.353627+0000","last_became_peered":"2024-03-04T10:05:57.353627+0000","last_unstale":"2024-03-04T10:05:57.355278+0000","last_undegraded":"2024-03-04T10:05:57.355278+0000","last_fullsized":"2024-03-04T10:05:57.355278+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:23.216649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339491+0000","last_change":"2024-03-04T10:05:57.339491+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339491+0000","last_undegraded":"2024-03-04T10:05:57.339491+0000","last_fullsized":"2024-03-04T10:05:57.339491+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.350556+0000","last_change":"2024-03-04T10:05:57.347633+0000","last_active":"2024-03-04T10:05:57.350556+0000","last_peered":"2024-03-04T10:05:57.350556+0000","last_clean":"2024-03-04T10:05:57.350556+0000","last_became_active":"2024-03-04T10:05:57.347321+0000","last_became_peered":"2024-03-04T10:05:57.347321+0000","last_unstale":"2024-03-04T10:05:57.350556+0000","last_undegraded":"2024-03-04T10:05:57.350556+0000","last_fullsized":"2024-03-04T10:05:57.350556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:27.622975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337788+0000","last_change":"2024-03-04T10:05:57.337788+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337788+0000","last_undegraded":"2024-03-04T10:05:57.337788+0000","last_fullsized":"2024-03-04T10:05:57.337788+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354379+0000","last_change":"2024-03-04T10:05:57.352257+0000","last_active":"2024-03-04T10:05:57.354379+0000","last_peered":"2024-03-04T10:05:57.354379+0000","last_clean":"2024-03-04T10:05:57.354379+0000","last_became_active":"2024-03-04T10:05:57.351978+0000","last_became_peered":"2024-03-04T10:05:57.351978+0000","last_unstale":"2024-03-04T10:05:57.354379+0000","last_undegraded":"2024-03-04T10:05:57.354379+0000","last_fullsized":"2024-03-04T10:05:57.354379+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:12.894467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340438+0000","last_change":"2024-03-04T10:05:57.340438+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340438+0000","last_undegraded":"2024-03-04T10:05:57.340438+0000","last_fullsized":"2024-03-04T10:05:57.340438+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353782+0000","last_change":"2024-03-04T10:05:57.347958+0000","last_active":"2024-03-04T10:05:57.353782+0000","last_peered":"2024-03-04T10:05:57.353782+0000","last_clean":"2024-03-04T10:05:57.353782+0000","last_became_active":"2024-03-04T10:05:57.347694+0000","last_became_peered":"2024-03-04T10:05:57.347694+0000","last_unstale":"2024-03-04T10:05:57.353782+0000","last_undegraded":"2024-03-04T10:05:57.353782+0000","last_fullsized":"2024-03-04T10:05:57.353782+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342652+0000","last_change":"2024-03-04T10:05:57.342652+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342652+0000","last_undegraded":"2024-03-04T10:05:57.342652+0000","last_fullsized":"2024-03-04T10:05:57.342652+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356093+0000","last_change":"2024-03-04T10:05:57.352564+0000","last_active":"2024-03-04T10:05:57.356093+0000","last_peered":"2024-03-04T10:05:57.356093+0000","last_clean":"2024-03-04T10:05:57.356093+0000","last_became_active":"2024-03-04T10:05:57.352324+0000","last_became_peered":"2024-03-04T10:05:57.352324+0000","last_unstale":"2024-03-04T10:05:57.356093+0000","last_undegraded":"2024-03-04T10:05:57.356093+0000","last_fullsized":"2024-03-04T10:05:57.356093+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342406+0000","last_change":"2024-03-04T10:05:57.342406+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342406+0000","last_undegraded":"2024-03-04T10:05:57.342406+0000","last_fullsized":"2024-03-04T10:05:57.342406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.345907+0000","last_change":"2024-03-04T10:05:57.341577+0000","last_active":"2024-03-04T10:05:57.345907+0000","last_peered":"2024-03-04T10:05:57.345907+0000","last_clean":"2024-03-04T10:05:57.345907+0000","last_became_active":"2024-03-04T10:05:57.341307+0000","last_became_peered":"2024-03-04T10:05:57.341307+0000","last_unstale":"2024-03-04T10:05:57.345907+0000","last_undegraded":"2024-03-04T10:05:57.345907+0000","last_fullsized":"2024-03-04T10:05:57.345907+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:23.216649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337281+0000","last_change":"2024-03-04T10:05:57.337281+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337281+0000","last_undegraded":"2024-03-04T10:05:57.337281+0000","last_fullsized":"2024-03-04T10:05:57.337281+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354473+0000","last_change":"2024-03-04T10:05:57.351387+0000","last_active":"2024-03-04T10:05:57.354473+0000","last_peered":"2024-03-04T10:05:57.354473+0000","last_clean":"2024-03-04T10:05:57.354473+0000","last_became_active":"2024-03-04T10:05:57.351012+0000","last_became_peered":"2024-03-04T10:05:57.351012+0000","last_unstale":"2024-03-04T10:05:57.354473+0000","last_undegraded":"2024-03-04T10:05:57.354473+0000","last_fullsized":"2024-03-04T10:05:57.354473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341292+0000","last_change":"2024-03-04T10:05:57.341292+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341292+0000","last_undegraded":"2024-03-04T10:05:57.341292+0000","last_fullsized":"2024-03-04T10:05:57.341292+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356535+0000","last_change":"2024-03-04T10:05:57.354681+0000","last_active":"2024-03-04T10:05:57.356535+0000","last_peered":"2024-03-04T10:05:57.356535+0000","last_clean":"2024-03-04T10:05:57.356535+0000","last_became_active":"2024-03-04T10:05:57.354415+0000","last_became_peered":"2024-03-04T10:05:57.354415+0000","last_unstale":"2024-03-04T10:05:57.356535+0000","last_undegraded":"2024-03-04T10:05:57.356535+0000","last_fullsized":"2024-03-04T10:05:57.356535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:53.178998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":231,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.338829+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:05:57.338829+0000","last_peered":"2024-03-04T10:05:57.338829+0000","last_clean":"2024-03-04T10:05:57.338829+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:05:57.338829+0000","last_undegraded":"2024-03-04T10:05:57.338829+0000","last_fullsized":"2024-03-04T10:05:57.338829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28176,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343664,"statfs":{"total":96636764160,"available":96607911936,"internally_reserved":0,"allocated":1175552,"data_stored":847263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":847263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:02.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:02.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.787+0000 7f07a10ab640 1 Processor -- start 2024-03-04T10:06:02.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.787+0000 7f07a10ab640 1 -- start start 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f07a10ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 0x7f079c0714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f07a10ab640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c15f020 con 0x7f079c0cf650 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f07a10ab640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0cd010 con 0x7f079c154a80 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f07a10ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0cff70 con 0x7f079c0cfae0 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f079a59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f079c0cf650 0x7f079c070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53312/0 (socket says 172.21.15.99:53312) 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f079a59c640 1 -- 172.21.15.99:0/3304020682 learned_addr learned my addr 172.21.15.99:0/3304020682 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691442251 0 0) 0x7f0790001110 con 0x7f079c0cf650 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784003520 con 0x7f079c0cf650 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f079b59e640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 0x7f079c0714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385434419 0 0) 0x7f0784003520 con 0x7f079c0cf650 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 msgr2=0x7f079c0714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 0x7f079c0714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 >> v1:172.21.15.162:6789/0 conn(0x7f079c154a80 legacy=0x7f079c15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c0cdaa0 con 0x7f079c0cf650 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f0790001840 con 0x7f079c0cf650 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 >> v1:172.21.15.99:6789/0 conn(0x7f079c0cf650 legacy=0x7f079c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07840039e0 0x7f0784003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0784004800 0x7f0784004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 0x7f07840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0cff70 con 0x7f07840039e0 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078400baa0 con 0x7f0784004800 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078400bca0 con 0x7f079c0cfae0 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.788+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07840039e0 0x7f0784003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079b59e640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 0x7f07840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079a59c640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0784004800 0x7f0784004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079ad9d640 1 -- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 msgr2=0x7f07840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c0cfae0 0x7f07840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079ad9d640 1 -- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0784004800 msgr2=0x7f0784004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0784004800 0x7f0784004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079ad9d640 1 -- 172.21.15.99:0/3304020682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784003520 con 0x7f07840039e0 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079a59c640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0784004800 0x7f0784004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.789+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07840039e0 0x7f0784003e00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0794001e20 tx=0x7f0784005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac65d843f333983 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794001a70 con 0x7f07840039e0 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794003b70 con 0x7f07840039e0 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f0799d9b640 1 -- 172.21.15.99:0/3304020682 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07940050d0 con 0x7f07840039e0 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f07a10ab640 1 -- 172.21.15.99:0/3304020682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07840039e0 msgr2=0x7f0784003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3304020682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07840039e0 0x7f0784003e00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0794001e20 tx=0x7f0784005830 comp rx=0 tx=0).stop 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f079b59e640 1 -- 172.21.15.99:0/3304020682 reap_dead start 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f07a10ab640 1 -- 172.21.15.99:0/3304020682 shutdown_connections 2024-03-04T10:06:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f07a10ab640 1 -- 172.21.15.99:0/3304020682 >> 172.21.15.99:0/3304020682 conn(0x7f079c06f5f0 msgr2=0x7f079c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:02.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f07a10ab640 1 -- 172.21.15.99:0/3304020682 shutdown_connections 2024-03-04T10:06:02.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.790+0000 7f07a10ab640 1 -- 172.21.15.99:0/3304020682 wait complete. 2024-03-04T10:06:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 Processor -- start 2024-03-04T10:06:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 -- start start 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07840039e0 0x7f079c0ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c1558b0 0x7f079c153d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 0x7f079c157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0ce340 con 0x7f079c1573d0 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0ce060 con 0x7f07840039e0 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f07a10ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0ce620 con 0x7f079c1558b0 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.792+0000 7f079b59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 0x7f079c157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 0x7f079c157850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35742/0 (socket says 172.21.15.99:35742) 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 -- 172.21.15.99:0/3495209154 learned_addr learned my addr 172.21.15.99:0/3495209154 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079a59c640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c1558b0 0x7f079c153d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07840039e0 0x7f079c0ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 -- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c1558b0 msgr2=0x7f079c153d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c1558b0 0x7f079c153d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 -- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07840039e0 msgr2=0x7f079c0ca4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07840039e0 0x7f079c0ca4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 -- 172.21.15.99:0/3495209154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c0cabd0 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079a59c640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c1558b0 0x7f079c153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07840039e0 0x7f079c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f079b59e640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 0x7f079c157850 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f079c06ea30 tx=0x7f078c011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c019020 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c004240 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c00f7a0 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c0caeb0 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.793+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c0b52a0 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.794+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c0cb3d0 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.795+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f078c01c020 con 0x7f079c1573d0 2024-03-04T10:06:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.795+0000 7f07837fe640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f078806b4d0 0x7f078806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.795+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f078c08a650 con 0x7f079c1573d0 2024-03-04T10:06:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.795+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f078806b4d0 0x7f078806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.796+0000 7f079ad9d640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f078806b4d0 0x7f078806d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0794034000 tx=0x7f0794009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.800+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f079c0cb3d0 con 0x7f079c1573d0 2024-03-04T10:06:02.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.965+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f079c06b8e0 con 0x7f078806b4d0 2024-03-04T10:06:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.972+0000 7f07837fe640 1 -- 172.21.15.99:0/3495209154 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246637 (secure 0 0 0) 0x7f079c06b8e0 con 0x7f078806b4d0 2024-03-04T10:06:02.977 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f078806b4d0 msgr2=0x7f078806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f078806b4d0 0x7f078806d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0794034000 tx=0x7f0794009030 comp rx=0 tx=0).stop 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 msgr2=0x7f079c157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 0x7f079c157850 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f079c06ea30 tx=0x7f078c011cf0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 shutdown_connections 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f078806b4d0 0x7f078806d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f079c1573d0 0x7f079c157850 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f079c1558b0 0x7f079c153d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 --2- 172.21.15.99:0/3495209154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07840039e0 0x7f079c0ca4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.974+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 >> 172.21.15.99:0/3495209154 conn(0x7f079c06f5f0 msgr2=0x7f079c0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.975+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 shutdown_connections 2024-03-04T10:06:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:02.975+0000 7f07a10ab640 1 -- 172.21.15.99:0/3495209154 wait complete. 2024-03-04T10:06:02.988 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-04T10:06:01.471627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221888,"kb_used_data":5704,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752832,"statfs":{"total":773094113280,"available":772866899968,"internally_reserved":0,"allocated":5840896,"data_stored":3235896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.132451"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342480+0000","last_change":"2024-03-04T10:05:57.342480+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342480+0000","last_undegraded":"2024-03-04T10:05:57.342480+0000","last_fullsized":"2024-03-04T10:05:57.342480+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338459+0000","last_change":"2024-03-04T10:05:57.338459+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338459+0000","last_undegraded":"2024-03-04T10:05:57.338459+0000","last_fullsized":"2024-03-04T10:05:57.338459+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341510+0000","last_change":"2024-03-04T10:05:57.341510+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341510+0000","last_undegraded":"2024-03-04T10:05:57.341510+0000","last_fullsized":"2024-03-04T10:05:57.341510+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339130+0000","last_change":"2024-03-04T10:05:57.339130+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339130+0000","last_undegraded":"2024-03-04T10:05:57.339130+0000","last_fullsized":"2024-03-04T10:05:57.339130+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338288+0000","last_change":"2024-03-04T10:05:57.338288+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338288+0000","last_undegraded":"2024-03-04T10:05:57.338288+0000","last_fullsized":"2024-03-04T10:05:57.338288+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343828+0000","last_change":"2024-03-04T10:05:57.343828+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343828+0000","last_undegraded":"2024-03-04T10:05:57.343828+0000","last_fullsized":"2024-03-04T10:05:57.343828+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340823+0000","last_change":"2024-03-04T10:05:57.340823+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340823+0000","last_undegraded":"2024-03-04T10:05:57.340823+0000","last_fullsized":"2024-03-04T10:05:57.340823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343184+0000","last_change":"2024-03-04T10:05:57.343184+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343184+0000","last_undegraded":"2024-03-04T10:05:57.343184+0000","last_fullsized":"2024-03-04T10:05:57.343184+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338669+0000","last_change":"2024-03-04T10:05:57.338669+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338669+0000","last_undegraded":"2024-03-04T10:05:57.338669+0000","last_fullsized":"2024-03-04T10:05:57.338669+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342054+0000","last_change":"2024-03-04T10:05:57.342054+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342054+0000","last_undegraded":"2024-03-04T10:05:57.342054+0000","last_fullsized":"2024-03-04T10:05:57.342054+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340974+0000","last_change":"2024-03-04T10:05:57.340974+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340974+0000","last_undegraded":"2024-03-04T10:05:57.340974+0000","last_fullsized":"2024-03-04T10:05:57.340974+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339260+0000","last_change":"2024-03-04T10:05:57.339260+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339260+0000","last_undegraded":"2024-03-04T10:05:57.339260+0000","last_fullsized":"2024-03-04T10:05:57.339260+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341698+0000","last_change":"2024-03-04T10:05:57.341698+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341698+0000","last_undegraded":"2024-03-04T10:05:57.341698+0000","last_fullsized":"2024-03-04T10:05:57.341698+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341083+0000","last_change":"2024-03-04T10:05:57.341083+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341083+0000","last_undegraded":"2024-03-04T10:05:57.341083+0000","last_fullsized":"2024-03-04T10:05:57.341083+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338937+0000","last_change":"2024-03-04T10:05:57.338937+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338937+0000","last_undegraded":"2024-03-04T10:05:57.338937+0000","last_fullsized":"2024-03-04T10:05:57.338937+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339257+0000","last_change":"2024-03-04T10:05:57.339257+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339257+0000","last_undegraded":"2024-03-04T10:05:57.339257+0000","last_fullsized":"2024-03-04T10:05:57.339257+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342715+0000","last_change":"2024-03-04T10:05:57.342715+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342715+0000","last_undegraded":"2024-03-04T10:05:57.342715+0000","last_fullsized":"2024-03-04T10:05:57.342715+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337971+0000","last_change":"2024-03-04T10:05:57.337971+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337971+0000","last_undegraded":"2024-03-04T10:05:57.337971+0000","last_fullsized":"2024-03-04T10:05:57.337971+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341139+0000","last_change":"2024-03-04T10:05:57.341139+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341139+0000","last_undegraded":"2024-03-04T10:05:57.341139+0000","last_fullsized":"2024-03-04T10:05:57.341139+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341520+0000","last_change":"2024-03-04T10:05:57.341520+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341520+0000","last_undegraded":"2024-03-04T10:05:57.341520+0000","last_fullsized":"2024-03-04T10:05:57.341520+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340984+0000","last_change":"2024-03-04T10:05:57.340984+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340984+0000","last_undegraded":"2024-03-04T10:05:57.340984+0000","last_fullsized":"2024-03-04T10:05:57.340984+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338695+0000","last_change":"2024-03-04T10:05:57.338695+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338695+0000","last_undegraded":"2024-03-04T10:05:57.338695+0000","last_fullsized":"2024-03-04T10:05:57.338695+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342052+0000","last_change":"2024-03-04T10:05:57.342052+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342052+0000","last_undegraded":"2024-03-04T10:05:57.342052+0000","last_fullsized":"2024-03-04T10:05:57.342052+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339928+0000","last_change":"2024-03-04T10:05:57.339928+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339928+0000","last_undegraded":"2024-03-04T10:05:57.339928+0000","last_fullsized":"2024-03-04T10:05:57.339928+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343322+0000","last_change":"2024-03-04T10:05:57.343322+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343322+0000","last_undegraded":"2024-03-04T10:05:57.343322+0000","last_fullsized":"2024-03-04T10:05:57.343322+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338555+0000","last_change":"2024-03-04T10:05:57.338555+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338555+0000","last_undegraded":"2024-03-04T10:05:57.338555+0000","last_fullsized":"2024-03-04T10:05:57.338555+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343243+0000","last_change":"2024-03-04T10:05:57.343243+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343243+0000","last_undegraded":"2024-03-04T10:05:57.343243+0000","last_fullsized":"2024-03-04T10:05:57.343243+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341030+0000","last_change":"2024-03-04T10:05:57.341030+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341030+0000","last_undegraded":"2024-03-04T10:05:57.341030+0000","last_fullsized":"2024-03-04T10:05:57.341030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342563+0000","last_change":"2024-03-04T10:05:57.342563+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342563+0000","last_undegraded":"2024-03-04T10:05:57.342563+0000","last_fullsized":"2024-03-04T10:05:57.342563+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339783+0000","last_change":"2024-03-04T10:05:57.339783+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339783+0000","last_undegraded":"2024-03-04T10:05:57.339783+0000","last_fullsized":"2024-03-04T10:05:57.339783+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340387+0000","last_change":"2024-03-04T10:05:57.340387+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340387+0000","last_undegraded":"2024-03-04T10:05:57.340387+0000","last_fullsized":"2024-03-04T10:05:57.340387+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342493+0000","last_change":"2024-03-04T10:05:57.342493+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342493+0000","last_undegraded":"2024-03-04T10:05:57.342493+0000","last_fullsized":"2024-03-04T10:05:57.342493+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338611+0000","last_change":"2024-03-04T10:05:57.338611+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338611+0000","last_undegraded":"2024-03-04T10:05:57.338611+0000","last_fullsized":"2024-03-04T10:05:57.338611+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.350566+0000","last_change":"2024-03-04T10:05:57.347237+0000","last_active":"2024-03-04T10:05:57.350566+0000","last_peered":"2024-03-04T10:05:57.350566+0000","last_clean":"2024-03-04T10:05:57.350566+0000","last_became_active":"2024-03-04T10:05:57.346945+0000","last_became_peered":"2024-03-04T10:05:57.346945+0000","last_unstale":"2024-03-04T10:05:57.350566+0000","last_undegraded":"2024-03-04T10:05:57.350566+0000","last_fullsized":"2024-03-04T10:05:57.350566+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:39:16.941315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339395+0000","last_change":"2024-03-04T10:05:57.339395+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339395+0000","last_undegraded":"2024-03-04T10:05:57.339395+0000","last_fullsized":"2024-03-04T10:05:57.339395+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356689+0000","last_change":"2024-03-04T10:05:57.354647+0000","last_active":"2024-03-04T10:05:57.356689+0000","last_peered":"2024-03-04T10:05:57.356689+0000","last_clean":"2024-03-04T10:05:57.356689+0000","last_became_active":"2024-03-04T10:05:57.354296+0000","last_became_peered":"2024-03-04T10:05:57.354296+0000","last_unstale":"2024-03-04T10:05:57.356689+0000","last_undegraded":"2024-03-04T10:05:57.356689+0000","last_fullsized":"2024-03-04T10:05:57.356689+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343540+0000","last_change":"2024-03-04T10:05:57.343540+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343540+0000","last_undegraded":"2024-03-04T10:05:57.343540+0000","last_fullsized":"2024-03-04T10:05:57.343540+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.349231+0000","last_change":"2024-03-04T10:05:57.348436+0000","last_active":"2024-03-04T10:05:57.349231+0000","last_peered":"2024-03-04T10:05:57.349231+0000","last_clean":"2024-03-04T10:05:57.349231+0000","last_became_active":"2024-03-04T10:05:57.348160+0000","last_became_peered":"2024-03-04T10:05:57.348160+0000","last_unstale":"2024-03-04T10:05:57.349231+0000","last_undegraded":"2024-03-04T10:05:57.349231+0000","last_fullsized":"2024-03-04T10:05:57.349231+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339087+0000","last_change":"2024-03-04T10:05:57.339087+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339087+0000","last_undegraded":"2024-03-04T10:05:57.339087+0000","last_fullsized":"2024-03-04T10:05:57.339087+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353404+0000","last_change":"2024-03-04T10:05:57.348949+0000","last_active":"2024-03-04T10:05:57.353404+0000","last_peered":"2024-03-04T10:05:57.353404+0000","last_clean":"2024-03-04T10:05:57.353404+0000","last_became_active":"2024-03-04T10:05:57.348244+0000","last_became_peered":"2024-03-04T10:05:57.348244+0000","last_unstale":"2024-03-04T10:05:57.353404+0000","last_undegraded":"2024-03-04T10:05:57.353404+0000","last_fullsized":"2024-03-04T10:05:57.353404+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338012+0000","last_change":"2024-03-04T10:05:57.338012+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338012+0000","last_undegraded":"2024-03-04T10:05:57.338012+0000","last_fullsized":"2024-03-04T10:05:57.338012+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355522+0000","last_change":"2024-03-04T10:05:57.353664+0000","last_active":"2024-03-04T10:05:57.355522+0000","last_peered":"2024-03-04T10:05:57.355522+0000","last_clean":"2024-03-04T10:05:57.355522+0000","last_became_active":"2024-03-04T10:05:57.353393+0000","last_became_peered":"2024-03-04T10:05:57.353393+0000","last_unstale":"2024-03-04T10:05:57.355522+0000","last_undegraded":"2024-03-04T10:05:57.355522+0000","last_fullsized":"2024-03-04T10:05:57.355522+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:27.622975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340503+0000","last_change":"2024-03-04T10:05:57.340503+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340503+0000","last_undegraded":"2024-03-04T10:05:57.340503+0000","last_fullsized":"2024-03-04T10:05:57.340503+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354018+0000","last_change":"2024-03-04T10:05:57.349177+0000","last_active":"2024-03-04T10:05:57.354018+0000","last_peered":"2024-03-04T10:05:57.354018+0000","last_clean":"2024-03-04T10:05:57.354018+0000","last_became_active":"2024-03-04T10:05:57.347881+0000","last_became_peered":"2024-03-04T10:05:57.347881+0000","last_unstale":"2024-03-04T10:05:57.354018+0000","last_undegraded":"2024-03-04T10:05:57.354018+0000","last_fullsized":"2024-03-04T10:05:57.354018+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337663+0000","last_change":"2024-03-04T10:05:57.337663+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337663+0000","last_undegraded":"2024-03-04T10:05:57.337663+0000","last_fullsized":"2024-03-04T10:05:57.337663+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353263+0000","last_change":"2024-03-04T10:05:57.348567+0000","last_active":"2024-03-04T10:05:57.353263+0000","last_peered":"2024-03-04T10:05:57.353263+0000","last_clean":"2024-03-04T10:05:57.353263+0000","last_became_active":"2024-03-04T10:05:57.348178+0000","last_became_peered":"2024-03-04T10:05:57.348178+0000","last_unstale":"2024-03-04T10:05:57.353263+0000","last_undegraded":"2024-03-04T10:05:57.353263+0000","last_fullsized":"2024-03-04T10:05:57.353263+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:01:12.651082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338926+0000","last_change":"2024-03-04T10:05:57.338926+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338926+0000","last_undegraded":"2024-03-04T10:05:57.338926+0000","last_fullsized":"2024-03-04T10:05:57.338926+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354413+0000","last_change":"2024-03-04T10:05:57.352999+0000","last_active":"2024-03-04T10:05:57.354413+0000","last_peered":"2024-03-04T10:05:57.354413+0000","last_clean":"2024-03-04T10:05:57.354413+0000","last_became_active":"2024-03-04T10:05:57.352754+0000","last_became_peered":"2024-03-04T10:05:57.352754+0000","last_unstale":"2024-03-04T10:05:57.354413+0000","last_undegraded":"2024-03-04T10:05:57.354413+0000","last_fullsized":"2024-03-04T10:05:57.354413+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:07.296828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.336768+0000","last_change":"2024-03-04T10:05:57.336768+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.336768+0000","last_undegraded":"2024-03-04T10:05:57.336768+0000","last_fullsized":"2024-03-04T10:05:57.336768+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353253+0000","last_change":"2024-03-04T10:05:57.349469+0000","last_active":"2024-03-04T10:05:57.353253+0000","last_peered":"2024-03-04T10:05:57.353253+0000","last_clean":"2024-03-04T10:05:57.353253+0000","last_became_active":"2024-03-04T10:05:57.349159+0000","last_became_peered":"2024-03-04T10:05:57.349159+0000","last_unstale":"2024-03-04T10:05:57.353253+0000","last_undegraded":"2024-03-04T10:05:57.353253+0000","last_fullsized":"2024-03-04T10:05:57.353253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:53.178998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339883+0000","last_change":"2024-03-04T10:05:57.339883+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339883+0000","last_undegraded":"2024-03-04T10:05:57.339883+0000","last_fullsized":"2024-03-04T10:05:57.339883+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355198+0000","last_change":"2024-03-04T10:05:57.354131+0000","last_active":"2024-03-04T10:05:57.355198+0000","last_peered":"2024-03-04T10:05:57.355198+0000","last_clean":"2024-03-04T10:05:57.355198+0000","last_became_active":"2024-03-04T10:05:57.353829+0000","last_became_peered":"2024-03-04T10:05:57.353829+0000","last_unstale":"2024-03-04T10:05:57.355198+0000","last_undegraded":"2024-03-04T10:05:57.355198+0000","last_fullsized":"2024-03-04T10:05:57.355198+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340720+0000","last_change":"2024-03-04T10:05:57.340720+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340720+0000","last_undegraded":"2024-03-04T10:05:57.340720+0000","last_fullsized":"2024-03-04T10:05:57.340720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353322+0000","last_change":"2024-03-04T10:05:57.349719+0000","last_active":"2024-03-04T10:05:57.353322+0000","last_peered":"2024-03-04T10:05:57.353322+0000","last_clean":"2024-03-04T10:05:57.353322+0000","last_became_active":"2024-03-04T10:05:57.349424+0000","last_became_peered":"2024-03-04T10:05:57.349424+0000","last_unstale":"2024-03-04T10:05:57.353322+0000","last_undegraded":"2024-03-04T10:05:57.353322+0000","last_fullsized":"2024-03-04T10:05:57.353322+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:22.446636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337789+0000","last_change":"2024-03-04T10:05:57.337789+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337789+0000","last_undegraded":"2024-03-04T10:05:57.337789+0000","last_fullsized":"2024-03-04T10:05:57.337789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357144+0000","last_change":"2024-03-04T10:05:57.353978+0000","last_active":"2024-03-04T10:05:57.357144+0000","last_peered":"2024-03-04T10:05:57.357144+0000","last_clean":"2024-03-04T10:05:57.357144+0000","last_became_active":"2024-03-04T10:05:57.347397+0000","last_became_peered":"2024-03-04T10:05:57.347397+0000","last_unstale":"2024-03-04T10:05:57.357144+0000","last_undegraded":"2024-03-04T10:05:57.357144+0000","last_fullsized":"2024-03-04T10:05:57.357144+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:39.719928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342228+0000","last_change":"2024-03-04T10:05:57.342228+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342228+0000","last_undegraded":"2024-03-04T10:05:57.342228+0000","last_fullsized":"2024-03-04T10:05:57.342228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357253+0000","last_change":"2024-03-04T10:05:57.355370+0000","last_active":"2024-03-04T10:05:57.357253+0000","last_peered":"2024-03-04T10:05:57.357253+0000","last_clean":"2024-03-04T10:05:57.357253+0000","last_became_active":"2024-03-04T10:05:57.355114+0000","last_became_peered":"2024-03-04T10:05:57.355114+0000","last_unstale":"2024-03-04T10:05:57.357253+0000","last_undegraded":"2024-03-04T10:05:57.357253+0000","last_fullsized":"2024-03-04T10:05:57.357253+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:08.382759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340369+0000","last_change":"2024-03-04T10:05:57.340369+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340369+0000","last_undegraded":"2024-03-04T10:05:57.340369+0000","last_fullsized":"2024-03-04T10:05:57.340369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355879+0000","last_change":"2024-03-04T10:05:57.354657+0000","last_active":"2024-03-04T10:05:57.355879+0000","last_peered":"2024-03-04T10:05:57.355879+0000","last_clean":"2024-03-04T10:05:57.355879+0000","last_became_active":"2024-03-04T10:05:57.354347+0000","last_became_peered":"2024-03-04T10:05:57.354347+0000","last_unstale":"2024-03-04T10:05:57.355879+0000","last_undegraded":"2024-03-04T10:05:57.355879+0000","last_fullsized":"2024-03-04T10:05:57.355879+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:33.770525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338070+0000","last_change":"2024-03-04T10:05:57.338070+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338070+0000","last_undegraded":"2024-03-04T10:05:57.338070+0000","last_fullsized":"2024-03-04T10:05:57.338070+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.349156+0000","last_change":"2024-03-04T10:05:57.346933+0000","last_active":"2024-03-04T10:05:57.349156+0000","last_peered":"2024-03-04T10:05:57.349156+0000","last_clean":"2024-03-04T10:05:57.349156+0000","last_became_active":"2024-03-04T10:05:57.346652+0000","last_became_peered":"2024-03-04T10:05:57.346652+0000","last_unstale":"2024-03-04T10:05:57.349156+0000","last_undegraded":"2024-03-04T10:05:57.349156+0000","last_fullsized":"2024-03-04T10:05:57.349156+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:28:07.296828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.343697+0000","last_change":"2024-03-04T10:05:57.343697+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.343697+0000","last_undegraded":"2024-03-04T10:05:57.343697+0000","last_fullsized":"2024-03-04T10:05:57.343697+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353593+0000","last_change":"2024-03-04T10:05:57.347278+0000","last_active":"2024-03-04T10:05:57.353593+0000","last_peered":"2024-03-04T10:05:57.353593+0000","last_clean":"2024-03-04T10:05:57.353593+0000","last_became_active":"2024-03-04T10:05:57.346983+0000","last_became_peered":"2024-03-04T10:05:57.346983+0000","last_unstale":"2024-03-04T10:05:57.353593+0000","last_undegraded":"2024-03-04T10:05:57.353593+0000","last_fullsized":"2024-03-04T10:05:57.353593+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337987+0000","last_change":"2024-03-04T10:05:57.337987+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337987+0000","last_undegraded":"2024-03-04T10:05:57.337987+0000","last_fullsized":"2024-03-04T10:05:57.337987+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357529+0000","last_change":"2024-03-04T10:05:57.356865+0000","last_active":"2024-03-04T10:05:57.357529+0000","last_peered":"2024-03-04T10:05:57.357529+0000","last_clean":"2024-03-04T10:05:57.357529+0000","last_became_active":"2024-03-04T10:05:57.356604+0000","last_became_peered":"2024-03-04T10:05:57.356604+0000","last_unstale":"2024-03-04T10:05:57.357529+0000","last_undegraded":"2024-03-04T10:05:57.357529+0000","last_fullsized":"2024-03-04T10:05:57.357529+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.338655+0000","last_change":"2024-03-04T10:05:57.338655+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.338655+0000","last_undegraded":"2024-03-04T10:05:57.338655+0000","last_fullsized":"2024-03-04T10:05:57.338655+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353131+0000","last_change":"2024-03-04T10:05:57.349592+0000","last_active":"2024-03-04T10:05:57.353131+0000","last_peered":"2024-03-04T10:05:57.353131+0000","last_clean":"2024-03-04T10:05:57.353131+0000","last_became_active":"2024-03-04T10:05:57.349329+0000","last_became_peered":"2024-03-04T10:05:57.349329+0000","last_unstale":"2024-03-04T10:05:57.353131+0000","last_undegraded":"2024-03-04T10:05:57.353131+0000","last_fullsized":"2024-03-04T10:05:57.353131+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:33:03.432962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337290+0000","last_change":"2024-03-04T10:05:57.337290+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337290+0000","last_undegraded":"2024-03-04T10:05:57.337290+0000","last_fullsized":"2024-03-04T10:05:57.337290+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353786+0000","last_change":"2024-03-04T10:05:57.350127+0000","last_active":"2024-03-04T10:05:57.353786+0000","last_peered":"2024-03-04T10:05:57.353786+0000","last_clean":"2024-03-04T10:05:57.353786+0000","last_became_active":"2024-03-04T10:05:57.349795+0000","last_became_peered":"2024-03-04T10:05:57.349795+0000","last_unstale":"2024-03-04T10:05:57.353786+0000","last_undegraded":"2024-03-04T10:05:57.353786+0000","last_fullsized":"2024-03-04T10:05:57.353786+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:20:58.362487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339440+0000","last_change":"2024-03-04T10:05:57.339440+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339440+0000","last_undegraded":"2024-03-04T10:05:57.339440+0000","last_fullsized":"2024-03-04T10:05:57.339440+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354102+0000","last_change":"2024-03-04T10:05:57.352501+0000","last_active":"2024-03-04T10:05:57.354102+0000","last_peered":"2024-03-04T10:05:57.354102+0000","last_clean":"2024-03-04T10:05:57.354102+0000","last_became_active":"2024-03-04T10:05:57.352263+0000","last_became_peered":"2024-03-04T10:05:57.352263+0000","last_unstale":"2024-03-04T10:05:57.354102+0000","last_undegraded":"2024-03-04T10:05:57.354102+0000","last_fullsized":"2024-03-04T10:05:57.354102+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:06.471395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340153+0000","last_change":"2024-03-04T10:05:57.340153+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340153+0000","last_undegraded":"2024-03-04T10:05:57.340153+0000","last_fullsized":"2024-03-04T10:05:57.340153+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.351766+0000","last_change":"2024-03-04T10:05:57.350188+0000","last_active":"2024-03-04T10:05:57.351766+0000","last_peered":"2024-03-04T10:05:57.351766+0000","last_clean":"2024-03-04T10:05:57.351766+0000","last_became_active":"2024-03-04T10:05:57.349869+0000","last_became_peered":"2024-03-04T10:05:57.349869+0000","last_unstale":"2024-03-04T10:05:57.351766+0000","last_undegraded":"2024-03-04T10:05:57.351766+0000","last_fullsized":"2024-03-04T10:05:57.351766+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341628+0000","last_change":"2024-03-04T10:05:57.341628+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341628+0000","last_undegraded":"2024-03-04T10:05:57.341628+0000","last_fullsized":"2024-03-04T10:05:57.341628+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353669+0000","last_change":"2024-03-04T10:05:57.348152+0000","last_active":"2024-03-04T10:05:57.353669+0000","last_peered":"2024-03-04T10:05:57.353669+0000","last_clean":"2024-03-04T10:05:57.353669+0000","last_became_active":"2024-03-04T10:05:57.347666+0000","last_became_peered":"2024-03-04T10:05:57.347666+0000","last_unstale":"2024-03-04T10:05:57.353669+0000","last_undegraded":"2024-03-04T10:05:57.353669+0000","last_fullsized":"2024-03-04T10:05:57.353669+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342010+0000","last_change":"2024-03-04T10:05:57.342010+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342010+0000","last_undegraded":"2024-03-04T10:05:57.342010+0000","last_fullsized":"2024-03-04T10:05:57.342010+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.357245+0000","last_change":"2024-03-04T10:05:57.355286+0000","last_active":"2024-03-04T10:05:57.357245+0000","last_peered":"2024-03-04T10:05:57.357245+0000","last_clean":"2024-03-04T10:05:57.357245+0000","last_became_active":"2024-03-04T10:05:57.354851+0000","last_became_peered":"2024-03-04T10:05:57.354851+0000","last_unstale":"2024-03-04T10:05:57.357245+0000","last_undegraded":"2024-03-04T10:05:57.357245+0000","last_fullsized":"2024-03-04T10:05:57.357245+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:22.446636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340221+0000","last_change":"2024-03-04T10:05:57.340221+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340221+0000","last_undegraded":"2024-03-04T10:05:57.340221+0000","last_fullsized":"2024-03-04T10:05:57.340221+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353590+0000","last_change":"2024-03-04T10:05:57.350358+0000","last_active":"2024-03-04T10:05:57.353590+0000","last_peered":"2024-03-04T10:05:57.353590+0000","last_clean":"2024-03-04T10:05:57.353590+0000","last_became_active":"2024-03-04T10:05:57.350087+0000","last_became_peered":"2024-03-04T10:05:57.350087+0000","last_unstale":"2024-03-04T10:05:57.353590+0000","last_undegraded":"2024-03-04T10:05:57.353590+0000","last_fullsized":"2024-03-04T10:05:57.353590+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:42.909689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339252+0000","last_change":"2024-03-04T10:05:57.339252+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339252+0000","last_undegraded":"2024-03-04T10:05:57.339252+0000","last_fullsized":"2024-03-04T10:05:57.339252+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.355278+0000","last_change":"2024-03-04T10:05:57.354106+0000","last_active":"2024-03-04T10:05:57.355278+0000","last_peered":"2024-03-04T10:05:57.355278+0000","last_clean":"2024-03-04T10:05:57.355278+0000","last_became_active":"2024-03-04T10:05:57.353627+0000","last_became_peered":"2024-03-04T10:05:57.353627+0000","last_unstale":"2024-03-04T10:05:57.355278+0000","last_undegraded":"2024-03-04T10:05:57.355278+0000","last_fullsized":"2024-03-04T10:05:57.355278+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:23.216649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.339491+0000","last_change":"2024-03-04T10:05:57.339491+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.339491+0000","last_undegraded":"2024-03-04T10:05:57.339491+0000","last_fullsized":"2024-03-04T10:05:57.339491+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.350556+0000","last_change":"2024-03-04T10:05:57.347633+0000","last_active":"2024-03-04T10:05:57.350556+0000","last_peered":"2024-03-04T10:05:57.350556+0000","last_clean":"2024-03-04T10:05:57.350556+0000","last_became_active":"2024-03-04T10:05:57.347321+0000","last_became_peered":"2024-03-04T10:05:57.347321+0000","last_unstale":"2024-03-04T10:05:57.350556+0000","last_undegraded":"2024-03-04T10:05:57.350556+0000","last_fullsized":"2024-03-04T10:05:57.350556+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:27.622975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337788+0000","last_change":"2024-03-04T10:05:57.337788+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337788+0000","last_undegraded":"2024-03-04T10:05:57.337788+0000","last_fullsized":"2024-03-04T10:05:57.337788+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354379+0000","last_change":"2024-03-04T10:05:57.352257+0000","last_active":"2024-03-04T10:05:57.354379+0000","last_peered":"2024-03-04T10:05:57.354379+0000","last_clean":"2024-03-04T10:05:57.354379+0000","last_became_active":"2024-03-04T10:05:57.351978+0000","last_became_peered":"2024-03-04T10:05:57.351978+0000","last_unstale":"2024-03-04T10:05:57.354379+0000","last_undegraded":"2024-03-04T10:05:57.354379+0000","last_fullsized":"2024-03-04T10:05:57.354379+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:12.894467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.340438+0000","last_change":"2024-03-04T10:05:57.340438+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.340438+0000","last_undegraded":"2024-03-04T10:05:57.340438+0000","last_fullsized":"2024-03-04T10:05:57.340438+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.353782+0000","last_change":"2024-03-04T10:05:57.347958+0000","last_active":"2024-03-04T10:05:57.353782+0000","last_peered":"2024-03-04T10:05:57.353782+0000","last_clean":"2024-03-04T10:05:57.353782+0000","last_became_active":"2024-03-04T10:05:57.347694+0000","last_became_peered":"2024-03-04T10:05:57.347694+0000","last_unstale":"2024-03-04T10:05:57.353782+0000","last_undegraded":"2024-03-04T10:05:57.353782+0000","last_fullsized":"2024-03-04T10:05:57.353782+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342652+0000","last_change":"2024-03-04T10:05:57.342652+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342652+0000","last_undegraded":"2024-03-04T10:05:57.342652+0000","last_fullsized":"2024-03-04T10:05:57.342652+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356093+0000","last_change":"2024-03-04T10:05:57.352564+0000","last_active":"2024-03-04T10:05:57.356093+0000","last_peered":"2024-03-04T10:05:57.356093+0000","last_clean":"2024-03-04T10:05:57.356093+0000","last_became_active":"2024-03-04T10:05:57.352324+0000","last_became_peered":"2024-03-04T10:05:57.352324+0000","last_unstale":"2024-03-04T10:05:57.356093+0000","last_undegraded":"2024-03-04T10:05:57.356093+0000","last_fullsized":"2024-03-04T10:05:57.356093+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.342406+0000","last_change":"2024-03-04T10:05:57.342406+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.342406+0000","last_undegraded":"2024-03-04T10:05:57.342406+0000","last_fullsized":"2024-03-04T10:05:57.342406+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.345907+0000","last_change":"2024-03-04T10:05:57.341577+0000","last_active":"2024-03-04T10:05:57.345907+0000","last_peered":"2024-03-04T10:05:57.345907+0000","last_clean":"2024-03-04T10:05:57.345907+0000","last_became_active":"2024-03-04T10:05:57.341307+0000","last_became_peered":"2024-03-04T10:05:57.341307+0000","last_unstale":"2024-03-04T10:05:57.345907+0000","last_undegraded":"2024-03-04T10:05:57.345907+0000","last_fullsized":"2024-03-04T10:05:57.345907+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:23.216649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.337281+0000","last_change":"2024-03-04T10:05:57.337281+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.337281+0000","last_undegraded":"2024-03-04T10:05:57.337281+0000","last_fullsized":"2024-03-04T10:05:57.337281+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.354473+0000","last_change":"2024-03-04T10:05:57.351387+0000","last_active":"2024-03-04T10:05:57.354473+0000","last_peered":"2024-03-04T10:05:57.354473+0000","last_clean":"2024-03-04T10:05:57.354473+0000","last_became_active":"2024-03-04T10:05:57.351012+0000","last_became_peered":"2024-03-04T10:05:57.351012+0000","last_unstale":"2024-03-04T10:05:57.354473+0000","last_undegraded":"2024-03-04T10:05:57.354473+0000","last_fullsized":"2024-03-04T10:05:57.354473+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":50,"state":"creating+peering","last_fresh":"2024-03-04T10:05:57.341292+0000","last_change":"2024-03-04T10:05:57.341292+0000","last_active":"2024-03-04T10:05:57.327086+0000","last_peered":"2024-03-04T10:05:57.327086+0000","last_clean":"2024-03-04T10:05:57.327086+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:05:57.341292+0000","last_undegraded":"2024-03-04T10:05:57.341292+0000","last_fullsized":"2024-03-04T10:05:57.341292+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.356535+0000","last_change":"2024-03-04T10:05:57.354681+0000","last_active":"2024-03-04T10:05:57.356535+0000","last_peered":"2024-03-04T10:05:57.356535+0000","last_clean":"2024-03-04T10:05:57.356535+0000","last_became_active":"2024-03-04T10:05:57.354415+0000","last_became_peered":"2024-03-04T10:05:57.354415+0000","last_unstale":"2024-03-04T10:05:57.356535+0000","last_undegraded":"2024-03-04T10:05:57.356535+0000","last_fullsized":"2024-03-04T10:05:57.356535+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:53.178998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":231,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-04T10:05:57.338829+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:05:57.338829+0000","last_peered":"2024-03-04T10:05:57.338829+0000","last_clean":"2024-03-04T10:05:57.338829+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:05:57.338829+0000","last_undegraded":"2024-03-04T10:05:57.338829+0000","last_fullsized":"2024-03-04T10:05:57.338829+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28176,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343664,"statfs":{"total":96636764160,"available":96607911936,"internally_reserved":0,"allocated":1175552,"data_stored":847263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":581632,"data_stored":256895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":847263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342480+0000', 'last_change': '2024-03-04T10:05:57.342480+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342480+0000', 'last_undegraded': '2024-03-04T10:05:57.342480+0000', 'last_fullsized': '2024-03-04T10:05:57.342480+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338459+0000', 'last_change': '2024-03-04T10:05:57.338459+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338459+0000', 'last_undegraded': '2024-03-04T10:05:57.338459+0000', 'last_fullsized': '2024-03-04T10:05:57.338459+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341510+0000', 'last_change': '2024-03-04T10:05:57.341510+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341510+0000', 'last_undegraded': '2024-03-04T10:05:57.341510+0000', 'last_fullsized': '2024-03-04T10:05:57.341510+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.992 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339130+0000', 'last_change': '2024-03-04T10:05:57.339130+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339130+0000', 'last_undegraded': '2024-03-04T10:05:57.339130+0000', 'last_fullsized': '2024-03-04T10:05:57.339130+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338288+0000', 'last_change': '2024-03-04T10:05:57.338288+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338288+0000', 'last_undegraded': '2024-03-04T10:05:57.338288+0000', 'last_fullsized': '2024-03-04T10:05:57.338288+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.343828+0000', 'last_change': '2024-03-04T10:05:57.343828+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.343828+0000', 'last_undegraded': '2024-03-04T10:05:57.343828+0000', 'last_fullsized': '2024-03-04T10:05:57.343828+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340823+0000', 'last_change': '2024-03-04T10:05:57.340823+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340823+0000', 'last_undegraded': '2024-03-04T10:05:57.340823+0000', 'last_fullsized': '2024-03-04T10:05:57.340823+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.343184+0000', 'last_change': '2024-03-04T10:05:57.343184+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.343184+0000', 'last_undegraded': '2024-03-04T10:05:57.343184+0000', 'last_fullsized': '2024-03-04T10:05:57.343184+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-03-04T10:06:02.993 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338669+0000', 'last_change': '2024-03-04T10:05:57.338669+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338669+0000', 'last_undegraded': '2024-03-04T10:05:57.338669+0000', 'last_fullsized': '2024-03-04T10:05:57.338669+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342054+0000', 'last_change': '2024-03-04T10:05:57.342054+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342054+0000', 'last_undegraded': '2024-03-04T10:05:57.342054+0000', 'last_fullsized': '2024-03-04T10:05:57.342054+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340974+0000', 'last_change': '2024-03-04T10:05:57.340974+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340974+0000', 'last_undegraded': '2024-03-04T10:05:57.340974+0000', 'last_fullsized': '2024-03-04T10:05:57.340974+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339260+0000', 'last_change': '2024-03-04T10:05:57.339260+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339260+0000', 'last_undegraded': '2024-03-04T10:05:57.339260+0000', 'last_fullsized': '2024-03-04T10:05:57.339260+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341698+0000', 'last_change': '2024-03-04T10:05:57.341698+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341698+0000', 'last_undegraded': '2024-03-04T10:05:57.341698+0000', 'last_fullsized': '2024-03-04T10:05:57.341698+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341083+0000', 'last_change': '2024-03-04T10:05:57.341083+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341083+0000', 'last_undegraded': '2024-03-04T10:05:57.341083+0000', 'last_fullsized': '2024-03-04T10:05:57.341083+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-03-04T10:06:02.994 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338937+0000', 'last_change': '2024-03-04T10:05:57.338937+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338937+0000', 'last_undegraded': '2024-03-04T10:05:57.338937+0000', 'last_fullsized': '2024-03-04T10:05:57.338937+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339257+0000', 'last_change': '2024-03-04T10:05:57.339257+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339257+0000', 'last_undegraded': '2024-03-04T10:05:57.339257+0000', 'last_fullsized': '2024-03-04T10:05:57.339257+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342715+0000', 'last_change': '2024-03-04T10:05:57.342715+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342715+0000', 'last_undegraded': '2024-03-04T10:05:57.342715+0000', 'last_fullsized': '2024-03-04T10:05:57.342715+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337971+0000', 'last_change': '2024-03-04T10:05:57.337971+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337971+0000', 'last_undegraded': '2024-03-04T10:05:57.337971+0000', 'last_fullsized': '2024-03-04T10:05:57.337971+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341139+0000', 'last_change': '2024-03-04T10:05:57.341139+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341139+0000', 'last_undegraded': '2024-03-04T10:05:57.341139+0000', 'last_fullsized': '2024-03-04T10:05:57.341139+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341520+0000', 'last_change': '2024-03-04T10:05:57.341520+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341520+0000', 'last_undegraded': '2024-03-04T10:05:57.341520+0000', 'last_fullsized': '2024-03-04T10:05:57.341520+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:06:02.995 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340984+0000', 'last_change': '2024-03-04T10:05:57.340984+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340984+0000', 'last_undegraded': '2024-03-04T10:05:57.340984+0000', 'last_fullsized': '2024-03-04T10:05:57.340984+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338695+0000', 'last_change': '2024-03-04T10:05:57.338695+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338695+0000', 'last_undegraded': '2024-03-04T10:05:57.338695+0000', 'last_fullsized': '2024-03-04T10:05:57.338695+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342052+0000', 'last_change': '2024-03-04T10:05:57.342052+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342052+0000', 'last_undegraded': '2024-03-04T10:05:57.342052+0000', 'last_fullsized': '2024-03-04T10:05:57.342052+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339928+0000', 'last_change': '2024-03-04T10:05:57.339928+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339928+0000', 'last_undegraded': '2024-03-04T10:05:57.339928+0000', 'last_fullsized': '2024-03-04T10:05:57.339928+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.343322+0000', 'last_change': '2024-03-04T10:05:57.343322+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.343322+0000', 'last_undegraded': '2024-03-04T10:05:57.343322+0000', 'last_fullsized': '2024-03-04T10:05:57.343322+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-03-04T10:06:02.996 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338555+0000', 'last_change': '2024-03-04T10:05:57.338555+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338555+0000', 'last_undegraded': '2024-03-04T10:05:57.338555+0000', 'last_fullsized': '2024-03-04T10:05:57.338555+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.343243+0000', 'last_change': '2024-03-04T10:05:57.343243+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.343243+0000', 'last_undegraded': '2024-03-04T10:05:57.343243+0000', 'last_fullsized': '2024-03-04T10:05:57.343243+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341030+0000', 'last_change': '2024-03-04T10:05:57.341030+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341030+0000', 'last_undegraded': '2024-03-04T10:05:57.341030+0000', 'last_fullsized': '2024-03-04T10:05:57.341030+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342563+0000', 'last_change': '2024-03-04T10:05:57.342563+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342563+0000', 'last_undegraded': '2024-03-04T10:05:57.342563+0000', 'last_fullsized': '2024-03-04T10:05:57.342563+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339783+0000', 'last_change': '2024-03-04T10:05:57.339783+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339783+0000', 'last_undegraded': '2024-03-04T10:05:57.339783+0000', 'last_fullsized': '2024-03-04T10:05:57.339783+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340387+0000', 'last_change': '2024-03-04T10:05:57.340387+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340387+0000', 'last_undegraded': '2024-03-04T10:05:57.340387+0000', 'last_fullsized': '2024-03-04T10:05:57.340387+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-03-04T10:06:02.997 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342493+0000', 'last_change': '2024-03-04T10:05:57.342493+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342493+0000', 'last_undegraded': '2024-03-04T10:05:57.342493+0000', 'last_fullsized': '2024-03-04T10:05:57.342493+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338611+0000', 'last_change': '2024-03-04T10:05:57.338611+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338611+0000', 'last_undegraded': '2024-03-04T10:05:57.338611+0000', 'last_fullsized': '2024-03-04T10:05:57.338611+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339395+0000', 'last_change': '2024-03-04T10:05:57.339395+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339395+0000', 'last_undegraded': '2024-03-04T10:05:57.339395+0000', 'last_fullsized': '2024-03-04T10:05:57.339395+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.343540+0000', 'last_change': '2024-03-04T10:05:57.343540+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.343540+0000', 'last_undegraded': '2024-03-04T10:05:57.343540+0000', 'last_fullsized': '2024-03-04T10:05:57.343540+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339087+0000', 'last_change': '2024-03-04T10:05:57.339087+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339087+0000', 'last_undegraded': '2024-03-04T10:05:57.339087+0000', 'last_fullsized': '2024-03-04T10:05:57.339087+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-03-04T10:06:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338012+0000', 'last_change': '2024-03-04T10:05:57.338012+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338012+0000', 'last_undegraded': '2024-03-04T10:05:57.338012+0000', 'last_fullsized': '2024-03-04T10:05:57.338012+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340503+0000', 'last_change': '2024-03-04T10:05:57.340503+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340503+0000', 'last_undegraded': '2024-03-04T10:05:57.340503+0000', 'last_fullsized': '2024-03-04T10:05:57.340503+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337663+0000', 'last_change': '2024-03-04T10:05:57.337663+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337663+0000', 'last_undegraded': '2024-03-04T10:05:57.337663+0000', 'last_fullsized': '2024-03-04T10:05:57.337663+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338926+0000', 'last_change': '2024-03-04T10:05:57.338926+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338926+0000', 'last_undegraded': '2024-03-04T10:05:57.338926+0000', 'last_fullsized': '2024-03-04T10:05:57.338926+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.336768+0000', 'last_change': '2024-03-04T10:05:57.336768+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.336768+0000', 'last_undegraded': '2024-03-04T10:05:57.336768+0000', 'last_fullsized': '2024-03-04T10:05:57.336768+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339883+0000', 'last_change': '2024-03-04T10:05:57.339883+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339883+0000', 'last_undegraded': '2024-03-04T10:05:57.339883+0000', 'last_fullsized': '2024-03-04T10:05:57.339883+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-04T10:06:02.999 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340720+0000', 'last_change': '2024-03-04T10:05:57.340720+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340720+0000', 'last_undegraded': '2024-03-04T10:05:57.340720+0000', 'last_fullsized': '2024-03-04T10:05:57.340720+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337789+0000', 'last_change': '2024-03-04T10:05:57.337789+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337789+0000', 'last_undegraded': '2024-03-04T10:05:57.337789+0000', 'last_fullsized': '2024-03-04T10:05:57.337789+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342228+0000', 'last_change': '2024-03-04T10:05:57.342228+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342228+0000', 'last_undegraded': '2024-03-04T10:05:57.342228+0000', 'last_fullsized': '2024-03-04T10:05:57.342228+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340369+0000', 'last_change': '2024-03-04T10:05:57.340369+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340369+0000', 'last_undegraded': '2024-03-04T10:05:57.340369+0000', 'last_fullsized': '2024-03-04T10:05:57.340369+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338070+0000', 'last_change': '2024-03-04T10:05:57.338070+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338070+0000', 'last_undegraded': '2024-03-04T10:05:57.338070+0000', 'last_fullsized': '2024-03-04T10:05:57.338070+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-04T10:06:03.000 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.343697+0000', 'last_change': '2024-03-04T10:05:57.343697+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.343697+0000', 'last_undegraded': '2024-03-04T10:05:57.343697+0000', 'last_fullsized': '2024-03-04T10:05:57.343697+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337987+0000', 'last_change': '2024-03-04T10:05:57.337987+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337987+0000', 'last_undegraded': '2024-03-04T10:05:57.337987+0000', 'last_fullsized': '2024-03-04T10:05:57.337987+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.338655+0000', 'last_change': '2024-03-04T10:05:57.338655+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.338655+0000', 'last_undegraded': '2024-03-04T10:05:57.338655+0000', 'last_fullsized': '2024-03-04T10:05:57.338655+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337290+0000', 'last_change': '2024-03-04T10:05:57.337290+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337290+0000', 'last_undegraded': '2024-03-04T10:05:57.337290+0000', 'last_fullsized': '2024-03-04T10:05:57.337290+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339440+0000', 'last_change': '2024-03-04T10:05:57.339440+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339440+0000', 'last_undegraded': '2024-03-04T10:05:57.339440+0000', 'last_fullsized': '2024-03-04T10:05:57.339440+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340153+0000', 'last_change': '2024-03-04T10:05:57.340153+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340153+0000', 'last_undegraded': '2024-03-04T10:05:57.340153+0000', 'last_fullsized': '2024-03-04T10:05:57.340153+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:03.001 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341628+0000', 'last_change': '2024-03-04T10:05:57.341628+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341628+0000', 'last_undegraded': '2024-03-04T10:05:57.341628+0000', 'last_fullsized': '2024-03-04T10:05:57.341628+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342010+0000', 'last_change': '2024-03-04T10:05:57.342010+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342010+0000', 'last_undegraded': '2024-03-04T10:05:57.342010+0000', 'last_fullsized': '2024-03-04T10:05:57.342010+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340221+0000', 'last_change': '2024-03-04T10:05:57.340221+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340221+0000', 'last_undegraded': '2024-03-04T10:05:57.340221+0000', 'last_fullsized': '2024-03-04T10:05:57.340221+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339252+0000', 'last_change': '2024-03-04T10:05:57.339252+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339252+0000', 'last_undegraded': '2024-03-04T10:05:57.339252+0000', 'last_fullsized': '2024-03-04T10:05:57.339252+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.339491+0000', 'last_change': '2024-03-04T10:05:57.339491+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.339491+0000', 'last_undegraded': '2024-03-04T10:05:57.339491+0000', 'last_fullsized': '2024-03-04T10:05:57.339491+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-04T10:06:03.002 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337788+0000', 'last_change': '2024-03-04T10:05:57.337788+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337788+0000', 'last_undegraded': '2024-03-04T10:05:57.337788+0000', 'last_fullsized': '2024-03-04T10:05:57.337788+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.340438+0000', 'last_change': '2024-03-04T10:05:57.340438+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.340438+0000', 'last_undegraded': '2024-03-04T10:05:57.340438+0000', 'last_fullsized': '2024-03-04T10:05:57.340438+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342652+0000', 'last_change': '2024-03-04T10:05:57.342652+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342652+0000', 'last_undegraded': '2024-03-04T10:05:57.342652+0000', 'last_fullsized': '2024-03-04T10:05:57.342652+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.342406+0000', 'last_change': '2024-03-04T10:05:57.342406+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.342406+0000', 'last_undegraded': '2024-03-04T10:05:57.342406+0000', 'last_fullsized': '2024-03-04T10:05:57.342406+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.337281+0000', 'last_change': '2024-03-04T10:05:57.337281+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.337281+0000', 'last_undegraded': '2024-03-04T10:05:57.337281+0000', 'last_fullsized': '2024-03-04T10:05:57.337281+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-03-04T10:06:03.003 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 50, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:05:57.341292+0000', 'last_change': '2024-03-04T10:05:57.341292+0000', 'last_active': '2024-03-04T10:05:57.327086+0000', 'last_peered': '2024-03-04T10:05:57.327086+0000', 'last_clean': '2024-03-04T10:05:57.327086+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:05:57.341292+0000', 'last_undegraded': '2024-03-04T10:05:57.341292+0000', 'last_fullsized': '2024-03-04T10:05:57.341292+0000', 'mapping_epoch': 50, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 50, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'last_clean_scrub_stamp': '2024-03-04T10:05:57.327086+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:06:03.004 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T10:06:03.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.125+0000 7f90924e8640 1 Processor -- start 2024-03-04T10:06:03.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.125+0000 7f90924e8640 1 -- start start 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f90924e8640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f908c0b0700 0x7f908c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f90924e8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0ccad0 con 0x7f908c15b960 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f90924e8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908c0cfab0 con 0x7f908c0b47c0 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f90924e8640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908c0cf860 con 0x7f908c0b0700 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f908bfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f908c15b960 0x7f908c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:53326/0 (socket says 172.21.15.99:53326) 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f908bfff640 1 -- 172.21.15.99:0/1887808369 learned_addr learned my addr 172.21.15.99:0/1887808369 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.126+0000 7f908b7fe640 1 --2- 172.21.15.99:0/1887808369 >> v2:172.21.15.162:3300/0 conn(0x7f908c0b0700 0x7f908c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.127+0000 7f908affd640 1 -- 172.21.15.99:0/1887808369 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508713582 0 0) 0x7f908c0ccad0 con 0x7f908c15b960 2024-03-04T10:06:03.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.127+0000 7f908affd640 1 -- 172.21.15.99:0/1887808369 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9074003520 con 0x7f908c15b960 2024-03-04T10:06:03.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.127+0000 7f908b7fe640 1 -- 172.21.15.99:0/1887808369 >> v1:172.21.15.162:6789/0 conn(0x7f908c0b47c0 legacy=0x7f908c0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.127+0000 7f908b7fe640 1 -- 172.21.15.99:0/1887808369 >> v1:172.21.15.99:6789/0 conn(0x7f908c15b960 legacy=0x7f908c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.127+0000 7f908affd640 1 -- 172.21.15.99:0/1887808369 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 152700749 0 0) 0x7f9074003520 con 0x7f908c15b960 2024-03-04T10:06:03.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.127+0000 7f908b7fe640 1 -- 172.21.15.99:0/1887808369 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0cfd90 con 0x7f908c0b0700 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f908b7fe640 1 --2- 172.21.15.99:0/1887808369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 0x7f908c0b0b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f90800097a0 tx=0x7f908002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36f3f472389bfc44 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f908affd640 1 -- 172.21.15.99:0/1887808369 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080037070 con 0x7f908c0b0700 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f908affd640 1 -- 172.21.15.99:0/1887808369 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9080030510 con 0x7f908c0b0700 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f908affd640 1 -- 172.21.15.99:0/1887808369 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c001670 con 0x7f908c0b0700 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 -- 172.21.15.99:0/1887808369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 msgr2=0x7f908c0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 --2- 172.21.15.99:0/1887808369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 0x7f908c0b0b90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f90800097a0 tx=0x7f908002b790 comp rx=0 tx=0).stop 2024-03-04T10:06:03.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 -- 172.21.15.99:0/1887808369 shutdown_connections 2024-03-04T10:06:03.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 --2- 172.21.15.99:0/1887808369 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 0x7f908c0b0b90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 -- 172.21.15.99:0/1887808369 >> 172.21.15.99:0/1887808369 conn(0x7f908c06f5f0 msgr2=0x7f908c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:03.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 -- 172.21.15.99:0/1887808369 shutdown_connections 2024-03-04T10:06:03.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.128+0000 7f90924e8640 1 -- 172.21.15.99:0/1887808369 wait complete. 2024-03-04T10:06:03.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.129+0000 7f90924e8640 1 Processor -- start 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 -- start start 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 0x7f908c0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 0x7f908c154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f908c15b960 0x7f908c070de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d0ca0 con 0x7f908c0b47c0 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d0100 con 0x7f908c0b0700 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f90924e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908c0d0e20 con 0x7f908c15b960 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f908b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 0x7f908c154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f908b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 0x7f908c154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35752/0 (socket says 172.21.15.99:35752) 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.130+0000 7f908b7fe640 1 -- 172.21.15.99:0/3508793316 learned_addr learned my addr 172.21.15.99:0/3508793316 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f908b7fe640 1 -- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f908c15b960 msgr2=0x7f908c070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f908b7fe640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f908c15b960 0x7f908c070de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f908b7fe640 1 -- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 msgr2=0x7f908c0c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f908b7fe640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 0x7f908c0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f908b7fe640 1 -- 172.21.15.99:0/3508793316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c0c6780 con 0x7f908c0b47c0 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f908b7fe640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 0x7f908c154e60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f908002b7d0 tx=0x7f90800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080042070 con 0x7f908c0b47c0 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f908003e870 con 0x7f908c0b47c0 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908c0c6a60 con 0x7f908c0b47c0 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9080037040 con 0x7f908c0b47c0 2024-03-04T10:06:03.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.131+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908c0c7020 con 0x7f908c0b47c0 2024-03-04T10:06:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.132+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908c074f60 con 0x7f908c0b47c0 2024-03-04T10:06:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.132+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9080045070 con 0x7f908c0b47c0 2024-03-04T10:06:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.133+0000 7f9088ff9640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f907806b4d0 0x7f907806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.133+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90800ac240 con 0x7f908c0b47c0 2024-03-04T10:06:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.133+0000 7f908bfff640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f907806b4d0 0x7f907806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.133+0000 7f908bfff640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f907806b4d0 0x7f907806d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f907c000c00 tx=0x7f907c0011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:03.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.137+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f908007b0e0 con 0x7f908c0b47c0 2024-03-04T10:06:03.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.376+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f908c0719b0 con 0x7f908c0b47c0 2024-03-04T10:06:03.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.376+0000 7f9088ff9640 1 -- 172.21.15.99:0/3508793316 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1035 (secure 0 0 0) 0x7f908007a9e0 con 0x7f908c0b47c0 2024-03-04T10:06:03.379 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:03.379 INFO:teuthology.orchestra.run.smithi099.stdout:{"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":168,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":52,"num_osds":8,"num_up_osds":8,"osd_up_since":1709546606,"num_in_osds":8,"osd_in_since":1709546597,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":227213312,"bytes_avail":772866899968,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":30,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5961}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-03-04T10:05:47.469111+0000","services":{}},"progress_events":{}} 2024-03-04T10:06:03.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.378+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f907806b4d0 msgr2=0x7f907806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.378+0000 7f90924e8640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f907806b4d0 0x7f907806d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f907c000c00 tx=0x7f907c0011a0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.378+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 msgr2=0x7f908c154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.378+0000 7f90924e8640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 0x7f908c154e60 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f908002b7d0 tx=0x7f90800046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 shutdown_connections 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f907806b4d0 0x7f907806d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f908c15b960 0x7f908c070de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f908c0b47c0 0x7f908c154e60 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 --2- 172.21.15.99:0/3508793316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f908c0b0700 0x7f908c0c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 >> 172.21.15.99:0/3508793316 conn(0x7f908c06f5f0 msgr2=0x7f908c0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 shutdown_connections 2024-03-04T10:06:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.379+0000 7f90924e8640 1 -- 172.21.15.99:0/3508793316 wait complete. 2024-03-04T10:06:03.392 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-04T10:06:03.392 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.489+0000 7f6cbc6dc640 1 Processor -- start 2024-03-04T10:06:03.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.489+0000 7f6cbc6dc640 1 -- start start 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cbc6dc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6cb40b39c0 0x7f6cb4161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cbc6dc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6cb406de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cbc6dc640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb406b500 con 0x7f6cb4158410 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cbc6dc640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb406b7e0 con 0x7f6cb40b39c0 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cbc6dc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb406bac0 con 0x7f6cb4161ca0 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cba478640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6cb4158410 0x7f6cb406acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55940/0 (socket says 172.21.15.99:55940) 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cba478640 1 -- 172.21.15.99:0/390536221 learned_addr learned my addr 172.21.15.99:0/390536221 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/390536221 >> v2:172.21.15.162:3300/0 conn(0x7f6cb40b39c0 0x7f6cb4161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cbac79640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6cb406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636588585 0 0) 0x7f6cb406b500 con 0x7f6cb4158410 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca0003520 con 0x7f6cb4158410 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369768361 0 0) 0x7f6ca0003520 con 0x7f6cb4158410 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 msgr2=0x7f6cb406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6cb406de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 >> v2:172.21.15.162:3300/0 conn(0x7f6cb40b39c0 msgr2=0x7f6cb4161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 --2- 172.21.15.99:0/390536221 >> v2:172.21.15.162:3300/0 conn(0x7f6cb40b39c0 0x7f6cb4161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb406bda0 con 0x7f6cb4158410 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6cb0001670 con 0x7f6cb4158410 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 >> v1:172.21.15.99:6789/0 conn(0x7f6cb4158410 legacy=0x7f6cb406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ca00039b0 0x7f6ca0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ca0004800 0x7f6ca0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6ca00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb406b7e0 con 0x7f6ca0004800 2024-03-04T10:06:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb406bac0 con 0x7f6ca00039b0 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.490+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca000b9c0 con 0x7f6cb4161ca0 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ca0004800 0x7f6ca0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cba478640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ca00039b0 0x7f6ca0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cbac79640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6ca00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 -- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 msgr2=0x7f6ca00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6ca00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 -- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ca00039b0 msgr2=0x7f6ca0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ca00039b0 0x7f6ca0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 -- 172.21.15.99:0/390536221 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0003520 con 0x7f6ca0004800 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cbac79640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6cb4161ca0 0x7f6ca00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.491+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ca0004800 0x7f6ca0004c80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009ac0 tx=0x7f6ca0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa48d6fe7f295f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4039070 con 0x7f6ca0004800 2024-03-04T10:06:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca402c710 con 0x7f6ca0004800 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cb9476640 1 -- 172.21.15.99:0/390536221 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4009690 con 0x7f6ca0004800 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/390536221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ca0004800 msgr2=0x7f6ca0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/390536221 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6ca0004800 0x7f6ca0004c80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009ac0 tx=0x7f6ca0005830 comp rx=0 tx=0).stop 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbac79640 1 -- 172.21.15.99:0/390536221 reap_dead start 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/390536221 shutdown_connections 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/390536221 >> 172.21.15.99:0/390536221 conn(0x7f6cb40cfef0 msgr2=0x7f6cb40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/390536221 shutdown_connections 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.492+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/390536221 wait complete. 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.493+0000 7f6cbc6dc640 1 Processor -- start 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.493+0000 7f6cbc6dc640 1 -- start start 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbc6dc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ca0004800 0x7f6cb415c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbc6dc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 0x7f6cb40cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbc6dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb41484f0 0x7f6cb4148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbc6dc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb406c5a0 con 0x7f6cb41f80b0 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbc6dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb40b3e50 con 0x7f6cb41484f0 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbc6dc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb406de50 con 0x7f6ca0004800 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 0x7f6cb40cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 0x7f6cb40cec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57286/0 (socket says 172.21.15.99:57286) 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 -- 172.21.15.99:0/3310308425 learned_addr learned my addr 172.21.15.99:0/3310308425 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cba478640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ca0004800 0x7f6cb415c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cbac79640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb41484f0 0x7f6cb4148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 -- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ca0004800 msgr2=0x7f6cb415c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ca0004800 0x7f6cb415c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 -- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb41484f0 msgr2=0x7f6cb4148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb41484f0 0x7f6cb4148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.494+0000 7f6cb9c77640 1 -- 172.21.15.99:0/3310308425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb41f8510 con 0x7f6cb41f80b0 2024-03-04T10:06:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.495+0000 7f6cb9c77640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 0x7f6cb40cec30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4000c00 tx=0x7f6ca4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.495+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4039050 con 0x7f6cb41f80b0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.495+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca4045070 con 0x7f6cb41f80b0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.495+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb415cbd0 con 0x7f6cb41f80b0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.495+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4003a50 con 0x7f6cb41f80b0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.495+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb415d190 con 0x7f6cb41f80b0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.496+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb40b1190 con 0x7f6cb41f80b0 2024-03-04T10:06:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.496+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6ca4003bf0 con 0x7f6cb41f80b0 2024-03-04T10:06:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.497+0000 7f6caaffd640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6c9806b4d0 0x7f6c9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.497+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6ca40abf10 con 0x7f6cb41f80b0 2024-03-04T10:06:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.497+0000 7f6cba478640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6c9806b4d0 0x7f6c9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:03.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.497+0000 7f6cba478640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6c9806b4d0 0x7f6c9806d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0001eb0 tx=0x7f6cb0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:03.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.501+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6cb40b1190 con 0x7f6cb41f80b0 2024-03-04T10:06:03.621 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:06:03.618+0000 7ff208ac5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:03.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.670+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6cb4066e10 con 0x7f6c9806b4d0 2024-03-04T10:06:03.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.676+0000 7f6caaffd640 1 -- 172.21.15.99:0/3310308425 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251957 (secure 0 0 0) 0x7f6cb4066e10 con 0x7f6c9806b4d0 2024-03-04T10:06:03.680 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:03.682 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:03.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6c9806b4d0 msgr2=0x7f6c9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6c9806b4d0 0x7f6c9806d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0001eb0 tx=0x7f6cb0009040 comp rx=0 tx=0).stop 2024-03-04T10:06:03.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 msgr2=0x7f6cb40cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 0x7f6cb40cec30 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4000c00 tx=0x7f6ca4004730 comp rx=0 tx=0).stop 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 shutdown_connections 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6c9806b4d0 0x7f6c9806d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6cb41484f0 0x7f6cb4148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6cb41f80b0 0x7f6cb40cec30 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 --2- 172.21.15.99:0/3310308425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6ca0004800 0x7f6cb415c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 >> 172.21.15.99:0/3310308425 conn(0x7f6cb40cfef0 msgr2=0x7f6cb40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 shutdown_connections 2024-03-04T10:06:03.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:03.679+0000 7f6cbc6dc640 1 -- 172.21.15.99:0/3310308425 wait complete. 2024-03-04T10:06:03.692 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-03-04T10:06:03.472114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222252,"kb_used_data":6056,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752468,"statfs":{"total":773094113280,"available":772866527232,"internally_reserved":0,"allocated":6201344,"data_stored":3404924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.138867"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347899+0000","last_change":"2024-03-04T10:05:58.358687+0000","last_active":"2024-03-04T10:05:59.347899+0000","last_peered":"2024-03-04T10:05:59.347899+0000","last_clean":"2024-03-04T10:05:59.347899+0000","last_became_active":"2024-03-04T10:05:58.358429+0000","last_became_peered":"2024-03-04T10:05:58.358429+0000","last_unstale":"2024-03-04T10:05:59.347899+0000","last_undegraded":"2024-03-04T10:05:59.347899+0000","last_fullsized":"2024-03-04T10:05:59.347899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:54.646485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350190+0000","last_change":"2024-03-04T10:05:58.351677+0000","last_active":"2024-03-04T10:05:59.350190+0000","last_peered":"2024-03-04T10:05:59.350190+0000","last_clean":"2024-03-04T10:05:59.350190+0000","last_became_active":"2024-03-04T10:05:58.351392+0000","last_became_peered":"2024-03-04T10:05:58.351392+0000","last_unstale":"2024-03-04T10:05:59.350190+0000","last_undegraded":"2024-03-04T10:05:59.350190+0000","last_fullsized":"2024-03-04T10:05:59.350190+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:01.921655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349965+0000","last_change":"2024-03-04T10:05:58.351831+0000","last_active":"2024-03-04T10:05:59.349965+0000","last_peered":"2024-03-04T10:05:59.349965+0000","last_clean":"2024-03-04T10:05:59.349965+0000","last_became_active":"2024-03-04T10:05:58.351295+0000","last_became_peered":"2024-03-04T10:05:58.351295+0000","last_unstale":"2024-03-04T10:05:59.349965+0000","last_undegraded":"2024-03-04T10:05:59.349965+0000","last_fullsized":"2024-03-04T10:05:59.349965+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:13.685578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350039+0000","last_change":"2024-03-04T10:05:58.346287+0000","last_active":"2024-03-04T10:05:59.350039+0000","last_peered":"2024-03-04T10:05:59.350039+0000","last_clean":"2024-03-04T10:05:59.350039+0000","last_became_active":"2024-03-04T10:05:58.346003+0000","last_became_peered":"2024-03-04T10:05:58.346003+0000","last_unstale":"2024-03-04T10:05:59.350039+0000","last_undegraded":"2024-03-04T10:05:59.350039+0000","last_fullsized":"2024-03-04T10:05:59.350039+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:56.686869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349113+0000","last_change":"2024-03-04T10:05:58.347170+0000","last_active":"2024-03-04T10:05:59.349113+0000","last_peered":"2024-03-04T10:05:59.349113+0000","last_clean":"2024-03-04T10:05:59.349113+0000","last_became_active":"2024-03-04T10:05:58.346819+0000","last_became_peered":"2024-03-04T10:05:58.346819+0000","last_unstale":"2024-03-04T10:05:59.349113+0000","last_undegraded":"2024-03-04T10:05:59.349113+0000","last_fullsized":"2024-03-04T10:05:59.349113+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:23.451619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347594+0000","last_change":"2024-03-04T10:05:58.354338+0000","last_active":"2024-03-04T10:05:59.347594+0000","last_peered":"2024-03-04T10:05:59.347594+0000","last_clean":"2024-03-04T10:05:59.347594+0000","last_became_active":"2024-03-04T10:05:58.353681+0000","last_became_peered":"2024-03-04T10:05:58.353681+0000","last_unstale":"2024-03-04T10:05:59.347594+0000","last_undegraded":"2024-03-04T10:05:59.347594+0000","last_fullsized":"2024-03-04T10:05:59.347594+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:42.586936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349547+0000","last_change":"2024-03-04T10:05:58.356631+0000","last_active":"2024-03-04T10:05:59.349547+0000","last_peered":"2024-03-04T10:05:59.349547+0000","last_clean":"2024-03-04T10:05:59.349547+0000","last_became_active":"2024-03-04T10:05:58.354506+0000","last_became_peered":"2024-03-04T10:05:58.354506+0000","last_unstale":"2024-03-04T10:05:59.349547+0000","last_undegraded":"2024-03-04T10:05:59.349547+0000","last_fullsized":"2024-03-04T10:05:59.349547+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:56.231659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542002+0000","last_change":"2024-03-04T10:05:58.358694+0000","last_active":"2024-03-04T10:05:59.542002+0000","last_peered":"2024-03-04T10:05:59.542002+0000","last_clean":"2024-03-04T10:05:59.542002+0000","last_became_active":"2024-03-04T10:05:58.358428+0000","last_became_peered":"2024-03-04T10:05:58.358428+0000","last_unstale":"2024-03-04T10:05:59.542002+0000","last_undegraded":"2024-03-04T10:05:59.542002+0000","last_fullsized":"2024-03-04T10:05:59.542002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:36.599089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542405+0000","last_change":"2024-03-04T10:05:58.360154+0000","last_active":"2024-03-04T10:05:59.542405+0000","last_peered":"2024-03-04T10:05:59.542405+0000","last_clean":"2024-03-04T10:05:59.542405+0000","last_became_active":"2024-03-04T10:05:58.359856+0000","last_became_peered":"2024-03-04T10:05:58.359856+0000","last_unstale":"2024-03-04T10:05:59.542405+0000","last_undegraded":"2024-03-04T10:05:59.542405+0000","last_fullsized":"2024-03-04T10:05:59.542405+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:25:39.593206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347899+0000","last_change":"2024-03-04T10:05:58.351885+0000","last_active":"2024-03-04T10:05:59.347899+0000","last_peered":"2024-03-04T10:05:59.347899+0000","last_clean":"2024-03-04T10:05:59.347899+0000","last_became_active":"2024-03-04T10:05:58.351627+0000","last_became_peered":"2024-03-04T10:05:58.351627+0000","last_unstale":"2024-03-04T10:05:59.347899+0000","last_undegraded":"2024-03-04T10:05:59.347899+0000","last_fullsized":"2024-03-04T10:05:59.347899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:55:38.424394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678432+0000","last_change":"2024-03-04T10:05:58.348999+0000","last_active":"2024-03-04T10:05:59.678432+0000","last_peered":"2024-03-04T10:05:59.678432+0000","last_clean":"2024-03-04T10:05:59.678432+0000","last_became_active":"2024-03-04T10:05:58.348768+0000","last_became_peered":"2024-03-04T10:05:58.348768+0000","last_unstale":"2024-03-04T10:05:59.678432+0000","last_undegraded":"2024-03-04T10:05:59.678432+0000","last_fullsized":"2024-03-04T10:05:59.678432+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:36:47.382317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542189+0000","last_change":"2024-03-04T10:05:58.352971+0000","last_active":"2024-03-04T10:05:59.542189+0000","last_peered":"2024-03-04T10:05:59.542189+0000","last_clean":"2024-03-04T10:05:59.542189+0000","last_became_active":"2024-03-04T10:05:58.352225+0000","last_became_peered":"2024-03-04T10:05:58.352225+0000","last_unstale":"2024-03-04T10:05:59.542189+0000","last_undegraded":"2024-03-04T10:05:59.542189+0000","last_fullsized":"2024-03-04T10:05:59.542189+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:27.532551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347604+0000","last_change":"2024-03-04T10:05:58.355130+0000","last_active":"2024-03-04T10:05:59.347604+0000","last_peered":"2024-03-04T10:05:59.347604+0000","last_clean":"2024-03-04T10:05:59.347604+0000","last_became_active":"2024-03-04T10:05:58.354862+0000","last_became_peered":"2024-03-04T10:05:58.354862+0000","last_unstale":"2024-03-04T10:05:59.347604+0000","last_undegraded":"2024-03-04T10:05:59.347604+0000","last_fullsized":"2024-03-04T10:05:59.347604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:28.642918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349735+0000","last_change":"2024-03-04T10:05:58.352249+0000","last_active":"2024-03-04T10:05:59.349735+0000","last_peered":"2024-03-04T10:05:59.349735+0000","last_clean":"2024-03-04T10:05:59.349735+0000","last_became_active":"2024-03-04T10:05:58.351883+0000","last_became_peered":"2024-03-04T10:05:58.351883+0000","last_unstale":"2024-03-04T10:05:59.349735+0000","last_undegraded":"2024-03-04T10:05:59.349735+0000","last_fullsized":"2024-03-04T10:05:59.349735+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:27.532551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349767+0000","last_change":"2024-03-04T10:05:58.353341+0000","last_active":"2024-03-04T10:05:59.349767+0000","last_peered":"2024-03-04T10:05:59.349767+0000","last_clean":"2024-03-04T10:05:59.349767+0000","last_became_active":"2024-03-04T10:05:58.353076+0000","last_became_peered":"2024-03-04T10:05:58.353076+0000","last_unstale":"2024-03-04T10:05:59.349767+0000","last_undegraded":"2024-03-04T10:05:59.349767+0000","last_fullsized":"2024-03-04T10:05:59.349767+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:55:38.424394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347474+0000","last_change":"2024-03-04T10:05:58.349430+0000","last_active":"2024-03-04T10:05:59.347474+0000","last_peered":"2024-03-04T10:05:59.347474+0000","last_clean":"2024-03-04T10:05:59.347474+0000","last_became_active":"2024-03-04T10:05:58.349179+0000","last_became_peered":"2024-03-04T10:05:58.349179+0000","last_unstale":"2024-03-04T10:05:59.347474+0000","last_undegraded":"2024-03-04T10:05:59.347474+0000","last_fullsized":"2024-03-04T10:05:59.347474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:18.755017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350265+0000","last_change":"2024-03-04T10:05:58.356090+0000","last_active":"2024-03-04T10:05:59.350265+0000","last_peered":"2024-03-04T10:05:59.350265+0000","last_clean":"2024-03-04T10:05:59.350265+0000","last_became_active":"2024-03-04T10:05:58.355492+0000","last_became_peered":"2024-03-04T10:05:58.355492+0000","last_unstale":"2024-03-04T10:05:59.350265+0000","last_undegraded":"2024-03-04T10:05:59.350265+0000","last_fullsized":"2024-03-04T10:05:59.350265+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:20.373338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350305+0000","last_change":"2024-03-04T10:05:58.355628+0000","last_active":"2024-03-04T10:05:59.350305+0000","last_peered":"2024-03-04T10:05:59.350305+0000","last_clean":"2024-03-04T10:05:59.350305+0000","last_became_active":"2024-03-04T10:05:58.355365+0000","last_became_peered":"2024-03-04T10:05:58.355365+0000","last_unstale":"2024-03-04T10:05:59.350305+0000","last_undegraded":"2024-03-04T10:05:59.350305+0000","last_fullsized":"2024-03-04T10:05:59.350305+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:34.135270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349954+0000","last_change":"2024-03-04T10:05:58.367494+0000","last_active":"2024-03-04T10:05:59.349954+0000","last_peered":"2024-03-04T10:05:59.349954+0000","last_clean":"2024-03-04T10:05:59.349954+0000","last_became_active":"2024-03-04T10:05:58.367205+0000","last_became_peered":"2024-03-04T10:05:58.367205+0000","last_unstale":"2024-03-04T10:05:59.349954+0000","last_undegraded":"2024-03-04T10:05:59.349954+0000","last_fullsized":"2024-03-04T10:05:59.349954+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:25:39.593206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678349+0000","last_change":"2024-03-04T10:05:58.356714+0000","last_active":"2024-03-04T10:05:59.678349+0000","last_peered":"2024-03-04T10:05:59.678349+0000","last_clean":"2024-03-04T10:05:59.678349+0000","last_became_active":"2024-03-04T10:05:58.356432+0000","last_became_peered":"2024-03-04T10:05:58.356432+0000","last_unstale":"2024-03-04T10:05:59.678349+0000","last_undegraded":"2024-03-04T10:05:59.678349+0000","last_fullsized":"2024-03-04T10:05:59.678349+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:04.417535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678300+0000","last_change":"2024-03-04T10:05:58.356136+0000","last_active":"2024-03-04T10:05:59.678300+0000","last_peered":"2024-03-04T10:05:59.678300+0000","last_clean":"2024-03-04T10:05:59.678300+0000","last_became_active":"2024-03-04T10:05:58.354709+0000","last_became_peered":"2024-03-04T10:05:58.354709+0000","last_unstale":"2024-03-04T10:05:59.678300+0000","last_undegraded":"2024-03-04T10:05:59.678300+0000","last_fullsized":"2024-03-04T10:05:59.678300+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:54.137325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.541926+0000","last_change":"2024-03-04T10:05:58.358660+0000","last_active":"2024-03-04T10:05:59.541926+0000","last_peered":"2024-03-04T10:05:59.541926+0000","last_clean":"2024-03-04T10:05:59.541926+0000","last_became_active":"2024-03-04T10:05:58.358391+0000","last_became_peered":"2024-03-04T10:05:58.358391+0000","last_unstale":"2024-03-04T10:05:59.541926+0000","last_undegraded":"2024-03-04T10:05:59.541926+0000","last_fullsized":"2024-03-04T10:05:59.541926+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:25:45.939247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678195+0000","last_change":"2024-03-04T10:05:58.347253+0000","last_active":"2024-03-04T10:05:59.678195+0000","last_peered":"2024-03-04T10:05:59.678195+0000","last_clean":"2024-03-04T10:05:59.678195+0000","last_became_active":"2024-03-04T10:05:58.347004+0000","last_became_peered":"2024-03-04T10:05:58.347004+0000","last_unstale":"2024-03-04T10:05:59.678195+0000","last_undegraded":"2024-03-04T10:05:59.678195+0000","last_fullsized":"2024-03-04T10:05:59.678195+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:32.545352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349519+0000","last_change":"2024-03-04T10:05:58.350917+0000","last_active":"2024-03-04T10:05:59.349519+0000","last_peered":"2024-03-04T10:05:59.349519+0000","last_clean":"2024-03-04T10:05:59.349519+0000","last_became_active":"2024-03-04T10:05:58.350629+0000","last_became_peered":"2024-03-04T10:05:58.350629+0000","last_unstale":"2024-03-04T10:05:59.349519+0000","last_undegraded":"2024-03-04T10:05:59.349519+0000","last_fullsized":"2024-03-04T10:05:59.349519+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:36.599089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350149+0000","last_change":"2024-03-04T10:05:58.362713+0000","last_active":"2024-03-04T10:05:59.350149+0000","last_peered":"2024-03-04T10:05:59.350149+0000","last_clean":"2024-03-04T10:05:59.350149+0000","last_became_active":"2024-03-04T10:05:58.362353+0000","last_became_peered":"2024-03-04T10:05:58.362353+0000","last_unstale":"2024-03-04T10:05:59.350149+0000","last_undegraded":"2024-03-04T10:05:59.350149+0000","last_fullsized":"2024-03-04T10:05:59.350149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:59:59.419648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542341+0000","last_change":"2024-03-04T10:05:58.343364+0000","last_active":"2024-03-04T10:05:59.542341+0000","last_peered":"2024-03-04T10:05:59.542341+0000","last_clean":"2024-03-04T10:05:59.542341+0000","last_became_active":"2024-03-04T10:05:58.343104+0000","last_became_peered":"2024-03-04T10:05:58.343104+0000","last_unstale":"2024-03-04T10:05:59.542341+0000","last_undegraded":"2024-03-04T10:05:59.542341+0000","last_fullsized":"2024-03-04T10:05:59.542341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:02:59.368526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347820+0000","last_change":"2024-03-04T10:05:58.363570+0000","last_active":"2024-03-04T10:05:59.347820+0000","last_peered":"2024-03-04T10:05:59.347820+0000","last_clean":"2024-03-04T10:05:59.347820+0000","last_became_active":"2024-03-04T10:05:58.363274+0000","last_became_peered":"2024-03-04T10:05:58.363274+0000","last_unstale":"2024-03-04T10:05:59.347820+0000","last_undegraded":"2024-03-04T10:05:59.347820+0000","last_fullsized":"2024-03-04T10:05:59.347820+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:08:18.698149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678606+0000","last_change":"2024-03-04T10:05:58.354742+0000","last_active":"2024-03-04T10:05:59.678606+0000","last_peered":"2024-03-04T10:05:59.678606+0000","last_clean":"2024-03-04T10:05:59.678606+0000","last_became_active":"2024-03-04T10:05:58.354479+0000","last_became_peered":"2024-03-04T10:05:58.354479+0000","last_unstale":"2024-03-04T10:05:59.678606+0000","last_undegraded":"2024-03-04T10:05:59.678606+0000","last_fullsized":"2024-03-04T10:05:59.678606+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:18.755017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349718+0000","last_change":"2024-03-04T10:05:58.367072+0000","last_active":"2024-03-04T10:05:59.349718+0000","last_peered":"2024-03-04T10:05:59.349718+0000","last_clean":"2024-03-04T10:05:59.349718+0000","last_became_active":"2024-03-04T10:05:58.366826+0000","last_became_peered":"2024-03-04T10:05:58.366826+0000","last_unstale":"2024-03-04T10:05:59.349718+0000","last_undegraded":"2024-03-04T10:05:59.349718+0000","last_fullsized":"2024-03-04T10:05:59.349718+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:16:47.573814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349896+0000","last_change":"2024-03-04T10:05:58.359029+0000","last_active":"2024-03-04T10:05:59.349896+0000","last_peered":"2024-03-04T10:05:59.349896+0000","last_clean":"2024-03-04T10:05:59.349896+0000","last_became_active":"2024-03-04T10:05:58.358757+0000","last_became_peered":"2024-03-04T10:05:58.358757+0000","last_unstale":"2024-03-04T10:05:59.349896+0000","last_undegraded":"2024-03-04T10:05:59.349896+0000","last_fullsized":"2024-03-04T10:05:59.349896+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:03:35.027129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349638+0000","last_change":"2024-03-04T10:05:58.355171+0000","last_active":"2024-03-04T10:05:59.349638+0000","last_peered":"2024-03-04T10:05:59.349638+0000","last_clean":"2024-03-04T10:05:59.349638+0000","last_became_active":"2024-03-04T10:05:58.353238+0000","last_became_peered":"2024-03-04T10:05:58.353238+0000","last_unstale":"2024-03-04T10:05:59.349638+0000","last_undegraded":"2024-03-04T10:05:59.349638+0000","last_fullsized":"2024-03-04T10:05:59.349638+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:02.511631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484369+0000","last_change":"2024-03-04T10:05:58.353446+0000","last_active":"2024-03-04T10:05:59.484369+0000","last_peered":"2024-03-04T10:05:59.484369+0000","last_clean":"2024-03-04T10:05:59.484369+0000","last_became_active":"2024-03-04T10:05:58.352953+0000","last_became_peered":"2024-03-04T10:05:58.352953+0000","last_unstale":"2024-03-04T10:05:59.484369+0000","last_undegraded":"2024-03-04T10:05:59.484369+0000","last_fullsized":"2024-03-04T10:05:59.484369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:55:38.424394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349460+0000","last_change":"2024-03-04T10:05:58.355426+0000","last_active":"2024-03-04T10:05:59.349460+0000","last_peered":"2024-03-04T10:05:59.349460+0000","last_clean":"2024-03-04T10:05:59.349460+0000","last_became_active":"2024-03-04T10:05:58.354992+0000","last_became_peered":"2024-03-04T10:05:58.354992+0000","last_unstale":"2024-03-04T10:05:59.349460+0000","last_undegraded":"2024-03-04T10:05:59.349460+0000","last_fullsized":"2024-03-04T10:05:59.349460+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:54.646485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678637+0000","last_change":"2024-03-04T10:05:57.347237+0000","last_active":"2024-03-04T10:05:59.678637+0000","last_peered":"2024-03-04T10:05:59.678637+0000","last_clean":"2024-03-04T10:05:59.678637+0000","last_became_active":"2024-03-04T10:05:57.346945+0000","last_became_peered":"2024-03-04T10:05:57.346945+0000","last_unstale":"2024-03-04T10:05:59.678637+0000","last_undegraded":"2024-03-04T10:05:59.678637+0000","last_fullsized":"2024-03-04T10:05:59.678637+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:51:42.577996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484434+0000","last_change":"2024-03-04T10:05:58.351073+0000","last_active":"2024-03-04T10:05:59.484434+0000","last_peered":"2024-03-04T10:05:59.484434+0000","last_clean":"2024-03-04T10:05:59.484434+0000","last_became_active":"2024-03-04T10:05:58.350745+0000","last_became_peered":"2024-03-04T10:05:58.350745+0000","last_unstale":"2024-03-04T10:05:59.484434+0000","last_undegraded":"2024-03-04T10:05:59.484434+0000","last_fullsized":"2024-03-04T10:05:59.484434+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:01.921655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347741+0000","last_change":"2024-03-04T10:05:57.354647+0000","last_active":"2024-03-04T10:05:59.347741+0000","last_peered":"2024-03-04T10:05:59.347741+0000","last_clean":"2024-03-04T10:05:59.347741+0000","last_became_active":"2024-03-04T10:05:57.354296+0000","last_became_peered":"2024-03-04T10:05:57.354296+0000","last_unstale":"2024-03-04T10:05:59.347741+0000","last_undegraded":"2024-03-04T10:05:59.347741+0000","last_fullsized":"2024-03-04T10:05:59.347741+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:59.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678228+0000","last_change":"2024-03-04T10:05:58.354235+0000","last_active":"2024-03-04T10:05:59.678228+0000","last_peered":"2024-03-04T10:05:59.678228+0000","last_clean":"2024-03-04T10:05:59.678228+0000","last_became_active":"2024-03-04T10:05:58.353913+0000","last_became_peered":"2024-03-04T10:05:58.353913+0000","last_unstale":"2024-03-04T10:05:59.678228+0000","last_undegraded":"2024-03-04T10:05:59.678228+0000","last_fullsized":"2024-03-04T10:05:59.678228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:09.387742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347892+0000","last_change":"2024-03-04T10:05:57.348436+0000","last_active":"2024-03-04T10:05:59.347892+0000","last_peered":"2024-03-04T10:05:59.347892+0000","last_clean":"2024-03-04T10:05:59.347892+0000","last_became_active":"2024-03-04T10:05:57.348160+0000","last_became_peered":"2024-03-04T10:05:57.348160+0000","last_unstale":"2024-03-04T10:05:59.347892+0000","last_undegraded":"2024-03-04T10:05:59.347892+0000","last_fullsized":"2024-03-04T10:05:59.347892+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:22.563194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542034+0000","last_change":"2024-03-04T10:05:58.345817+0000","last_active":"2024-03-04T10:05:59.542034+0000","last_peered":"2024-03-04T10:05:59.542034+0000","last_clean":"2024-03-04T10:05:59.542034+0000","last_became_active":"2024-03-04T10:05:58.345535+0000","last_became_peered":"2024-03-04T10:05:58.345535+0000","last_unstale":"2024-03-04T10:05:59.542034+0000","last_undegraded":"2024-03-04T10:05:59.542034+0000","last_fullsized":"2024-03-04T10:05:59.542034+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:02.511631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484549+0000","last_change":"2024-03-04T10:05:57.348949+0000","last_active":"2024-03-04T10:05:59.484549+0000","last_peered":"2024-03-04T10:05:59.484549+0000","last_clean":"2024-03-04T10:05:59.484549+0000","last_became_active":"2024-03-04T10:05:57.348244+0000","last_became_peered":"2024-03-04T10:05:57.348244+0000","last_unstale":"2024-03-04T10:05:59.484549+0000","last_undegraded":"2024-03-04T10:05:59.484549+0000","last_fullsized":"2024-03-04T10:05:59.484549+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349061+0000","last_change":"2024-03-04T10:05:58.353589+0000","last_active":"2024-03-04T10:05:59.349061+0000","last_peered":"2024-03-04T10:05:59.349061+0000","last_clean":"2024-03-04T10:05:59.349061+0000","last_became_active":"2024-03-04T10:05:58.353321+0000","last_became_peered":"2024-03-04T10:05:58.353321+0000","last_unstale":"2024-03-04T10:05:59.349061+0000","last_undegraded":"2024-03-04T10:05:59.349061+0000","last_fullsized":"2024-03-04T10:05:59.349061+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:54.183981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349728+0000","last_change":"2024-03-04T10:05:57.353664+0000","last_active":"2024-03-04T10:05:59.349728+0000","last_peered":"2024-03-04T10:05:59.349728+0000","last_clean":"2024-03-04T10:05:59.349728+0000","last_became_active":"2024-03-04T10:05:57.353393+0000","last_became_peered":"2024-03-04T10:05:57.353393+0000","last_unstale":"2024-03-04T10:05:59.349728+0000","last_undegraded":"2024-03-04T10:05:59.349728+0000","last_fullsized":"2024-03-04T10:05:59.349728+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350292+0000","last_change":"2024-03-04T10:05:58.360294+0000","last_active":"2024-03-04T10:05:59.350292+0000","last_peered":"2024-03-04T10:05:59.350292+0000","last_clean":"2024-03-04T10:05:59.350292+0000","last_became_active":"2024-03-04T10:05:58.359904+0000","last_became_peered":"2024-03-04T10:05:58.359904+0000","last_unstale":"2024-03-04T10:05:59.350292+0000","last_undegraded":"2024-03-04T10:05:59.350292+0000","last_fullsized":"2024-03-04T10:05:59.350292+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:23.925984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349189+0000","last_change":"2024-03-04T10:05:57.349177+0000","last_active":"2024-03-04T10:05:59.349189+0000","last_peered":"2024-03-04T10:05:59.349189+0000","last_clean":"2024-03-04T10:05:59.349189+0000","last_became_active":"2024-03-04T10:05:57.347881+0000","last_became_peered":"2024-03-04T10:05:57.347881+0000","last_unstale":"2024-03-04T10:05:59.349189+0000","last_undegraded":"2024-03-04T10:05:59.349189+0000","last_fullsized":"2024-03-04T10:05:59.349189+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:51:42.577996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542488+0000","last_change":"2024-03-04T10:05:58.345154+0000","last_active":"2024-03-04T10:05:59.542488+0000","last_peered":"2024-03-04T10:05:59.542488+0000","last_clean":"2024-03-04T10:05:59.542488+0000","last_became_active":"2024-03-04T10:05:58.344897+0000","last_became_peered":"2024-03-04T10:05:58.344897+0000","last_unstale":"2024-03-04T10:05:59.542488+0000","last_undegraded":"2024-03-04T10:05:59.542488+0000","last_fullsized":"2024-03-04T10:05:59.542488+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:10.508566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678177+0000","last_change":"2024-03-04T10:05:57.348567+0000","last_active":"2024-03-04T10:05:59.678177+0000","last_peered":"2024-03-04T10:05:59.678177+0000","last_clean":"2024-03-04T10:05:59.678177+0000","last_became_active":"2024-03-04T10:05:57.348178+0000","last_became_peered":"2024-03-04T10:05:57.348178+0000","last_unstale":"2024-03-04T10:05:59.678177+0000","last_undegraded":"2024-03-04T10:05:59.678177+0000","last_fullsized":"2024-03-04T10:05:59.678177+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:32.544831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350151+0000","last_change":"2024-03-04T10:05:58.352562+0000","last_active":"2024-03-04T10:05:59.350151+0000","last_peered":"2024-03-04T10:05:59.350151+0000","last_clean":"2024-03-04T10:05:59.350151+0000","last_became_active":"2024-03-04T10:05:58.352288+0000","last_became_peered":"2024-03-04T10:05:58.352288+0000","last_unstale":"2024-03-04T10:05:59.350151+0000","last_undegraded":"2024-03-04T10:05:59.350151+0000","last_fullsized":"2024-03-04T10:05:59.350151+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:48.573686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349516+0000","last_change":"2024-03-04T10:05:57.352999+0000","last_active":"2024-03-04T10:05:59.349516+0000","last_peered":"2024-03-04T10:05:59.349516+0000","last_clean":"2024-03-04T10:05:59.349516+0000","last_became_active":"2024-03-04T10:05:57.352754+0000","last_became_peered":"2024-03-04T10:05:57.352754+0000","last_unstale":"2024-03-04T10:05:59.349516+0000","last_undegraded":"2024-03-04T10:05:59.349516+0000","last_fullsized":"2024-03-04T10:05:59.349516+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.348913+0000","last_change":"2024-03-04T10:05:58.342473+0000","last_active":"2024-03-04T10:05:59.348913+0000","last_peered":"2024-03-04T10:05:59.348913+0000","last_clean":"2024-03-04T10:05:59.348913+0000","last_became_active":"2024-03-04T10:05:58.342185+0000","last_became_peered":"2024-03-04T10:05:58.342185+0000","last_unstale":"2024-03-04T10:05:59.348913+0000","last_undegraded":"2024-03-04T10:05:59.348913+0000","last_fullsized":"2024-03-04T10:05:59.348913+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:28.627958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350099+0000","last_change":"2024-03-04T10:05:57.349469+0000","last_active":"2024-03-04T10:05:59.350099+0000","last_peered":"2024-03-04T10:05:59.350099+0000","last_clean":"2024-03-04T10:05:59.350099+0000","last_became_active":"2024-03-04T10:05:57.349159+0000","last_became_peered":"2024-03-04T10:05:57.349159+0000","last_unstale":"2024-03-04T10:05:59.350099+0000","last_undegraded":"2024-03-04T10:05:59.350099+0000","last_fullsized":"2024-03-04T10:05:59.350099+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484375+0000","last_change":"2024-03-04T10:05:58.352743+0000","last_active":"2024-03-04T10:05:59.484375+0000","last_peered":"2024-03-04T10:05:59.484375+0000","last_clean":"2024-03-04T10:05:59.484375+0000","last_became_active":"2024-03-04T10:05:58.351308+0000","last_became_peered":"2024-03-04T10:05:58.351308+0000","last_unstale":"2024-03-04T10:05:59.484375+0000","last_undegraded":"2024-03-04T10:05:59.484375+0000","last_fullsized":"2024-03-04T10:05:59.484375+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:56.686869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349492+0000","last_change":"2024-03-04T10:05:57.354131+0000","last_active":"2024-03-04T10:05:59.349492+0000","last_peered":"2024-03-04T10:05:59.349492+0000","last_clean":"2024-03-04T10:05:59.349492+0000","last_became_active":"2024-03-04T10:05:57.353829+0000","last_became_peered":"2024-03-04T10:05:57.353829+0000","last_unstale":"2024-03-04T10:05:59.349492+0000","last_undegraded":"2024-03-04T10:05:59.349492+0000","last_fullsized":"2024-03-04T10:05:59.349492+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349742+0000","last_change":"2024-03-04T10:05:58.358486+0000","last_active":"2024-03-04T10:05:59.349742+0000","last_peered":"2024-03-04T10:05:59.349742+0000","last_clean":"2024-03-04T10:05:59.349742+0000","last_became_active":"2024-03-04T10:05:58.358223+0000","last_became_peered":"2024-03-04T10:05:58.358223+0000","last_unstale":"2024-03-04T10:05:59.349742+0000","last_undegraded":"2024-03-04T10:05:59.349742+0000","last_fullsized":"2024-03-04T10:05:59.349742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:02:59.368526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350365+0000","last_change":"2024-03-04T10:05:57.349719+0000","last_active":"2024-03-04T10:05:59.350365+0000","last_peered":"2024-03-04T10:05:59.350365+0000","last_clean":"2024-03-04T10:05:59.350365+0000","last_became_active":"2024-03-04T10:05:57.349424+0000","last_became_peered":"2024-03-04T10:05:57.349424+0000","last_unstale":"2024-03-04T10:05:59.350365+0000","last_undegraded":"2024-03-04T10:05:59.350365+0000","last_fullsized":"2024-03-04T10:05:59.350365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349270+0000","last_change":"2024-03-04T10:05:58.343323+0000","last_active":"2024-03-04T10:05:59.349270+0000","last_peered":"2024-03-04T10:05:59.349270+0000","last_clean":"2024-03-04T10:05:59.349270+0000","last_became_active":"2024-03-04T10:05:58.341469+0000","last_became_peered":"2024-03-04T10:05:58.341469+0000","last_unstale":"2024-03-04T10:05:59.349270+0000","last_undegraded":"2024-03-04T10:05:59.349270+0000","last_fullsized":"2024-03-04T10:05:59.349270+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:11.644158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542221+0000","last_change":"2024-03-04T10:05:57.353978+0000","last_active":"2024-03-04T10:05:59.542221+0000","last_peered":"2024-03-04T10:05:59.542221+0000","last_clean":"2024-03-04T10:05:59.542221+0000","last_became_active":"2024-03-04T10:05:57.347397+0000","last_became_peered":"2024-03-04T10:05:57.347397+0000","last_unstale":"2024-03-04T10:05:59.542221+0000","last_undegraded":"2024-03-04T10:05:59.542221+0000","last_fullsized":"2024-03-04T10:05:59.542221+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:09.058096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484412+0000","last_change":"2024-03-04T10:05:58.351909+0000","last_active":"2024-03-04T10:05:59.484412+0000","last_peered":"2024-03-04T10:05:59.484412+0000","last_clean":"2024-03-04T10:05:59.484412+0000","last_became_active":"2024-03-04T10:05:58.351264+0000","last_became_peered":"2024-03-04T10:05:58.351264+0000","last_unstale":"2024-03-04T10:05:59.484412+0000","last_undegraded":"2024-03-04T10:05:59.484412+0000","last_fullsized":"2024-03-04T10:05:59.484412+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:48.573686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349820+0000","last_change":"2024-03-04T10:05:57.355370+0000","last_active":"2024-03-04T10:05:59.349820+0000","last_peered":"2024-03-04T10:05:59.349820+0000","last_clean":"2024-03-04T10:05:59.349820+0000","last_became_active":"2024-03-04T10:05:57.355114+0000","last_became_peered":"2024-03-04T10:05:57.355114+0000","last_unstale":"2024-03-04T10:05:59.349820+0000","last_undegraded":"2024-03-04T10:05:59.349820+0000","last_fullsized":"2024-03-04T10:05:59.349820+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350084+0000","last_change":"2024-03-04T10:05:58.362098+0000","last_active":"2024-03-04T10:05:59.350084+0000","last_peered":"2024-03-04T10:05:59.350084+0000","last_clean":"2024-03-04T10:05:59.350084+0000","last_became_active":"2024-03-04T10:05:58.361794+0000","last_became_peered":"2024-03-04T10:05:58.361794+0000","last_unstale":"2024-03-04T10:05:59.350084+0000","last_undegraded":"2024-03-04T10:05:59.350084+0000","last_fullsized":"2024-03-04T10:05:59.350084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:35.113488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678374+0000","last_change":"2024-03-04T10:05:57.354657+0000","last_active":"2024-03-04T10:05:59.678374+0000","last_peered":"2024-03-04T10:05:59.678374+0000","last_clean":"2024-03-04T10:05:59.678374+0000","last_became_active":"2024-03-04T10:05:57.354347+0000","last_became_peered":"2024-03-04T10:05:57.354347+0000","last_unstale":"2024-03-04T10:05:59.678374+0000","last_undegraded":"2024-03-04T10:05:59.678374+0000","last_fullsized":"2024-03-04T10:05:59.678374+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:59.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542370+0000","last_change":"2024-03-04T10:05:58.346966+0000","last_active":"2024-03-04T10:05:59.542370+0000","last_peered":"2024-03-04T10:05:59.542370+0000","last_clean":"2024-03-04T10:05:59.542370+0000","last_became_active":"2024-03-04T10:05:58.346721+0000","last_became_peered":"2024-03-04T10:05:58.346721+0000","last_unstale":"2024-03-04T10:05:59.542370+0000","last_undegraded":"2024-03-04T10:05:59.542370+0000","last_fullsized":"2024-03-04T10:05:59.542370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:13.698347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349769+0000","last_change":"2024-03-04T10:05:57.346933+0000","last_active":"2024-03-04T10:05:59.349769+0000","last_peered":"2024-03-04T10:05:59.349769+0000","last_clean":"2024-03-04T10:05:59.349769+0000","last_became_active":"2024-03-04T10:05:57.346652+0000","last_became_peered":"2024-03-04T10:05:57.346652+0000","last_unstale":"2024-03-04T10:05:59.349769+0000","last_undegraded":"2024-03-04T10:05:59.349769+0000","last_fullsized":"2024-03-04T10:05:59.349769+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:08.382759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350120+0000","last_change":"2024-03-04T10:05:58.356292+0000","last_active":"2024-03-04T10:05:59.350120+0000","last_peered":"2024-03-04T10:05:59.350120+0000","last_clean":"2024-03-04T10:05:59.350120+0000","last_became_active":"2024-03-04T10:05:58.355832+0000","last_became_peered":"2024-03-04T10:05:58.355832+0000","last_unstale":"2024-03-04T10:05:59.350120+0000","last_undegraded":"2024-03-04T10:05:59.350120+0000","last_fullsized":"2024-03-04T10:05:59.350120+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:53.906371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542094+0000","last_change":"2024-03-04T10:05:57.347278+0000","last_active":"2024-03-04T10:05:59.542094+0000","last_peered":"2024-03-04T10:05:59.542094+0000","last_clean":"2024-03-04T10:05:59.542094+0000","last_became_active":"2024-03-04T10:05:57.346983+0000","last_became_peered":"2024-03-04T10:05:57.346983+0000","last_unstale":"2024-03-04T10:05:59.542094+0000","last_undegraded":"2024-03-04T10:05:59.542094+0000","last_fullsized":"2024-03-04T10:05:59.542094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:16.800199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542035+0000","last_change":"2024-03-04T10:05:58.355905+0000","last_active":"2024-03-04T10:05:59.542035+0000","last_peered":"2024-03-04T10:05:59.542035+0000","last_clean":"2024-03-04T10:05:59.542035+0000","last_became_active":"2024-03-04T10:05:58.355644+0000","last_became_peered":"2024-03-04T10:05:58.355644+0000","last_unstale":"2024-03-04T10:05:59.542035+0000","last_undegraded":"2024-03-04T10:05:59.542035+0000","last_fullsized":"2024-03-04T10:05:59.542035+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:56.231659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350176+0000","last_change":"2024-03-04T10:05:57.356865+0000","last_active":"2024-03-04T10:05:59.350176+0000","last_peered":"2024-03-04T10:05:59.350176+0000","last_clean":"2024-03-04T10:05:59.350176+0000","last_became_active":"2024-03-04T10:05:57.356604+0000","last_became_peered":"2024-03-04T10:05:57.356604+0000","last_unstale":"2024-03-04T10:05:59.350176+0000","last_undegraded":"2024-03-04T10:05:59.350176+0000","last_fullsized":"2024-03-04T10:05:59.350176+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:41.581953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347473+0000","last_change":"2024-03-04T10:05:58.366747+0000","last_active":"2024-03-04T10:05:59.347473+0000","last_peered":"2024-03-04T10:05:59.347473+0000","last_clean":"2024-03-04T10:05:59.347473+0000","last_became_active":"2024-03-04T10:05:58.366008+0000","last_became_peered":"2024-03-04T10:05:58.366008+0000","last_unstale":"2024-03-04T10:05:59.347473+0000","last_undegraded":"2024-03-04T10:05:59.347473+0000","last_fullsized":"2024-03-04T10:05:59.347473+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:40.724911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678444+0000","last_change":"2024-03-04T10:05:57.349592+0000","last_active":"2024-03-04T10:05:59.678444+0000","last_peered":"2024-03-04T10:05:59.678444+0000","last_clean":"2024-03-04T10:05:59.678444+0000","last_became_active":"2024-03-04T10:05:57.349329+0000","last_became_peered":"2024-03-04T10:05:57.349329+0000","last_unstale":"2024-03-04T10:05:59.678444+0000","last_undegraded":"2024-03-04T10:05:59.678444+0000","last_fullsized":"2024-03-04T10:05:59.678444+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:55:20.154042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349580+0000","last_change":"2024-03-04T10:05:58.360272+0000","last_active":"2024-03-04T10:05:59.349580+0000","last_peered":"2024-03-04T10:05:59.349580+0000","last_clean":"2024-03-04T10:05:59.349580+0000","last_became_active":"2024-03-04T10:05:58.359306+0000","last_became_peered":"2024-03-04T10:05:58.359306+0000","last_unstale":"2024-03-04T10:05:59.349580+0000","last_undegraded":"2024-03-04T10:05:59.349580+0000","last_fullsized":"2024-03-04T10:05:59.349580+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:09.387742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678628+0000","last_change":"2024-03-04T10:05:57.350127+0000","last_active":"2024-03-04T10:05:59.678628+0000","last_peered":"2024-03-04T10:05:59.678628+0000","last_clean":"2024-03-04T10:05:59.678628+0000","last_became_active":"2024-03-04T10:05:57.349795+0000","last_became_peered":"2024-03-04T10:05:57.349795+0000","last_unstale":"2024-03-04T10:05:59.678628+0000","last_undegraded":"2024-03-04T10:05:59.678628+0000","last_fullsized":"2024-03-04T10:05:59.678628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347796+0000","last_change":"2024-03-04T10:05:58.357135+0000","last_active":"2024-03-04T10:05:59.347796+0000","last_peered":"2024-03-04T10:05:59.347796+0000","last_clean":"2024-03-04T10:05:59.347796+0000","last_became_active":"2024-03-04T10:05:58.356883+0000","last_became_peered":"2024-03-04T10:05:58.356883+0000","last_unstale":"2024-03-04T10:05:59.347796+0000","last_undegraded":"2024-03-04T10:05:59.347796+0000","last_fullsized":"2024-03-04T10:05:59.347796+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:31.546202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678524+0000","last_change":"2024-03-04T10:05:57.352501+0000","last_active":"2024-03-04T10:05:59.678524+0000","last_peered":"2024-03-04T10:05:59.678524+0000","last_clean":"2024-03-04T10:05:59.678524+0000","last_became_active":"2024-03-04T10:05:57.352263+0000","last_became_peered":"2024-03-04T10:05:57.352263+0000","last_unstale":"2024-03-04T10:05:59.678524+0000","last_undegraded":"2024-03-04T10:05:59.678524+0000","last_fullsized":"2024-03-04T10:05:59.678524+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484355+0000","last_change":"2024-03-04T10:05:58.352055+0000","last_active":"2024-03-04T10:05:59.484355+0000","last_peered":"2024-03-04T10:05:59.484355+0000","last_clean":"2024-03-04T10:05:59.484355+0000","last_became_active":"2024-03-04T10:05:58.350986+0000","last_became_peered":"2024-03-04T10:05:58.350986+0000","last_unstale":"2024-03-04T10:05:59.484355+0000","last_undegraded":"2024-03-04T10:05:59.484355+0000","last_fullsized":"2024-03-04T10:05:59.484355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:03:35.027129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347697+0000","last_change":"2024-03-04T10:05:57.350188+0000","last_active":"2024-03-04T10:05:59.347697+0000","last_peered":"2024-03-04T10:05:59.347697+0000","last_clean":"2024-03-04T10:05:59.347697+0000","last_became_active":"2024-03-04T10:05:57.349869+0000","last_became_peered":"2024-03-04T10:05:57.349869+0000","last_unstale":"2024-03-04T10:05:59.347697+0000","last_undegraded":"2024-03-04T10:05:59.347697+0000","last_fullsized":"2024-03-04T10:05:59.347697+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:51:42.577996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678635+0000","last_change":"2024-03-04T10:05:58.353685+0000","last_active":"2024-03-04T10:05:59.678635+0000","last_peered":"2024-03-04T10:05:59.678635+0000","last_clean":"2024-03-04T10:05:59.678635+0000","last_became_active":"2024-03-04T10:05:58.353389+0000","last_became_peered":"2024-03-04T10:05:58.353389+0000","last_unstale":"2024-03-04T10:05:59.678635+0000","last_undegraded":"2024-03-04T10:05:59.678635+0000","last_fullsized":"2024-03-04T10:05:59.678635+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:40.724911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.541896+0000","last_change":"2024-03-04T10:05:57.348152+0000","last_active":"2024-03-04T10:05:59.541896+0000","last_peered":"2024-03-04T10:05:59.541896+0000","last_clean":"2024-03-04T10:05:59.541896+0000","last_became_active":"2024-03-04T10:05:57.347666+0000","last_became_peered":"2024-03-04T10:05:57.347666+0000","last_unstale":"2024-03-04T10:05:59.541896+0000","last_undegraded":"2024-03-04T10:05:59.541896+0000","last_fullsized":"2024-03-04T10:05:59.541896+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:41.581953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350051+0000","last_change":"2024-03-04T10:05:58.367059+0000","last_active":"2024-03-04T10:05:59.350051+0000","last_peered":"2024-03-04T10:05:59.350051+0000","last_clean":"2024-03-04T10:05:59.350051+0000","last_became_active":"2024-03-04T10:05:58.366804+0000","last_became_peered":"2024-03-04T10:05:58.366804+0000","last_unstale":"2024-03-04T10:05:59.350051+0000","last_undegraded":"2024-03-04T10:05:59.350051+0000","last_fullsized":"2024-03-04T10:05:59.350051+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:46.683655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349634+0000","last_change":"2024-03-04T10:05:57.355286+0000","last_active":"2024-03-04T10:05:59.349634+0000","last_peered":"2024-03-04T10:05:59.349634+0000","last_clean":"2024-03-04T10:05:59.349634+0000","last_became_active":"2024-03-04T10:05:57.354851+0000","last_became_peered":"2024-03-04T10:05:57.354851+0000","last_unstale":"2024-03-04T10:05:59.349634+0000","last_undegraded":"2024-03-04T10:05:59.349634+0000","last_fullsized":"2024-03-04T10:05:59.349634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:09.058096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347817+0000","last_change":"2024-03-04T10:05:58.355678+0000","last_active":"2024-03-04T10:05:59.347817+0000","last_peered":"2024-03-04T10:05:59.347817+0000","last_clean":"2024-03-04T10:05:59.347817+0000","last_became_active":"2024-03-04T10:05:58.355422+0000","last_became_peered":"2024-03-04T10:05:58.355422+0000","last_unstale":"2024-03-04T10:05:59.347817+0000","last_undegraded":"2024-03-04T10:05:59.347817+0000","last_fullsized":"2024-03-04T10:05:59.347817+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:17.805182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678119+0000","last_change":"2024-03-04T10:05:57.350358+0000","last_active":"2024-03-04T10:05:59.678119+0000","last_peered":"2024-03-04T10:05:59.678119+0000","last_clean":"2024-03-04T10:05:59.678119+0000","last_became_active":"2024-03-04T10:05:57.350087+0000","last_became_peered":"2024-03-04T10:05:57.350087+0000","last_unstale":"2024-03-04T10:05:59.678119+0000","last_undegraded":"2024-03-04T10:05:59.678119+0000","last_fullsized":"2024-03-04T10:05:59.678119+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:01.967647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484482+0000","last_change":"2024-03-04T10:05:58.353238+0000","last_active":"2024-03-04T10:05:59.484482+0000","last_peered":"2024-03-04T10:05:59.484482+0000","last_clean":"2024-03-04T10:05:59.484482+0000","last_became_active":"2024-03-04T10:05:58.352609+0000","last_became_peered":"2024-03-04T10:05:58.352609+0000","last_unstale":"2024-03-04T10:05:59.484482+0000","last_undegraded":"2024-03-04T10:05:59.484482+0000","last_fullsized":"2024-03-04T10:05:59.484482+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:34.135270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678640+0000","last_change":"2024-03-04T10:05:57.354106+0000","last_active":"2024-03-04T10:05:59.678640+0000","last_peered":"2024-03-04T10:05:59.678640+0000","last_clean":"2024-03-04T10:05:59.678640+0000","last_became_active":"2024-03-04T10:05:57.353627+0000","last_became_peered":"2024-03-04T10:05:57.353627+0000","last_unstale":"2024-03-04T10:05:59.678640+0000","last_undegraded":"2024-03-04T10:05:59.678640+0000","last_fullsized":"2024-03-04T10:05:59.678640+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:22.563194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347541+0000","last_change":"2024-03-04T10:05:58.354558+0000","last_active":"2024-03-04T10:05:59.347541+0000","last_peered":"2024-03-04T10:05:59.347541+0000","last_clean":"2024-03-04T10:05:59.347541+0000","last_became_active":"2024-03-04T10:05:58.354088+0000","last_became_peered":"2024-03-04T10:05:58.354088+0000","last_unstale":"2024-03-04T10:05:59.347541+0000","last_undegraded":"2024-03-04T10:05:59.347541+0000","last_fullsized":"2024-03-04T10:05:59.347541+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:10.063079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350403+0000","last_change":"2024-03-04T10:05:57.347633+0000","last_active":"2024-03-04T10:05:59.350403+0000","last_peered":"2024-03-04T10:05:59.350403+0000","last_clean":"2024-03-04T10:05:59.350403+0000","last_became_active":"2024-03-04T10:05:57.347321+0000","last_became_peered":"2024-03-04T10:05:57.347321+0000","last_unstale":"2024-03-04T10:05:59.350403+0000","last_undegraded":"2024-03-04T10:05:59.350403+0000","last_fullsized":"2024-03-04T10:05:59.350403+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542104+0000","last_change":"2024-03-04T10:05:58.345239+0000","last_active":"2024-03-04T10:05:59.542104+0000","last_peered":"2024-03-04T10:05:59.542104+0000","last_clean":"2024-03-04T10:05:59.542104+0000","last_became_active":"2024-03-04T10:05:58.344934+0000","last_became_peered":"2024-03-04T10:05:58.344934+0000","last_unstale":"2024-03-04T10:05:59.542104+0000","last_undegraded":"2024-03-04T10:05:59.542104+0000","last_fullsized":"2024-03-04T10:05:59.542104+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:16:47.573814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349837+0000","last_change":"2024-03-04T10:05:57.352257+0000","last_active":"2024-03-04T10:05:59.349837+0000","last_peered":"2024-03-04T10:05:59.349837+0000","last_clean":"2024-03-04T10:05:59.349837+0000","last_became_active":"2024-03-04T10:05:57.351978+0000","last_became_peered":"2024-03-04T10:05:57.351978+0000","last_unstale":"2024-03-04T10:05:59.349837+0000","last_undegraded":"2024-03-04T10:05:59.349837+0000","last_fullsized":"2024-03-04T10:05:59.349837+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:39.719928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349862+0000","last_change":"2024-03-04T10:05:58.365197+0000","last_active":"2024-03-04T10:05:59.349862+0000","last_peered":"2024-03-04T10:05:59.349862+0000","last_clean":"2024-03-04T10:05:59.349862+0000","last_became_active":"2024-03-04T10:05:58.364963+0000","last_became_peered":"2024-03-04T10:05:58.364963+0000","last_unstale":"2024-03-04T10:05:59.349862+0000","last_undegraded":"2024-03-04T10:05:59.349862+0000","last_fullsized":"2024-03-04T10:05:59.349862+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:13.698347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542132+0000","last_change":"2024-03-04T10:05:57.347958+0000","last_active":"2024-03-04T10:05:59.542132+0000","last_peered":"2024-03-04T10:05:59.542132+0000","last_clean":"2024-03-04T10:05:59.542132+0000","last_became_active":"2024-03-04T10:05:57.347694+0000","last_became_peered":"2024-03-04T10:05:57.347694+0000","last_unstale":"2024-03-04T10:05:59.542132+0000","last_undegraded":"2024-03-04T10:05:59.542132+0000","last_fullsized":"2024-03-04T10:05:59.542132+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:30.541219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484498+0000","last_change":"2024-03-04T10:05:58.352431+0000","last_active":"2024-03-04T10:05:59.484498+0000","last_peered":"2024-03-04T10:05:59.484498+0000","last_clean":"2024-03-04T10:05:59.484498+0000","last_became_active":"2024-03-04T10:05:58.352188+0000","last_became_peered":"2024-03-04T10:05:58.352188+0000","last_unstale":"2024-03-04T10:05:59.484498+0000","last_undegraded":"2024-03-04T10:05:59.484498+0000","last_fullsized":"2024-03-04T10:05:59.484498+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:23.925984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542005+0000","last_change":"2024-03-04T10:05:57.352564+0000","last_active":"2024-03-04T10:05:59.542005+0000","last_peered":"2024-03-04T10:05:59.542005+0000","last_clean":"2024-03-04T10:05:59.542005+0000","last_became_active":"2024-03-04T10:05:57.352324+0000","last_became_peered":"2024-03-04T10:05:57.352324+0000","last_unstale":"2024-03-04T10:05:59.542005+0000","last_undegraded":"2024-03-04T10:05:59.542005+0000","last_fullsized":"2024-03-04T10:05:59.542005+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:27.637935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350005+0000","last_change":"2024-03-04T10:05:58.364902+0000","last_active":"2024-03-04T10:05:59.350005+0000","last_peered":"2024-03-04T10:05:59.350005+0000","last_clean":"2024-03-04T10:05:59.350005+0000","last_became_active":"2024-03-04T10:05:58.364648+0000","last_became_peered":"2024-03-04T10:05:58.364648+0000","last_unstale":"2024-03-04T10:05:59.350005+0000","last_undegraded":"2024-03-04T10:05:59.350005+0000","last_fullsized":"2024-03-04T10:05:59.350005+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:10.508566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.348695+0000","last_change":"2024-03-04T10:05:57.341577+0000","last_active":"2024-03-04T10:05:59.348695+0000","last_peered":"2024-03-04T10:05:59.348695+0000","last_clean":"2024-03-04T10:05:59.348695+0000","last_became_active":"2024-03-04T10:05:57.341307+0000","last_became_peered":"2024-03-04T10:05:57.341307+0000","last_unstale":"2024-03-04T10:05:59.348695+0000","last_undegraded":"2024-03-04T10:05:59.348695+0000","last_fullsized":"2024-03-04T10:05:59.348695+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:32.544831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542432+0000","last_change":"2024-03-04T10:05:58.345986+0000","last_active":"2024-03-04T10:05:59.542432+0000","last_peered":"2024-03-04T10:05:59.542432+0000","last_clean":"2024-03-04T10:05:59.542432+0000","last_became_active":"2024-03-04T10:05:58.345499+0000","last_became_peered":"2024-03-04T10:05:58.345499+0000","last_unstale":"2024-03-04T10:05:59.542432+0000","last_undegraded":"2024-03-04T10:05:59.542432+0000","last_fullsized":"2024-03-04T10:05:59.542432+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:13.685578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484518+0000","last_change":"2024-03-04T10:05:57.351387+0000","last_active":"2024-03-04T10:05:59.484518+0000","last_peered":"2024-03-04T10:05:59.484518+0000","last_clean":"2024-03-04T10:05:59.484518+0000","last_became_active":"2024-03-04T10:05:57.351012+0000","last_became_peered":"2024-03-04T10:05:57.351012+0000","last_unstale":"2024-03-04T10:05:59.484518+0000","last_undegraded":"2024-03-04T10:05:59.484518+0000","last_fullsized":"2024-03-04T10:05:59.484518+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678251+0000","last_change":"2024-03-04T10:05:58.358932+0000","last_active":"2024-03-04T10:05:59.678251+0000","last_peered":"2024-03-04T10:05:59.678251+0000","last_clean":"2024-03-04T10:05:59.678251+0000","last_became_active":"2024-03-04T10:05:58.358232+0000","last_became_peered":"2024-03-04T10:05:58.358232+0000","last_unstale":"2024-03-04T10:05:59.678251+0000","last_undegraded":"2024-03-04T10:05:59.678251+0000","last_fullsized":"2024-03-04T10:05:59.678251+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:11.644158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350134+0000","last_change":"2024-03-04T10:05:57.354681+0000","last_active":"2024-03-04T10:05:59.350134+0000","last_peered":"2024-03-04T10:05:59.350134+0000","last_clean":"2024-03-04T10:05:59.350134+0000","last_became_active":"2024-03-04T10:05:57.354415+0000","last_became_peered":"2024-03-04T10:05:57.354415+0000","last_unstale":"2024-03-04T10:05:59.350134+0000","last_undegraded":"2024-03-04T10:05:59.350134+0000","last_fullsized":"2024-03-04T10:05:59.350134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:27.637935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349516+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:05:59.349516+0000","last_peered":"2024-03-04T10:05:59.349516+0000","last_clean":"2024-03-04T10:05:59.349516+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:05:59.349516+0000","last_undegraded":"2024-03-04T10:05:59.349516+0000","last_fullsized":"2024-03-04T10:05:59.349516+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1220608,"data_stored":867913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":278073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":277421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":278003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":279055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":279033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":277581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":867845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:06.696 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:06.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.806+0000 7f85ff910640 1 Processor -- start 2024-03-04T10:06:06.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.807+0000 7f85ff910640 1 -- start start 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.807+0000 7f85ff910640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f85f815de00 0x7f85f80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.807+0000 7f85ff910640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f815e1e0 con 0x7f85f80b9380 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.807+0000 7f85ff910640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f80d1c10 con 0x7f85f80b5470 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.807+0000 7f85ff910640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f80d2820 con 0x7f85f815de00 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.807+0000 7f85fd6ac640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f85f815de00 0x7f85f80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.808+0000 7f85fceab640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f85f80b9380 0x7f85f80b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55956/0 (socket says 172.21.15.99:55956) 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.808+0000 7f85fceab640 1 -- 172.21.15.99:0/1942970258 learned_addr learned my addr 172.21.15.99:0/1942970258 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.808+0000 7f85fd6ac640 1 -- 172.21.15.99:0/1942970258 >> v1:172.21.15.162:6789/0 conn(0x7f85f80b5470 legacy=0x7f85f80d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.808+0000 7f85fd6ac640 1 -- 172.21.15.99:0/1942970258 >> v1:172.21.15.99:6789/0 conn(0x7f85f80b9380 legacy=0x7f85f80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.808+0000 7f85fd6ac640 1 -- 172.21.15.99:0/1942970258 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80d1ef0 con 0x7f85f815de00 2024-03-04T10:06:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.808+0000 7f85effff640 1 -- 172.21.15.99:0/1942970258 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331620501 0 0) 0x7f85f815e1e0 con 0x7f85f80b9380 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85fd6ac640 1 --2- 172.21.15.99:0/1942970258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f815de00 0x7f85f80b2e60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009d90 tx=0x7f85f402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87875b0970b883cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85effff640 1 -- 172.21.15.99:0/1942970258 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4039070 con 0x7f85f815de00 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85effff640 1 -- 172.21.15.99:0/1942970258 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f402cc60 con 0x7f85f815de00 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85effff640 1 -- 172.21.15.99:0/1942970258 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f4034660 con 0x7f85f815de00 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 -- 172.21.15.99:0/1942970258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f815de00 msgr2=0x7f85f80b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 --2- 172.21.15.99:0/1942970258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f815de00 0x7f85f80b2e60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f85f4009d90 tx=0x7f85f402beb0 comp rx=0 tx=0).stop 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 -- 172.21.15.99:0/1942970258 shutdown_connections 2024-03-04T10:06:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 --2- 172.21.15.99:0/1942970258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f815de00 0x7f85f80b2e60 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:06.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 -- 172.21.15.99:0/1942970258 >> 172.21.15.99:0/1942970258 conn(0x7f85f80d0470 msgr2=0x7f85f80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:06.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 -- 172.21.15.99:0/1942970258 shutdown_connections 2024-03-04T10:06:06.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.809+0000 7f85ff910640 1 -- 172.21.15.99:0/1942970258 wait complete. 2024-03-04T10:06:06.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.810+0000 7f85ff910640 1 Processor -- start 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 -- start start 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85f80b5470 0x7f85f80cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 0x7f85f80c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f815de00 0x7f85f8156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f80b97e0 con 0x7f85f80b9380 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f80d24b0 con 0x7f85f815de00 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85ff910640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f80b72a0 con 0x7f85f80b5470 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85fceab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 0x7f85f80c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85fceab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 0x7f85f80c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57310/0 (socket says 172.21.15.99:57310) 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.811+0000 7f85fceab640 1 -- 172.21.15.99:0/596624449 learned_addr learned my addr 172.21.15.99:0/596624449 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85fceab640 1 -- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85f80b5470 msgr2=0x7f85f80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85fceab640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85f80b5470 0x7f85f80cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85fceab640 1 -- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f815de00 msgr2=0x7f85f8156560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:06.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85fceab640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f815de00 0x7f85f8156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85fceab640 1 -- 172.21.15.99:0/596624449 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80cc010 con 0x7f85f80b9380 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85fceab640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 0x7f85f80c9c90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f85e800c4a0 tx=0x7f85e800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8018070 con 0x7f85f80b9380 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e80074e0 con 0x7f85f80b9380 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f80cc2f0 con 0x7f85f80b9380 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8004a20 con 0x7f85f80b9380 2024-03-04T10:06:06.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.812+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f80cc8b0 con 0x7f85f80b9380 2024-03-04T10:06:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.813+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f8074f60 con 0x7f85f80b9380 2024-03-04T10:06:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.813+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f85e801b070 con 0x7f85f80b9380 2024-03-04T10:06:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.814+0000 7f85edffb640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85dc06b4d0 0x7f85dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.814+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f85e80896c0 con 0x7f85f80b9380 2024-03-04T10:06:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.814+0000 7f85fd6ac640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85dc06b4d0 0x7f85dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:06.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.815+0000 7f85fd6ac640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85dc06b4d0 0x7f85dc06d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f85f4036000 tx=0x7f85f402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:06.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.818+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f85e80583b0 con 0x7f85f80b9380 2024-03-04T10:06:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.985+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85f806c370 con 0x7f85dc06b4d0 2024-03-04T10:06:06.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.992+0000 7f85edffb640 1 -- 172.21.15.99:0/596624449 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251957 (secure 0 0 0) 0x7f85f806c370 con 0x7f85dc06b4d0 2024-03-04T10:06:06.997 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:06.998 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:06.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85dc06b4d0 msgr2=0x7f85dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85dc06b4d0 0x7f85dc06d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f85f4036000 tx=0x7f85f402e000 comp rx=0 tx=0).stop 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 msgr2=0x7f85f80c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 0x7f85f80c9c90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f85e800c4a0 tx=0x7f85e800c960 comp rx=0 tx=0).stop 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 shutdown_connections 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85dc06b4d0 0x7f85dc06d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f815de00 0x7f85f8156560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f80b9380 0x7f85f80c9c90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 --2- 172.21.15.99:0/596624449 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85f80b5470 0x7f85f80cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 >> 172.21.15.99:0/596624449 conn(0x7f85f80d0470 msgr2=0x7f85f80b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 shutdown_connections 2024-03-04T10:06:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:06.995+0000 7f85ff910640 1 -- 172.21.15.99:0/596624449 wait complete. 2024-03-04T10:06:07.008 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-03-04T10:06:05.472742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":191,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222252,"kb_used_data":6056,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752468,"statfs":{"total":773094113280,"available":772866527232,"internally_reserved":0,"allocated":6201344,"data_stored":3404924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002062"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347899+0000","last_change":"2024-03-04T10:05:58.358687+0000","last_active":"2024-03-04T10:05:59.347899+0000","last_peered":"2024-03-04T10:05:59.347899+0000","last_clean":"2024-03-04T10:05:59.347899+0000","last_became_active":"2024-03-04T10:05:58.358429+0000","last_became_peered":"2024-03-04T10:05:58.358429+0000","last_unstale":"2024-03-04T10:05:59.347899+0000","last_undegraded":"2024-03-04T10:05:59.347899+0000","last_fullsized":"2024-03-04T10:05:59.347899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:54.646485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350190+0000","last_change":"2024-03-04T10:05:58.351677+0000","last_active":"2024-03-04T10:05:59.350190+0000","last_peered":"2024-03-04T10:05:59.350190+0000","last_clean":"2024-03-04T10:05:59.350190+0000","last_became_active":"2024-03-04T10:05:58.351392+0000","last_became_peered":"2024-03-04T10:05:58.351392+0000","last_unstale":"2024-03-04T10:05:59.350190+0000","last_undegraded":"2024-03-04T10:05:59.350190+0000","last_fullsized":"2024-03-04T10:05:59.350190+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:01.921655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349965+0000","last_change":"2024-03-04T10:05:58.351831+0000","last_active":"2024-03-04T10:05:59.349965+0000","last_peered":"2024-03-04T10:05:59.349965+0000","last_clean":"2024-03-04T10:05:59.349965+0000","last_became_active":"2024-03-04T10:05:58.351295+0000","last_became_peered":"2024-03-04T10:05:58.351295+0000","last_unstale":"2024-03-04T10:05:59.349965+0000","last_undegraded":"2024-03-04T10:05:59.349965+0000","last_fullsized":"2024-03-04T10:05:59.349965+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:13.685578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350039+0000","last_change":"2024-03-04T10:05:58.346287+0000","last_active":"2024-03-04T10:05:59.350039+0000","last_peered":"2024-03-04T10:05:59.350039+0000","last_clean":"2024-03-04T10:05:59.350039+0000","last_became_active":"2024-03-04T10:05:58.346003+0000","last_became_peered":"2024-03-04T10:05:58.346003+0000","last_unstale":"2024-03-04T10:05:59.350039+0000","last_undegraded":"2024-03-04T10:05:59.350039+0000","last_fullsized":"2024-03-04T10:05:59.350039+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:56.686869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349113+0000","last_change":"2024-03-04T10:05:58.347170+0000","last_active":"2024-03-04T10:05:59.349113+0000","last_peered":"2024-03-04T10:05:59.349113+0000","last_clean":"2024-03-04T10:05:59.349113+0000","last_became_active":"2024-03-04T10:05:58.346819+0000","last_became_peered":"2024-03-04T10:05:58.346819+0000","last_unstale":"2024-03-04T10:05:59.349113+0000","last_undegraded":"2024-03-04T10:05:59.349113+0000","last_fullsized":"2024-03-04T10:05:59.349113+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:23.451619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347594+0000","last_change":"2024-03-04T10:05:58.354338+0000","last_active":"2024-03-04T10:05:59.347594+0000","last_peered":"2024-03-04T10:05:59.347594+0000","last_clean":"2024-03-04T10:05:59.347594+0000","last_became_active":"2024-03-04T10:05:58.353681+0000","last_became_peered":"2024-03-04T10:05:58.353681+0000","last_unstale":"2024-03-04T10:05:59.347594+0000","last_undegraded":"2024-03-04T10:05:59.347594+0000","last_fullsized":"2024-03-04T10:05:59.347594+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:42.586936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349547+0000","last_change":"2024-03-04T10:05:58.356631+0000","last_active":"2024-03-04T10:05:59.349547+0000","last_peered":"2024-03-04T10:05:59.349547+0000","last_clean":"2024-03-04T10:05:59.349547+0000","last_became_active":"2024-03-04T10:05:58.354506+0000","last_became_peered":"2024-03-04T10:05:58.354506+0000","last_unstale":"2024-03-04T10:05:59.349547+0000","last_undegraded":"2024-03-04T10:05:59.349547+0000","last_fullsized":"2024-03-04T10:05:59.349547+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:56.231659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542002+0000","last_change":"2024-03-04T10:05:58.358694+0000","last_active":"2024-03-04T10:05:59.542002+0000","last_peered":"2024-03-04T10:05:59.542002+0000","last_clean":"2024-03-04T10:05:59.542002+0000","last_became_active":"2024-03-04T10:05:58.358428+0000","last_became_peered":"2024-03-04T10:05:58.358428+0000","last_unstale":"2024-03-04T10:05:59.542002+0000","last_undegraded":"2024-03-04T10:05:59.542002+0000","last_fullsized":"2024-03-04T10:05:59.542002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:36.599089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542405+0000","last_change":"2024-03-04T10:05:58.360154+0000","last_active":"2024-03-04T10:05:59.542405+0000","last_peered":"2024-03-04T10:05:59.542405+0000","last_clean":"2024-03-04T10:05:59.542405+0000","last_became_active":"2024-03-04T10:05:58.359856+0000","last_became_peered":"2024-03-04T10:05:58.359856+0000","last_unstale":"2024-03-04T10:05:59.542405+0000","last_undegraded":"2024-03-04T10:05:59.542405+0000","last_fullsized":"2024-03-04T10:05:59.542405+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:25:39.593206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347899+0000","last_change":"2024-03-04T10:05:58.351885+0000","last_active":"2024-03-04T10:05:59.347899+0000","last_peered":"2024-03-04T10:05:59.347899+0000","last_clean":"2024-03-04T10:05:59.347899+0000","last_became_active":"2024-03-04T10:05:58.351627+0000","last_became_peered":"2024-03-04T10:05:58.351627+0000","last_unstale":"2024-03-04T10:05:59.347899+0000","last_undegraded":"2024-03-04T10:05:59.347899+0000","last_fullsized":"2024-03-04T10:05:59.347899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:55:38.424394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678432+0000","last_change":"2024-03-04T10:05:58.348999+0000","last_active":"2024-03-04T10:05:59.678432+0000","last_peered":"2024-03-04T10:05:59.678432+0000","last_clean":"2024-03-04T10:05:59.678432+0000","last_became_active":"2024-03-04T10:05:58.348768+0000","last_became_peered":"2024-03-04T10:05:58.348768+0000","last_unstale":"2024-03-04T10:05:59.678432+0000","last_undegraded":"2024-03-04T10:05:59.678432+0000","last_fullsized":"2024-03-04T10:05:59.678432+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:36:47.382317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542189+0000","last_change":"2024-03-04T10:05:58.352971+0000","last_active":"2024-03-04T10:05:59.542189+0000","last_peered":"2024-03-04T10:05:59.542189+0000","last_clean":"2024-03-04T10:05:59.542189+0000","last_became_active":"2024-03-04T10:05:58.352225+0000","last_became_peered":"2024-03-04T10:05:58.352225+0000","last_unstale":"2024-03-04T10:05:59.542189+0000","last_undegraded":"2024-03-04T10:05:59.542189+0000","last_fullsized":"2024-03-04T10:05:59.542189+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:27.532551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347604+0000","last_change":"2024-03-04T10:05:58.355130+0000","last_active":"2024-03-04T10:05:59.347604+0000","last_peered":"2024-03-04T10:05:59.347604+0000","last_clean":"2024-03-04T10:05:59.347604+0000","last_became_active":"2024-03-04T10:05:58.354862+0000","last_became_peered":"2024-03-04T10:05:58.354862+0000","last_unstale":"2024-03-04T10:05:59.347604+0000","last_undegraded":"2024-03-04T10:05:59.347604+0000","last_fullsized":"2024-03-04T10:05:59.347604+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:28.642918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349735+0000","last_change":"2024-03-04T10:05:58.352249+0000","last_active":"2024-03-04T10:05:59.349735+0000","last_peered":"2024-03-04T10:05:59.349735+0000","last_clean":"2024-03-04T10:05:59.349735+0000","last_became_active":"2024-03-04T10:05:58.351883+0000","last_became_peered":"2024-03-04T10:05:58.351883+0000","last_unstale":"2024-03-04T10:05:59.349735+0000","last_undegraded":"2024-03-04T10:05:59.349735+0000","last_fullsized":"2024-03-04T10:05:59.349735+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:37:27.532551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349767+0000","last_change":"2024-03-04T10:05:58.353341+0000","last_active":"2024-03-04T10:05:59.349767+0000","last_peered":"2024-03-04T10:05:59.349767+0000","last_clean":"2024-03-04T10:05:59.349767+0000","last_became_active":"2024-03-04T10:05:58.353076+0000","last_became_peered":"2024-03-04T10:05:58.353076+0000","last_unstale":"2024-03-04T10:05:59.349767+0000","last_undegraded":"2024-03-04T10:05:59.349767+0000","last_fullsized":"2024-03-04T10:05:59.349767+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:55:38.424394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347474+0000","last_change":"2024-03-04T10:05:58.349430+0000","last_active":"2024-03-04T10:05:59.347474+0000","last_peered":"2024-03-04T10:05:59.347474+0000","last_clean":"2024-03-04T10:05:59.347474+0000","last_became_active":"2024-03-04T10:05:58.349179+0000","last_became_peered":"2024-03-04T10:05:58.349179+0000","last_unstale":"2024-03-04T10:05:59.347474+0000","last_undegraded":"2024-03-04T10:05:59.347474+0000","last_fullsized":"2024-03-04T10:05:59.347474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:18.755017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350265+0000","last_change":"2024-03-04T10:05:58.356090+0000","last_active":"2024-03-04T10:05:59.350265+0000","last_peered":"2024-03-04T10:05:59.350265+0000","last_clean":"2024-03-04T10:05:59.350265+0000","last_became_active":"2024-03-04T10:05:58.355492+0000","last_became_peered":"2024-03-04T10:05:58.355492+0000","last_unstale":"2024-03-04T10:05:59.350265+0000","last_undegraded":"2024-03-04T10:05:59.350265+0000","last_fullsized":"2024-03-04T10:05:59.350265+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:20.373338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350305+0000","last_change":"2024-03-04T10:05:58.355628+0000","last_active":"2024-03-04T10:05:59.350305+0000","last_peered":"2024-03-04T10:05:59.350305+0000","last_clean":"2024-03-04T10:05:59.350305+0000","last_became_active":"2024-03-04T10:05:58.355365+0000","last_became_peered":"2024-03-04T10:05:58.355365+0000","last_unstale":"2024-03-04T10:05:59.350305+0000","last_undegraded":"2024-03-04T10:05:59.350305+0000","last_fullsized":"2024-03-04T10:05:59.350305+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:34.135270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349954+0000","last_change":"2024-03-04T10:05:58.367494+0000","last_active":"2024-03-04T10:05:59.349954+0000","last_peered":"2024-03-04T10:05:59.349954+0000","last_clean":"2024-03-04T10:05:59.349954+0000","last_became_active":"2024-03-04T10:05:58.367205+0000","last_became_peered":"2024-03-04T10:05:58.367205+0000","last_unstale":"2024-03-04T10:05:59.349954+0000","last_undegraded":"2024-03-04T10:05:59.349954+0000","last_fullsized":"2024-03-04T10:05:59.349954+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:25:39.593206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678349+0000","last_change":"2024-03-04T10:05:58.356714+0000","last_active":"2024-03-04T10:05:59.678349+0000","last_peered":"2024-03-04T10:05:59.678349+0000","last_clean":"2024-03-04T10:05:59.678349+0000","last_became_active":"2024-03-04T10:05:58.356432+0000","last_became_peered":"2024-03-04T10:05:58.356432+0000","last_unstale":"2024-03-04T10:05:59.678349+0000","last_undegraded":"2024-03-04T10:05:59.678349+0000","last_fullsized":"2024-03-04T10:05:59.678349+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:04.417535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678300+0000","last_change":"2024-03-04T10:05:58.356136+0000","last_active":"2024-03-04T10:05:59.678300+0000","last_peered":"2024-03-04T10:05:59.678300+0000","last_clean":"2024-03-04T10:05:59.678300+0000","last_became_active":"2024-03-04T10:05:58.354709+0000","last_became_peered":"2024-03-04T10:05:58.354709+0000","last_unstale":"2024-03-04T10:05:59.678300+0000","last_undegraded":"2024-03-04T10:05:59.678300+0000","last_fullsized":"2024-03-04T10:05:59.678300+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:54.137325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.541926+0000","last_change":"2024-03-04T10:05:58.358660+0000","last_active":"2024-03-04T10:05:59.541926+0000","last_peered":"2024-03-04T10:05:59.541926+0000","last_clean":"2024-03-04T10:05:59.541926+0000","last_became_active":"2024-03-04T10:05:58.358391+0000","last_became_peered":"2024-03-04T10:05:58.358391+0000","last_unstale":"2024-03-04T10:05:59.541926+0000","last_undegraded":"2024-03-04T10:05:59.541926+0000","last_fullsized":"2024-03-04T10:05:59.541926+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:25:45.939247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678195+0000","last_change":"2024-03-04T10:05:58.347253+0000","last_active":"2024-03-04T10:05:59.678195+0000","last_peered":"2024-03-04T10:05:59.678195+0000","last_clean":"2024-03-04T10:05:59.678195+0000","last_became_active":"2024-03-04T10:05:58.347004+0000","last_became_peered":"2024-03-04T10:05:58.347004+0000","last_unstale":"2024-03-04T10:05:59.678195+0000","last_undegraded":"2024-03-04T10:05:59.678195+0000","last_fullsized":"2024-03-04T10:05:59.678195+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:32.545352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349519+0000","last_change":"2024-03-04T10:05:58.350917+0000","last_active":"2024-03-04T10:05:59.349519+0000","last_peered":"2024-03-04T10:05:59.349519+0000","last_clean":"2024-03-04T10:05:59.349519+0000","last_became_active":"2024-03-04T10:05:58.350629+0000","last_became_peered":"2024-03-04T10:05:58.350629+0000","last_unstale":"2024-03-04T10:05:59.349519+0000","last_undegraded":"2024-03-04T10:05:59.349519+0000","last_fullsized":"2024-03-04T10:05:59.349519+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:36.599089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350149+0000","last_change":"2024-03-04T10:05:58.362713+0000","last_active":"2024-03-04T10:05:59.350149+0000","last_peered":"2024-03-04T10:05:59.350149+0000","last_clean":"2024-03-04T10:05:59.350149+0000","last_became_active":"2024-03-04T10:05:58.362353+0000","last_became_peered":"2024-03-04T10:05:58.362353+0000","last_unstale":"2024-03-04T10:05:59.350149+0000","last_undegraded":"2024-03-04T10:05:59.350149+0000","last_fullsized":"2024-03-04T10:05:59.350149+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:59:59.419648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542341+0000","last_change":"2024-03-04T10:05:58.343364+0000","last_active":"2024-03-04T10:05:59.542341+0000","last_peered":"2024-03-04T10:05:59.542341+0000","last_clean":"2024-03-04T10:05:59.542341+0000","last_became_active":"2024-03-04T10:05:58.343104+0000","last_became_peered":"2024-03-04T10:05:58.343104+0000","last_unstale":"2024-03-04T10:05:59.542341+0000","last_undegraded":"2024-03-04T10:05:59.542341+0000","last_fullsized":"2024-03-04T10:05:59.542341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:02:59.368526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347820+0000","last_change":"2024-03-04T10:05:58.363570+0000","last_active":"2024-03-04T10:05:59.347820+0000","last_peered":"2024-03-04T10:05:59.347820+0000","last_clean":"2024-03-04T10:05:59.347820+0000","last_became_active":"2024-03-04T10:05:58.363274+0000","last_became_peered":"2024-03-04T10:05:58.363274+0000","last_unstale":"2024-03-04T10:05:59.347820+0000","last_undegraded":"2024-03-04T10:05:59.347820+0000","last_fullsized":"2024-03-04T10:05:59.347820+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:08:18.698149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678606+0000","last_change":"2024-03-04T10:05:58.354742+0000","last_active":"2024-03-04T10:05:59.678606+0000","last_peered":"2024-03-04T10:05:59.678606+0000","last_clean":"2024-03-04T10:05:59.678606+0000","last_became_active":"2024-03-04T10:05:58.354479+0000","last_became_peered":"2024-03-04T10:05:58.354479+0000","last_unstale":"2024-03-04T10:05:59.678606+0000","last_undegraded":"2024-03-04T10:05:59.678606+0000","last_fullsized":"2024-03-04T10:05:59.678606+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:18.755017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349718+0000","last_change":"2024-03-04T10:05:58.367072+0000","last_active":"2024-03-04T10:05:59.349718+0000","last_peered":"2024-03-04T10:05:59.349718+0000","last_clean":"2024-03-04T10:05:59.349718+0000","last_became_active":"2024-03-04T10:05:58.366826+0000","last_became_peered":"2024-03-04T10:05:58.366826+0000","last_unstale":"2024-03-04T10:05:59.349718+0000","last_undegraded":"2024-03-04T10:05:59.349718+0000","last_fullsized":"2024-03-04T10:05:59.349718+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:16:47.573814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349896+0000","last_change":"2024-03-04T10:05:58.359029+0000","last_active":"2024-03-04T10:05:59.349896+0000","last_peered":"2024-03-04T10:05:59.349896+0000","last_clean":"2024-03-04T10:05:59.349896+0000","last_became_active":"2024-03-04T10:05:58.358757+0000","last_became_peered":"2024-03-04T10:05:58.358757+0000","last_unstale":"2024-03-04T10:05:59.349896+0000","last_undegraded":"2024-03-04T10:05:59.349896+0000","last_fullsized":"2024-03-04T10:05:59.349896+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:03:35.027129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349638+0000","last_change":"2024-03-04T10:05:58.355171+0000","last_active":"2024-03-04T10:05:59.349638+0000","last_peered":"2024-03-04T10:05:59.349638+0000","last_clean":"2024-03-04T10:05:59.349638+0000","last_became_active":"2024-03-04T10:05:58.353238+0000","last_became_peered":"2024-03-04T10:05:58.353238+0000","last_unstale":"2024-03-04T10:05:59.349638+0000","last_undegraded":"2024-03-04T10:05:59.349638+0000","last_fullsized":"2024-03-04T10:05:59.349638+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:02.511631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484369+0000","last_change":"2024-03-04T10:05:58.353446+0000","last_active":"2024-03-04T10:05:59.484369+0000","last_peered":"2024-03-04T10:05:59.484369+0000","last_clean":"2024-03-04T10:05:59.484369+0000","last_became_active":"2024-03-04T10:05:58.352953+0000","last_became_peered":"2024-03-04T10:05:58.352953+0000","last_unstale":"2024-03-04T10:05:59.484369+0000","last_undegraded":"2024-03-04T10:05:59.484369+0000","last_fullsized":"2024-03-04T10:05:59.484369+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:55:38.424394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349460+0000","last_change":"2024-03-04T10:05:58.355426+0000","last_active":"2024-03-04T10:05:59.349460+0000","last_peered":"2024-03-04T10:05:59.349460+0000","last_clean":"2024-03-04T10:05:59.349460+0000","last_became_active":"2024-03-04T10:05:58.354992+0000","last_became_peered":"2024-03-04T10:05:58.354992+0000","last_unstale":"2024-03-04T10:05:59.349460+0000","last_undegraded":"2024-03-04T10:05:59.349460+0000","last_fullsized":"2024-03-04T10:05:59.349460+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:46:54.646485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"51'4","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678637+0000","last_change":"2024-03-04T10:05:57.347237+0000","last_active":"2024-03-04T10:05:59.678637+0000","last_peered":"2024-03-04T10:05:59.678637+0000","last_clean":"2024-03-04T10:05:59.678637+0000","last_became_active":"2024-03-04T10:05:57.346945+0000","last_became_peered":"2024-03-04T10:05:57.346945+0000","last_unstale":"2024-03-04T10:05:59.678637+0000","last_undegraded":"2024-03-04T10:05:59.678637+0000","last_fullsized":"2024-03-04T10:05:59.678637+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:51:42.577996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484434+0000","last_change":"2024-03-04T10:05:58.351073+0000","last_active":"2024-03-04T10:05:59.484434+0000","last_peered":"2024-03-04T10:05:59.484434+0000","last_clean":"2024-03-04T10:05:59.484434+0000","last_became_active":"2024-03-04T10:05:58.350745+0000","last_became_peered":"2024-03-04T10:05:58.350745+0000","last_unstale":"2024-03-04T10:05:59.484434+0000","last_undegraded":"2024-03-04T10:05:59.484434+0000","last_fullsized":"2024-03-04T10:05:59.484434+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:24:01.921655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347741+0000","last_change":"2024-03-04T10:05:57.354647+0000","last_active":"2024-03-04T10:05:59.347741+0000","last_peered":"2024-03-04T10:05:59.347741+0000","last_clean":"2024-03-04T10:05:59.347741+0000","last_became_active":"2024-03-04T10:05:57.354296+0000","last_became_peered":"2024-03-04T10:05:57.354296+0000","last_unstale":"2024-03-04T10:05:59.347741+0000","last_undegraded":"2024-03-04T10:05:59.347741+0000","last_fullsized":"2024-03-04T10:05:59.347741+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:59.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678228+0000","last_change":"2024-03-04T10:05:58.354235+0000","last_active":"2024-03-04T10:05:59.678228+0000","last_peered":"2024-03-04T10:05:59.678228+0000","last_clean":"2024-03-04T10:05:59.678228+0000","last_became_active":"2024-03-04T10:05:58.353913+0000","last_became_peered":"2024-03-04T10:05:58.353913+0000","last_unstale":"2024-03-04T10:05:59.678228+0000","last_undegraded":"2024-03-04T10:05:59.678228+0000","last_fullsized":"2024-03-04T10:05:59.678228+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:09.387742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347892+0000","last_change":"2024-03-04T10:05:57.348436+0000","last_active":"2024-03-04T10:05:59.347892+0000","last_peered":"2024-03-04T10:05:59.347892+0000","last_clean":"2024-03-04T10:05:59.347892+0000","last_became_active":"2024-03-04T10:05:57.348160+0000","last_became_peered":"2024-03-04T10:05:57.348160+0000","last_unstale":"2024-03-04T10:05:59.347892+0000","last_undegraded":"2024-03-04T10:05:59.347892+0000","last_fullsized":"2024-03-04T10:05:59.347892+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:22.563194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542034+0000","last_change":"2024-03-04T10:05:58.345817+0000","last_active":"2024-03-04T10:05:59.542034+0000","last_peered":"2024-03-04T10:05:59.542034+0000","last_clean":"2024-03-04T10:05:59.542034+0000","last_became_active":"2024-03-04T10:05:58.345535+0000","last_became_peered":"2024-03-04T10:05:58.345535+0000","last_unstale":"2024-03-04T10:05:59.542034+0000","last_undegraded":"2024-03-04T10:05:59.542034+0000","last_fullsized":"2024-03-04T10:05:59.542034+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:26:02.511631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484549+0000","last_change":"2024-03-04T10:05:57.348949+0000","last_active":"2024-03-04T10:05:59.484549+0000","last_peered":"2024-03-04T10:05:59.484549+0000","last_clean":"2024-03-04T10:05:59.484549+0000","last_became_active":"2024-03-04T10:05:57.348244+0000","last_became_peered":"2024-03-04T10:05:57.348244+0000","last_unstale":"2024-03-04T10:05:59.484549+0000","last_undegraded":"2024-03-04T10:05:59.484549+0000","last_fullsized":"2024-03-04T10:05:59.484549+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349061+0000","last_change":"2024-03-04T10:05:58.353589+0000","last_active":"2024-03-04T10:05:59.349061+0000","last_peered":"2024-03-04T10:05:59.349061+0000","last_clean":"2024-03-04T10:05:59.349061+0000","last_became_active":"2024-03-04T10:05:58.353321+0000","last_became_peered":"2024-03-04T10:05:58.353321+0000","last_unstale":"2024-03-04T10:05:59.349061+0000","last_undegraded":"2024-03-04T10:05:59.349061+0000","last_fullsized":"2024-03-04T10:05:59.349061+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:42:54.183981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349728+0000","last_change":"2024-03-04T10:05:57.353664+0000","last_active":"2024-03-04T10:05:59.349728+0000","last_peered":"2024-03-04T10:05:59.349728+0000","last_clean":"2024-03-04T10:05:59.349728+0000","last_became_active":"2024-03-04T10:05:57.353393+0000","last_became_peered":"2024-03-04T10:05:57.353393+0000","last_unstale":"2024-03-04T10:05:59.349728+0000","last_undegraded":"2024-03-04T10:05:59.349728+0000","last_fullsized":"2024-03-04T10:05:59.349728+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350292+0000","last_change":"2024-03-04T10:05:58.360294+0000","last_active":"2024-03-04T10:05:59.350292+0000","last_peered":"2024-03-04T10:05:59.350292+0000","last_clean":"2024-03-04T10:05:59.350292+0000","last_became_active":"2024-03-04T10:05:58.359904+0000","last_became_peered":"2024-03-04T10:05:58.359904+0000","last_unstale":"2024-03-04T10:05:59.350292+0000","last_undegraded":"2024-03-04T10:05:59.350292+0000","last_fullsized":"2024-03-04T10:05:59.350292+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:23.925984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349189+0000","last_change":"2024-03-04T10:05:57.349177+0000","last_active":"2024-03-04T10:05:59.349189+0000","last_peered":"2024-03-04T10:05:59.349189+0000","last_clean":"2024-03-04T10:05:59.349189+0000","last_became_active":"2024-03-04T10:05:57.347881+0000","last_became_peered":"2024-03-04T10:05:57.347881+0000","last_unstale":"2024-03-04T10:05:59.349189+0000","last_undegraded":"2024-03-04T10:05:59.349189+0000","last_fullsized":"2024-03-04T10:05:59.349189+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:51:42.577996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542488+0000","last_change":"2024-03-04T10:05:58.345154+0000","last_active":"2024-03-04T10:05:59.542488+0000","last_peered":"2024-03-04T10:05:59.542488+0000","last_clean":"2024-03-04T10:05:59.542488+0000","last_became_active":"2024-03-04T10:05:58.344897+0000","last_became_peered":"2024-03-04T10:05:58.344897+0000","last_unstale":"2024-03-04T10:05:59.542488+0000","last_undegraded":"2024-03-04T10:05:59.542488+0000","last_fullsized":"2024-03-04T10:05:59.542488+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:10.508566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678177+0000","last_change":"2024-03-04T10:05:57.348567+0000","last_active":"2024-03-04T10:05:59.678177+0000","last_peered":"2024-03-04T10:05:59.678177+0000","last_clean":"2024-03-04T10:05:59.678177+0000","last_became_active":"2024-03-04T10:05:57.348178+0000","last_became_peered":"2024-03-04T10:05:57.348178+0000","last_unstale":"2024-03-04T10:05:59.678177+0000","last_undegraded":"2024-03-04T10:05:59.678177+0000","last_fullsized":"2024-03-04T10:05:59.678177+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:32.544831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350151+0000","last_change":"2024-03-04T10:05:58.352562+0000","last_active":"2024-03-04T10:05:59.350151+0000","last_peered":"2024-03-04T10:05:59.350151+0000","last_clean":"2024-03-04T10:05:59.350151+0000","last_became_active":"2024-03-04T10:05:58.352288+0000","last_became_peered":"2024-03-04T10:05:58.352288+0000","last_unstale":"2024-03-04T10:05:59.350151+0000","last_undegraded":"2024-03-04T10:05:59.350151+0000","last_fullsized":"2024-03-04T10:05:59.350151+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:48.573686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349516+0000","last_change":"2024-03-04T10:05:57.352999+0000","last_active":"2024-03-04T10:05:59.349516+0000","last_peered":"2024-03-04T10:05:59.349516+0000","last_clean":"2024-03-04T10:05:59.349516+0000","last_became_active":"2024-03-04T10:05:57.352754+0000","last_became_peered":"2024-03-04T10:05:57.352754+0000","last_unstale":"2024-03-04T10:05:59.349516+0000","last_undegraded":"2024-03-04T10:05:59.349516+0000","last_fullsized":"2024-03-04T10:05:59.349516+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.348913+0000","last_change":"2024-03-04T10:05:58.342473+0000","last_active":"2024-03-04T10:05:59.348913+0000","last_peered":"2024-03-04T10:05:59.348913+0000","last_clean":"2024-03-04T10:05:59.348913+0000","last_became_active":"2024-03-04T10:05:58.342185+0000","last_became_peered":"2024-03-04T10:05:58.342185+0000","last_unstale":"2024-03-04T10:05:59.348913+0000","last_undegraded":"2024-03-04T10:05:59.348913+0000","last_fullsized":"2024-03-04T10:05:59.348913+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:21:28.627958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350099+0000","last_change":"2024-03-04T10:05:57.349469+0000","last_active":"2024-03-04T10:05:59.350099+0000","last_peered":"2024-03-04T10:05:59.350099+0000","last_clean":"2024-03-04T10:05:59.350099+0000","last_became_active":"2024-03-04T10:05:57.349159+0000","last_became_peered":"2024-03-04T10:05:57.349159+0000","last_unstale":"2024-03-04T10:05:59.350099+0000","last_undegraded":"2024-03-04T10:05:59.350099+0000","last_fullsized":"2024-03-04T10:05:59.350099+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484375+0000","last_change":"2024-03-04T10:05:58.352743+0000","last_active":"2024-03-04T10:05:59.484375+0000","last_peered":"2024-03-04T10:05:59.484375+0000","last_clean":"2024-03-04T10:05:59.484375+0000","last_became_active":"2024-03-04T10:05:58.351308+0000","last_became_peered":"2024-03-04T10:05:58.351308+0000","last_unstale":"2024-03-04T10:05:59.484375+0000","last_undegraded":"2024-03-04T10:05:59.484375+0000","last_fullsized":"2024-03-04T10:05:59.484375+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:01:56.686869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349492+0000","last_change":"2024-03-04T10:05:57.354131+0000","last_active":"2024-03-04T10:05:59.349492+0000","last_peered":"2024-03-04T10:05:59.349492+0000","last_clean":"2024-03-04T10:05:59.349492+0000","last_became_active":"2024-03-04T10:05:57.353829+0000","last_became_peered":"2024-03-04T10:05:57.353829+0000","last_unstale":"2024-03-04T10:05:59.349492+0000","last_undegraded":"2024-03-04T10:05:59.349492+0000","last_fullsized":"2024-03-04T10:05:59.349492+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349742+0000","last_change":"2024-03-04T10:05:58.358486+0000","last_active":"2024-03-04T10:05:59.349742+0000","last_peered":"2024-03-04T10:05:59.349742+0000","last_clean":"2024-03-04T10:05:59.349742+0000","last_became_active":"2024-03-04T10:05:58.358223+0000","last_became_peered":"2024-03-04T10:05:58.358223+0000","last_unstale":"2024-03-04T10:05:59.349742+0000","last_undegraded":"2024-03-04T10:05:59.349742+0000","last_fullsized":"2024-03-04T10:05:59.349742+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:02:59.368526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"51'3","reported_seq":17,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350365+0000","last_change":"2024-03-04T10:05:57.349719+0000","last_active":"2024-03-04T10:05:59.350365+0000","last_peered":"2024-03-04T10:05:59.350365+0000","last_clean":"2024-03-04T10:05:59.350365+0000","last_became_active":"2024-03-04T10:05:57.349424+0000","last_became_peered":"2024-03-04T10:05:57.349424+0000","last_unstale":"2024-03-04T10:05:59.350365+0000","last_undegraded":"2024-03-04T10:05:59.350365+0000","last_fullsized":"2024-03-04T10:05:59.350365+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:10.639175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349270+0000","last_change":"2024-03-04T10:05:58.343323+0000","last_active":"2024-03-04T10:05:59.349270+0000","last_peered":"2024-03-04T10:05:59.349270+0000","last_clean":"2024-03-04T10:05:59.349270+0000","last_became_active":"2024-03-04T10:05:58.341469+0000","last_became_peered":"2024-03-04T10:05:58.341469+0000","last_unstale":"2024-03-04T10:05:59.349270+0000","last_undegraded":"2024-03-04T10:05:59.349270+0000","last_fullsized":"2024-03-04T10:05:59.349270+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:11.644158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"51'6","reported_seq":20,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542221+0000","last_change":"2024-03-04T10:05:57.353978+0000","last_active":"2024-03-04T10:05:59.542221+0000","last_peered":"2024-03-04T10:05:59.542221+0000","last_clean":"2024-03-04T10:05:59.542221+0000","last_became_active":"2024-03-04T10:05:57.347397+0000","last_became_peered":"2024-03-04T10:05:57.347397+0000","last_unstale":"2024-03-04T10:05:59.542221+0000","last_undegraded":"2024-03-04T10:05:59.542221+0000","last_fullsized":"2024-03-04T10:05:59.542221+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:09.058096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484412+0000","last_change":"2024-03-04T10:05:58.351909+0000","last_active":"2024-03-04T10:05:59.484412+0000","last_peered":"2024-03-04T10:05:59.484412+0000","last_clean":"2024-03-04T10:05:59.484412+0000","last_became_active":"2024-03-04T10:05:58.351264+0000","last_became_peered":"2024-03-04T10:05:58.351264+0000","last_unstale":"2024-03-04T10:05:59.484412+0000","last_undegraded":"2024-03-04T10:05:59.484412+0000","last_fullsized":"2024-03-04T10:05:59.484412+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:48.573686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349820+0000","last_change":"2024-03-04T10:05:57.355370+0000","last_active":"2024-03-04T10:05:59.349820+0000","last_peered":"2024-03-04T10:05:59.349820+0000","last_clean":"2024-03-04T10:05:59.349820+0000","last_became_active":"2024-03-04T10:05:57.355114+0000","last_became_peered":"2024-03-04T10:05:57.355114+0000","last_unstale":"2024-03-04T10:05:59.349820+0000","last_undegraded":"2024-03-04T10:05:59.349820+0000","last_fullsized":"2024-03-04T10:05:59.349820+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:19:03.412552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350084+0000","last_change":"2024-03-04T10:05:58.362098+0000","last_active":"2024-03-04T10:05:59.350084+0000","last_peered":"2024-03-04T10:05:59.350084+0000","last_clean":"2024-03-04T10:05:59.350084+0000","last_became_active":"2024-03-04T10:05:58.361794+0000","last_became_peered":"2024-03-04T10:05:58.361794+0000","last_unstale":"2024-03-04T10:05:59.350084+0000","last_undegraded":"2024-03-04T10:05:59.350084+0000","last_fullsized":"2024-03-04T10:05:59.350084+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:35.113488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678374+0000","last_change":"2024-03-04T10:05:57.354657+0000","last_active":"2024-03-04T10:05:59.678374+0000","last_peered":"2024-03-04T10:05:59.678374+0000","last_clean":"2024-03-04T10:05:59.678374+0000","last_became_active":"2024-03-04T10:05:57.354347+0000","last_became_peered":"2024-03-04T10:05:57.354347+0000","last_unstale":"2024-03-04T10:05:59.678374+0000","last_undegraded":"2024-03-04T10:05:59.678374+0000","last_fullsized":"2024-03-04T10:05:59.678374+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:22:59.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542370+0000","last_change":"2024-03-04T10:05:58.346966+0000","last_active":"2024-03-04T10:05:59.542370+0000","last_peered":"2024-03-04T10:05:59.542370+0000","last_clean":"2024-03-04T10:05:59.542370+0000","last_became_active":"2024-03-04T10:05:58.346721+0000","last_became_peered":"2024-03-04T10:05:58.346721+0000","last_unstale":"2024-03-04T10:05:59.542370+0000","last_undegraded":"2024-03-04T10:05:59.542370+0000","last_fullsized":"2024-03-04T10:05:59.542370+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:13.698347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349769+0000","last_change":"2024-03-04T10:05:57.346933+0000","last_active":"2024-03-04T10:05:59.349769+0000","last_peered":"2024-03-04T10:05:59.349769+0000","last_clean":"2024-03-04T10:05:59.349769+0000","last_became_active":"2024-03-04T10:05:57.346652+0000","last_became_peered":"2024-03-04T10:05:57.346652+0000","last_unstale":"2024-03-04T10:05:59.349769+0000","last_undegraded":"2024-03-04T10:05:59.349769+0000","last_fullsized":"2024-03-04T10:05:59.349769+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:08.382759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350120+0000","last_change":"2024-03-04T10:05:58.356292+0000","last_active":"2024-03-04T10:05:59.350120+0000","last_peered":"2024-03-04T10:05:59.350120+0000","last_clean":"2024-03-04T10:05:59.350120+0000","last_became_active":"2024-03-04T10:05:58.355832+0000","last_became_peered":"2024-03-04T10:05:58.355832+0000","last_unstale":"2024-03-04T10:05:59.350120+0000","last_undegraded":"2024-03-04T10:05:59.350120+0000","last_fullsized":"2024-03-04T10:05:59.350120+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:39:53.906371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"51'8","reported_seq":22,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542094+0000","last_change":"2024-03-04T10:05:57.347278+0000","last_active":"2024-03-04T10:05:59.542094+0000","last_peered":"2024-03-04T10:05:59.542094+0000","last_clean":"2024-03-04T10:05:59.542094+0000","last_became_active":"2024-03-04T10:05:57.346983+0000","last_became_peered":"2024-03-04T10:05:57.346983+0000","last_unstale":"2024-03-04T10:05:59.542094+0000","last_undegraded":"2024-03-04T10:05:59.542094+0000","last_fullsized":"2024-03-04T10:05:59.542094+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:16.800199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542035+0000","last_change":"2024-03-04T10:05:58.355905+0000","last_active":"2024-03-04T10:05:59.542035+0000","last_peered":"2024-03-04T10:05:59.542035+0000","last_clean":"2024-03-04T10:05:59.542035+0000","last_became_active":"2024-03-04T10:05:58.355644+0000","last_became_peered":"2024-03-04T10:05:58.355644+0000","last_unstale":"2024-03-04T10:05:59.542035+0000","last_undegraded":"2024-03-04T10:05:59.542035+0000","last_fullsized":"2024-03-04T10:05:59.542035+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:56.231659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350176+0000","last_change":"2024-03-04T10:05:57.356865+0000","last_active":"2024-03-04T10:05:59.350176+0000","last_peered":"2024-03-04T10:05:59.350176+0000","last_clean":"2024-03-04T10:05:59.350176+0000","last_became_active":"2024-03-04T10:05:57.356604+0000","last_became_peered":"2024-03-04T10:05:57.356604+0000","last_unstale":"2024-03-04T10:05:59.350176+0000","last_undegraded":"2024-03-04T10:05:59.350176+0000","last_fullsized":"2024-03-04T10:05:59.350176+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:41.581953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347473+0000","last_change":"2024-03-04T10:05:58.366747+0000","last_active":"2024-03-04T10:05:59.347473+0000","last_peered":"2024-03-04T10:05:59.347473+0000","last_clean":"2024-03-04T10:05:59.347473+0000","last_became_active":"2024-03-04T10:05:58.366008+0000","last_became_peered":"2024-03-04T10:05:58.366008+0000","last_unstale":"2024-03-04T10:05:59.347473+0000","last_undegraded":"2024-03-04T10:05:59.347473+0000","last_fullsized":"2024-03-04T10:05:59.347473+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:40.724911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678444+0000","last_change":"2024-03-04T10:05:57.349592+0000","last_active":"2024-03-04T10:05:59.678444+0000","last_peered":"2024-03-04T10:05:59.678444+0000","last_clean":"2024-03-04T10:05:59.678444+0000","last_became_active":"2024-03-04T10:05:57.349329+0000","last_became_peered":"2024-03-04T10:05:57.349329+0000","last_unstale":"2024-03-04T10:05:59.678444+0000","last_undegraded":"2024-03-04T10:05:59.678444+0000","last_fullsized":"2024-03-04T10:05:59.678444+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:55:20.154042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349580+0000","last_change":"2024-03-04T10:05:58.360272+0000","last_active":"2024-03-04T10:05:59.349580+0000","last_peered":"2024-03-04T10:05:59.349580+0000","last_clean":"2024-03-04T10:05:59.349580+0000","last_became_active":"2024-03-04T10:05:58.359306+0000","last_became_peered":"2024-03-04T10:05:58.359306+0000","last_unstale":"2024-03-04T10:05:59.349580+0000","last_undegraded":"2024-03-04T10:05:59.349580+0000","last_fullsized":"2024-03-04T10:05:59.349580+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:09.387742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"51'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678628+0000","last_change":"2024-03-04T10:05:57.350127+0000","last_active":"2024-03-04T10:05:59.678628+0000","last_peered":"2024-03-04T10:05:59.678628+0000","last_clean":"2024-03-04T10:05:59.678628+0000","last_became_active":"2024-03-04T10:05:57.349795+0000","last_became_peered":"2024-03-04T10:05:57.349795+0000","last_unstale":"2024-03-04T10:05:59.678628+0000","last_undegraded":"2024-03-04T10:05:59.678628+0000","last_fullsized":"2024-03-04T10:05:59.678628+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:39:31.674401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347796+0000","last_change":"2024-03-04T10:05:58.357135+0000","last_active":"2024-03-04T10:05:59.347796+0000","last_peered":"2024-03-04T10:05:59.347796+0000","last_clean":"2024-03-04T10:05:59.347796+0000","last_became_active":"2024-03-04T10:05:58.356883+0000","last_became_peered":"2024-03-04T10:05:58.356883+0000","last_unstale":"2024-03-04T10:05:59.347796+0000","last_undegraded":"2024-03-04T10:05:59.347796+0000","last_fullsized":"2024-03-04T10:05:59.347796+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:31.546202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678524+0000","last_change":"2024-03-04T10:05:57.352501+0000","last_active":"2024-03-04T10:05:59.678524+0000","last_peered":"2024-03-04T10:05:59.678524+0000","last_clean":"2024-03-04T10:05:59.678524+0000","last_became_active":"2024-03-04T10:05:57.352263+0000","last_became_peered":"2024-03-04T10:05:57.352263+0000","last_unstale":"2024-03-04T10:05:59.678524+0000","last_undegraded":"2024-03-04T10:05:59.678524+0000","last_fullsized":"2024-03-04T10:05:59.678524+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:50:36.862297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484355+0000","last_change":"2024-03-04T10:05:58.352055+0000","last_active":"2024-03-04T10:05:59.484355+0000","last_peered":"2024-03-04T10:05:59.484355+0000","last_clean":"2024-03-04T10:05:59.484355+0000","last_became_active":"2024-03-04T10:05:58.350986+0000","last_became_peered":"2024-03-04T10:05:58.350986+0000","last_unstale":"2024-03-04T10:05:59.484355+0000","last_undegraded":"2024-03-04T10:05:59.484355+0000","last_fullsized":"2024-03-04T10:05:59.484355+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:03:35.027129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347697+0000","last_change":"2024-03-04T10:05:57.350188+0000","last_active":"2024-03-04T10:05:59.347697+0000","last_peered":"2024-03-04T10:05:59.347697+0000","last_clean":"2024-03-04T10:05:59.347697+0000","last_became_active":"2024-03-04T10:05:57.349869+0000","last_became_peered":"2024-03-04T10:05:57.349869+0000","last_unstale":"2024-03-04T10:05:59.347697+0000","last_undegraded":"2024-03-04T10:05:59.347697+0000","last_fullsized":"2024-03-04T10:05:59.347697+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:51:42.577996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678635+0000","last_change":"2024-03-04T10:05:58.353685+0000","last_active":"2024-03-04T10:05:59.678635+0000","last_peered":"2024-03-04T10:05:59.678635+0000","last_clean":"2024-03-04T10:05:59.678635+0000","last_became_active":"2024-03-04T10:05:58.353389+0000","last_became_peered":"2024-03-04T10:05:58.353389+0000","last_unstale":"2024-03-04T10:05:59.678635+0000","last_undegraded":"2024-03-04T10:05:59.678635+0000","last_fullsized":"2024-03-04T10:05:59.678635+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:40.724911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.541896+0000","last_change":"2024-03-04T10:05:57.348152+0000","last_active":"2024-03-04T10:05:59.541896+0000","last_peered":"2024-03-04T10:05:59.541896+0000","last_clean":"2024-03-04T10:05:59.541896+0000","last_became_active":"2024-03-04T10:05:57.347666+0000","last_became_peered":"2024-03-04T10:05:57.347666+0000","last_unstale":"2024-03-04T10:05:59.541896+0000","last_undegraded":"2024-03-04T10:05:59.541896+0000","last_fullsized":"2024-03-04T10:05:59.541896+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:41.581953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350051+0000","last_change":"2024-03-04T10:05:58.367059+0000","last_active":"2024-03-04T10:05:59.350051+0000","last_peered":"2024-03-04T10:05:59.350051+0000","last_clean":"2024-03-04T10:05:59.350051+0000","last_became_active":"2024-03-04T10:05:58.366804+0000","last_became_peered":"2024-03-04T10:05:58.366804+0000","last_unstale":"2024-03-04T10:05:59.350051+0000","last_undegraded":"2024-03-04T10:05:59.350051+0000","last_fullsized":"2024-03-04T10:05:59.350051+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:46.683655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349634+0000","last_change":"2024-03-04T10:05:57.355286+0000","last_active":"2024-03-04T10:05:59.349634+0000","last_peered":"2024-03-04T10:05:59.349634+0000","last_clean":"2024-03-04T10:05:59.349634+0000","last_became_active":"2024-03-04T10:05:57.354851+0000","last_became_peered":"2024-03-04T10:05:57.354851+0000","last_unstale":"2024-03-04T10:05:59.349634+0000","last_undegraded":"2024-03-04T10:05:59.349634+0000","last_fullsized":"2024-03-04T10:05:59.349634+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:09.058096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347817+0000","last_change":"2024-03-04T10:05:58.355678+0000","last_active":"2024-03-04T10:05:59.347817+0000","last_peered":"2024-03-04T10:05:59.347817+0000","last_clean":"2024-03-04T10:05:59.347817+0000","last_became_active":"2024-03-04T10:05:58.355422+0000","last_became_peered":"2024-03-04T10:05:58.355422+0000","last_unstale":"2024-03-04T10:05:59.347817+0000","last_undegraded":"2024-03-04T10:05:59.347817+0000","last_fullsized":"2024-03-04T10:05:59.347817+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:17.805182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678119+0000","last_change":"2024-03-04T10:05:57.350358+0000","last_active":"2024-03-04T10:05:59.678119+0000","last_peered":"2024-03-04T10:05:59.678119+0000","last_clean":"2024-03-04T10:05:59.678119+0000","last_became_active":"2024-03-04T10:05:57.350087+0000","last_became_peered":"2024-03-04T10:05:57.350087+0000","last_unstale":"2024-03-04T10:05:59.678119+0000","last_undegraded":"2024-03-04T10:05:59.678119+0000","last_fullsized":"2024-03-04T10:05:59.678119+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:01.967647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484482+0000","last_change":"2024-03-04T10:05:58.353238+0000","last_active":"2024-03-04T10:05:59.484482+0000","last_peered":"2024-03-04T10:05:59.484482+0000","last_clean":"2024-03-04T10:05:59.484482+0000","last_became_active":"2024-03-04T10:05:58.352609+0000","last_became_peered":"2024-03-04T10:05:58.352609+0000","last_unstale":"2024-03-04T10:05:59.484482+0000","last_undegraded":"2024-03-04T10:05:59.484482+0000","last_fullsized":"2024-03-04T10:05:59.484482+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:34.135270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"51'2","reported_seq":16,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678640+0000","last_change":"2024-03-04T10:05:57.354106+0000","last_active":"2024-03-04T10:05:59.678640+0000","last_peered":"2024-03-04T10:05:59.678640+0000","last_clean":"2024-03-04T10:05:59.678640+0000","last_became_active":"2024-03-04T10:05:57.353627+0000","last_became_peered":"2024-03-04T10:05:57.353627+0000","last_unstale":"2024-03-04T10:05:59.678640+0000","last_undegraded":"2024-03-04T10:05:59.678640+0000","last_fullsized":"2024-03-04T10:05:59.678640+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:12:22.563194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.347541+0000","last_change":"2024-03-04T10:05:58.354558+0000","last_active":"2024-03-04T10:05:59.347541+0000","last_peered":"2024-03-04T10:05:59.347541+0000","last_clean":"2024-03-04T10:05:59.347541+0000","last_became_active":"2024-03-04T10:05:58.354088+0000","last_became_peered":"2024-03-04T10:05:58.354088+0000","last_unstale":"2024-03-04T10:05:59.347541+0000","last_undegraded":"2024-03-04T10:05:59.347541+0000","last_fullsized":"2024-03-04T10:05:59.347541+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:10.063079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350403+0000","last_change":"2024-03-04T10:05:57.347633+0000","last_active":"2024-03-04T10:05:59.350403+0000","last_peered":"2024-03-04T10:05:59.350403+0000","last_clean":"2024-03-04T10:05:59.350403+0000","last_became_active":"2024-03-04T10:05:57.347321+0000","last_became_peered":"2024-03-04T10:05:57.347321+0000","last_unstale":"2024-03-04T10:05:59.350403+0000","last_undegraded":"2024-03-04T10:05:59.350403+0000","last_fullsized":"2024-03-04T10:05:59.350403+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:53.132342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542104+0000","last_change":"2024-03-04T10:05:58.345239+0000","last_active":"2024-03-04T10:05:59.542104+0000","last_peered":"2024-03-04T10:05:59.542104+0000","last_clean":"2024-03-04T10:05:59.542104+0000","last_became_active":"2024-03-04T10:05:58.344934+0000","last_became_peered":"2024-03-04T10:05:58.344934+0000","last_unstale":"2024-03-04T10:05:59.542104+0000","last_undegraded":"2024-03-04T10:05:59.542104+0000","last_fullsized":"2024-03-04T10:05:59.542104+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:16:47.573814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349837+0000","last_change":"2024-03-04T10:05:57.352257+0000","last_active":"2024-03-04T10:05:59.349837+0000","last_peered":"2024-03-04T10:05:59.349837+0000","last_clean":"2024-03-04T10:05:59.349837+0000","last_became_active":"2024-03-04T10:05:57.351978+0000","last_became_peered":"2024-03-04T10:05:57.351978+0000","last_unstale":"2024-03-04T10:05:59.349837+0000","last_undegraded":"2024-03-04T10:05:59.349837+0000","last_fullsized":"2024-03-04T10:05:59.349837+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:39.719928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349862+0000","last_change":"2024-03-04T10:05:58.365197+0000","last_active":"2024-03-04T10:05:59.349862+0000","last_peered":"2024-03-04T10:05:59.349862+0000","last_clean":"2024-03-04T10:05:59.349862+0000","last_became_active":"2024-03-04T10:05:58.364963+0000","last_became_peered":"2024-03-04T10:05:58.364963+0000","last_unstale":"2024-03-04T10:05:59.349862+0000","last_undegraded":"2024-03-04T10:05:59.349862+0000","last_fullsized":"2024-03-04T10:05:59.349862+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:13.698347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542132+0000","last_change":"2024-03-04T10:05:57.347958+0000","last_active":"2024-03-04T10:05:59.542132+0000","last_peered":"2024-03-04T10:05:59.542132+0000","last_clean":"2024-03-04T10:05:59.542132+0000","last_became_active":"2024-03-04T10:05:57.347694+0000","last_became_peered":"2024-03-04T10:05:57.347694+0000","last_unstale":"2024-03-04T10:05:59.542132+0000","last_undegraded":"2024-03-04T10:05:59.542132+0000","last_fullsized":"2024-03-04T10:05:59.542132+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:30.541219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484498+0000","last_change":"2024-03-04T10:05:58.352431+0000","last_active":"2024-03-04T10:05:59.484498+0000","last_peered":"2024-03-04T10:05:59.484498+0000","last_clean":"2024-03-04T10:05:59.484498+0000","last_became_active":"2024-03-04T10:05:58.352188+0000","last_became_peered":"2024-03-04T10:05:58.352188+0000","last_unstale":"2024-03-04T10:05:59.484498+0000","last_undegraded":"2024-03-04T10:05:59.484498+0000","last_fullsized":"2024-03-04T10:05:59.484498+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:44:23.925984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542005+0000","last_change":"2024-03-04T10:05:57.352564+0000","last_active":"2024-03-04T10:05:59.542005+0000","last_peered":"2024-03-04T10:05:59.542005+0000","last_clean":"2024-03-04T10:05:59.542005+0000","last_became_active":"2024-03-04T10:05:57.352324+0000","last_became_peered":"2024-03-04T10:05:57.352324+0000","last_unstale":"2024-03-04T10:05:59.542005+0000","last_undegraded":"2024-03-04T10:05:59.542005+0000","last_fullsized":"2024-03-04T10:05:59.542005+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:27.637935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350005+0000","last_change":"2024-03-04T10:05:58.364902+0000","last_active":"2024-03-04T10:05:59.350005+0000","last_peered":"2024-03-04T10:05:59.350005+0000","last_clean":"2024-03-04T10:05:59.350005+0000","last_became_active":"2024-03-04T10:05:58.364648+0000","last_became_peered":"2024-03-04T10:05:58.364648+0000","last_unstale":"2024-03-04T10:05:59.350005+0000","last_undegraded":"2024-03-04T10:05:59.350005+0000","last_fullsized":"2024-03-04T10:05:59.350005+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:10.508566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.348695+0000","last_change":"2024-03-04T10:05:57.341577+0000","last_active":"2024-03-04T10:05:59.348695+0000","last_peered":"2024-03-04T10:05:59.348695+0000","last_clean":"2024-03-04T10:05:59.348695+0000","last_became_active":"2024-03-04T10:05:57.341307+0000","last_became_peered":"2024-03-04T10:05:57.341307+0000","last_unstale":"2024-03-04T10:05:59.348695+0000","last_undegraded":"2024-03-04T10:05:59.348695+0000","last_fullsized":"2024-03-04T10:05:59.348695+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:32.544831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.542432+0000","last_change":"2024-03-04T10:05:58.345986+0000","last_active":"2024-03-04T10:05:59.542432+0000","last_peered":"2024-03-04T10:05:59.542432+0000","last_clean":"2024-03-04T10:05:59.542432+0000","last_became_active":"2024-03-04T10:05:58.345499+0000","last_became_peered":"2024-03-04T10:05:58.345499+0000","last_unstale":"2024-03-04T10:05:59.542432+0000","last_undegraded":"2024-03-04T10:05:59.542432+0000","last_fullsized":"2024-03-04T10:05:59.542432+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:13.685578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.484518+0000","last_change":"2024-03-04T10:05:57.351387+0000","last_active":"2024-03-04T10:05:59.484518+0000","last_peered":"2024-03-04T10:05:59.484518+0000","last_clean":"2024-03-04T10:05:59.484518+0000","last_became_active":"2024-03-04T10:05:57.351012+0000","last_became_peered":"2024-03-04T10:05:57.351012+0000","last_unstale":"2024-03-04T10:05:59.484518+0000","last_undegraded":"2024-03-04T10:05:59.484518+0000","last_fullsized":"2024-03-04T10:05:59.484518+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:17.750034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.678251+0000","last_change":"2024-03-04T10:05:58.358932+0000","last_active":"2024-03-04T10:05:59.678251+0000","last_peered":"2024-03-04T10:05:59.678251+0000","last_clean":"2024-03-04T10:05:59.678251+0000","last_became_active":"2024-03-04T10:05:58.358232+0000","last_became_peered":"2024-03-04T10:05:58.358232+0000","last_unstale":"2024-03-04T10:05:59.678251+0000","last_undegraded":"2024-03-04T10:05:59.678251+0000","last_fullsized":"2024-03-04T10:05:59.678251+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:57.327086+0000","last_clean_scrub_stamp":"2024-03-04T10:05:57.327086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:11.644158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"52'1","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.350134+0000","last_change":"2024-03-04T10:05:57.354681+0000","last_active":"2024-03-04T10:05:59.350134+0000","last_peered":"2024-03-04T10:05:59.350134+0000","last_clean":"2024-03-04T10:05:59.350134+0000","last_became_active":"2024-03-04T10:05:57.354415+0000","last_became_peered":"2024-03-04T10:05:57.354415+0000","last_unstale":"2024-03-04T10:05:59.350134+0000","last_undegraded":"2024-03-04T10:05:59.350134+0000","last_fullsized":"2024-03-04T10:05:59.350134+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":49,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:05:56.322103+0000","last_clean_scrub_stamp":"2024-03-04T10:05:56.322103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:37:27.637935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":235,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-04T10:05:59.349516+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:05:59.349516+0000","last_peered":"2024-03-04T10:05:59.349516+0000","last_clean":"2024-03-04T10:05:59.349516+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:05:59.349516+0000","last_undegraded":"2024-03-04T10:05:59.349516+0000","last_fullsized":"2024-03-04T10:05:59.349516+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1220608,"data_stored":867913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":278073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":277421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":278003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":634880,"data_stored":279055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":279033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":277581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":867845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:07.012 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T10:06:07.012 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T10:06:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.152+0000 7f5c905f1640 1 Processor -- start 2024-03-04T10:06:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.152+0000 7f5c905f1640 1 -- start start 2024-03-04T10:06:07.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c905f1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5c88158230 0x7f5c880cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c905f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880d1e80 0x7f5c880cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c905f1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c880cc350 con 0x7f5c880cd110 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c905f1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c880cc630 con 0x7f5c88158230 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c905f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880cc910 con 0x7f5c880d1e80 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c8e38d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5c88158230 0x7f5c880cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c8db8c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5c880cd110 0x7f5c880d1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55958/0 (socket says 172.21.15.99:55958) 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c8db8c640 1 -- 172.21.15.99:0/1251749092 learned_addr learned my addr 172.21.15.99:0/1251749092 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.153+0000 7f5c8e38d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5c88158230 0x7f5c880cf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57324/0 (socket says 172.21.15.99:57324) 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8eb8e640 1 --2- 172.21.15.99:0/1251749092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880d1e80 0x7f5c880cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8e38d640 1 -- 172.21.15.99:0/1251749092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880d1e80 msgr2=0x7f5c880cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8e38d640 1 --2- 172.21.15.99:0/1251749092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880d1e80 0x7f5c880cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8e38d640 1 -- 172.21.15.99:0/1251749092 >> v1:172.21.15.99:6789/0 conn(0x7f5c880cd110 legacy=0x7f5c880d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8e38d640 1 -- 172.21.15.99:0/1251749092 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880ccbf0 con 0x7f5c88158230 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8d38b640 1 -- 172.21.15.99:0/1251749092 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811571220 0 0) 0x7f5c880cc350 con 0x7f5c880cd110 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.154+0000 7f5c8e38d640 1 --2- 172.21.15.99:0/1251749092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c88158230 0x7f5c880cf6e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009d90 tx=0x7f5c8402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7739ee291bc8ded server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c8d38b640 1 -- 172.21.15.99:0/1251749092 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84039070 con 0x7f5c88158230 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c8d38b640 1 -- 172.21.15.99:0/1251749092 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8402cc50 con 0x7f5c88158230 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c8d38b640 1 -- 172.21.15.99:0/1251749092 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c84034440 con 0x7f5c88158230 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 -- 172.21.15.99:0/1251749092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c88158230 msgr2=0x7f5c880cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 --2- 172.21.15.99:0/1251749092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c88158230 0x7f5c880cf6e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009d90 tx=0x7f5c8402bea0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 -- 172.21.15.99:0/1251749092 shutdown_connections 2024-03-04T10:06:07.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 --2- 172.21.15.99:0/1251749092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880d1e80 0x7f5c880cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 --2- 172.21.15.99:0/1251749092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c88158230 0x7f5c880cf6e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 -- 172.21.15.99:0/1251749092 >> 172.21.15.99:0/1251749092 conn(0x7f5c880b3330 msgr2=0x7f5c880b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 -- 172.21.15.99:0/1251749092 shutdown_connections 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.155+0000 7f5c905f1640 1 -- 172.21.15.99:0/1251749092 wait complete. 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.156+0000 7f5c905f1640 1 Processor -- start 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.156+0000 7f5c905f1640 1 -- start start 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c905f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880cd110 0x7f5c881529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c905f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 0x7f5c880c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c905f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88158230 0x7f5c8806d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c905f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880cc0f0 con 0x7f5c880d1e80 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c905f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880cbf70 con 0x7f5c88158230 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c905f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880cdc10 con 0x7f5c880cd110 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 0x7f5c880c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 0x7f5c880c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57330/0 (socket says 172.21.15.99:57330) 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 -- 172.21.15.99:0/3508817475 learned_addr learned my addr 172.21.15.99:0/3508817475 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8e38d640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880cd110 0x7f5c881529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 -- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880cd110 msgr2=0x7f5c881529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880cd110 0x7f5c881529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 -- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88158230 msgr2=0x7f5c8806d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8eb8e640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88158230 0x7f5c8806d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88158230 0x7f5c8806d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.157+0000 7f5c8db8c640 1 -- 172.21.15.99:0/3508817475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880ca8a0 con 0x7f5c880d1e80 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.158+0000 7f5c8db8c640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 0x7f5c880c88b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c460 tx=0x7f5c7800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.158+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78018070 con 0x7f5c880d1e80 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.158+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c78007520 con 0x7f5c880d1e80 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.158+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78004a20 con 0x7f5c880d1e80 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.158+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c880cab80 con 0x7f5c880d1e80 2024-03-04T10:06:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.158+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c880c8df0 con 0x7f5c880d1e80 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.159+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c88074f60 con 0x7f5c880d1e80 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.159+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5c78004bc0 con 0x7f5c880d1e80 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.160+0000 7f5c7effd640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.160+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c780895f0 con 0x7f5c880d1e80 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.160+0000 7f5c8e38d640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.160+0000 7f5c8e38d640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5c84036000 tx=0x7f5c84030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.164+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5c78057c90 con 0x7f5c880d1e80 2024-03-04T10:06:07.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.399+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f5c880cb0a0 con 0x7f5c880d1e80 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.402+0000 7f5c7effd640 1 -- 172.21.15.99:0/3508817475 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5c7805bd30 con 0x7f5c880d1e80 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stdout: caps mds = "allow" 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stdout: caps mon = "allow r" 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T10:06:07.404 INFO:teuthology.orchestra.run.smithi099.stderr:updated caps for client.0 2024-03-04T10:06:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c6c06b4d0 msgr2=0x7f5c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5c84036000 tx=0x7f5c84030000 comp rx=0 tx=0).stop 2024-03-04T10:06:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 msgr2=0x7f5c880c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 0x7f5c880c88b0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c460 tx=0x7f5c7800c920 comp rx=0 tx=0).stop 2024-03-04T10:06:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 shutdown_connections 2024-03-04T10:06:07.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c88158230 0x7f5c8806d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c880d1e80 0x7f5c880c88b0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 --2- 172.21.15.99:0/3508817475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c880cd110 0x7f5c881529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 >> 172.21.15.99:0/3508817475 conn(0x7f5c880b3330 msgr2=0x7f5c88154cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:07.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 shutdown_connections 2024-03-04T10:06:07.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.404+0000 7f5c905f1640 1 -- 172.21.15.99:0/3508817475 wait complete. 2024-03-04T10:06:07.417 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.515+0000 7fcbfb4fc640 1 Processor -- start 2024-03-04T10:06:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.515+0000 7fcbfb4fc640 1 -- start start 2024-03-04T10:06:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbfb4fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcbf415dea0 0x7fcbf40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbfb4fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf40cc8d0 0x7fcbf406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbfb4fc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcbf40d1200 0x7fcbf406b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbfb4fc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf406fad0 con 0x7fcbf415dea0 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbfb4fc640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf406fd20 con 0x7fcbf40d1200 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbfb4fc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf4070000 con 0x7fcbf40cc8d0 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcbf415dea0 0x7fcbf40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcbf415dea0 0x7fcbf40cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57346/0 (socket says 172.21.15.99:57346) 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 -- 172.21.15.99:0/1836902607 learned_addr learned my addr 172.21.15.99:0/1836902607 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 -- 172.21.15.99:0/1836902607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf40cc8d0 msgr2=0x7fcbf406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 --2- 172.21.15.99:0/1836902607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf40cc8d0 0x7fcbf406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 -- 172.21.15.99:0/1836902607 >> v2:172.21.15.162:3300/0 conn(0x7fcbf40d1200 msgr2=0x7fcbf406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 --2- 172.21.15.99:0/1836902607 >> v2:172.21.15.162:3300/0 conn(0x7fcbf40d1200 0x7fcbf406b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.516+0000 7fcbf9298640 1 -- 172.21.15.99:0/1836902607 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf40702e0 con 0x7fcbf415dea0 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbf9298640 1 --2- 172.21.15.99:0/1836902607 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf415dea0 0x7fcbf40cad60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40b0120 tx=0x7fcbf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55fcd1913a975d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbebfff640 1 -- 172.21.15.99:0/1836902607 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0035020 con 0x7fcbf415dea0 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbebfff640 1 -- 172.21.15.99:0/1836902607 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf0031870 con 0x7fcbf415dea0 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbebfff640 1 -- 172.21.15.99:0/1836902607 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf0031c40 con 0x7fcbf415dea0 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/1836902607 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf415dea0 msgr2=0x7fcbf40cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/1836902607 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf415dea0 0x7fcbf40cad60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40b0120 tx=0x7fcbf002b910 comp rx=0 tx=0).stop 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/1836902607 shutdown_connections 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/1836902607 >> v2:172.21.15.162:3300/0 conn(0x7fcbf40d1200 0x7fcbf406b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/1836902607 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf40cc8d0 0x7fcbf406acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/1836902607 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf415dea0 0x7fcbf40cad60 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.517+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/1836902607 >> 172.21.15.99:0/1836902607 conn(0x7fcbf40d0710 msgr2=0x7fcbf40d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.518+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/1836902607 shutdown_connections 2024-03-04T10:06:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.518+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/1836902607 wait complete. 2024-03-04T10:06:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 Processor -- start 2024-03-04T10:06:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 -- start start 2024-03-04T10:06:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbf40cc8d0 0x7fcbf4159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 0x7fcbf40c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf415dea0 0x7fcbf41f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40cd480 con 0x7fcbf40d1200 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40cd300 con 0x7fcbf40cc8d0 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbfb4fc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf40cd600 con 0x7fcbf415dea0 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbf8a97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 0x7fcbf40c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbf8a97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 0x7fcbf40c6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57348/0 (socket says 172.21.15.99:57348) 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.519+0000 7fcbf8a97640 1 -- 172.21.15.99:0/2145184138 learned_addr learned my addr 172.21.15.99:0/2145184138 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf9298640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbf40cc8d0 0x7fcbf4159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf9a99640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf415dea0 0x7fcbf41f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf8a97640 1 -- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf415dea0 msgr2=0x7fcbf41f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf8a97640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf415dea0 0x7fcbf41f7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf8a97640 1 -- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbf40cc8d0 msgr2=0x7fcbf4159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf8a97640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbf40cc8d0 0x7fcbf4159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf8a97640 1 -- 172.21.15.99:0/2145184138 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf415a280 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf9298640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbf40cc8d0 0x7fcbf4159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbf8a97640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 0x7fcbf40c6b40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b4a0 tx=0x7fcbe400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4014070 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe4010610 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf40c4370 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4010a00 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.520+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf40c4930 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.521+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf4074f60 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.521+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcbe4017070 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.522+0000 7fcbe9ffb640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbd806b4d0 0x7fcbd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.522+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcbe4089c50 con 0x7fcbf40d1200 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.522+0000 7fcbf9298640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbd806b4d0 0x7fcbd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.523+0000 7fcbf9298640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbd806b4d0 0x7fcbd806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0036000 tx=0x7fcbf003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.526+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcbe408f020 con 0x7fcbf40d1200 2024-03-04T10:06:07.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.730+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbf40cb850 con 0x7fcbf40d1200 2024-03-04T10:06:07.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.730+0000 7fcbe9ffb640 1 -- 172.21.15.99:0/2145184138 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fcbe4058380 con 0x7fcbf40d1200 2024-03-04T10:06:07.733 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:07.733 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:00.693772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5961},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5961":{"gid":5961,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6833/370337902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":370337902},{"type":"v1","addr":"172.21.15.99:6833","nonce":370337902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:07.733 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 30 2024-03-04T10:06:07.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbd806b4d0 msgr2=0x7fcbd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbd806b4d0 0x7fcbd806d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0036000 tx=0x7fcbf003c040 comp rx=0 tx=0).stop 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 msgr2=0x7fcbf40c6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 0x7fcbf40c6b40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b4a0 tx=0x7fcbe400b960 comp rx=0 tx=0).stop 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 shutdown_connections 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbd806b4d0 0x7fcbd806d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcbf415dea0 0x7fcbf41f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcbf40d1200 0x7fcbf40c6b40 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 --2- 172.21.15.99:0/2145184138 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcbf40cc8d0 0x7fcbf4159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 >> 172.21.15.99:0/2145184138 conn(0x7fcbf40d0710 msgr2=0x7fcbf40cbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 shutdown_connections 2024-03-04T10:06:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:07.732+0000 7fcbfb4fc640 1 -- 172.21.15.99:0/2145184138 wait complete. 2024-03-04T10:06:08.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:05:57.681103+0000', 'modified': '2024-03-04T10:06:00.693772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5961}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5961': {'gid': 5961, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6833/370337902', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 370337902}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 370337902}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:06:08.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:06:08.148 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T10:06:08.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.250+0000 7f1b3c516640 1 Processor -- start 2024-03-04T10:06:08.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.250+0000 7f1b3c516640 1 -- start start 2024-03-04T10:06:08.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3c516640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1b3415b960 0x7f1b340d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3c516640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340ccad0 con 0x7f1b340b0e10 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3c516640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340cfab0 con 0x7f1b340d0ad0 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3c516640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b340cf860 con 0x7f1b3415b960 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3a2b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1b3415b960 0x7f1b340d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3aab3640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1b340b0e10 0x7f1b340b47c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55966/0 (socket says 172.21.15.99:55966) 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3a2b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1b3415b960 0x7f1b340d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57356/0 (socket says 172.21.15.99:57356) 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3aab3640 1 -- 172.21.15.99:0/3909158042 learned_addr learned my addr 172.21.15.99:0/3909158042 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3909158042 >> v1:172.21.15.162:6789/0 conn(0x7f1b340d0ad0 legacy=0x7f1b340b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3909158042 >> v1:172.21.15.99:6789/0 conn(0x7f1b340b0e10 legacy=0x7f1b340b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3909158042 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b340cfd90 con 0x7f1b3415b960 2024-03-04T10:06:08.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.251+0000 7f1b392b0640 1 -- 172.21.15.99:0/3909158042 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265921701 0 0) 0x7f1b340ccad0 con 0x7f1b340b0e10 2024-03-04T10:06:08.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.252+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3909158042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b3415b960 0x7f1b340d0590 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1b340cf9e0 tx=0x7f1b3002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8369d0e9227ff0b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.252+0000 7f1b392b0640 1 -- 172.21.15.99:0/3909158042 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30035020 con 0x7f1b3415b960 2024-03-04T10:06:08.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.252+0000 7f1b392b0640 1 -- 172.21.15.99:0/3909158042 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b30031830 con 0x7f1b3415b960 2024-03-04T10:06:08.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.252+0000 7f1b392b0640 1 -- 172.21.15.99:0/3909158042 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30031c00 con 0x7f1b3415b960 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 -- 172.21.15.99:0/3909158042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b3415b960 msgr2=0x7f1b340d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3909158042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b3415b960 0x7f1b340d0590 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f1b340cf9e0 tx=0x7f1b3002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 -- 172.21.15.99:0/3909158042 shutdown_connections 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3909158042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b3415b960 0x7f1b340d0590 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 -- 172.21.15.99:0/3909158042 >> 172.21.15.99:0/3909158042 conn(0x7f1b3406f5f0 msgr2=0x7f1b3406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 -- 172.21.15.99:0/3909158042 shutdown_connections 2024-03-04T10:06:08.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.253+0000 7f1b3c516640 1 -- 172.21.15.99:0/3909158042 wait complete. 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.254+0000 7f1b3c516640 1 Processor -- start 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.254+0000 7f1b3c516640 1 -- start start 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.254+0000 7f1b3c516640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 0x7f1b341f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b340d0ad0 0x7f1b340a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b3415b960 0x7f1b340a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340b4ed0 con 0x7f1b340b0e10 2024-03-04T10:06:08.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340d0100 con 0x7f1b3415b960 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340b5050 con 0x7f1b340d0ad0 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 0x7f1b341f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 0x7f1b341f0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57362/0 (socket says 172.21.15.99:57362) 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3371333008 learned_addr learned my addr 172.21.15.99:0/3371333008 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b340d0ad0 0x7f1b340a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b3415b960 0x7f1b340a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b340d0ad0 msgr2=0x7f1b340a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b340d0ad0 0x7f1b340a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b3415b960 msgr2=0x7f1b340a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b3415b960 0x7f1b340a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/3371333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b340caca0 con 0x7f1b340b0e10 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b340d0ad0 0x7f1b340a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b3415b960 0x7f1b340a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:08.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 0x7f1b341f0a60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009890 tx=0x7f1b300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b2affd640 1 -- 172.21.15.99:0/3371333008 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30041070 con 0x7f1b340b0e10 2024-03-04T10:06:08.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b2affd640 1 -- 172.21.15.99:0/3371333008 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3003a950 con 0x7f1b340b0e10 2024-03-04T10:06:08.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b340c9f60 con 0x7f1b340b0e10 2024-03-04T10:06:08.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b2affd640 1 -- 172.21.15.99:0/3371333008 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30035040 con 0x7f1b340b0e10 2024-03-04T10:06:08.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.255+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b340cb1f0 con 0x7f1b340b0e10 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.257+0000 7f1b2affd640 1 -- 172.21.15.99:0/3371333008 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1b30038020 con 0x7f1b340b0e10 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.257+0000 7f1b2affd640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1b1806b4d0 0x7f1b1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.257+0000 7f1b2affd640 1 -- 172.21.15.99:0/3371333008 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b300abe50 con 0x7f1b340b0e10 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.257+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1b1806b4d0 0x7f1b1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.258+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1b1806b4d0 0x7f1b1806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1b340cab90 tx=0x7f1b24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.258+0000 7f1b3f1d0740 1 Processor -- start 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.258+0000 7f1b3f1d0740 1 -- start start 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3596ea0 0x561ed3569e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 0x561ed359a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ed34d47c0 con 0x561ed356a380 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ed347db50 con 0x561ed3596ea0 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ed34d0860 con 0x561ed3597280 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 0x561ed359a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 0x561ed359a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57368/0 (socket says 172.21.15.99:57368) 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 -- 172.21.15.99:0/1322311071 learned_addr learned my addr 172.21.15.99:0/1322311071 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3596ea0 0x561ed3569e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 -- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 msgr2=0x561ed3599c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 -- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3596ea0 msgr2=0x561ed3569e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3596ea0 0x561ed3569e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.259+0000 7f1b3aab3640 1 -- 172.21.15.99:0/1322311071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ed34919a0 con 0x561ed356a380 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 0x561ed359a170 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c001e10 tx=0x7f1b2c00d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70186967e5006b4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b17fff640 1 -- 172.21.15.99:0/1322311071 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00ddd0 con 0x561ed356a380 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b17fff640 1 -- 172.21.15.99:0/1322311071 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c004b60 con 0x561ed356a380 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b17fff640 1 -- 172.21.15.99:0/1322311071 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0070d0 con 0x561ed356a380 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/1322311071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 msgr2=0x561ed359a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 0x561ed359a170 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c001e10 tx=0x7f1b2c00d520 comp rx=0 tx=0).stop 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/1322311071 shutdown_connections 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed356a380 0x561ed359a170 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/1322311071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3596ea0 0x561ed3569e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.260+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/1322311071 >> 172.21.15.99:0/1322311071 conn(0x561ed356c0b0 msgr2=0x561ed35b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.261+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/1322311071 shutdown_connections 2024-03-04T10:06:08.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.261+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/1322311071 wait complete. 2024-03-04T10:06:08.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.261+0000 7f1b3f1d0740 1 -- start start 2024-03-04T10:06:08.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.261+0000 7f1b3f1d0740 1 Processor -- start 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed358a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3597280 0x561ed35a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 0x561ed35c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ed356e750 con 0x561ed359bab0 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ed3490f90 con 0x561ed3597280 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ed3570f90 con 0x561ed3596ea0 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3f1d0740 10 client.0 _ll_register_callbacks cb 0x561ed34b2d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:08.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 0x561ed35c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 0x561ed35c45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57376/0 (socket says 172.21.15.99:57376) 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 -- 172.21.15.99:0/3945905492 learned_addr learned my addr 172.21.15.99:0/3945905492 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed358a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3597280 0x561ed35a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 -- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 msgr2=0x561ed358a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed358a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 -- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3597280 msgr2=0x561ed35a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3597280 0x561ed35a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.262+0000 7f1b3aab3640 1 -- 172.21.15.99:0/3945905492 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ed31e92d0 con 0x561ed359bab0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 0x561ed35c45e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009010 tx=0x7f1b2c007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e163fe95537fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b157fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b157fa640 1 -- 172.21.15.99:0/3945905492 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c023030 con 0x561ed359bab0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b157fa640 1 -- 172.21.15.99:0/3945905492 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b2c01a040 con 0x561ed359bab0 2024-03-04T10:06:08.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b2c004b60 con 0x561ed359bab0 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b157fa640 1 -- 172.21.15.99:0/3945905492 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b2c02c050 con 0x561ed359bab0 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 10 client.6144 fetch_fsmap learned FSMap version 30 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x561ed32b36b0 con 0x561ed359bab0 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b157fa640 1 -- 172.21.15.99:0/3945905492 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c004b60 con 0x561ed359bab0 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b157fa640 1 -- 172.21.15.99:0/3945905492 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1b2c02f020 con 0x561ed359bab0 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 10 client.6144 fetch_fsmap finished waiting for FSMap version 30 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 10 client.6144 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 20 client.6144 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed359ee80 0x561ed359f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 4 client.6144 mds_command: new command op to 5961 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 --> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ed34e1d00 con 0x561ed359ee80 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.263+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed359ee80 0x561ed359f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b157fa640 10 client.6144 ms_handle_connect on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:08.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed359ee80 0x561ed359f260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ba54ab8b30e719 server_cookie=fc72fc9bd084ae2e in_seq=0 out_seq=0 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b157fa640 1 -- 172.21.15.99:0/3945905492 <== mds.0 v2:172.21.15.99:6832/370337902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561ed34e1d00 con 0x561ed359ee80 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b157fa640 10 client.6144 handle_command_reply: tid=0 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 client.6144 shutdown 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 msgr2=0x561ed35c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 0x561ed35c45e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009010 tx=0x7f1b2c007cc0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 shutdown_connections 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed359ee80 0x561ed359f260 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed359bab0 0x561ed35c45e0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed3597280 0x561ed35a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b157fa640 0 client.6144 ms_handle_reset on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/3945905492 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed358a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 >> 172.21.15.99:0/3945905492 conn(0x561ed356a380 msgr2=0x561ed35b7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 shutdown_connections 2024-03-04T10:06:08.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.264+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/3945905492 wait complete. 2024-03-04T10:06:08.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.265+0000 7f1b3f1d0740 20 client.6144 trim_cache size 0 max 16384 2024-03-04T10:06:08.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.280+0000 7f1b3f1d0740 1 Processor -- start 2024-03-04T10:06:08.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.280+0000 7f1b3f1d0740 1 -- start start 2024-03-04T10:06:08.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed359d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 0x561ed356eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed356f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b341fd450 con 0x561ed3597280 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b340cc860 con 0x561ed359bab0 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b341fce80 con 0x561ed3596ea0 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 0x561ed356eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 0x561ed356eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57390/0 (socket says 172.21.15.99:57390) 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 -- 172.21.15.99:0/433591530 learned_addr learned my addr 172.21.15.99:0/433591530 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed359d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed356f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 -- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 msgr2=0x561ed359d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed359d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 -- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 msgr2=0x561ed356f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed356f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 -- 172.21.15.99:0/433591530 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ed359ca50 con 0x561ed3597280 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 0x561ed356eb00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1b240054d0 tx=0x7f1b24013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351e17f3c0f58251 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b177fe640 1 -- 172.21.15.99:0/433591530 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24007440 con 0x561ed3597280 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b177fe640 1 -- 172.21.15.99:0/433591530 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b240075e0 con 0x561ed3597280 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.281+0000 7f1b177fe640 1 -- 172.21.15.99:0/433591530 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240069d0 con 0x561ed3597280 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/433591530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 msgr2=0x561ed356eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 0x561ed356eb00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1b240054d0 tx=0x7f1b24013000 comp rx=0 tx=0).stop 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/433591530 shutdown_connections 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed356f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3597280 0x561ed356eb00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/433591530 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3596ea0 0x561ed359d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/433591530 >> 172.21.15.99:0/433591530 conn(0x561ed356a380 msgr2=0x561ed356ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/433591530 shutdown_connections 2024-03-04T10:06:08.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/433591530 wait complete. 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.282+0000 7f1b3f1d0740 1 -- start start 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 Processor -- start 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 0x561ed3599670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed359a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ed359fba0 con 0x561ed3596ea0 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ed359c770 con 0x561ed359bab0 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.283+0000 7f1b3f1d0740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ed35c8e60 con 0x561ed3597280 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3f1d0740 10 client.0 _ll_register_callbacks cb 0x7f1b2c0015c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 0x561ed3599670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 0x561ed3599670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57402/0 (socket says 172.21.15.99:57402) 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/318284800 learned_addr learned my addr 172.21.15.99:0/318284800 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed359a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 msgr2=0x561ed3599bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 msgr2=0x561ed359a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed359a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 -- 172.21.15.99:0/318284800 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ed35cca60 con 0x561ed3596ea0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3aab3640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed359a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3a2b2640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 0x561ed3599670 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1b3002b910 tx=0x7f1b30076810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219304a3efdfa77f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b157fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b157fa640 1 -- 172.21.15.99:0/318284800 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30076ab0 con 0x561ed3596ea0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b157fa640 1 -- 172.21.15.99:0/318284800 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b3003ad00 con 0x561ed3596ea0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b240075e0 con 0x561ed3596ea0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b157fa640 1 -- 172.21.15.99:0/318284800 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b30076c50 con 0x561ed3596ea0 2024-03-04T10:06:08.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3f1d0740 10 client.6150 fetch_fsmap learned FSMap version 30 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x561ed35ce520 con 0x561ed3596ea0 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b157fa640 1 -- 172.21.15.99:0/318284800 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b240075e0 con 0x561ed3596ea0 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.284+0000 7f1b157fa640 1 -- 172.21.15.99:0/318284800 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1b300826c0 con 0x561ed3596ea0 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 10 client.6150 fetch_fsmap finished waiting for FSMap version 30 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 10 client.6150 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 20 client.6150 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed35c7b60 0x561ed3624740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 4 client.6150 mds_command: new command op to 5961 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 --> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561ed34dc050 con 0x561ed35c7b60 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed35c7b60 0x561ed3624740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b39ab1640 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed35c7b60 0x561ed3624740 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=748c6d8d8ca8d34 server_cookie=a567a5a6d9bccc7b in_seq=0 out_seq=0 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b157fa640 10 client.6150 ms_handle_connect on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b157fa640 1 -- 172.21.15.99:0/318284800 <== mds.0 v2:172.21.15.99:6832/370337902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x561ed34dc050 con 0x561ed35c7b60 2024-03-04T10:06:08.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b157fa640 10 client.6150 handle_command_reply: tid=0 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 client.6150 shutdown 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 msgr2=0x561ed3599670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 0x561ed3599670 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1b3002b910 tx=0x7f1b30076810 comp rx=0 tx=0).stop 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 shutdown_connections 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x561ed35c7b60 0x561ed3624740 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561ed359bab0 0x561ed359a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561ed3597280 0x561ed3599bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 --2- 172.21.15.99:0/318284800 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561ed3596ea0 0x561ed3599670 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b157fa640 0 client.6150 ms_handle_reset on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 >> 172.21.15.99:0/318284800 conn(0x561ed356a380 msgr2=0x561ed3568db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 shutdown_connections 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.285+0000 7f1b3f1d0740 1 -- 172.21.15.99:0/318284800 wait complete. 2024-03-04T10:06:08.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3f1d0740 20 client.6150 trim_cache size 0 max 16384 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 5961, 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 9.9423006399999991, 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 30, 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 52, 2024-03-04T10:06:08.296 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 52, 2024-03-04T10:06:08.297 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 14.986674124 2024-03-04T10:06:08.297 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:08.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1b1806b4d0 msgr2=0x7f1b1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1b1806b4d0 0x7f1b1806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1b340cab90 tx=0x7f1b24009040 comp rx=0 tx=0).stop 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 msgr2=0x7f1b341f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 0x7f1b341f0a60 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009890 tx=0x7f1b300046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 shutdown_connections 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1b1806b4d0 0x7f1b1806d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1b3415b960 0x7f1b340a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1b340d0ad0 0x7f1b340a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 --2- 172.21.15.99:0/3371333008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1b340b0e10 0x7f1b341f0a60 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 >> 172.21.15.99:0/3371333008 conn(0x7f1b3406f5f0 msgr2=0x7f1b340b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 shutdown_connections 2024-03-04T10:06:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.286+0000 7f1b3c516640 1 -- 172.21.15.99:0/3371333008 wait complete. 2024-03-04T10:06:08.299 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:06:08.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 Processor -- start 2024-03-04T10:06:08.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 -- start start 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efff80cfd60 0x7efff80cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7efff80cf0d0 0x7efff80cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efff80d0140 con 0x7efff806d3b0 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff815cbb0 con 0x7efff80cfd60 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.397+0000 7efffcbb2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efff815c280 con 0x7efff80cf0d0 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efff80cfd60 0x7efff80cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7efff80cfd60 0x7efff80cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57408/0 (socket says 172.21.15.99:57408) 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 -- 172.21.15.99:0/2728135736 learned_addr learned my addr 172.21.15.99:0/2728135736 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff6d9d640 1 --2- 172.21.15.99:0/2728135736 >> v2:172.21.15.162:3300/0 conn(0x7efff80cf0d0 0x7efff80cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 -- 172.21.15.99:0/2728135736 >> v2:172.21.15.162:3300/0 conn(0x7efff80cf0d0 msgr2=0x7efff80cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 --2- 172.21.15.99:0/2728135736 >> v2:172.21.15.162:3300/0 conn(0x7efff80cf0d0 0x7efff80cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 -- 172.21.15.99:0/2728135736 >> v1:172.21.15.162:6789/0 conn(0x7efff806d3b0 legacy=0x7efff80ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.398+0000 7efff659c640 1 -- 172.21.15.99:0/2728135736 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff815c560 con 0x7efff80cfd60 2024-03-04T10:06:08.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efff659c640 1 --2- 172.21.15.99:0/2728135736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff80cbdf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efff815cd30 tx=0x7effe802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa97467cb8ce91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efff5d9b640 1 -- 172.21.15.99:0/2728135736 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8034070 con 0x7efff80cfd60 2024-03-04T10:06:08.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efff5d9b640 1 -- 172.21.15.99:0/2728135736 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe8030860 con 0x7efff80cfd60 2024-03-04T10:06:08.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efff5d9b640 1 -- 172.21.15.99:0/2728135736 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe8030c30 con 0x7efff80cfd60 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 -- 172.21.15.99:0/2728135736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 msgr2=0x7efff80cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 --2- 172.21.15.99:0/2728135736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff80cbdf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7efff815cd30 tx=0x7effe802b9a0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 -- 172.21.15.99:0/2728135736 shutdown_connections 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 --2- 172.21.15.99:0/2728135736 >> v2:172.21.15.162:3300/0 conn(0x7efff80cf0d0 0x7efff80cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 --2- 172.21.15.99:0/2728135736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff80cbdf0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 -- 172.21.15.99:0/2728135736 >> 172.21.15.99:0/2728135736 conn(0x7efff806acd0 msgr2=0x7efff806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 -- 172.21.15.99:0/2728135736 shutdown_connections 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.399+0000 7efffcbb2640 1 -- 172.21.15.99:0/2728135736 wait complete. 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 Processor -- start 2024-03-04T10:06:08.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 -- start start 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff806d3b0 0x7efff8152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efff80cf0d0 0x7efff815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff815e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff815f520 con 0x7efff80cfd60 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff815bf30 con 0x7efff806d3b0 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efffcbb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff815f6a0 con 0x7efff80cf0d0 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.401+0000 7efff6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff815e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff815e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57410/0 (socket says 172.21.15.99:57410) 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 -- 172.21.15.99:0/1079812450 learned_addr learned my addr 172.21.15.99:0/1079812450 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff659c640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff806d3b0 0x7efff8152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7effeffff640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efff80cf0d0 0x7efff815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 -- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efff80cf0d0 msgr2=0x7efff815e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efff80cf0d0 0x7efff815e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 -- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff806d3b0 msgr2=0x7efff8152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff806d3b0 0x7efff8152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 -- 172.21.15.99:0/1079812450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff81574a0 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff659c640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff806d3b0 0x7efff8152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efff6d9d640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff815e5f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efff000efc0 tx=0x7efff000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0018020 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0014480 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efff8156840 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0014870 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.402+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff8152cd0 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.403+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efff8074f60 con 0x7efff80cfd60 2024-03-04T10:06:08.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.404+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7efff0005d10 con 0x7efff80cfd60 2024-03-04T10:06:08.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.404+0000 7effeeffd640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7effd806b500 0x7effd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:08.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.405+0000 7efff659c640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7effd806b500 0x7effd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:08.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.405+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efff0057550 con 0x7efff80cfd60 2024-03-04T10:06:08.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.405+0000 7efff659c640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7effd806b500 0x7effd806d9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7effe8036000 tx=0x7effe803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:08.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.409+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7efff0022070 con 0x7efff80cfd60 2024-03-04T10:06:08.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.617+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efff80cc290 con 0x7efff80cfd60 2024-03-04T10:06:08.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.618+0000 7effeeffd640 1 -- 172.21.15.99:0/1079812450 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) v1 ==== 53+0+73 (secure 0 0 0) 0x7efff0010040 con 0x7efff80cfd60 2024-03-04T10:06:08.620 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:06:08.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7effd806b500 msgr2=0x7effd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7effd806b500 0x7effd806d9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7effe8036000 tx=0x7effe803b040 comp rx=0 tx=0).stop 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 msgr2=0x7efff815e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff815e5f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efff000efc0 tx=0x7efff000c490 comp rx=0 tx=0).stop 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 shutdown_connections 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7effd806b500 0x7effd806d9c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7efff80cfd60 0x7efff815e5f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7efff80cf0d0 0x7efff815e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 --2- 172.21.15.99:0/1079812450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efff806d3b0 0x7efff8152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 >> 172.21.15.99:0/1079812450 conn(0x7efff806acd0 msgr2=0x7efff806ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 shutdown_connections 2024-03-04T10:06:08.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:08.620+0000 7efffcbb2640 1 -- 172.21.15.99:0/1079812450 wait complete. 2024-03-04T10:06:08.633 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:06:08.634 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:08.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:08.634 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:08.650 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42617sec preferred_lft 42617sec 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.651 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:06:08.652 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:06:08.653 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:08.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:08.653 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:06:08.716 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:06:08.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:08.717 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:06:08.772 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:06:08.773 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:06:08.773 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:06:08.773 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:06:08.773 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:06:08.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:08.773 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:06:08.773 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:06:08.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:06:08.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:06:08.773 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:06:08.773 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:06:08.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:08.868 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:06:08.882 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:06:08.882 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:06:08.882 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:08.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:08.883 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:06:08.949 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:06:08.949 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:09.003 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:09.004 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:06:09.004 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:09.058 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:06:09.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:09.058 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:09.141 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:09.141 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:06:09.148 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:09.148 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:06:09.148 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:06:09.148 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:06:09.148 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:06:09.149 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:06:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:06:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:06:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:06:09.149 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:06:09.151 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:06:09.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:09.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:09.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:09.218 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:06:09.277 INFO:teuthology.orchestra.run.smithi099.stdout:client6159 2024-03-04T10:06:09.278 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:09.278 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:09.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:09.333 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mkdir -p a/b') 2024-03-04T10:06:09.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:09.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:09.397 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T10:06:09.397 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-03-04T10:06:09.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.562+0000 7fa5e7505640 1 Processor -- start 2024-03-04T10:06:09.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e7505640 1 -- start start 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e7505640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e006b700 0x7fa5e006fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e7505640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa5e00cfa40 0x7fa5e00cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e7505640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e006bb60 con 0x7fa5e015df00 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e7505640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e00701d0 con 0x7fa5e00cfa40 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e7505640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00704b0 con 0x7fa5e006b700 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.563+0000 7fa5e5aa2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa5e00cfa40 0x7fa5e00cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e52a1640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa5e015df00 0x7fa5e00cad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55982/0 (socket says 172.21.15.99:55982) 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e52a1640 1 -- 172.21.15.99:0/3376787440 learned_addr learned my addr 172.21.15.99:0/3376787440 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3376787440 >> v2:172.21.15.99:3300/0 conn(0x7fa5e00cfa40 0x7fa5e00cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57424/0 (socket says 172.21.15.99:57424) 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3376787440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e006b700 msgr2=0x7fa5e006fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e006b700 0x7fa5e006fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5d7fff640 1 -- 172.21.15.99:0/3376787440 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345786191 0 0) 0x7fa5e006bb60 con 0x7fa5e015df00 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e006b700 0x7fa5e006fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3376787440 >> v1:172.21.15.99:6789/0 conn(0x7fa5e015df00 legacy=0x7fa5e00cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3376787440 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00cc8e0 con 0x7fa5e00cfa40 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.564+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e006b700 0x7fa5e006fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e00cfa40 0x7fa5e00cfed0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80097a0 tx=0x7fa5d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee61059f8df00542 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5d7fff640 1 -- 172.21.15.99:0/3376787440 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8035020 con 0x7fa5e00cfa40 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5d7fff640 1 -- 172.21.15.99:0/3376787440 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8031830 con 0x7fa5e00cfa40 2024-03-04T10:06:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5d7fff640 1 -- 172.21.15.99:0/3376787440 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8031c00 con 0x7fa5e00cfa40 2024-03-04T10:06:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 -- 172.21.15.99:0/3376787440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e00cfa40 msgr2=0x7fa5e00cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e00cfa40 0x7fa5e00cfed0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80097a0 tx=0x7fa5d802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 -- 172.21.15.99:0/3376787440 shutdown_connections 2024-03-04T10:06:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e00cfa40 0x7fa5e00cfed0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3376787440 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e006b700 0x7fa5e006fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 -- 172.21.15.99:0/3376787440 >> 172.21.15.99:0/3376787440 conn(0x7fa5e00b2250 msgr2=0x7fa5e00b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 -- 172.21.15.99:0/3376787440 shutdown_connections 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.565+0000 7fa5e7505640 1 -- 172.21.15.99:0/3376787440 wait complete. 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.566+0000 7fa5e7505640 1 Processor -- start 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 -- start start 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e006b700 0x7fa5e00b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e00cfa40 0x7fa5e014cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 0x7fa5e00b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00d0410 con 0x7fa5e015df00 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00ccea0 con 0x7fa5e006b700 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e7505640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00cd3c0 con 0x7fa5e00cfa40 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e5aa2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 0x7fa5e00b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e5aa2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 0x7fa5e00b5b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57428/0 (socket says 172.21.15.99:57428) 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.567+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3260046750 learned_addr learned my addr 172.21.15.99:0/3260046750 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e00cfa40 msgr2=0x7fa5e014cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e006b700 0x7fa5e00b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e00cfa40 0x7fa5e014cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e006b700 msgr2=0x7fa5e00b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e006b700 0x7fa5e00b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3260046750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00b60a0 con 0x7fa5e015df00 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e006b700 0x7fa5e00b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 0x7fa5e00b5b60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80098c0 tx=0x7fa5d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5d5ffb640 1 -- 172.21.15.99:0/3260046750 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8041070 con 0x7fa5e015df00 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5d5ffb640 1 -- 172.21.15.99:0/3260046750 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d803a8d0 con 0x7fa5e015df00 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e0159340 con 0x7fa5e015df00 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5d5ffb640 1 -- 172.21.15.99:0/3260046750 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8035040 con 0x7fa5e015df00 2024-03-04T10:06:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.568+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e00c52b0 con 0x7fa5e015df00 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.570+0000 7fa5d5ffb640 1 -- 172.21.15.99:0/3260046750 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa5d8038020 con 0x7fa5e015df00 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.570+0000 7fa5d5ffb640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa5c406b4d0 0x7fa5c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.570+0000 7fa5d5ffb640 1 -- 172.21.15.99:0/3260046750 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa5d80abdb0 con 0x7fa5e015df00 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.570+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa5c406b4d0 0x7fa5c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.571+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa5c406b4d0 0x7fa5c406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc007fd0 tx=0x7fa5dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.571+0000 7fa5ea1b7740 1 Processor -- start 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.571+0000 7fa5ea1b7740 1 -- start start 2024-03-04T10:06:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655296ff800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x56552972f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655296ffd40 0x56552972fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x565529668e00 con 0x56552972c860 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x565529611b10 con 0x5655296ffd40 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x565529664ea0 con 0x56552972cc40 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655296ff800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655296ff800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57438/0 (socket says 172.21.15.99:57438) 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 -- 172.21.15.99:0/1113342862 learned_addr learned my addr 172.21.15.99:0/1113342862 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655296ffd40 0x56552972fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x56552972f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 -- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 msgr2=0x56552972f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x56552972f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 -- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655296ffd40 msgr2=0x56552972fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655296ffd40 0x56552972fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 -- 172.21.15.99:0/1113342862 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5655294477d0 con 0x56552972c860 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655296ffd40 0x56552972fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x56552972f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.572+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655296ff800 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc005fd0 tx=0x7fa5dc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213964a5f9186db7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5caffd640 1 -- 172.21.15.99:0/1113342862 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc007850 con 0x56552972c860 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5caffd640 1 -- 172.21.15.99:0/1113342862 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc020450 con 0x56552972c860 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5caffd640 1 -- 172.21.15.99:0/1113342862 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc0079f0 con 0x56552972c860 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/1113342862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 msgr2=0x5655296ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655296ff800 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc005fd0 tx=0x7fa5dc010000 comp rx=0 tx=0).stop 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/1113342862 shutdown_connections 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655296ffd40 0x56552972fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x56552972f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/1113342862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655296ff800 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/1113342862 >> 172.21.15.99:0/1113342862 conn(0x565529701a70 msgr2=0x56552974a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/1113342862 shutdown_connections 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.573+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/1113342862 wait complete. 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.574+0000 7fa5ea1b7740 1 -- start start 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.574+0000 7fa5ea1b7740 1 Processor -- start 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655297281a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x5655297324d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655297288c0 0x565529732a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x565529704010 con 0x56552972c860 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655296250a0 con 0x5655297288c0 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x565529703400 con 0x56552972cc40 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 10 client.0 _ll_register_callbacks cb 0x565529624a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655297281a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655297281a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57446/0 (socket says 172.21.15.99:57446) 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 -- 172.21.15.99:0/3004323610 learned_addr learned my addr 172.21.15.99:0/3004323610 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655297288c0 0x565529732a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x5655297324d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 -- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 msgr2=0x5655297324d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x5655297324d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 -- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655297288c0 msgr2=0x565529732a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655297288c0 0x565529732a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 -- 172.21.15.99:0/3004323610 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56552937d3f0 con 0x56552972c860 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655297288c0 0x565529732a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x5655297324d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655297281a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0012e0 tx=0x7fa5dc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230dce4d328fb083 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5bffff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5bffff640 1 -- 172.21.15.99:0/3004323610 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc030060 con 0x56552972c860 2024-03-04T10:06:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5bffff640 1 -- 172.21.15.99:0/3004323610 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa5dc0223a0 con 0x56552972c860 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.575+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5dc020450 con 0x56552972c860 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5bffff640 1 -- 172.21.15.99:0/3004323610 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5dc036050 con 0x56552972c860 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 10 client.6174 fetch_fsmap learned FSMap version 30 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x56552933d0f0 con 0x56552972c860 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5bffff640 1 -- 172.21.15.99:0/3004323610 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc020450 con 0x56552972c860 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5bffff640 1 -- 172.21.15.99:0/3004323610 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa5dc039020 con 0x56552972c860 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 10 client.6174 fetch_fsmap finished waiting for FSMap version 30 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 10 client.6174 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 20 client.6174 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x565529713550 0x565529715ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 4 client.6174 mds_command: new command op to 5961 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 --> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565529676340 con 0x565529713550 2024-03-04T10:06:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x565529713550 0x565529715ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5bffff640 10 client.6174 ms_handle_connect on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.576+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x565529713550 0x565529715ab0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10c57934661b3988 server_cookie=cd294ecaf1460a9d in_seq=0 out_seq=0 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5bffff640 1 -- 172.21.15.99:0/3004323610 <== mds.0 v2:172.21.15.99:6832/370337902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x565529676340 con 0x565529713550 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5bffff640 10 client.6174 handle_command_reply: tid=0 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 client.6174 shutdown 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 msgr2=0x5655297281a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655297281a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0012e0 tx=0x7fa5dc001250 comp rx=0 tx=0).stop 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 shutdown_connections 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x565529713550 0x565529715ab0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5655297288c0 0x565529732a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972cc40 0x5655297324d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5bffff640 0 client.6174 ms_handle_reset on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3004323610 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972c860 0x5655297281a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 >> 172.21.15.99:0/3004323610 conn(0x5655296ffd40 msgr2=0x56552972f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 shutdown_connections 2024-03-04T10:06:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3004323610 wait complete. 2024-03-04T10:06:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.577+0000 7fa5ea1b7740 20 client.6174 trim_cache size 0 max 16384 2024-03-04T10:06:09.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.590+0000 7fa5ea1b7740 1 Processor -- start 2024-03-04T10:06:09.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.590+0000 7fa5ea1b7740 1 -- start start 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 0x565529723fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972c860 0x565529706820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972cc40 0x5655297324b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e02003f0 con 0x5655297288c0 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e01ff5c0 con 0x56552972cc40 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e00c6a20 con 0x56552972c860 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 0x565529723fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 0x565529723fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57458/0 (socket says 172.21.15.99:57458) 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 -- 172.21.15.99:0/2100487079 learned_addr learned my addr 172.21.15.99:0/2100487079 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 -- 172.21.15.99:0/2100487079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972c860 msgr2=0x565529706820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972c860 0x565529706820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 -- 172.21.15.99:0/2100487079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972cc40 msgr2=0x5655297324b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972cc40 0x5655297324b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.591+0000 7fa5e52a1640 1 -- 172.21.15.99:0/2100487079 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5655297278c0 con 0x5655297288c0 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 0x565529723fc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc001540 tx=0x7fa5dc024050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1c061e353c766a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ca7fc640 1 -- 172.21.15.99:0/2100487079 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc03f070 con 0x5655297288c0 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ca7fc640 1 -- 172.21.15.99:0/2100487079 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc022890 con 0x5655297288c0 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ca7fc640 1 -- 172.21.15.99:0/2100487079 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc022e00 con 0x5655297288c0 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/2100487079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 msgr2=0x565529723fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 0x565529723fc0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc001540 tx=0x7fa5dc024050 comp rx=0 tx=0).stop 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/2100487079 shutdown_connections 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972cc40 0x5655297324b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56552972c860 0x565529706820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/2100487079 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5655297288c0 0x565529723fc0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/2100487079 >> 172.21.15.99:0/2100487079 conn(0x5655296ffd40 msgr2=0x565529727fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/2100487079 shutdown_connections 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.592+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/2100487079 wait complete. 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.593+0000 7fa5ea1b7740 1 -- start start 2024-03-04T10:06:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.593+0000 7fa5ea1b7740 1 Processor -- start 2024-03-04T10:06:09.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5655297288c0 0x5655297270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972c860 0x5655297b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 0x5655297323b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56552972c100 con 0x56552972cc40 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e01ff160 con 0x56552972c860 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56552961fe20 con 0x5655297288c0 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 10 client.0 _ll_register_callbacks cb 0x7fa5dc007960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 0x5655297323b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 0x5655297323b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57464/0 (socket says 172.21.15.99:57464) 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3924490395 learned_addr learned my addr 172.21.15.99:0/3924490395 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5655297288c0 0x5655297270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972c860 0x5655297b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5655297288c0 msgr2=0x5655297270c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5655297288c0 0x5655297270c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972c860 msgr2=0x5655297b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972c860 0x5655297b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 -- 172.21.15.99:0/3924490395 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565529724540 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5655297288c0 0x5655297270c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e4aa0640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972c860 0x5655297b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5e5aa2640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 0x5655297323b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009890 tx=0x7fa5d807ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efad92f51a677a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5bffff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5bffff640 1 -- 172.21.15.99:0/3924490395 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d80a8060 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5bffff640 1 -- 172.21.15.99:0/3924490395 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa5d8076a10 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.594+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5dc022890 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5bffff640 1 -- 172.21.15.99:0/3924490395 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5d803aca0 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 10 client.6180 fetch_fsmap learned FSMap version 30 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x565529763e50 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5bffff640 1 -- 172.21.15.99:0/3924490395 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc022890 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5bffff640 1 -- 172.21.15.99:0/3924490395 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa5d80797f0 con 0x56552972cc40 2024-03-04T10:06:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 10 client.6180 fetch_fsmap finished waiting for FSMap version 30 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 10 client.6180 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 20 client.6180 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x5655297328f0 0x5655297b7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 4 client.6180 mds_command: new command op to 5961 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 --> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565529670690 con 0x5655297328f0 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x5655297328f0 0x5655297b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5bffff640 10 client.6180 ms_handle_connect on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:09.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.595+0000 7fa5e52a1640 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x5655297328f0 0x5655297b7180 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae199a596d66727 server_cookie=796e576d90a412f4 in_seq=0 out_seq=0 2024-03-04T10:06:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.606+0000 7fa5bffff640 1 -- 172.21.15.99:0/3924490395 <== mds.0 v2:172.21.15.99:6832/370337902 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565529670690 con 0x5655297328f0 2024-03-04T10:06:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.606+0000 7fa5bffff640 10 client.6180 handle_command_reply: tid=0 2024-03-04T10:06:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.606+0000 7fa5ea1b7740 1 client.6180 shutdown 2024-03-04T10:06:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.606+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 msgr2=0x5655297323b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.606+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 0x5655297323b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009890 tx=0x7fa5d807ac10 comp rx=0 tx=0).stop 2024-03-04T10:06:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 shutdown_connections 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:6832/370337902,v1:172.21.15.99:6833/370337902] conn(0x5655297328f0 0x5655297b7180 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56552972cc40 0x5655297323b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56552972c860 0x5655297b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5bffff640 0 client.6180 ms_handle_reset on v2:172.21.15.99:6832/370337902 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 --2- 172.21.15.99:0/3924490395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5655297288c0 0x5655297270c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 >> 172.21.15.99:0/3924490395 conn(0x5655296ffd40 msgr2=0x565529723b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 shutdown_connections 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 1 -- 172.21.15.99:0/3924490395 wait complete. 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5ea1b7740 20 client.6180 trim_cache size 0 max 16384 2024-03-04T10:06:09.610 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "message": "", 2024-03-04T10:06:09.611 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0 2024-03-04T10:06:09.611 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:09.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa5c406b4d0 msgr2=0x7fa5c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa5c406b4d0 0x7fa5c406d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc007fd0 tx=0x7fa5dc009040 comp rx=0 tx=0).stop 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 msgr2=0x7fa5e00b5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 0x7fa5e00b5b60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80098c0 tx=0x7fa5d80046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 shutdown_connections 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa5c406b4d0 0x7fa5c406d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5e015df00 0x7fa5e00b5b60 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5e00cfa40 0x7fa5e014cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 --2- 172.21.15.99:0/3260046750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5e006b700 0x7fa5e00b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 >> 172.21.15.99:0/3260046750 conn(0x7fa5e00b2250 msgr2=0x7fa5e00ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 shutdown_connections 2024-03-04T10:06:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.607+0000 7fa5e7505640 1 -- 172.21.15.99:0/3260046750 wait complete. 2024-03-04T10:06:09.620 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.721+0000 7fb7acd59640 1 Processor -- start 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb7acd59640 1 -- start start 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb7acd59640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb7a815a810 0x7fb7a80b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb7acd59640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a806d8a0 con 0x7fb7a806d3c0 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb7acd59640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a806db80 con 0x7fb7a80cbfb0 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb7acd59640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a806de60 con 0x7fb7a815a810 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb7a659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb7a815a810 0x7fb7a80b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.722+0000 7fb79ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb7a806d3c0 0x7fb7a815abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:55988/0 (socket says 172.21.15.99:55988) 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.723+0000 7fb79ffff640 1 -- 172.21.15.99:0/50872485 learned_addr learned my addr 172.21.15.99:0/50872485 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.723+0000 7fb7a659c640 1 -- 172.21.15.99:0/50872485 >> v1:172.21.15.162:6789/0 conn(0x7fb7a80cbfb0 legacy=0x7fb7a815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.723+0000 7fb7a5d9b640 1 -- 172.21.15.99:0/50872485 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751290219 0 0) 0x7fb7a806d8a0 con 0x7fb7a806d3c0 2024-03-04T10:06:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.723+0000 7fb7a659c640 1 -- 172.21.15.99:0/50872485 >> v1:172.21.15.99:6789/0 conn(0x7fb7a806d3c0 legacy=0x7fb7a815abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.723+0000 7fb7a659c640 1 -- 172.21.15.99:0/50872485 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a806e140 con 0x7fb7a815a810 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7a659c640 1 --2- 172.21.15.99:0/50872485 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a80b4190 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb7980097a0 tx=0x7fb79802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7f19b504e6e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7a5d9b640 1 -- 172.21.15.99:0/50872485 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798034070 con 0x7fb7a815a810 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7a5d9b640 1 -- 172.21.15.99:0/50872485 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798030610 con 0x7fb7a815a810 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7a5d9b640 1 -- 172.21.15.99:0/50872485 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7980309e0 con 0x7fb7a815a810 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 -- 172.21.15.99:0/50872485 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 msgr2=0x7fb7a80b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 --2- 172.21.15.99:0/50872485 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a80b4190 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fb7980097a0 tx=0x7fb79802b960 comp rx=0 tx=0).stop 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 -- 172.21.15.99:0/50872485 shutdown_connections 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 --2- 172.21.15.99:0/50872485 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a80b4190 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 -- 172.21.15.99:0/50872485 >> 172.21.15.99:0/50872485 conn(0x7fb7a806acd0 msgr2=0x7fb7a806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 -- 172.21.15.99:0/50872485 shutdown_connections 2024-03-04T10:06:09.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.724+0000 7fb7acd59640 1 -- 172.21.15.99:0/50872485 wait complete. 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 Processor -- start 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 -- start start 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7a806d3c0 0x7fb7a81e6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7a80cbfb0 0x7fb7a814df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a8156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80b3a00 con 0x7fb7a815a810 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a815bc60 con 0x7fb7a80cbfb0 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7acd59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80b3b80 con 0x7fb7a806d3c0 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7a6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a8156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.726+0000 7fb7a6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a8156aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57484/0 (socket says 172.21.15.99:57484) 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 -- 172.21.15.99:0/1805631067 learned_addr learned my addr 172.21.15.99:0/1805631067 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a659c640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7a806d3c0 0x7fb7a81e6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb79ffff640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7a80cbfb0 0x7fb7a814df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 -- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7a806d3c0 msgr2=0x7fb7a81e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7a806d3c0 0x7fb7a81e6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 -- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7a80cbfb0 msgr2=0x7fb7a814df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7a80cbfb0 0x7fb7a814df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 -- 172.21.15.99:0/1805631067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a8156fe0 con 0x7fb7a815a810 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a659c640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7a806d3c0 0x7fb7a81e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7a6d9d640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a8156aa0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000c7d0 tx=0x7fb7a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0003b70 con 0x7fb7a815a810 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a0003d30 con 0x7fb7a815a810 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a000aa90 con 0x7fb7a815a810 2024-03-04T10:06:09.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a8157260 con 0x7fb7a815a810 2024-03-04T10:06:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.727+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a8157820 con 0x7fb7a815a810 2024-03-04T10:06:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.728+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a8074f60 con 0x7fb7a815a810 2024-03-04T10:06:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.728+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb7a000ac30 con 0x7fb7a815a810 2024-03-04T10:06:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.729+0000 7fb79effd640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb78806b4d0 0x7fb78806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:09.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.729+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7a008a2c0 con 0x7fb7a815a810 2024-03-04T10:06:09.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.729+0000 7fb7a659c640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb78806b4d0 0x7fb78806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:09.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.730+0000 7fb7a659c640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb78806b4d0 0x7fb78806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb798036000 tx=0x7fb798008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:09.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.733+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb7a8156fe0 con 0x7fb7a815a810 2024-03-04T10:06:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.904+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) v1 -- 0x7fb7a8159f90 con 0x7fb7a815a810 2024-03-04T10:06:09.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.908+0000 7fb79effd640 1 -- 172.21.15.99:0/1805631067 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v13)=0 v13) v1 ==== 135+0+0 (secure 0 0 0) 0x7fb7a0058fb0 con 0x7fb7a815a810 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb78806b4d0 msgr2=0x7fb78806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb78806b4d0 0x7fb78806d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb798036000 tx=0x7fb798008030 comp rx=0 tx=0).stop 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 msgr2=0x7fb7a8156aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a8156aa0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000c7d0 tx=0x7fb7a000cc90 comp rx=0 tx=0).stop 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 shutdown_connections 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb78806b4d0 0x7fb78806d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7a815a810 0x7fb7a8156aa0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7a80cbfb0 0x7fb7a814df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 --2- 172.21.15.99:0/1805631067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7a806d3c0 0x7fb7a81e6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:09.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 >> 172.21.15.99:0/1805631067 conn(0x7fb7a806acd0 msgr2=0x7fb7a80b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:09.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 shutdown_connections 2024-03-04T10:06:09.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:09.910+0000 7fb7acd59640 1 -- 172.21.15.99:0/1805631067 wait complete. 2024-03-04T10:06:09.925 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-03-04T10:06:10.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.026+0000 7f7eb4d16640 1 Processor -- start 2024-03-04T10:06:10.028 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.026+0000 7f7eb4d16640 1 -- start start 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eb4d16640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7eb00ae5a0 0x7f7eb00ae980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eb4d16640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7eb015e840 0x7f7eb00cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eb4d16640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 0x7f7eb00cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eb4d16640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb00aeec0 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eb4d16640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb01617b0 con 0x7f7eb015e840 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eb4d16640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb0161a90 con 0x7f7eb015eca0 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7eb00ae5a0 0x7f7eb00ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7eb00ae5a0 0x7f7eb00ae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57492/0 (socket says 172.21.15.99:57492) 2024-03-04T10:06:10.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 -- 172.21.15.99:0/2507124559 learned_addr learned my addr 172.21.15.99:0/2507124559 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 -- 172.21.15.99:0/2507124559 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 msgr2=0x7f7eb00cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 --2- 172.21.15.99:0/2507124559 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 0x7f7eb00cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 -- 172.21.15.99:0/2507124559 >> v2:172.21.15.162:3300/0 conn(0x7f7eb015e840 msgr2=0x7f7eb00cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7ea7fff640 1 --2- 172.21.15.99:0/2507124559 >> v2:172.21.15.162:3300/0 conn(0x7f7eb015e840 0x7f7eb00cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 --2- 172.21.15.99:0/2507124559 >> v2:172.21.15.162:3300/0 conn(0x7f7eb015e840 0x7f7eb00cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.027+0000 7f7eae59c640 1 -- 172.21.15.99:0/2507124559 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb0161d70 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.028+0000 7f7eae59c640 1 --2- 172.21.15.99:0/2507124559 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb00ae980 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00725d0 tx=0x7f7ea002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f473e7fff94f954 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.028+0000 7f7eadd9b640 1 -- 172.21.15.99:0/2507124559 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0038070 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.028+0000 7f7eadd9b640 1 -- 172.21.15.99:0/2507124559 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea002f050 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.028+0000 7f7eadd9b640 1 -- 172.21.15.99:0/2507124559 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0031970 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.028+0000 7f7eb4d16640 1 -- 172.21.15.99:0/2507124559 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 msgr2=0x7f7eb00ae980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.028+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/2507124559 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb00ae980 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7eb00725d0 tx=0x7f7ea002b7f0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 -- 172.21.15.99:0/2507124559 shutdown_connections 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/2507124559 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 0x7f7eb00cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/2507124559 >> v2:172.21.15.162:3300/0 conn(0x7f7eb015e840 0x7f7eb00cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/2507124559 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb00ae980 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 -- 172.21.15.99:0/2507124559 >> 172.21.15.99:0/2507124559 conn(0x7f7eb006acd0 msgr2=0x7f7eb006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 -- 172.21.15.99:0/2507124559 shutdown_connections 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.029+0000 7f7eb4d16640 1 -- 172.21.15.99:0/2507124559 wait complete. 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.030+0000 7f7eb4d16640 1 Processor -- start 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.030+0000 7f7eb4d16640 1 -- start start 2024-03-04T10:06:10.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb0147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7eb015e840 0x7f7eb00cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 0x7f7eb00c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb015e2a0 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb0162330 con 0x7f7eb015e840 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb015e420 con 0x7f7eb015eca0 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb0147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb0147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57500/0 (socket says 172.21.15.99:57500) 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 -- 172.21.15.99:0/1305242884 learned_addr learned my addr 172.21.15.99:0/1305242884 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 -- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 msgr2=0x7f7eb00c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 0x7f7eb00c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 -- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7eb015e840 msgr2=0x7f7eb00cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7eb015e840 0x7f7eb00cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 -- 172.21.15.99:0/1305242884 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb00cc950 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eae59c640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb0147090 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0000c00 tx=0x7f7ea00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0042070 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea003d870 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0038040 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb00ccc30 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.031+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb00cd1f0 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.032+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb0074f60 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.033+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7ea0045070 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.033+0000 7f7ea6ffd640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e9006b4d0 0x7f7e9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.033+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ea00ac320 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.033+0000 7f7ea7fff640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e9006b4d0 0x7f7e9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:10.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.034+0000 7f7ea7fff640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e9006b4d0 0x7f7e9006d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7eb01f6fb0 tx=0x7f7e98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:10.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.038+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7ea007b130 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.208+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) v1 -- 0x7f7eb00af9a0 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.211+0000 7f7ea6ffd640 1 -- 172.21.15.99:0/1305242884 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v14)=0 v14) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7ea007aa30 con 0x7f7eb00ae5a0 2024-03-04T10:06:10.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e9006b4d0 msgr2=0x7f7e9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e9006b4d0 0x7f7e9006d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7eb01f6fb0 tx=0x7f7e98008040 comp rx=0 tx=0).stop 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 msgr2=0x7f7eb0147090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb0147090 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0000c00 tx=0x7f7ea00046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 shutdown_connections 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e9006b4d0 0x7f7e9006d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7eb015eca0 0x7f7eb00c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7eb015e840 0x7f7eb00cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 --2- 172.21.15.99:0/1305242884 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7eb00ae5a0 0x7f7eb0147090 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 >> 172.21.15.99:0/1305242884 conn(0x7f7eb006acd0 msgr2=0x7f7eb0071780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 shutdown_connections 2024-03-04T10:06:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:10.214+0000 7f7eb4d16640 1 -- 172.21.15.99:0/1305242884 wait complete. 2024-03-04T10:06:15.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:15.227 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mv a/b a/c; sync .') 2024-03-04T10:06:15.248 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:06:15.247+0000 7ff20b2ca640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x5630574a6d80) (dversion lock w=1 last_client=6159) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x563057572000] 2024-03-04T10:06:15.250 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:15.250 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:15.309 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:15.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.458+0000 7f987c0d1640 1 Processor -- start 2024-03-04T10:06:15.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.458+0000 7f987c0d1640 1 -- start start 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987c0d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f98740cfd60 0x7f98740cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987c0d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f987406d3b0 0x7f98740cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987c0d1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f98740ce9c0 0x7f98740cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987c0d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987415cbb0 con 0x7f987406d3b0 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987c0d1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98740d0140 con 0x7f98740ce9c0 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987c0d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987415c280 con 0x7f98740cfd60 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987966c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f987406d3b0 0x7f98740cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987966c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f987406d3b0 0x7f98740cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56478/0 (socket says 172.21.15.99:56478) 2024-03-04T10:06:15.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.459+0000 7f987966c640 1 -- 172.21.15.99:0/3293300867 learned_addr learned my addr 172.21.15.99:0/3293300867 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f9879e6d640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f98740cfd60 0x7f98740cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987a66e640 1 --2- 172.21.15.99:0/3293300867 >> v2:172.21.15.162:3300/0 conn(0x7f98740ce9c0 0x7f98740cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987966c640 1 -- 172.21.15.99:0/3293300867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f98740cfd60 msgr2=0x7f98740cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987966c640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f98740cfd60 0x7f98740cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987966c640 1 -- 172.21.15.99:0/3293300867 >> v2:172.21.15.162:3300/0 conn(0x7f98740ce9c0 msgr2=0x7f98740cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987966c640 1 --2- 172.21.15.99:0/3293300867 >> v2:172.21.15.162:3300/0 conn(0x7f98740ce9c0 0x7f98740cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987966c640 1 -- 172.21.15.99:0/3293300867 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987415c560 con 0x7f987406d3b0 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f9879e6d640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f98740cfd60 0x7f98740cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987a66e640 1 --2- 172.21.15.99:0/3293300867 >> v2:172.21.15.162:3300/0 conn(0x7f98740ce9c0 0x7f98740cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.460+0000 7f987966c640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f987406d3b0 0x7f98740cc330 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f98740d04c0 tx=0x7f986402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4604d758b79ca9b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f9878e6b640 1 -- 172.21.15.99:0/3293300867 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864035020 con 0x7f987406d3b0 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f9878e6b640 1 -- 172.21.15.99:0/3293300867 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986402d040 con 0x7f987406d3b0 2024-03-04T10:06:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f9878e6b640 1 -- 172.21.15.99:0/3293300867 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864031a80 con 0x7f987406d3b0 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 -- 172.21.15.99:0/3293300867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f987406d3b0 msgr2=0x7f98740cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f987406d3b0 0x7f98740cc330 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f98740d04c0 tx=0x7f986402b900 comp rx=0 tx=0).stop 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 -- 172.21.15.99:0/3293300867 shutdown_connections 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 --2- 172.21.15.99:0/3293300867 >> v2:172.21.15.162:3300/0 conn(0x7f98740ce9c0 0x7f98740cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f987406d3b0 0x7f98740cc330 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 --2- 172.21.15.99:0/3293300867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f98740cfd60 0x7f98740cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 -- 172.21.15.99:0/3293300867 >> 172.21.15.99:0/3293300867 conn(0x7f987406acd0 msgr2=0x7f987406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 -- 172.21.15.99:0/3293300867 shutdown_connections 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.461+0000 7f987c0d1640 1 -- 172.21.15.99:0/3293300867 wait complete. 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 Processor -- start 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 -- start start 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f987406d3b0 0x7f9874158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 0x7f98741f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98740cfd60 0x7f98741448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987415bf30 con 0x7f98740ce9c0 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98740cf390 con 0x7f98740cfd60 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987c0d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9874071350 con 0x7f987406d3b0 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987966c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 0x7f98741f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987966c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 0x7f98741f4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56488/0 (socket says 172.21.15.99:56488) 2024-03-04T10:06:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.463+0000 7f987966c640 1 -- 172.21.15.99:0/2445262439 learned_addr learned my addr 172.21.15.99:0/2445262439 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987966c640 1 -- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f987406d3b0 msgr2=0x7f9874158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f9879e6d640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f987406d3b0 0x7f9874158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987a66e640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98740cfd60 0x7f98741448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987966c640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f987406d3b0 0x7f9874158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987966c640 1 -- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98740cfd60 msgr2=0x7f98741448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987966c640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98740cfd60 0x7f98741448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987966c640 1 -- 172.21.15.99:0/2445262439 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98741f5520 con 0x7f98740ce9c0 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987a66e640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98740cfd60 0x7f98741448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f9879e6d640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f987406d3b0 0x7f9874158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987966c640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 0x7f98741f4fe0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9864000c00 tx=0x7f98640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864040070 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9864039bf0 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864035040 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9874157c30 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.464+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98741581f0 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.465+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9874074f60 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.465+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9864038020 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.466+0000 7f986a7fc640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f985806b4d0 0x7f985806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.466+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f98640ac190 con 0x7f98740ce9c0 2024-03-04T10:06:15.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.466+0000 7f9879e6d640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f985806b4d0 0x7f985806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.466+0000 7f9879e6d640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f985806b4d0 0x7f985806d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9870007f40 tx=0x7f9870007490 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:15.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.470+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f98741f5520 con 0x7f98740ce9c0 2024-03-04T10:06:15.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.642+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f987406c310 con 0x7f98740ce9c0 2024-03-04T10:06:15.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.643+0000 7f986a7fc640 1 -- 172.21.15.99:0/2445262439 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16906 (secure 0 0 0) 0x7f986407ae80 con 0x7f98740ce9c0 2024-03-04T10:06:15.644 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:15.644 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":52,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:05:59.341669+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:15.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f985806b4d0 msgr2=0x7f985806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f985806b4d0 0x7f985806d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9870007f40 tx=0x7f9870007490 comp rx=0 tx=0).stop 2024-03-04T10:06:15.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 msgr2=0x7f98741f4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 0x7f98741f4fe0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9864000c00 tx=0x7f98640046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 shutdown_connections 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f985806b4d0 0x7f985806d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f98740cfd60 0x7f98741448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f98740ce9c0 0x7f98741f4fe0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 --2- 172.21.15.99:0/2445262439 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f987406d3b0 0x7f9874158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 >> 172.21.15.99:0/2445262439 conn(0x7f987406acd0 msgr2=0x7f987415a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 shutdown_connections 2024-03-04T10:06:15.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.645+0000 7f987c0d1640 1 -- 172.21.15.99:0/2445262439 wait complete. 2024-03-04T10:06:15.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:05:55.899275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:05:56.653255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:06:15.657 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:06:15.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.759+0000 7f285fd45640 1 Processor -- start 2024-03-04T10:06:15.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.759+0000 7f285fd45640 1 -- start start 2024-03-04T10:06:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285fd45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 0x7f28580b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285fd45640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f28580cf020 0x7f28580d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285fd45640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28580ca970 con 0x7f28580cec40 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285fd45640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2858161d90 con 0x7f28580cf020 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285fd45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2858162070 con 0x7f285815de40 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285dae1640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f28580cec40 0x7f285815d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37052/0 (socket says 172.21.15.99:37052) 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285dae1640 1 -- 172.21.15.99:0/1142193270 learned_addr learned my addr 172.21.15.99:0/1142193270 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285d2e0640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 0x7f28580b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285e2e2640 1 --2- 172.21.15.99:0/1142193270 >> v2:172.21.15.162:3300/0 conn(0x7f28580cf020 0x7f28580d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959185955 0 0) 0x7f28580ca970 con 0x7f28580cec40 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2844003520 con 0x7f28580cec40 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594545430 0 0) 0x7f2844003520 con 0x7f28580cec40 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 msgr2=0x7f28580b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 0x7f28580b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 >> v2:172.21.15.162:3300/0 conn(0x7f28580cf020 msgr2=0x7f28580d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 --2- 172.21.15.99:0/1142193270 >> v2:172.21.15.162:3300/0 conn(0x7f28580cf020 0x7f28580d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858162350 con 0x7f28580cec40 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f2854001a10 con 0x7f28580cec40 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285e2e2640 1 --2- 172.21.15.99:0/1142193270 >> v2:172.21.15.162:3300/0 conn(0x7f28580cf020 0x7f28580d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 >> v1:172.21.15.99:6789/0 conn(0x7f28580cec40 legacy=0x7f285815d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28440039b0 0x7f2844003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 0x7f2844004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2844004d40 0x7f28440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2858161d90 con 0x7f28440039b0 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2858162070 con 0x7f2844004d40 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f284400b9c0 con 0x7f285815de40 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285dae1640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28440039b0 0x7f2844003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.760+0000 7f285d2e0640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 0x7f2844004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285e2e2640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2844004d40 0x7f28440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285dae1640 1 -- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 msgr2=0x7f2844004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285dae1640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f285815de40 0x7f2844004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285dae1640 1 -- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2844004d40 msgr2=0x7f28440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285dae1640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2844004d40 0x7f28440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285dae1640 1 -- 172.21.15.99:0/1142193270 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844003520 con 0x7f28440039b0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285e2e2640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2844004d40 0x7f28440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.761+0000 7f285dae1640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28440039b0 0x7f2844003dd0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f285400a8a0 tx=0x7f2844005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02b60e57f22e945 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285400a4c0 con 0x7f28440039b0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2854034630 con 0x7f28440039b0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285cadf640 1 -- 172.21.15.99:0/1142193270 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854034a20 con 0x7f28440039b0 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285fd45640 1 -- 172.21.15.99:0/1142193270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28440039b0 msgr2=0x7f2844003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285fd45640 1 --2- 172.21.15.99:0/1142193270 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28440039b0 0x7f2844003dd0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f285400a8a0 tx=0x7f2844005830 comp rx=0 tx=0).stop 2024-03-04T10:06:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285e2e2640 1 -- 172.21.15.99:0/1142193270 reap_dead start 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285fd45640 1 -- 172.21.15.99:0/1142193270 shutdown_connections 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285fd45640 1 -- 172.21.15.99:0/1142193270 >> 172.21.15.99:0/1142193270 conn(0x7f28580d0580 msgr2=0x7f28580d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285fd45640 1 -- 172.21.15.99:0/1142193270 shutdown_connections 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.762+0000 7f285fd45640 1 -- 172.21.15.99:0/1142193270 wait complete. 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.763+0000 7f285fd45640 1 Processor -- start 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.763+0000 7f285fd45640 1 -- start start 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285fd45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28440039b0 0x7f2858154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285fd45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28581e7580 0x7f28581e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285fd45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 0x7f285815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285fd45640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580d3300 con 0x7f285815a720 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285fd45640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580d1bf0 con 0x7f28440039b0 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285fd45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28580d3480 con 0x7f28581e7580 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 0x7f285815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 0x7f285815aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56500/0 (socket says 172.21.15.99:56500) 2024-03-04T10:06:15.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 -- 172.21.15.99:0/2596037305 learned_addr learned my addr 172.21.15.99:0/2596037305 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285d2e0640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28581e7580 0x7f28581e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285dae1640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28440039b0 0x7f2858154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 -- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28581e7580 msgr2=0x7f28581e7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28581e7580 0x7f28581e7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 -- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28440039b0 msgr2=0x7f2858154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28440039b0 0x7f2858154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.764+0000 7f285e2e2640 1 -- 172.21.15.99:0/2596037305 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285815b0e0 con 0x7f285815a720 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.765+0000 7f285e2e2640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 0x7f285815aba0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f28580d1900 tx=0x7f2850011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.765+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850019020 con 0x7f285815a720 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.765+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2850004440 con 0x7f285815a720 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.765+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850003850 con 0x7f285815a720 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.765+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28581e8040 con 0x7f285815a720 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.765+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28581e8600 con 0x7f285815a720 2024-03-04T10:06:15.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.766+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28581e8c10 con 0x7f285815a720 2024-03-04T10:06:15.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.766+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f285001c020 con 0x7f285815a720 2024-03-04T10:06:15.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.766+0000 7f284e7fc640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f283c06b4d0 0x7f283c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:15.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.766+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f285008a420 con 0x7f285815a720 2024-03-04T10:06:15.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.767+0000 7f285dae1640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f283c06b4d0 0x7f283c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:15.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.767+0000 7f285dae1640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f283c06b4d0 0x7f283c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2854034000 tx=0x7f2854009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:15.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.771+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2850023030 con 0x7f285815a720 2024-03-04T10:06:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.940+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28580d1930 con 0x7f285815a720 2024-03-04T10:06:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.940+0000 7f284e7fc640 1 -- 172.21.15.99:0/2596037305 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28500591a0 con 0x7f285815a720 2024-03-04T10:06:15.941 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f283c06b4d0 msgr2=0x7f283c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f283c06b4d0 0x7f283c06d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2854034000 tx=0x7f2854009030 comp rx=0 tx=0).stop 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 msgr2=0x7f285815aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 0x7f285815aba0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f28580d1900 tx=0x7f2850011930 comp rx=0 tx=0).stop 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 shutdown_connections 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f283c06b4d0 0x7f283c06d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f285815a720 0x7f285815aba0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28581e7580 0x7f28581e7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 --2- 172.21.15.99:0/2596037305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28440039b0 0x7f2858154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 >> 172.21.15.99:0/2596037305 conn(0x7f28580d0580 msgr2=0x7f2858157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 shutdown_connections 2024-03-04T10:06:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:15.942+0000 7f285fd45640 1 -- 172.21.15.99:0/2596037305 wait complete. 2024-03-04T10:06:15.954 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 Processor -- start 2024-03-04T10:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 -- start start 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2b80d34a0 0x7fd2b8161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd2b80b9410 0x7fd2b8162120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b80d2400 con 0x7fd2b80b5700 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b80d26e0 con 0x7fd2b80d34a0 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.057+0000 7fd2bd480640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b80d29c0 con 0x7fd2b80b9410 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2b80d34a0 0x7fd2b8161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2b80d34a0 0x7fd2b8161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56504/0 (socket says 172.21.15.99:56504) 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 -- 172.21.15.99:0/3835200276 learned_addr learned my addr 172.21.15.99:0/3835200276 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b77fe640 1 --2- 172.21.15.99:0/3835200276 >> v2:172.21.15.162:3300/0 conn(0x7fd2b80b9410 0x7fd2b8162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 -- 172.21.15.99:0/3835200276 >> v2:172.21.15.162:3300/0 conn(0x7fd2b80b9410 msgr2=0x7fd2b8162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 --2- 172.21.15.99:0/3835200276 >> v2:172.21.15.162:3300/0 conn(0x7fd2b80b9410 0x7fd2b8162120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 -- 172.21.15.99:0/3835200276 >> v1:172.21.15.162:6789/0 conn(0x7fd2b80b5700 legacy=0x7fd2b80d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b67fc640 1 -- 172.21.15.99:0/3835200276 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80d2ca0 con 0x7fd2b80d34a0 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.058+0000 7fd2b77fe640 1 --2- 172.21.15.99:0/3835200276 >> v2:172.21.15.162:3300/0 conn(0x7fd2b80b9410 0x7fd2b8162120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2b67fc640 1 --2- 172.21.15.99:0/3835200276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80d34a0 0x7fd2b8161be0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80ca850 tx=0x7fd2ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d597fefa86ce6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2b5ffb640 1 -- 172.21.15.99:0/3835200276 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac02e070 con 0x7fd2b80d34a0 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2b5ffb640 1 -- 172.21.15.99:0/3835200276 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac037070 con 0x7fd2b80d34a0 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2b5ffb640 1 -- 172.21.15.99:0/3835200276 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac031a00 con 0x7fd2b80d34a0 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 -- 172.21.15.99:0/3835200276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80d34a0 msgr2=0x7fd2b8161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 --2- 172.21.15.99:0/3835200276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80d34a0 0x7fd2b8161be0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80ca850 tx=0x7fd2ac02b880 comp rx=0 tx=0).stop 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 -- 172.21.15.99:0/3835200276 shutdown_connections 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 --2- 172.21.15.99:0/3835200276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80d34a0 0x7fd2b8161be0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 --2- 172.21.15.99:0/3835200276 >> v2:172.21.15.162:3300/0 conn(0x7fd2b80b9410 0x7fd2b8162120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 -- 172.21.15.99:0/3835200276 >> 172.21.15.99:0/3835200276 conn(0x7fd2b80d0490 msgr2=0x7fd2b80d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 -- 172.21.15.99:0/3835200276 shutdown_connections 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.059+0000 7fd2bd480640 1 -- 172.21.15.99:0/3835200276 wait complete. 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 Processor -- start 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 -- start start 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 0x7fd2b8160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b80b9410 0x7fd2b80c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2b80d34a0 0x7fd2b80cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8162660 con 0x7fd2b80b5700 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80b5cf0 con 0x7fd2b80b9410 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2bd480640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b81627e0 con 0x7fd2b80d34a0 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2b6ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 0x7fd2b8160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2b6ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 0x7fd2b8160390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56506/0 (socket says 172.21.15.99:56506) 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.061+0000 7fd2b6ffd640 1 -- 172.21.15.99:0/2067263105 learned_addr learned my addr 172.21.15.99:0/2067263105 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b77fe640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2b80d34a0 0x7fd2b80cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b67fc640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b80b9410 0x7fd2b80c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b6ffd640 1 -- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2b80d34a0 msgr2=0x7fd2b80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b6ffd640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2b80d34a0 0x7fd2b80cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b6ffd640 1 -- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b80b9410 msgr2=0x7fd2b80c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b6ffd640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b80b9410 0x7fd2b80c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b6ffd640 1 -- 172.21.15.99:0/2067263105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80c85d0 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2b6ffd640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 0x7fd2b8160390 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800b3e0 tx=0x7fd2a800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8015020 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a8011850 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b80c88b0 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8011c40 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.062+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b80c8e70 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.063+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b8074f60 con 0x7fd2b80b5700 2024-03-04T10:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.063+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd2a8018020 con 0x7fd2b80b5700 2024-03-04T10:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.064+0000 7fd29f7fe640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd2a406b4d0 0x7fd2a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.064+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2a808a270 con 0x7fd2b80b5700 2024-03-04T10:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.064+0000 7fd2b67fc640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd2a406b4d0 0x7fd2a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.064+0000 7fd2b67fc640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd2a406b4d0 0x7fd2a406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac036000 tx=0x7fd2ac03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.068+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd2a801f020 con 0x7fd2b80b5700 2024-03-04T10:06:16.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.238+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2b80b6180 con 0x7fd2b80b5700 2024-03-04T10:06:16.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.238+0000 7fd29f7fe640 1 -- 172.21.15.99:0/2067263105 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd2a80172c0 con 0x7fd2b80b5700 2024-03-04T10:06:16.239 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:06:16.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd2a406b4d0 msgr2=0x7fd2a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd2a406b4d0 0x7fd2a406d990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac036000 tx=0x7fd2ac03d040 comp rx=0 tx=0).stop 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 msgr2=0x7fd2b8160390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 0x7fd2b8160390 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd2a800b3e0 tx=0x7fd2a800b8a0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 shutdown_connections 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd2a406b4d0 0x7fd2a406d990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd2b80d34a0 0x7fd2b80cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2b80b9410 0x7fd2b80c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 --2- 172.21.15.99:0/2067263105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2b80b5700 0x7fd2b8160390 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 >> 172.21.15.99:0/2067263105 conn(0x7fd2b80d0490 msgr2=0x7fd2b80d39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:16.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 shutdown_connections 2024-03-04T10:06:16.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.240+0000 7fd2bd480640 1 -- 172.21.15.99:0/2067263105 wait complete. 2024-03-04T10:06:16.252 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:06:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.354+0000 7f0cd27b9640 1 Processor -- start 2024-03-04T10:06:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.354+0000 7f0cd27b9640 1 -- start start 2024-03-04T10:06:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.354+0000 7f0cd27b9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0ccc0cf660 0x7f0ccc0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0cd27b9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc162450 con 0x7f0ccc15da20 2024-03-04T10:06:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0cd27b9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc0cc2d0 con 0x7f0ccc1616e0 2024-03-04T10:06:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0cd27b9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ccc0d0000 con 0x7f0ccc0cf660 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccbfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0ccc15da20 0x7f0ccc0ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37064/0 (socket says 172.21.15.99:37064) 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccbfff640 1 -- 172.21.15.99:0/4062191863 learned_addr learned my addr 172.21.15.99:0/4062191863 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437078135 0 0) 0x7f0ccc162450 con 0x7f0ccc15da20 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb4003520 con 0x7f0ccc15da20 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccb7fe640 1 --2- 172.21.15.99:0/4062191863 >> v2:172.21.15.162:3300/0 conn(0x7f0ccc0cf660 0x7f0ccc0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2351448177 0 0) 0x7f0cb4003520 con 0x7f0ccc15da20 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 >> v2:172.21.15.162:3300/0 conn(0x7f0ccc0cf660 msgr2=0x7f0ccc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 --2- 172.21.15.99:0/4062191863 >> v2:172.21.15.162:3300/0 conn(0x7f0ccc0cf660 0x7f0ccc0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 >> v1:172.21.15.162:6789/0 conn(0x7f0ccc1616e0 legacy=0x7f0ccc161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0cc5b0 con 0x7f0ccc15da20 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f0cbc001670 con 0x7f0ccc15da20 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 >> v1:172.21.15.99:6789/0 conn(0x7f0ccc15da20 legacy=0x7f0ccc0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40039b0 0x7f0cb4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb4004800 0x7f0cb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0cb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400c180 con 0x7f0cb40052e0 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400c330 con 0x7f0cb4004800 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400c530 con 0x7f0cb40039b0 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.355+0000 7f0cd0d56640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0cb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0ccbfff640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40039b0 0x7f0cb4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0ccb7fe640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb4004800 0x7f0cb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 -- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40039b0 msgr2=0x7f0cb4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0cb40039b0 0x7f0cb4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 -- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb4004800 msgr2=0x7f0cb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb4004800 0x7f0cb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 -- 172.21.15.99:0/4062191863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4003520 con 0x7f0cb40052e0 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0ccb7fe640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0cb4004800 0x7f0cb4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 -- 172.21.15.99:0/4062191863 reap_dead start 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0cd0d56640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0cb400b9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4001e20 tx=0x7f0cb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f087ede91618d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.356+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4001a70 con 0x7f0cb40052e0 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4003b70 con 0x7f0cb40052e0 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0ccaffd640 1 -- 172.21.15.99:0/4062191863 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc40050d0 con 0x7f0cb40052e0 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 -- 172.21.15.99:0/4062191863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 msgr2=0x7f0cb400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0cb400b9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4001e20 tx=0x7f0cb4005830 comp rx=0 tx=0).stop 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 -- 172.21.15.99:0/4062191863 shutdown_connections 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/4062191863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0cb400b9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 -- 172.21.15.99:0/4062191863 >> 172.21.15.99:0/4062191863 conn(0x7f0ccc0b33a0 msgr2=0x7f0ccc0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 -- 172.21.15.99:0/4062191863 shutdown_connections 2024-03-04T10:06:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.357+0000 7f0cd27b9640 1 -- 172.21.15.99:0/4062191863 wait complete. 2024-03-04T10:06:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.358+0000 7f0cd27b9640 1 Processor -- start 2024-03-04T10:06:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.358+0000 7f0cd27b9640 1 -- start start 2024-03-04T10:06:16.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd27b9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0ccc0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd27b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ccc1f2cc0 0x7f0ccc15c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd27b9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0ccc1f3120 0x7f0ccc15ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd27b9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc15de00 con 0x7f0cb40052e0 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd27b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0cc920 con 0x7f0ccc1f2cc0 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd27b9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0cfac0 con 0x7f0ccc1f3120 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0ccc0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0ccc0cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56528/0 (socket says 172.21.15.99:56528) 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 -- 172.21.15.99:0/891881535 learned_addr learned my addr 172.21.15.99:0/891881535 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0cd0d56640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0ccc1f3120 0x7f0ccc15ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccb7fe640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ccc1f2cc0 0x7f0ccc15c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 -- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0ccc1f3120 msgr2=0x7f0ccc15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0ccc1f3120 0x7f0ccc15ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 -- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ccc1f2cc0 msgr2=0x7f0ccc15c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ccc1f2cc0 0x7f0ccc15c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.359+0000 7f0ccbfff640 1 -- 172.21.15.99:0/891881535 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0c9ab0 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.360+0000 7f0ccbfff640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0ccc0cb1d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0072c0 tx=0x7f0cbc007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.360+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00f960 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.360+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc018070 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.360+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc0c9d90 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.360+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00fd60 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.360+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc0ca350 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.361+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc074f60 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.361+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0cbc01d460 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.362+0000 7f0cc8ff9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0cb806b4d0 0x7f0cb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.362+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0cbc08a270 con 0x7f0cb40052e0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.362+0000 7f0ccb7fe640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0cb806b4d0 0x7f0cb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.363+0000 7f0ccb7fe640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0cb806b4d0 0x7f0cb806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0ce060 tx=0x7f0cc0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.366+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0cbc058ff0 con 0x7f0cb40052e0 2024-03-04T10:06:16.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.536+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ccc06e620 con 0x7f0cb40052e0 2024-03-04T10:06:16.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.537+0000 7f0cc8ff9640 1 -- 172.21.15.99:0/891881535 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0cbc0588f0 con 0x7f0cb40052e0 2024-03-04T10:06:16.538 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0cb806b4d0 msgr2=0x7f0cb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0cb806b4d0 0x7f0cb806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0ce060 tx=0x7f0cc0008040 comp rx=0 tx=0).stop 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 msgr2=0x7f0ccc0cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0ccc0cb1d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0072c0 tx=0x7f0cbc007780 comp rx=0 tx=0).stop 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 shutdown_connections 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0cb806b4d0 0x7f0cb806d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0ccc1f3120 0x7f0ccc15ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ccc1f2cc0 0x7f0ccc15c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 --2- 172.21.15.99:0/891881535 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0cb40052e0 0x7f0ccc0cb1d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.538+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 >> 172.21.15.99:0/891881535 conn(0x7f0ccc0b33a0 msgr2=0x7f0ccc0ce2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:16.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.539+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 shutdown_connections 2024-03-04T10:06:16.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.539+0000 7f0cd27b9640 1 -- 172.21.15.99:0/891881535 wait complete. 2024-03-04T10:06:16.550 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:06:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.650+0000 7f03d829a640 1 Processor -- start 2024-03-04T10:06:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.650+0000 7f03d829a640 1 -- start start 2024-03-04T10:06:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d829a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f03d006c470 0x7f03d006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d829a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f03d0066de0 0x7f03d015f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d829a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d00ad6c0 con 0x7f03d006c090 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d829a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00ad960 con 0x7f03d0066de0 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d829a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d006ebc0 con 0x7f03d006c470 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6837640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f03d0066de0 0x7f03d015f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6036640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f03d006c090 0x7f03d00af220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37080/0 (socket says 172.21.15.99:37080) 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6036640 1 -- 172.21.15.99:0/476944943 learned_addr learned my addr 172.21.15.99:0/476944943 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d5835640 1 --2- 172.21.15.99:0/476944943 >> v2:172.21.15.162:3300/0 conn(0x7f03d006c470 0x7f03d006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6837640 1 -- 172.21.15.99:0/476944943 >> v2:172.21.15.162:3300/0 conn(0x7f03d006c470 msgr2=0x7f03d006f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6837640 1 --2- 172.21.15.99:0/476944943 >> v2:172.21.15.162:3300/0 conn(0x7f03d006c470 0x7f03d006f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6837640 1 -- 172.21.15.99:0/476944943 >> v1:172.21.15.99:6789/0 conn(0x7f03d006c090 legacy=0x7f03d00af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d6837640 1 -- 172.21.15.99:0/476944943 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d006eea0 con 0x7f03d0066de0 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d5034640 1 -- 172.21.15.99:0/476944943 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369341984 0 0) 0x7f03d00ad6c0 con 0x7f03d006c090 2024-03-04T10:06:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.651+0000 7f03d5835640 1 --2- 172.21.15.99:0/476944943 >> v2:172.21.15.162:3300/0 conn(0x7f03d006c470 0x7f03d006f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.652+0000 7f03d6837640 1 --2- 172.21.15.99:0/476944943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d015f560 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f03d01565c0 tx=0x7f03c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5319f21bcc65abef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.652+0000 7f03d5034640 1 -- 172.21.15.99:0/476944943 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8039070 con 0x7f03d0066de0 2024-03-04T10:06:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.652+0000 7f03d5034640 1 -- 172.21.15.99:0/476944943 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c802cbf0 con 0x7f03d0066de0 2024-03-04T10:06:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.652+0000 7f03d5034640 1 -- 172.21.15.99:0/476944943 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8034440 con 0x7f03d0066de0 2024-03-04T10:06:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.652+0000 7f03d829a640 1 -- 172.21.15.99:0/476944943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 msgr2=0x7f03d015f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.652+0000 7f03d829a640 1 --2- 172.21.15.99:0/476944943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d015f560 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f03d01565c0 tx=0x7f03c802be40 comp rx=0 tx=0).stop 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.653+0000 7f03d829a640 1 -- 172.21.15.99:0/476944943 shutdown_connections 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.653+0000 7f03d829a640 1 --2- 172.21.15.99:0/476944943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d015f560 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.653+0000 7f03d829a640 1 --2- 172.21.15.99:0/476944943 >> v2:172.21.15.162:3300/0 conn(0x7f03d006c470 0x7f03d006f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.653+0000 7f03d829a640 1 -- 172.21.15.99:0/476944943 >> 172.21.15.99:0/476944943 conn(0x7f03d00cfef0 msgr2=0x7f03d00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.653+0000 7f03d829a640 1 -- 172.21.15.99:0/476944943 shutdown_connections 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.653+0000 7f03d829a640 1 -- 172.21.15.99:0/476944943 wait complete. 2024-03-04T10:06:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 Processor -- start 2024-03-04T10:06:16.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 -- start start 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d00cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03d006c090 0x7f03d0143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03d00cbf30 0x7f03d0146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00addc0 con 0x7f03d0066de0 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00adc40 con 0x7f03d00cbf30 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.654+0000 7f03d829a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d015faa0 con 0x7f03d006c090 2024-03-04T10:06:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d00cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d00cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56540/0 (socket says 172.21.15.99:56540) 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 -- 172.21.15.99:0/3798369304 learned_addr learned my addr 172.21.15.99:0/3798369304 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d5835640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03d006c090 0x7f03d0143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6837640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03d00cbf30 0x7f03d0146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 -- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03d006c090 msgr2=0x7f03d0143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03d006c090 0x7f03d0143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 -- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03d00cbf30 msgr2=0x7f03d0146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03d00cbf30 0x7f03d0146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 -- 172.21.15.99:0/3798369304 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00ced70 con 0x7f03d0066de0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d6036640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d00cd470 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c4a0 tx=0x7f03cc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc017070 con 0x7f03d0066de0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc005ce0 con 0x7f03d0066de0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0072b0 con 0x7f03d0066de0 2024-03-04T10:06:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d0146f80 con 0x7f03d0066de0 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.655+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d00cc6b0 con 0x7f03d0066de0 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.656+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0074f60 con 0x7f03d0066de0 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.657+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f03cc01a070 con 0x7f03d0066de0 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.657+0000 7f03c6ffd640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f03b406b4d0 0x7f03b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.657+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f03cc089710 con 0x7f03d0066de0 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.657+0000 7f03d5835640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f03b406b4d0 0x7f03b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.658+0000 7f03d5835640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f03b406b4d0 0x7f03b406d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f03d01f6510 tx=0x7f03c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:16.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.661+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f03cc058400 con 0x7f03d0066de0 2024-03-04T10:06:16.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.844+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f03d00ad100 con 0x7f03d0066de0 2024-03-04T10:06:16.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.844+0000 7f03c6ffd640 1 -- 172.21.15.99:0/3798369304 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v52) v1 ==== 80+0+0 (secure 0 0 0) 0x7f03cc057d00 con 0x7f03d0066de0 2024-03-04T10:06:16.846 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f03b406b4d0 msgr2=0x7f03b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f03b406b4d0 0x7f03b406d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f03d01f6510 tx=0x7f03c0008040 comp rx=0 tx=0).stop 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 msgr2=0x7f03d00cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d00cd470 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00c4a0 tx=0x7f03cc00c960 comp rx=0 tx=0).stop 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 shutdown_connections 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f03b406b4d0 0x7f03b406d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03d00cbf30 0x7f03d0146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03d006c090 0x7f03d0143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 --2- 172.21.15.99:0/3798369304 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03d0066de0 0x7f03d00cd470 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:16.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 >> 172.21.15.99:0/3798369304 conn(0x7f03d00cfef0 msgr2=0x7f03d00b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:16.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 shutdown_connections 2024-03-04T10:06:16.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:16.846+0000 7f03d829a640 1 -- 172.21.15.99:0/3798369304 wait complete. 2024-03-04T10:06:16.858 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:06:16.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:16.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:16.890 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:06:16.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:16.890 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:06:16.945 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.085+0000 7ff3bba81640 1 Processor -- start 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.085+0000 7ff3bba81640 1 -- start start 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3bba81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 0x7ff3b40b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3bba81640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b406b500 con 0x7ff3b406acd0 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3bba81640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b406b7e0 con 0x7ff3b4161e70 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3bba81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b40b3f00 con 0x7ff3b4158410 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b901c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff3b406acd0 0x7ff3b4161760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37088/0 (socket says 172.21.15.99:37088) 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b901c640 1 -- 172.21.15.99:0/3863109461 learned_addr learned my addr 172.21.15.99:0/3863109461 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b981d640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 0x7ff3b40b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234755644 0 0) 0x7ff3b406b500 con 0x7ff3b406acd0 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a0003520 con 0x7ff3b406acd0 2024-03-04T10:06:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287353722 0 0) 0x7ff3a0003520 con 0x7ff3b406acd0 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 msgr2=0x7ff3b40b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 0x7ff3b40b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 >> v1:172.21.15.162:6789/0 conn(0x7ff3b4161e70 legacy=0x7ff3b406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b406bac0 con 0x7ff3b406acd0 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7ff3a4001670 con 0x7ff3b406acd0 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 >> v1:172.21.15.99:6789/0 conn(0x7ff3b406acd0 legacy=0x7ff3b4161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a00039e0 0x7ff3a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.086+0000 7ff3b881b640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 0x7ff3a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3b881b640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3a0004d40 0x7ff3a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a000b9c0 con 0x7ff3a0004d40 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a000bb40 con 0x7ff3a00039e0 2024-03-04T10:06:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a000bd40 con 0x7ff3b4158410 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3a0004d40 0x7ff3a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3b981d640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a00039e0 0x7ff3a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3b901c640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 0x7ff3a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 msgr2=0x7ff3a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4158410 0x7ff3a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a00039e0 msgr2=0x7ff3a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a00039e0 0x7ff3a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.087+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3863109461 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a0003520 con 0x7ff3a0004d40 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3a0004d40 0x7ff3a00051c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40719c0 tx=0x7ff3a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260db88405638c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac039070 con 0x7ff3a0004d40 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac02c710 con 0x7ff3a0004d40 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3b881b640 1 -- 172.21.15.99:0/3863109461 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac001be0 con 0x7ff3a0004d40 2024-03-04T10:06:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3bba81640 1 -- 172.21.15.99:0/3863109461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3a0004d40 msgr2=0x7ff3a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3863109461 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3a0004d40 0x7ff3a00051c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40719c0 tx=0x7ff3a0005830 comp rx=0 tx=0).stop 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3863109461 reap_dead start 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3bba81640 1 -- 172.21.15.99:0/3863109461 shutdown_connections 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3bba81640 1 -- 172.21.15.99:0/3863109461 >> 172.21.15.99:0/3863109461 conn(0x7ff3b40cfef0 msgr2=0x7ff3b40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3bba81640 1 -- 172.21.15.99:0/3863109461 shutdown_connections 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.088+0000 7ff3bba81640 1 -- 172.21.15.99:0/3863109461 wait complete. 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 Processor -- start 2024-03-04T10:06:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 -- start start 2024-03-04T10:06:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a0004d40 0x7ff3b4146970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4150980 0x7ff3b40cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 0x7ff3b40cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b406c5a0 con 0x7ff3b4152080 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b406c080 con 0x7ff3a0004d40 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3bba81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b406b130 con 0x7ff3b4150980 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3ba01e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 0x7ff3b40cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3ba01e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 0x7ff3b40cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56558/0 (socket says 172.21.15.99:56558) 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.090+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3574162035 learned_addr learned my addr 172.21.15.99:0/3574162035 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3b901c640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4150980 0x7ff3b40cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3b981d640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a0004d40 0x7ff3b4146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4150980 msgr2=0x7ff3b40cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4150980 0x7ff3b40cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a0004d40 msgr2=0x7ff3b4146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a0004d40 0x7ff3b4146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3ba01e640 1 -- 172.21.15.99:0/3574162035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b4146eb0 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3ba01e640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 0x7ff3b40cc5c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0020d0 tx=0x7ff3ac001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac042020 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac02dd90 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac039040 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b4154d30 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.091+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b4154fd0 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.092+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b4074f60 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.092+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff3ac045020 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.093+0000 7ff3aa7fc640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff39806b500 0x7ff39806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.093+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff3ac0abd00 con 0x7ff3b4152080 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.093+0000 7ff3b981d640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff39806b500 0x7ff39806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.094+0000 7ff3b981d640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff39806b500 0x7ff39806d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40cc570 tx=0x7ff3b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.097+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff3ac04c020 con 0x7ff3b4152080 2024-03-04T10:06:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.302+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3b4066e10 con 0x7ff3b4152080 2024-03-04T10:06:17.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.302+0000 7ff3aa7fc640 1 -- 172.21.15.99:0/3574162035 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4938 (secure 0 0 0) 0x7ff3b4146eb0 con 0x7ff3b4152080 2024-03-04T10:06:17.304 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:17.304 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:00.693772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5961},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5961":{"gid":5961,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6833/370337902","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":370337902},{"type":"v1","addr":"172.21.15.99:6833","nonce":370337902}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:17.304 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 30 2024-03-04T10:06:17.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff39806b500 msgr2=0x7ff39806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff39806b500 0x7ff39806d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40cc570 tx=0x7ff3b0008040 comp rx=0 tx=0).stop 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 msgr2=0x7ff3b40cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 0x7ff3b40cc5c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0020d0 tx=0x7ff3ac001630 comp rx=0 tx=0).stop 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 shutdown_connections 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff39806b500 0x7ff39806d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff3b4152080 0x7ff3b40cc5c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff3b4150980 0x7ff3b40cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 --2- 172.21.15.99:0/3574162035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff3a0004d40 0x7ff3b4146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 >> 172.21.15.99:0/3574162035 conn(0x7ff3b40cfef0 msgr2=0x7ff3b406f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 shutdown_connections 2024-03-04T10:06:17.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.304+0000 7ff3bba81640 1 -- 172.21.15.99:0/3574162035 wait complete. 2024-03-04T10:06:17.316 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /a 0 2024-03-04T10:06:17.428 INFO:teuthology.orchestra.run.smithi099.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-03-04T10:06:15.246473+0000","btime":"2024-03-04T10:06:09.390614+0000","mode":16877,"uid":1000,"gid":1301,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-03-04T10:06:15.246473+0000","atime":"2024-03-04T10:06:09.390614+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-03-04T10:06:15.246473+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-03-04T10:06:15.246473+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-03-04T10:06:15.246473+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"dirfrag":1,"caps":0,"dirtyparent":0,"dirty":1,"authpin":0},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"3","committed_version":"3","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"a/c","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":0,"dirty":1,"authpin":0},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-03-04T10:06:17.429 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-03-04T10:06:15.246473+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-03-04T10:06:09.390614+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-03-04T10:06:09.390614+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-03-04T10:06:15.246473+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "3", "committing_version": "3", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 0, "lock": 0, "request": 0 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/a", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-03-04T10:06:15.246473+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1301, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-03-04T10:06:15.246473+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 0, "caps": 0, "dirfrag": 1, "dirty": 1, "dirtyparent": 0, "lock": 0, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-03-04T10:06:15.246473+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } ] 2024-03-04T10:06:17.429 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-03-04T10:06:15.246473+0000', 'btime': '2024-03-04T10:06:09.390614+0000', 'mode': 16877, 'uid': 1000, 'gid': 1301, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-03-04T10:06:15.246473+0000', 'atime': '2024-03-04T10:06:09.390614+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-03-04T10:06:15.246473+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-03-04T10:06:15.246473+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-03-04T10:06:15.246473+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [{'key': 'security.selinux', 'val': 'unconfined_u:object_r:unlabeled_t:s0\x00'}], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'dirfrag': 1, 'caps': 0, 'dirtyparent': 0, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {'gather_set': [], 'state': 'lock', 'type': 'inest', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'flocklock': {}, 'policylock': {}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '3', 'committed_version': '3', 'is_rep': False, 'dir_auth': '', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'dentries': [{'path': 'a/c', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 1, 'dirty': 1, 'authpin': 0}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'request': 0, 'lock': 0, 'inodepin': 0, 'dirty': 1, 'authpin': 0}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-03-04T10:06:17.430 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:06:17.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.529+0000 7f358890a640 1 Processor -- start 2024-03-04T10:06:17.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.529+0000 7f358890a640 1 -- start start 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f358890a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f35840b0700 0x7f35840b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f358890a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f35840b47c0 0x7f35840b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f358890a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35840ccad0 con 0x7f358415b960 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f358890a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35840b5190 con 0x7f35840b47c0 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f358890a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35840cf860 con 0x7f35840b0700 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f3582d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f35840b47c0 0x7f35840b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f3582d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f35840b47c0 0x7f35840b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56560/0 (socket says 172.21.15.99:56560) 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.530+0000 7f3582d9d640 1 -- 172.21.15.99:0/2632587836 learned_addr learned my addr 172.21.15.99:0/2632587836 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.531+0000 7f3581d9b640 1 --2- 172.21.15.99:0/2632587836 >> v2:172.21.15.162:3300/0 conn(0x7f35840b0700 0x7f35840b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.531+0000 7f3582d9d640 1 -- 172.21.15.99:0/2632587836 >> v2:172.21.15.162:3300/0 conn(0x7f35840b0700 msgr2=0x7f35840b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.531+0000 7f3582d9d640 1 --2- 172.21.15.99:0/2632587836 >> v2:172.21.15.162:3300/0 conn(0x7f35840b0700 0x7f35840b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.531+0000 7f3582d9d640 1 -- 172.21.15.99:0/2632587836 >> v1:172.21.15.162:6789/0 conn(0x7f358415b960 legacy=0x7f35840d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.531+0000 7f3582d9d640 1 -- 172.21.15.99:0/2632587836 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35840cfab0 con 0x7f35840b47c0 2024-03-04T10:06:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.531+0000 7f3582d9d640 1 --2- 172.21.15.99:0/2632587836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840b4c50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f35840b73c0 tx=0x7f357c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf2152d8ad3c850 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358159a640 1 -- 172.21.15.99:0/2632587836 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c035020 con 0x7f35840b47c0 2024-03-04T10:06:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358159a640 1 -- 172.21.15.99:0/2632587836 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c031870 con 0x7f35840b47c0 2024-03-04T10:06:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358159a640 1 -- 172.21.15.99:0/2632587836 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c031c40 con 0x7f35840b47c0 2024-03-04T10:06:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 -- 172.21.15.99:0/2632587836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 msgr2=0x7f35840b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 --2- 172.21.15.99:0/2632587836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840b4c50 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f35840b73c0 tx=0x7f357c02b910 comp rx=0 tx=0).stop 2024-03-04T10:06:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 -- 172.21.15.99:0/2632587836 shutdown_connections 2024-03-04T10:06:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 --2- 172.21.15.99:0/2632587836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840b4c50 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 --2- 172.21.15.99:0/2632587836 >> v2:172.21.15.162:3300/0 conn(0x7f35840b0700 0x7f35840b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 -- 172.21.15.99:0/2632587836 >> 172.21.15.99:0/2632587836 conn(0x7f358406f5f0 msgr2=0x7f358406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 -- 172.21.15.99:0/2632587836 shutdown_connections 2024-03-04T10:06:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.532+0000 7f358890a640 1 -- 172.21.15.99:0/2632587836 wait complete. 2024-03-04T10:06:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 Processor -- start 2024-03-04T10:06:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 -- start start 2024-03-04T10:06:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35840b0700 0x7f35840ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f358415b960 0x7f35841f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358415bd40 con 0x7f35840b47c0 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35840d0100 con 0x7f35840b0700 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.534+0000 7f358890a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358415bec0 con 0x7f358415b960 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840a8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56576/0 (socket says 172.21.15.99:56576) 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 -- 172.21.15.99:0/1012435624 learned_addr learned my addr 172.21.15.99:0/1012435624 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3582d9d640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f358415b960 0x7f35841f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f358259c640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35840b0700 0x7f35840ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 -- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f358415b960 msgr2=0x7f35841f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f358415b960 0x7f35841f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 -- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35840b0700 msgr2=0x7f35840ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35840b0700 0x7f35840ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 -- 172.21.15.99:0/1012435624 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35840a93c0 con 0x7f35840b47c0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f358259c640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35840b0700 0x7f35840ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3582d9d640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f358415b960 0x7f35841f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3581d9b640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840a8e80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f357800d800 tx=0x7f357800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357800b6c0 con 0x7f35840b47c0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3578019070 con 0x7f35840b47c0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35840a9660 con 0x7f35840b47c0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357800ba80 con 0x7f35840b47c0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.535+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35840a9c20 con 0x7f35840b47c0 2024-03-04T10:06:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.536+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3584074f60 con 0x7f35840b47c0 2024-03-04T10:06:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.537+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f357800bc20 con 0x7f35840b47c0 2024-03-04T10:06:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.537+0000 7f3572ffd640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f356406b4d0 0x7f356406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.537+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f357808a840 con 0x7f35840b47c0 2024-03-04T10:06:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.537+0000 7f358259c640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f356406b4d0 0x7f356406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.538+0000 7f358259c640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f356406b4d0 0x7f356406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3574001d60 tx=0x7f3574009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.541+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3578059530 con 0x7f35840b47c0 2024-03-04T10:06:17.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.746+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f35840b1f10 con 0x7f35840b47c0 2024-03-04T10:06:17.747 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:06:17.746+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:06:17.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.748+0000 7f3572ffd640 1 -- 172.21.15.99:0/1012435624 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3578058e30 con 0x7f35840b47c0 2024-03-04T10:06:17.749 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f356406b4d0 msgr2=0x7f356406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f356406b4d0 0x7f356406d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3574001d60 tx=0x7f3574009040 comp rx=0 tx=0).stop 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 msgr2=0x7f35840a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840a8e80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f357800d800 tx=0x7f357800dcc0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 shutdown_connections 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f356406b4d0 0x7f356406d990 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3574001d60 tx=0x7f3574009040 comp rx=0 tx=0).stop 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f358415b960 0x7f35841f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35840b47c0 0x7f35840a8e80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 --2- 172.21.15.99:0/1012435624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35840b0700 0x7f35840ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 >> 172.21.15.99:0/1012435624 conn(0x7f358406f5f0 msgr2=0x7f35840b1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:17.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 shutdown_connections 2024-03-04T10:06:17.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.750+0000 7f358890a640 1 -- 172.21.15.99:0/1012435624 wait complete. 2024-03-04T10:06:17.753 INFO:tasks.ceph.mds.a.smithi099.stderr: -3851> 2024-03-04T10:06:03.618+0000 7ff208ac5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:17.758 INFO:tasks.ceph.mds.a.smithi099.stderr: -370> 2024-03-04T10:06:15.247+0000 7ff20b2ca640 -1 mds.0.cache.den(0x10000000000 c) newly corrupt dentry to be committed: [dentry #0x1/a/c [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x5630574a6d80) (dversion lock w=1 last_client=6159) pv=4 v=3 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x563057572000] 2024-03-04T10:06:17.763 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-03-04T10:06:17.782 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:06:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 Processor -- start 2024-03-04T10:06:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 -- start start 2024-03-04T10:06:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7b400b48a0 0x7f7b400b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b400b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4006d8a0 con 0x7f7b4015a810 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4006db80 con 0x7f7b400b48a0 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.865+0000 7f7b46fdf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4006de60 con 0x7f7b4006d3c0 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7b400b48a0 0x7f7b400b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b4557c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b400b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7b400b48a0 0x7f7b400b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:54352/0 (socket says 172.21.15.99:54352) 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b44d7b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7b4015a810 0x7f7b400b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:46640/0 (socket says 172.21.15.99:46640) 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 -- 172.21.15.99:0/2620370760 learned_addr learned my addr 172.21.15.99:0/2620370760 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b4557c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b400b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:59348/0 (socket says 172.21.15.99:59348) 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 -- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 msgr2=0x7f7b400b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 --2- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b400b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 -- 172.21.15.99:0/2620370760 >> v1:172.21.15.162:6789/0 conn(0x7f7b4015a810 legacy=0x7f7b400b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3ffff640 1 -- 172.21.15.99:0/2620370760 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4006e140 con 0x7f7b400b48a0 2024-03-04T10:06:17.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b4557c640 1 --2- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b400b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:17.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.866+0000 7f7b3f7fe640 1 -- 172.21.15.99:0/2620370760 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860374719 0 0) 0x7f7b4006d8a0 con 0x7f7b4015a810 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.867+0000 7f7b3ffff640 1 --2- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b400b29f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009870 tx=0x7f7b2802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8585d7a5960c5ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.867+0000 7f7b3f7fe640 1 -- 172.21.15.99:0/2620370760 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28034070 con 0x7f7b400b48a0 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.867+0000 7f7b3f7fe640 1 -- 172.21.15.99:0/2620370760 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28030840 con 0x7f7b400b48a0 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.867+0000 7f7b3f7fe640 1 -- 172.21.15.99:0/2620370760 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b28030c10 con 0x7f7b400b48a0 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 -- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 msgr2=0x7f7b400b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b400b29f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b28009870 tx=0x7f7b2802b980 comp rx=0 tx=0).stop 2024-03-04T10:06:17.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 -- 172.21.15.99:0/2620370760 shutdown_connections 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b400b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/2620370760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b400b29f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 -- 172.21.15.99:0/2620370760 >> 172.21.15.99:0/2620370760 conn(0x7f7b4006acd0 msgr2=0x7f7b4006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 -- 172.21.15.99:0/2620370760 shutdown_connections 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.868+0000 7f7b46fdf640 1 -- 172.21.15.99:0/2620370760 wait complete. 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.869+0000 7f7b46fdf640 1 Processor -- start 2024-03-04T10:06:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.869+0000 7f7b46fdf640 1 -- start start 2024-03-04T10:06:17.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b40146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b40147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 0x7f7b401491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400cc130 con 0x7f7b4015a810 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b400cbfb0 con 0x7f7b400b48a0 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b40147be0 con 0x7f7b4006d3c0 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 0x7f7b401491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 0x7f7b401491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56614/0 (socket says 172.21.15.99:56614) 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 -- 172.21.15.99:0/623970555 learned_addr learned my addr 172.21.15.99:0/623970555 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b3ffff640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b40147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b44d7b640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b40146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 -- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 msgr2=0x7f7b40146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b40146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 -- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 msgr2=0x7f7b40147120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b40147120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 -- 172.21.15.99:0/623970555 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40147ec0 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b3ffff640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b40147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b4557c640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 0x7f7b401491b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000e890 tx=0x7f7b3000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30017070 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3000c940 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b401481a0 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3000cd10 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.870+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b40146d50 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.871+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b40074f60 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.872+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7b3001a070 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.872+0000 7f7b3d7fa640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7b2006b4d0 0x7f7b2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.872+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7b3008a620 con 0x7f7b4015a810 2024-03-04T10:06:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.872+0000 7f7b44d7b640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7b2006b4d0 0x7f7b2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:17.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.873+0000 7f7b44d7b640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7b2006b4d0 0x7f7b2006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7b38007ed0 tx=0x7f7b38007e60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:17.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:17.876+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7b300592c0 con 0x7f7b4015a810 2024-03-04T10:06:18.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.049+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) v1 -- 0x7f7b4006ec30 con 0x7f7b4015a810 2024-03-04T10:06:18.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.052+0000 7f7b3d7fa640 1 -- 172.21.15.99:0/623970555 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v15) v1 ==== 121+0+0 (secure 0 0 0) 0x7f7b30058bc0 con 0x7f7b4015a810 2024-03-04T10:06:18.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7b2006b4d0 msgr2=0x7f7b2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7b2006b4d0 0x7f7b2006d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7b38007ed0 tx=0x7f7b38007e60 comp rx=0 tx=0).stop 2024-03-04T10:06:18.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 msgr2=0x7f7b401491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 0x7f7b401491b0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000e890 tx=0x7f7b3000ed50 comp rx=0 tx=0).stop 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 shutdown_connections 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7b2006b4d0 0x7f7b2006d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7b4015a810 0x7f7b401491b0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7b400b48a0 0x7f7b40147120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 --2- 172.21.15.99:0/623970555 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7b4006d3c0 0x7f7b40146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 >> 172.21.15.99:0/623970555 conn(0x7f7b4006acd0 msgr2=0x7f7b400cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 shutdown_connections 2024-03-04T10:06:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.054+0000 7f7b46fdf640 1 -- 172.21.15.99:0/623970555 wait complete. 2024-03-04T10:06:18.066 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_abort_on_newly_corrupt_dentry False 2024-03-04T10:06:18.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.211+0000 7f2b89697640 1 Processor -- start 2024-03-04T10:06:18.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.211+0000 7f2b89697640 1 -- start start 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b89697640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2b840ccbd0 0x7f2b840b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b89697640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b89697640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2b840cecf0 0x7f2b840b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b89697640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8415d2f0 con 0x7f2b840cecf0 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b89697640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8415d6d0 con 0x7f2b840ccbd0 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b89697640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8415ca50 con 0x7f2b8415c2e0 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b837fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2b840cecf0 0x7f2b840b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b837fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2b840cecf0 0x7f2b840b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56620/0 (socket says 172.21.15.99:56620) 2024-03-04T10:06:18.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b837fe640 1 -- 172.21.15.99:0/1697601935 learned_addr learned my addr 172.21.15.99:0/1697601935 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b827fc640 1 --2- 172.21.15.99:0/1697601935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.212+0000 7f2b82ffd640 1 --2- 172.21.15.99:0/1697601935 >> v2:172.21.15.162:3300/0 conn(0x7f2b840ccbd0 0x7f2b840b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b837fe640 1 -- 172.21.15.99:0/1697601935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 msgr2=0x7f2b840ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b837fe640 1 --2- 172.21.15.99:0/1697601935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b837fe640 1 -- 172.21.15.99:0/1697601935 >> v2:172.21.15.162:3300/0 conn(0x7f2b840ccbd0 msgr2=0x7f2b840b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b837fe640 1 --2- 172.21.15.99:0/1697601935 >> v2:172.21.15.162:3300/0 conn(0x7f2b840ccbd0 0x7f2b840b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b837fe640 1 -- 172.21.15.99:0/1697601935 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8415cd30 con 0x7f2b840cecf0 2024-03-04T10:06:18.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b837fe640 1 --2- 172.21.15.99:0/1697601935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840cecf0 0x7f2b840b0560 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2b840b38b0 tx=0x7f2b7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cabd52617f6c85 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b81ffb640 1 -- 172.21.15.99:0/1697601935 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c035020 con 0x7f2b840cecf0 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b81ffb640 1 -- 172.21.15.99:0/1697601935 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7c031870 con 0x7f2b840cecf0 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.213+0000 7f2b81ffb640 1 -- 172.21.15.99:0/1697601935 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c031c40 con 0x7f2b840cecf0 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 -- 172.21.15.99:0/1697601935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840cecf0 msgr2=0x7f2b840b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 --2- 172.21.15.99:0/1697601935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840cecf0 0x7f2b840b0560 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2b840b38b0 tx=0x7f2b7c02b910 comp rx=0 tx=0).stop 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 -- 172.21.15.99:0/1697601935 shutdown_connections 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 --2- 172.21.15.99:0/1697601935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840cecf0 0x7f2b840b0560 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 --2- 172.21.15.99:0/1697601935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840ce7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 --2- 172.21.15.99:0/1697601935 >> v2:172.21.15.162:3300/0 conn(0x7f2b840ccbd0 0x7f2b840b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 -- 172.21.15.99:0/1697601935 >> 172.21.15.99:0/1697601935 conn(0x7f2b84066dc0 msgr2=0x7f2b840671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 -- 172.21.15.99:0/1697601935 shutdown_connections 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.214+0000 7f2b89697640 1 -- 172.21.15.99:0/1697601935 wait complete. 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.215+0000 7f2b89697640 1 Processor -- start 2024-03-04T10:06:18.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.215+0000 7f2b89697640 1 -- start start 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b89697640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 0x7f2b840d1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b89697640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b840cecf0 0x7f2b840c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b89697640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b89697640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840ccfb0 con 0x7f2b840ccbd0 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b89697640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8415c760 con 0x7f2b840cecf0 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b89697640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b840b0aa0 con 0x7f2b8415c2e0 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 0x7f2b840d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 0x7f2b840d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56628/0 (socket says 172.21.15.99:56628) 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 -- 172.21.15.99:0/1070592832 learned_addr learned my addr 172.21.15.99:0/1070592832 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b837fe640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b827fc640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b840cecf0 0x7f2b840c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 -- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 msgr2=0x7f2b840cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 -- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b840cecf0 msgr2=0x7f2b840c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b840cecf0 0x7f2b840c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b82ffd640 1 -- 172.21.15.99:0/1070592832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b840d05a0 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.216+0000 7f2b837fe640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.217+0000 7f2b82ffd640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 0x7f2b840d1640 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2b74007e60 tx=0x7f2b7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.217+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74014620 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.217+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b74019070 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.217+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b840d0880 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.217+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74014a00 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.217+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b840d0e40 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.218+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b84074f60 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.218+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2b74014ba0 con 0x7f2b840ccbd0 2024-03-04T10:06:18.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.219+0000 7f2b6b7fe640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2b7006b4d0 0x7f2b7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.219+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2b7408ad40 con 0x7f2b840ccbd0 2024-03-04T10:06:18.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.219+0000 7f2b827fc640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2b7006b4d0 0x7f2b7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.219+0000 7f2b827fc640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2b7006b4d0 0x7f2b7006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006fd0 tx=0x7f2b78008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.223+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2b74059960 con 0x7f2b840ccbd0 2024-03-04T10:06:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.393+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}] v 0) v1 -- 0x7f2b84159940 con 0x7f2b840ccbd0 2024-03-04T10:06:18.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.393+0000 7f2b6b7fe640 1 -- 172.21.15.99:0/1070592832 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_abort_on_newly_corrupt_dentry}]=0 v15)=0 v15) v1 ==== 135+0+0 (secure 0 0 0) 0x7f2b74059260 con 0x7f2b840ccbd0 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2b7006b4d0 msgr2=0x7f2b7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2b7006b4d0 0x7f2b7006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2b78006fd0 tx=0x7f2b78008040 comp rx=0 tx=0).stop 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 msgr2=0x7f2b840d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 0x7f2b840d1640 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2b74007e60 tx=0x7f2b7400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 shutdown_connections 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2b7006b4d0 0x7f2b7006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2b8415c2e0 0x7f2b840cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b840cecf0 0x7f2b840c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 --2- 172.21.15.99:0/1070592832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2b840ccbd0 0x7f2b840d1640 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.395+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 >> 172.21.15.99:0/1070592832 conn(0x7f2b84066dc0 msgr2=0x7f2b8415da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:18.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.396+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 shutdown_connections 2024-03-04T10:06:18.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.396+0000 7f2b89697640 1 -- 172.21.15.99:0/1070592832 wait complete. 2024-03-04T10:06:18.407 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:06:18.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.509+0000 7fea1c989640 1 Processor -- start 2024-03-04T10:06:18.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.509+0000 7fea1c989640 1 -- start start 2024-03-04T10:06:18.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1c989640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fea18161390 0x7fea1806c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1c989640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea180d20c0 0x7fea180ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1c989640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180cec20 0x7fea18161aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1c989640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1806c5a0 con 0x7fea180d20c0 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1c989640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1806d3e0 con 0x7fea18161390 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1c989640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea180cf740 con 0x7fea180cec20 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea15d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea180d20c0 0x7fea180ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea15d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea180d20c0 0x7fea180ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56634/0 (socket says 172.21.15.99:56634) 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea15d9b640 1 -- 172.21.15.99:0/3925678357 learned_addr learned my addr 172.21.15.99:0/3925678357 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea16d9d640 1 --2- 172.21.15.99:0/3925678357 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180cec20 0x7fea18161aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.510+0000 7fea1659c640 1 --2- 172.21.15.99:0/3925678357 >> v2:172.21.15.162:3300/0 conn(0x7fea18161390 0x7fea1806c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea15d9b640 1 -- 172.21.15.99:0/3925678357 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180cec20 msgr2=0x7fea18161aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea15d9b640 1 --2- 172.21.15.99:0/3925678357 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180cec20 0x7fea18161aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea15d9b640 1 -- 172.21.15.99:0/3925678357 >> v2:172.21.15.162:3300/0 conn(0x7fea18161390 msgr2=0x7fea1806c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea15d9b640 1 --2- 172.21.15.99:0/3925678357 >> v2:172.21.15.162:3300/0 conn(0x7fea18161390 0x7fea1806c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea15d9b640 1 -- 172.21.15.99:0/3925678357 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1806c7d0 con 0x7fea180d20c0 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea15d9b640 1 --2- 172.21.15.99:0/3925678357 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180d20c0 0x7fea180ce6e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fea18071290 tx=0x7fea0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b211c1a68f45eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.511+0000 7fea1559a640 1 -- 172.21.15.99:0/3925678357 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c035020 con 0x7fea180d20c0 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1559a640 1 -- 172.21.15.99:0/3925678357 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c031870 con 0x7fea180d20c0 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1559a640 1 -- 172.21.15.99:0/3925678357 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c031c40 con 0x7fea180d20c0 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 -- 172.21.15.99:0/3925678357 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180d20c0 msgr2=0x7fea180ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 --2- 172.21.15.99:0/3925678357 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180d20c0 0x7fea180ce6e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fea18071290 tx=0x7fea0c02b910 comp rx=0 tx=0).stop 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 -- 172.21.15.99:0/3925678357 shutdown_connections 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 --2- 172.21.15.99:0/3925678357 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180cec20 0x7fea18161aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 --2- 172.21.15.99:0/3925678357 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180d20c0 0x7fea180ce6e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 --2- 172.21.15.99:0/3925678357 >> v2:172.21.15.162:3300/0 conn(0x7fea18161390 0x7fea1806c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 -- 172.21.15.99:0/3925678357 >> 172.21.15.99:0/3925678357 conn(0x7fea18066dc0 msgr2=0x7fea180671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:18.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 -- 172.21.15.99:0/3925678357 shutdown_connections 2024-03-04T10:06:18.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.512+0000 7fea1c989640 1 -- 172.21.15.99:0/3925678357 wait complete. 2024-03-04T10:06:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.513+0000 7fea1c989640 1 Processor -- start 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 -- start start 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 0x7fea1815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180d20c0 0x7fea181f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea18161390 0x7fea18160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1806d070 con 0x7fea180cec20 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1806cd90 con 0x7fea18161390 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1c989640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea18161770 con 0x7fea180d20c0 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 0x7fea1815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 0x7fea1815e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56636/0 (socket says 172.21.15.99:56636) 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.514+0000 7fea1659c640 1 -- 172.21.15.99:0/1091432173 learned_addr learned my addr 172.21.15.99:0/1091432173 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea15d9b640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180d20c0 0x7fea181f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea16d9d640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea18161390 0x7fea18160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1659c640 1 -- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180d20c0 msgr2=0x7fea181f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1659c640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180d20c0 0x7fea181f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1659c640 1 -- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea18161390 msgr2=0x7fea18160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1659c640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea18161390 0x7fea18160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1659c640 1 -- 172.21.15.99:0/1091432173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea181f4b70 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea16d9d640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea18161390 0x7fea18160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1659c640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 0x7fea1815e7d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fea0800efc0 tx=0x7fea0800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08018020 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08014480 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea181f4e50 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08014850 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.515+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1815a270 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.516+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea180d2520 con 0x7fea180cec20 2024-03-04T10:06:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.517+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fea08005d10 con 0x7fea180cec20 2024-03-04T10:06:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.517+0000 7fea06ffd640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe9f806b4d0 0x7fe9f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.517+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fea0808ad10 con 0x7fea180cec20 2024-03-04T10:06:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.517+0000 7fea15d9b640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe9f806b4d0 0x7fe9f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.518+0000 7fea15d9b640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe9f806b4d0 0x7fe9f806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fea0c036000 tx=0x7fea0c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.521+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fea180d2520 con 0x7fea180cec20 2024-03-04T10:06:18.728 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.727+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fea18074f60 con 0x7fea180cec20 2024-03-04T10:06:18.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.749+0000 7fea06ffd640 1 -- 172.21.15.99:0/1091432173 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v32) v1 ==== 162+0+0 (secure 0 0 0) 0x7fea08022070 con 0x7fea180cec20 2024-03-04T10:06:18.750 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:06:18.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe9f806b4d0 msgr2=0x7fe9f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe9f806b4d0 0x7fe9f806d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fea0c036000 tx=0x7fea0c03c040 comp rx=0 tx=0).stop 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 msgr2=0x7fea1815e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 0x7fea1815e7d0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fea0800efc0 tx=0x7fea0800c490 comp rx=0 tx=0).stop 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 shutdown_connections 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe9f806b4d0 0x7fe9f806d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea18161390 0x7fea18160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea180d20c0 0x7fea181f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 --2- 172.21.15.99:0/1091432173 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea180cec20 0x7fea1815e7d0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 >> 172.21.15.99:0/1091432173 conn(0x7fea18066dc0 msgr2=0x7fea180c8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 shutdown_connections 2024-03-04T10:06:18.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.751+0000 7fea1c989640 1 -- 172.21.15.99:0/1091432173 wait complete. 2024-03-04T10:06:18.765 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:18.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.863+0000 7fdfe74b2640 1 Processor -- start 2024-03-04T10:06:18.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.863+0000 7fdfe74b2640 1 -- start start 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe74b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe0154a80 0x7fdfe015e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe74b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdfe00cf650 0x7fdfe00cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe74b2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe00cdaa0 con 0x7fdfe01616c0 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe74b2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe00cfff0 con 0x7fdfe00cf650 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe74b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe0071df0 con 0x7fdfe0154a80 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe4a4d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdfe00cf650 0x7fdfe00cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe5a4f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdfe01616c0 0x7fdfe0161b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37106/0 (socket says 172.21.15.99:37106) 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe5a4f640 1 -- 172.21.15.99:0/4101195087 learned_addr learned my addr 172.21.15.99:0/4101195087 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:18.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/4101195087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe0154a80 msgr2=0x7fdfe015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe4a4d640 1 --2- 172.21.15.99:0/4101195087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe0154a80 0x7fdfe015e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/4101195087 >> v1:172.21.15.99:6789/0 conn(0x7fdfe01616c0 legacy=0x7fdfe0161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/4101195087 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe00cd010 con 0x7fdfe00cf650 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.864+0000 7fdfd7fff640 1 -- 172.21.15.99:0/4101195087 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425915190 0 0) 0x7fdfe00cdaa0 con 0x7fdfe01616c0 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe4a4d640 1 --2- 172.21.15.99:0/4101195087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe00cf650 0x7fdfe00cfab0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00097d0 tx=0x7fdfd002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b149418e3bbba1bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfd7fff640 1 -- 172.21.15.99:0/4101195087 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0035020 con 0x7fdfe00cf650 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfd7fff640 1 -- 172.21.15.99:0/4101195087 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd0031800 con 0x7fdfe00cf650 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfd7fff640 1 -- 172.21.15.99:0/4101195087 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0031bd0 con 0x7fdfe00cf650 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe74b2640 1 -- 172.21.15.99:0/4101195087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe00cf650 msgr2=0x7fdfe00cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/4101195087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe00cf650 0x7fdfe00cfab0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00097d0 tx=0x7fdfd002b8a0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe74b2640 1 -- 172.21.15.99:0/4101195087 shutdown_connections 2024-03-04T10:06:18.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/4101195087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe00cf650 0x7fdfe00cfab0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/4101195087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe0154a80 0x7fdfe015e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.865+0000 7fdfe74b2640 1 -- 172.21.15.99:0/4101195087 >> 172.21.15.99:0/4101195087 conn(0x7fdfe006f5f0 msgr2=0x7fdfe006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:18.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.866+0000 7fdfe74b2640 1 -- 172.21.15.99:0/4101195087 shutdown_connections 2024-03-04T10:06:18.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.866+0000 7fdfe74b2640 1 -- 172.21.15.99:0/4101195087 wait complete. 2024-03-04T10:06:18.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 Processor -- start 2024-03-04T10:06:18.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 -- start start 2024-03-04T10:06:18.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfe00cf650 0x7fdfe0152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 0x7fdfe015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe01616c0 0x7fdfe015b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00ce340 con 0x7fdfe0154a80 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00ce060 con 0x7fdfe00cf650 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe74b2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00ce620 con 0x7fdfe01616c0 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe4a4d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 0x7fdfe015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe4a4d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 0x7fdfe015ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56642/0 (socket says 172.21.15.99:56642) 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.867+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/3786534430 learned_addr learned my addr 172.21.15.99:0/3786534430 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe5a4f640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe01616c0 0x7fdfe015b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe524e640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfe00cf650 0x7fdfe0152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe01616c0 msgr2=0x7fdfe015b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe4a4d640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe01616c0 0x7fdfe015b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfe00cf650 msgr2=0x7fdfe0152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:18.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe4a4d640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfe00cf650 0x7fdfe0152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe4a4d640 1 -- 172.21.15.99:0/3786534430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe0153830 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe524e640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfe00cf650 0x7fdfe0152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe5a4f640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe01616c0 0x7fdfe015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe4a4d640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 0x7fdfe015ac70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0000c00 tx=0x7fdfd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0035050 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd003bd70 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe015e720 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0004270 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.868+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe015ece0 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.869+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe0074f60 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.870+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdfd0038020 con 0x7fdfe0154a80 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.870+0000 7fdfd5ffb640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfc406b4d0 0x7fdfc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.870+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdfd00ac4f0 con 0x7fdfe0154a80 2024-03-04T10:06:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.870+0000 7fdfe524e640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfc406b4d0 0x7fdfc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.871+0000 7fdfe524e640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfc406b4d0 0x7fdfc406d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009930 tx=0x7fdfdc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:18.874+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdfd0040070 con 0x7fdfe0154a80 2024-03-04T10:06:19.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.084+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfe00c86b0 con 0x7fdfe0154a80 2024-03-04T10:06:19.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.084+0000 7fdfd5ffb640 1 -- 172.21.15.99:0/3786534430 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4939 (secure 0 0 0) 0x7fdfd0043340 con 0x7fdfe0154a80 2024-03-04T10:06:19.086 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:19.086 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:18.751810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:19.086 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 33 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.086+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfc406b4d0 msgr2=0x7fdfc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.086+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfc406b4d0 0x7fdfc406d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009930 tx=0x7fdfdc008040 comp rx=0 tx=0).stop 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.086+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 msgr2=0x7fdfe015ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.086+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 0x7fdfe015ac70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0000c00 tx=0x7fdfd00046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 shutdown_connections 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfc406b4d0 0x7fdfc406d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfe01616c0 0x7fdfe015b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfe0154a80 0x7fdfe015ac70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 --2- 172.21.15.99:0/3786534430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfe00cf650 0x7fdfe0152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 >> 172.21.15.99:0/3786534430 conn(0x7fdfe006f5f0 msgr2=0x7fdfe0161b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 shutdown_connections 2024-03-04T10:06:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.087+0000 7fdfe74b2640 1 -- 172.21.15.99:0/3786534430 wait complete. 2024-03-04T10:06:19.098 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T10:06:19.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 flush journal 2024-03-04T10:06:19.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.199+0000 7f764dd9e640 1 Processor -- start 2024-03-04T10:06:19.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.199+0000 7f764dd9e640 1 -- start start 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f764dd9e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 0x7f76480b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f764dd9e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480cebd0 con 0x7f76481450e0 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f764dd9e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76480ceeb0 con 0x7f76480cfc70 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f764dd9e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480cf7e0 con 0x7f76480cc510 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76477fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f76481450e0 0x7f764815c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37114/0 (socket says 172.21.15.99:37114) 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76477fe640 1 -- 172.21.15.99:0/2391690962 learned_addr learned my addr 172.21.15.99:0/2391690962 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019129943 0 0) 0x7f76480cebd0 con 0x7f76481450e0 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7634003520 con 0x7f76481450e0 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f7647fff640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 0x7f76480b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238257763 0 0) 0x7f7634003520 con 0x7f76481450e0 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 msgr2=0x7f76480b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 0x7f76480b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 >> v1:172.21.15.162:6789/0 conn(0x7f76480cfc70 legacy=0x7f76480cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76480cf190 con 0x7f76481450e0 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7630001670 con 0x7f76481450e0 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 >> v1:172.21.15.99:6789/0 conn(0x7f76481450e0 legacy=0x7f764815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f76340039e0 0x7f7634003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7634004800 0x7f7634004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 0x7f76340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480cf7e0 con 0x7f76340039e0 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763400baa0 con 0x7f7634004800 2024-03-04T10:06:19.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.200+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763400bca0 con 0x7f76480cc510 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f76477fe640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f76340039e0 0x7f7634003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f7647fff640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 0x7f76340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f7646ffd640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7634004800 0x7f7634004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f76477fe640 1 -- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 msgr2=0x7f76340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f76477fe640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76480cc510 0x7f76340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f76477fe640 1 -- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7634004800 msgr2=0x7f7634004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f76477fe640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7634004800 0x7f7634004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f76477fe640 1 -- 172.21.15.99:0/2391690962 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7634003520 con 0x7f76340039e0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.201+0000 7f7646ffd640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7634004800 0x7f7634004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f76477fe640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f76340039e0 0x7f7634003e00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f763000c210 tx=0x7f7634005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5980dbcacfb9cc16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630007cf0 con 0x7f76340039e0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7630030070 con 0x7f76340039e0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f76467fc640 1 -- 172.21.15.99:0/2391690962 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630005d30 con 0x7f76340039e0 2024-03-04T10:06:19.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f764dd9e640 1 -- 172.21.15.99:0/2391690962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f76340039e0 msgr2=0x7f7634003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2391690962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f76340039e0 0x7f7634003e00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f763000c210 tx=0x7f7634005830 comp rx=0 tx=0).stop 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f7647fff640 1 -- 172.21.15.99:0/2391690962 reap_dead start 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f764dd9e640 1 -- 172.21.15.99:0/2391690962 shutdown_connections 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f764dd9e640 1 -- 172.21.15.99:0/2391690962 >> 172.21.15.99:0/2391690962 conn(0x7f764806acd0 msgr2=0x7f764806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f764dd9e640 1 -- 172.21.15.99:0/2391690962 shutdown_connections 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.202+0000 7f764dd9e640 1 -- 172.21.15.99:0/2391690962 wait complete. 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.203+0000 7f764dd9e640 1 Processor -- start 2024-03-04T10:06:19.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 -- start start 2024-03-04T10:06:19.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76340039e0 0x7f764815b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 0x7f7648132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480b3250 0x7f76480b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76480ccd10 con 0x7f7648131e10 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481454c0 con 0x7f76480b3250 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f764dd9e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76480ce760 con 0x7f76340039e0 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 0x7f7648132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 0x7f7648132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56650/0 (socket says 172.21.15.99:56650) 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.204+0000 7f7646ffd640 1 -- 172.21.15.99:0/2664428111 learned_addr learned my addr 172.21.15.99:0/2664428111 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f76477fe640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76340039e0 0x7f764815b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7647fff640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480b3250 0x7f76480b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7646ffd640 1 -- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76340039e0 msgr2=0x7f764815b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7646ffd640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76340039e0 0x7f764815b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7646ffd640 1 -- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480b3250 msgr2=0x7f76480b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7646ffd640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480b3250 0x7f76480b36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7646ffd640 1 -- 172.21.15.99:0/2664428111 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76481328d0 con 0x7f7648131e10 2024-03-04T10:06:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f76477fe640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76340039e0 0x7f764815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7647fff640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480b3250 0x7f76480b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f7646ffd640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 0x7f7648132270 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f763800e4d0 tx=0x7f763800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f763ffff640 1 -- 172.21.15.99:0/2664428111 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380052e0 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f763ffff640 1 -- 172.21.15.99:0/2664428111 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7638018070 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7648132bb0 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f763ffff640 1 -- 172.21.15.99:0/2664428111 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638003810 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.205+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7648130270 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.206+0000 7f763ffff640 1 -- 172.21.15.99:0/2664428111 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f76380039b0 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.207+0000 7f763ffff640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f762806b500 0x7f762806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.207+0000 7f763ffff640 1 -- 172.21.15.99:0/2664428111 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f763808a0c0 con 0x7f7648131e10 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.207+0000 7f76477fe640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f762806b500 0x7f762806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.207+0000 7f76477fe640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f762806b500 0x7f762806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7630034000 tx=0x7f7630007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.208+0000 7f765097c740 1 Processor -- start 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.208+0000 7f765097c740 1 -- start start 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506860 0x56512b4d9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f765097c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 0x56512b5095f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b4d9d40 0x56512b509b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f765097c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512b442e00 con 0x56512b506c40 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512b3ebb10 con 0x56512b4d9d40 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56512b43eea0 con 0x56512b506860 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 0x56512b5095f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 0x56512b5095f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56660/0 (socket says 172.21.15.99:56660) 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 -- 172.21.15.99:0/596693312 learned_addr learned my addr 172.21.15.99:0/596693312 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f76477fe640 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506860 0x56512b4d9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7647fff640 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b4d9d40 0x56512b509b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 -- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506860 msgr2=0x56512b4d9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506860 0x56512b4d9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 -- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b4d9d40 msgr2=0x56512b509b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b4d9d40 0x56512b509b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 -- 172.21.15.99:0/596693312 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56512b2217d0 con 0x56512b506c40 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f7646ffd640 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 0x56512b5095f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f76380011d0 tx=0x7f7638057f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b3689e6c66722 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f763d7fa640 1 -- 172.21.15.99:0/596693312 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638086070 con 0x56512b506c40 2024-03-04T10:06:19.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f763d7fa640 1 -- 172.21.15.99:0/596693312 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763805c720 con 0x56512b506c40 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.209+0000 7f763d7fa640 1 -- 172.21.15.99:0/596693312 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638060d70 con 0x56512b506c40 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 -- 172.21.15.99:0/596693312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 msgr2=0x56512b5095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 0x56512b5095f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f76380011d0 tx=0x7f7638057f30 comp rx=0 tx=0).stop 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 -- 172.21.15.99:0/596693312 shutdown_connections 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b4d9d40 0x56512b509b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506c40 0x56512b5095f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 --2- 172.21.15.99:0/596693312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506860 0x56512b4d9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 -- 172.21.15.99:0/596693312 >> 172.21.15.99:0/596693312 conn(0x56512b4dba70 msgr2=0x56512b5245f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 -- 172.21.15.99:0/596693312 shutdown_connections 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.210+0000 7f765097c740 1 -- 172.21.15.99:0/596693312 wait complete. 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 -- start start 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 Processor -- start 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b50cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506c40 0x56512b500ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b501000 0x56512b508020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512b4de010 con 0x56512b506860 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512b3ff0a0 con 0x56512b506c40 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56512b4dd400 con 0x56512b501000 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f76477fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b50cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.211+0000 7f765097c740 10 client.0 _ll_register_callbacks cb 0x56512b3fea70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b50cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56666/0 (socket says 172.21.15.99:56666) 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 -- 172.21.15.99:0/1999955077 learned_addr learned my addr 172.21.15.99:0/1999955077 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506c40 0x56512b500ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f7647fff640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b501000 0x56512b508020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 -- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b501000 msgr2=0x56512b508020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b501000 0x56512b508020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 -- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506c40 msgr2=0x56512b500ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506c40 0x56512b500ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 -- 172.21.15.99:0/1999955077 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56512b1573f0 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506c40 0x56512b500ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76477fe640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b50cd70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f763002c4e0 tx=0x7f763002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5377c8ffec7c7637 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76267fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76267fc640 1 -- 172.21.15.99:0/1999955077 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630007540 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76267fc640 1 -- 172.21.15.99:0/1999955077 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f76300076e0 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f763805c720 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76267fc640 1 -- 172.21.15.99:0/1999955077 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7630035470 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 10 client.6285 fetch_fsmap learned FSMap version 33 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x56512b1170f0 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76267fc640 1 -- 172.21.15.99:0/1999955077 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763805c720 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f76267fc640 1 -- 172.21.15.99:0/1999955077 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7630035610 con 0x56512b506860 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 10 client.6285 fetch_fsmap finished waiting for FSMap version 33 2024-03-04T10:06:19.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 10 client.6285 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 20 client.6285 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b50a5e0 0x56512b50a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 4 client.6285 mds_command: new command op to 6249 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.212+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 --> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56512b450340 con 0x56512b50a5e0 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.213+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b50a5e0 0x56512b50a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.213+0000 7f76267fc640 10 client.6285 ms_handle_connect on v2:172.21.15.99:6832/628973925 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.213+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b50a5e0 0x56512b50a9e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd37afa7f0ff31f7 server_cookie=37b1a5990117e5a7 in_seq=0 out_seq=0 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.213+0000 7f76267fc640 1 -- 172.21.15.99:0/1999955077 <== mds.0 v2:172.21.15.99:6832/628973925 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56512b450340 con 0x56512b50a5e0 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.213+0000 7f76267fc640 10 client.6285 handle_command_reply: tid=0 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.213+0000 7f765097c740 1 client.6285 shutdown 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 msgr2=0x56512b50cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b50cd70 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f763002c4e0 tx=0x7f763002e070 comp rx=0 tx=0).stop 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 shutdown_connections 2024-03-04T10:06:19.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b501000 0x56512b508020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b50a5e0 0x56512b50a9e0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506c40 0x56512b500ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 --2- 172.21.15.99:0/1999955077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b50cd70 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f76267fc640 0 client.6285 ms_handle_reset on v2:172.21.15.99:6832/628973925 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 >> 172.21.15.99:0/1999955077 conn(0x56512b4d9d40 msgr2=0x56512b527b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 shutdown_connections 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 1 -- 172.21.15.99:0/1999955077 wait complete. 2024-03-04T10:06:19.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.214+0000 7f765097c740 20 client.6285 trim_cache size 0 max 16384 2024-03-04T10:06:19.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.227+0000 7f765097c740 1 Processor -- start 2024-03-04T10:06:19.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.227+0000 7f765097c740 1 -- start start 2024-03-04T10:06:19.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b501000 0x56512b5000e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f765097c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b507e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b4dd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f765097c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648201000 con 0x56512b506860 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7648200200 con 0x56512b501000 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7648131920 con 0x56512b506c40 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b507e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b507e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56678/0 (socket says 172.21.15.99:56678) 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 -- 172.21.15.99:0/1714863528 learned_addr learned my addr 172.21.15.99:0/1714863528 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 -- 172.21.15.99:0/1714863528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 msgr2=0x56512b4dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b4dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 -- 172.21.15.99:0/1714863528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b501000 msgr2=0x56512b5000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b501000 0x56512b5000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 -- 172.21.15.99:0/1714863528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56512b508b50 con 0x56512b506860 2024-03-04T10:06:19.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f7646ffd640 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b507e70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7638088460 tx=0x7f763806e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99898473c0c6d54 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f763cff9640 1 -- 172.21.15.99:0/1714863528 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7638086040 con 0x56512b506860 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f763cff9640 1 -- 172.21.15.99:0/1714863528 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763805c720 con 0x56512b506860 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.228+0000 7f763cff9640 1 -- 172.21.15.99:0/1714863528 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76380701f0 con 0x56512b506860 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 -- 172.21.15.99:0/1714863528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 msgr2=0x56512b507e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b507e70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f7638088460 tx=0x7f763806e000 comp rx=0 tx=0).stop 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 -- 172.21.15.99:0/1714863528 shutdown_connections 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b4dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b506860 0x56512b507e70 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 --2- 172.21.15.99:0/1714863528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b501000 0x56512b5000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 -- 172.21.15.99:0/1714863528 >> 172.21.15.99:0/1714863528 conn(0x56512b4d9d40 msgr2=0x56512b4e0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 -- 172.21.15.99:0/1714863528 shutdown_connections 2024-03-04T10:06:19.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 -- 172.21.15.99:0/1714863528 wait complete. 2024-03-04T10:06:19.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 -- start start 2024-03-04T10:06:19.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.229+0000 7f765097c740 1 Processor -- start 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 0x56512b4d8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506860 0x56512b50c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b50cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56512b504d80 con 0x56512b501000 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76481ffc30 con 0x56512b506860 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56512b533cd0 con 0x56512b506c40 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f765097c740 10 client.0 _ll_register_callbacks cb 0x7f763000c0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 0x56512b4d8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 0x56512b4d8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56692/0 (socket says 172.21.15.99:56692) 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 -- 172.21.15.99:0/83533180 learned_addr learned my addr 172.21.15.99:0/83533180 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f7647fff640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b50cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f7646ffd640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506860 0x56512b50c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 -- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 msgr2=0x56512b50cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b50cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 -- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506860 msgr2=0x56512b50c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.230+0000 7f76477fe640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506860 0x56512b50c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76477fe640 1 -- 172.21.15.99:0/83533180 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56512b4fd2c0 con 0x56512b501000 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76477fe640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 0x56512b4d8f50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f76300358c0 tx=0x7f76480cf080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71369c8dac23ead server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76267fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76267fc640 1 -- 172.21.15.99:0/83533180 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630060060 con 0x56512b501000 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76267fc640 1 -- 172.21.15.99:0/83533180 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7630035d60 con 0x56512b501000 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f763805c720 con 0x56512b501000 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76267fc640 1 -- 172.21.15.99:0/83533180 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7630066050 con 0x56512b501000 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 10 client.6291 fetch_fsmap learned FSMap version 33 2024-03-04T10:06:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x56512b519f80 con 0x56512b501000 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76267fc640 1 -- 172.21.15.99:0/83533180 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763805c720 con 0x56512b501000 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f76267fc640 1 -- 172.21.15.99:0/83533180 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 33) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7630069020 con 0x56512b501000 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 10 client.6291 fetch_fsmap finished waiting for FSMap version 33 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 10 client.6291 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 20 client.6291 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b4d89b0 0x56512b501440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 4 client.6291 mds_command: new command op to 6249 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 --> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56512b44a690 con 0x56512b4d89b0 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.231+0000 7f7646ffd640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b4d89b0 0x56512b501440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f76267fc640 10 client.6291 ms_handle_connect on v2:172.21.15.99:6832/628973925 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f7646ffd640 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b4d89b0 0x56512b501440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=145070b588edcb74 server_cookie=1c03feeca49b55e6 in_seq=0 out_seq=0 2024-03-04T10:06:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f76267fc640 1 -- 172.21.15.99:0/83533180 <== mds.0 v2:172.21.15.99:6832/628973925 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56512b44a690 con 0x56512b4d89b0 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f76267fc640 10 client.6291 handle_command_reply: tid=0 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 client.6291 shutdown 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 msgr2=0x56512b4d8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 0x56512b4d8f50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f76300358c0 tx=0x7f76480cf080 comp rx=0 tx=0).stop 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 shutdown_connections 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56512b506c40 0x56512b50cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:6832/628973925,v1:172.21.15.99:6833/628973925] conn(0x56512b4d89b0 0x56512b501440 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56512b506860 0x56512b50c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 --2- 172.21.15.99:0/83533180 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56512b501000 0x56512b4d8f50 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f76267fc640 0 client.6291 ms_handle_reset on v2:172.21.15.99:6832/628973925 2024-03-04T10:06:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 >> 172.21.15.99:0/83533180 conn(0x56512b4d9d40 msgr2=0x56512b4dd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 shutdown_connections 2024-03-04T10:06:19.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 1 -- 172.21.15.99:0/83533180 wait complete. 2024-03-04T10:06:19.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f765097c740 20 client.6291 trim_cache size 0 max 16384 2024-03-04T10:06:19.242 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:19.242 INFO:teuthology.orchestra.run.smithi099.stdout: "message": "", 2024-03-04T10:06:19.242 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0 2024-03-04T10:06:19.243 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.232+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f762806b500 msgr2=0x7f762806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f762806b500 0x7f762806d9c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7630034000 tx=0x7f7630007000 comp rx=0 tx=0).stop 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 msgr2=0x7f7648132270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 0x7f7648132270 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f763800e4d0 tx=0x7f763800e990 comp rx=0 tx=0).stop 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 shutdown_connections 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f762806b500 0x7f762806d9c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f76480b3250 0x7f76480b36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7648131e10 0x7f7648132270 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 --2- 172.21.15.99:0/2664428111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f76340039e0 0x7f764815b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 >> 172.21.15.99:0/2664428111 conn(0x7f764806acd0 msgr2=0x7f76480cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 shutdown_connections 2024-03-04T10:06:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.233+0000 7f764dd9e640 1 -- 172.21.15.99:0/2664428111 wait complete. 2024-03-04T10:06:19.245 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.344+0000 7fe124d7e640 1 Processor -- start 2024-03-04T10:06:19.346 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.344+0000 7fe124d7e640 1 -- start start 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe124d7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe12006b3e0 0x7fe1200cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe124d7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1200cc8d0 0x7fe1200cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe124d7e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12006fd20 con 0x7fe12015dea0 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe124d7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12006fad0 con 0x7fe12006b3e0 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe124d7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe120070000 con 0x7fe1200cc8d0 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe11e59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe12015dea0 0x7fe12006acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37116/0 (socket says 172.21.15.99:37116) 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe11e59c640 1 -- 172.21.15.99:0/3775281126 learned_addr learned my addr 172.21.15.99:0/3775281126 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe11ed9d640 1 --2- 172.21.15.99:0/3775281126 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1200cc8d0 0x7fe1200cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe11dd9b640 1 --2- 172.21.15.99:0/3775281126 >> v2:172.21.15.162:3300/0 conn(0x7fe12006b3e0 0x7fe1200cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe11d59a640 1 -- 172.21.15.99:0/3775281126 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171431590 0 0) 0x7fe12006fd20 con 0x7fe12015dea0 2024-03-04T10:06:19.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.345+0000 7fe11d59a640 1 -- 172.21.15.99:0/3775281126 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe108003520 con 0x7fe12015dea0 2024-03-04T10:06:19.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.346+0000 7fe11d59a640 1 -- 172.21.15.99:0/3775281126 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 459731351 0 0) 0x7fe108003520 con 0x7fe12015dea0 2024-03-04T10:06:19.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.346+0000 7fe11ed9d640 1 -- 172.21.15.99:0/3775281126 >> v2:172.21.15.162:3300/0 conn(0x7fe12006b3e0 msgr2=0x7fe1200cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.346+0000 7fe11ed9d640 1 --2- 172.21.15.99:0/3775281126 >> v2:172.21.15.162:3300/0 conn(0x7fe12006b3e0 0x7fe1200cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.346+0000 7fe11ed9d640 1 -- 172.21.15.99:0/3775281126 >> v1:172.21.15.99:6789/0 conn(0x7fe12015dea0 legacy=0x7fe12006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.346+0000 7fe11ed9d640 1 -- 172.21.15.99:0/3775281126 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200702e0 con 0x7fe1200cc8d0 2024-03-04T10:06:19.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe11ed9d640 1 --2- 172.21.15.99:0/3775281126 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1200cc8d0 0x7fe1200cd300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe1180097a0 tx=0x7fe11802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70a7dc687650eb82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe11d59a640 1 -- 172.21.15.99:0/3775281126 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe118037070 con 0x7fe1200cc8d0 2024-03-04T10:06:19.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe11d59a640 1 -- 172.21.15.99:0/3775281126 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118030510 con 0x7fe1200cc8d0 2024-03-04T10:06:19.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe11d59a640 1 -- 172.21.15.99:0/3775281126 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110001a10 con 0x7fe1200cc8d0 2024-03-04T10:06:19.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe124d7e640 1 -- 172.21.15.99:0/3775281126 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1200cc8d0 msgr2=0x7fe1200cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe124d7e640 1 --2- 172.21.15.99:0/3775281126 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1200cc8d0 0x7fe1200cd300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe1180097a0 tx=0x7fe11802b790 comp rx=0 tx=0).stop 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe124d7e640 1 -- 172.21.15.99:0/3775281126 shutdown_connections 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe124d7e640 1 --2- 172.21.15.99:0/3775281126 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe1200cc8d0 0x7fe1200cd300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe124d7e640 1 --2- 172.21.15.99:0/3775281126 >> v2:172.21.15.162:3300/0 conn(0x7fe12006b3e0 0x7fe1200cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.347+0000 7fe124d7e640 1 -- 172.21.15.99:0/3775281126 >> 172.21.15.99:0/3775281126 conn(0x7fe1200d0710 msgr2=0x7fe1200d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.348+0000 7fe124d7e640 1 -- 172.21.15.99:0/3775281126 shutdown_connections 2024-03-04T10:06:19.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.348+0000 7fe124d7e640 1 -- 172.21.15.99:0/3775281126 wait complete. 2024-03-04T10:06:19.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.349+0000 7fe124d7e640 1 Processor -- start 2024-03-04T10:06:19.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.349+0000 7fe124d7e640 1 -- start start 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 0x7fe1200c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cc8d0 0x7fe1200c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015dea0 0x7fe1200b5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200d1380 con 0x7fe12006b3e0 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200d1200 con 0x7fe1200cc8d0 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1200d1500 con 0x7fe12015dea0 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 0x7fe1200c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 0x7fe1200c81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56694/0 (socket says 172.21.15.99:56694) 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 -- 172.21.15.99:0/762403586 learned_addr learned my addr 172.21.15.99:0/762403586 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11ed9d640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015dea0 0x7fe1200b5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11dd9b640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cc8d0 0x7fe1200c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 -- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015dea0 msgr2=0x7fe1200b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015dea0 0x7fe1200b5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 -- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cc8d0 msgr2=0x7fe1200c5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cc8d0 0x7fe1200c5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 -- 172.21.15.99:0/762403586 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1200c4970 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11ed9d640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015dea0 0x7fe1200b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe11e59c640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 0x7fe1200c81c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe11000caa0 tx=0x7fe11000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110018070 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11000a6d0 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1200c4c50 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11000aac0 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.350+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1200c5210 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.351+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1200c5410 con 0x7fe12006b3e0 2024-03-04T10:06:19.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.352+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe11001b070 con 0x7fe12006b3e0 2024-03-04T10:06:19.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.352+0000 7fe10effd640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10006b4d0 0x7fe10006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.352+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe110089b80 con 0x7fe12006b3e0 2024-03-04T10:06:19.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.352+0000 7fe11dd9b640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10006b4d0 0x7fe10006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.353+0000 7fe11dd9b640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10006b4d0 0x7fe10006d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe1200c4570 tx=0x7fe114008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.357+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe1100588b0 con 0x7fe12006b3e0 2024-03-04T10:06:19.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.561+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe120074f60 con 0x7fe12006b3e0 2024-03-04T10:06:19.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.561+0000 7fe10effd640 1 -- 172.21.15.99:0/762403586 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4939 (secure 0 0 0) 0x7fe1100581b0 con 0x7fe12006b3e0 2024-03-04T10:06:19.563 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:19.563 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":33,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:18.751810+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:19.563 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 33 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10006b4d0 msgr2=0x7fe10006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10006b4d0 0x7fe10006d990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe1200c4570 tx=0x7fe114008040 comp rx=0 tx=0).stop 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 msgr2=0x7fe1200c81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 0x7fe1200c81c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe11000caa0 tx=0x7fe11000cf60 comp rx=0 tx=0).stop 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 shutdown_connections 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe10006b4d0 0x7fe10006d990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe12015dea0 0x7fe1200b5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe1200cc8d0 0x7fe1200c5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 --2- 172.21.15.99:0/762403586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe12006b3e0 0x7fe1200c81c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 >> 172.21.15.99:0/762403586 conn(0x7fe1200d0710 msgr2=0x7fe1200d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 shutdown_connections 2024-03-04T10:06:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.563+0000 7fe124d7e640 1 -- 172.21.15.99:0/762403586 wait complete. 2024-03-04T10:06:19.575 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:19.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 Processor -- start 2024-03-04T10:06:19.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 -- start start 2024-03-04T10:06:19.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fee080b4570 0x7fee0806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fee080cc140 0x7fee080b4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0806bff0 con 0x7fee0806c2d0 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080ae100 con 0x7fee080cc140 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.674+0000 7fee0df37640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080b4010 con 0x7fee080b4570 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fee080cc140 0x7fee080b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fee080cc140 0x7fee080b4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56706/0 (socket says 172.21.15.99:56706) 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee07fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fee0806c2d0 0x7fee080aeb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37128/0 (socket says 172.21.15.99:37128) 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 -- 172.21.15.99:0/1786599810 learned_addr learned my addr 172.21.15.99:0/1786599810 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 -- 172.21.15.99:0/1786599810 >> v2:172.21.15.162:3300/0 conn(0x7fee080b4570 msgr2=0x7fee0806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee077fe640 1 --2- 172.21.15.99:0/1786599810 >> v2:172.21.15.162:3300/0 conn(0x7fee080b4570 0x7fee0806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 --2- 172.21.15.99:0/1786599810 >> v2:172.21.15.162:3300/0 conn(0x7fee080b4570 0x7fee0806dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 -- 172.21.15.99:0/1786599810 >> v1:172.21.15.99:6789/0 conn(0x7fee0806c2d0 legacy=0x7fee080aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee06ffd640 1 -- 172.21.15.99:0/1786599810 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080b4e90 con 0x7fee080cc140 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee067fc640 1 -- 172.21.15.99:0/1786599810 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611678787 0 0) 0x7fee0806bff0 con 0x7fee0806c2d0 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.675+0000 7fee077fe640 1 --2- 172.21.15.99:0/1786599810 >> v2:172.21.15.162:3300/0 conn(0x7fee080b4570 0x7fee0806dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee06ffd640 1 --2- 172.21.15.99:0/1786599810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee080cc140 0x7fee080b4950 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fee08072c10 tx=0x7fedf002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e1a89cd684d692 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee067fc640 1 -- 172.21.15.99:0/1786599810 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0034070 con 0x7fee080cc140 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee067fc640 1 -- 172.21.15.99:0/1786599810 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf0030860 con 0x7fee080cc140 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee067fc640 1 -- 172.21.15.99:0/1786599810 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0030c30 con 0x7fee080cc140 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 -- 172.21.15.99:0/1786599810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee080cc140 msgr2=0x7fee080b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 --2- 172.21.15.99:0/1786599810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee080cc140 0x7fee080b4950 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fee08072c10 tx=0x7fedf002b9a0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 -- 172.21.15.99:0/1786599810 shutdown_connections 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 --2- 172.21.15.99:0/1786599810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee080cc140 0x7fee080b4950 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 --2- 172.21.15.99:0/1786599810 >> v2:172.21.15.162:3300/0 conn(0x7fee080b4570 0x7fee0806dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 -- 172.21.15.99:0/1786599810 >> 172.21.15.99:0/1786599810 conn(0x7fee080cfef0 msgr2=0x7fee080d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 -- 172.21.15.99:0/1786599810 shutdown_connections 2024-03-04T10:06:19.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.676+0000 7fee0df37640 1 -- 172.21.15.99:0/1786599810 wait complete. 2024-03-04T10:06:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.677+0000 7fee0df37640 1 Processor -- start 2024-03-04T10:06:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 -- start start 2024-03-04T10:06:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 0x7fee08153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee080b4570 0x7fee08140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee080cc140 0x7fee080cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee080b3d30 con 0x7fee0806c2d0 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee080b3a50 con 0x7fee080cc140 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee0df37640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080ae330 con 0x7fee080b4570 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee077fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 0x7fee08153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee077fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 0x7fee08153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56708/0 (socket says 172.21.15.99:56708) 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.678+0000 7fee077fe640 1 -- 172.21.15.99:0/3821227879 learned_addr learned my addr 172.21.15.99:0/3821227879 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee06ffd640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee080b4570 0x7fee08140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee077fe640 1 -- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee080b4570 msgr2=0x7fee08140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee077fe640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee080b4570 0x7fee08140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee07fff640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee080cc140 0x7fee080cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee077fe640 1 -- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee080cc140 msgr2=0x7fee080cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee077fe640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee080cc140 0x7fee080cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee077fe640 1 -- 172.21.15.99:0/3821227879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080cf600 con 0x7fee0806c2d0 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee06ffd640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee080b4570 0x7fee08140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:19.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee07fff640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee080cc140 0x7fee080cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee077fe640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 0x7fee08153a10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fedf800cf60 tx=0x7fedf800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf801a070 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf800d840 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee08153ff0 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf800dc10 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.679+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee081545b0 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.680+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee08074f60 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.680+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fedf801d070 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.681+0000 7fedfffff640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fede806b500 0x7fede806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.681+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fedf808aaa0 con 0x7fee0806c2d0 2024-03-04T10:06:19.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.681+0000 7fee06ffd640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fede806b500 0x7fede806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:19.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.682+0000 7fee06ffd640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fede806b500 0x7fede806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fedf0036000 tx=0x7fedf0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:19.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.685+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fedf8059180 con 0x7fee0806c2d0 2024-03-04T10:06:19.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.891+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee08066d20 con 0x7fee0806c2d0 2024-03-04T10:06:19.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.892+0000 7fedfffff640 1 -- 172.21.15.99:0/3821227879 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fedf805d220 con 0x7fee0806c2d0 2024-03-04T10:06:19.893 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:19.893 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:18.760829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:19.893 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 34 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fede806b500 msgr2=0x7fede806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fede806b500 0x7fede806d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fedf0036000 tx=0x7fedf0008030 comp rx=0 tx=0).stop 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 msgr2=0x7fee08153a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 0x7fee08153a10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fedf800cf60 tx=0x7fedf800cf90 comp rx=0 tx=0).stop 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 shutdown_connections 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fede806b500 0x7fede806d9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee080cc140 0x7fee080cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fee080b4570 0x7fee08140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.893+0000 7fee0df37640 1 --2- 172.21.15.99:0/3821227879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee0806c2d0 0x7fee08153a10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.894+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 >> 172.21.15.99:0/3821227879 conn(0x7fee080cfef0 msgr2=0x7fee08159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.894+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 shutdown_connections 2024-03-04T10:06:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:19.894+0000 7fee0df37640 1 -- 172.21.15.99:0/3821227879 wait complete. 2024-03-04T10:06:19.906 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /a 0 2024-03-04T10:06:20.019 INFO:teuthology.orchestra.run.smithi099.stdout:[{"ino":1099511627776,"rdev":0,"ctime":"2024-03-04T10:06:15.246473+0000","btime":"2024-03-04T10:06:09.390614+0000","mode":16877,"uid":1000,"gid":1301,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-03-04T10:06:15.246473+0000","atime":"2024-03-04T10:06:09.390614+0000","time_warp_seq":0,"change_attr":3,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-03-04T10:06:15.246473+0000","num_files":0,"num_subdirs":1,"change_attr":3},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-03-04T10:06:15.246473+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":2,"rsnaps":0,"rctime":"2024-03-04T10:06:15.246473+0000"},"version":9,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirfrag":1,"dirty":1},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth","dirty"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/a","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1},"nref":1,"dentries":[{"path":"a/c","path_ino":1,"snap_first":18446744073709551606,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511627777,"remote_type":"","version":4,"projected_version":4,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"inodepin":1,"dirty":1},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]},{"path":"a/b","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":false,"is_remote":false,"is_null":true,"is_new":false,"inode":0,"remote_type":"","version":5,"projected_version":5,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirty":1},"nref":1,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-03-04T10:06:20.020 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-03-04T10:06:15.246473+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "atime": "2024-03-04T10:06:09.390614+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-03-04T10:06:09.390614+0000", "change_attr": 3, "client_caps": [], "client_ranges": [], "ctime": "2024-03-04T10:06:15.246473+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511627777, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "a/c", "path_ino": 1, "pins": { "dirty": 1, "inodepin": 1 }, "projected_version": 4, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 18446744073709551606, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } }, { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 0, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": true, "is_primary": false, "is_remote": false, "lock": {}, "nref": 1, "path": "a/b", "path_ino": 1, "pins": { "dirty": 1 }, "projected_version": 5, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 5, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/a", "pins": { "child": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 3, "mtime": "2024-03-04T10:06:15.246473+0000", "num_files": 0, "num_subdirs": 1, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1301, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-03-04T10:06:15.246473+0000", "nestlock": {}, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "dirfrag": 1, "dirty": 1 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-03-04T10:06:15.246473+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 2, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 9, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } ] 2024-03-04T10:06:20.021 DEBUG:tasks.cephfs.test_damage:[{'ino': 1099511627776, 'rdev': 0, 'ctime': '2024-03-04T10:06:15.246473+0000', 'btime': '2024-03-04T10:06:09.390614+0000', 'mode': 16877, 'uid': 1000, 'gid': 1301, 'nlink': 1, 'dir_layout': {'dir_hash': 2, 'unused1': 0, 'unused2': 0, 'unused3': 0}, 'layout': {'stripe_unit': 0, 'stripe_count': 0, 'object_size': 0, 'pool_id': -1, 'pool_ns': ''}, 'old_pools': [], 'size': 0, 'truncate_seq': 1, 'truncate_size': 18446744073709551615, 'truncate_from': 0, 'truncate_pending': 0, 'mtime': '2024-03-04T10:06:15.246473+0000', 'atime': '2024-03-04T10:06:09.390614+0000', 'time_warp_seq': 0, 'change_attr': 3, 'export_pin': -1, 'export_ephemeral_random_pin': 0, 'export_ephemeral_distributed_pin': False, 'client_ranges': [], 'dirstat': {'version': 0, 'mtime': '2024-03-04T10:06:15.246473+0000', 'num_files': 0, 'num_subdirs': 1, 'change_attr': 3}, 'rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-03-04T10:06:15.246473+0000'}, 'accounted_rstat': {'version': 0, 'rbytes': 0, 'rfiles': 0, 'rsubdirs': 2, 'rsnaps': 0, 'rctime': '2024-03-04T10:06:15.246473+0000'}, 'version': 9, 'file_data_version': 0, 'xattr_version': 1, 'backtrace_version': 2, 'stray_prior_path': '', 'max_size_ever': 0, 'quota': {'max_bytes': 0, 'max_files': 0}, 'last_scrub_stamp': '0.000000', 'last_scrub_version': 0, 'symlink': '', 'xattrs': [{'key': 'security.selinux', 'val': 'unconfined_u:object_r:unlabeled_t:s0\x00'}], 'dirfragtree': {'splits': []}, 'old_inodes': [], 'oldest_snap': 18446744073709551614, 'damage_flags': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirfrag': 1, 'dirty': 1}, 'nref': 2, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'iversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'authlock': {}, 'linklock': {}, 'dirfragtreelock': {}, 'filelock': {}, 'xattrlock': {}, 'snaplock': {}, 'nestlock': {}, 'flocklock': {}, 'policylock': {}, 'states': ['auth', 'dirty'], 'client_caps': [], 'loner': -1, 'want_loner': -1, 'mds_caps_wanted': [], 'dirfrags': [{'path': '/a', 'dirfrag': '0x10000000000', 'snapid_first': 2, 'projected_version': '7', 'version': '7', 'committing_version': '0', 'committed_version': '0', 'is_rep': False, 'dir_auth': '', 'states': ['auth'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1}, 'nref': 1, 'dentries': [{'path': 'a/c', 'path_ino': 1, 'snap_first': 18446744073709551606, 'snap_last': 18446744073709551614, 'is_primary': True, 'is_remote': False, 'is_null': False, 'is_new': False, 'inode': 1099511627777, 'remote_type': '', 'version': 4, 'projected_version': 4, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'inodepin': 1, 'dirty': 1}, 'nref': 2, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}, {'path': 'a/b', 'path_ino': 1, 'snap_first': 2, 'snap_last': 18446744073709551614, 'is_primary': False, 'is_remote': False, 'is_null': True, 'is_new': False, 'inode': 0, 'remote_type': '', 'version': 5, 'projected_version': 5, 'auth_pins': 0, 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'is_frozen': False, 'is_freezing': False, 'pins': {'dirty': 1}, 'nref': 1, 'lock': {}, 'versionlock': {'gather_set': [], 'state': 'lock', 'type': 'dversion', 'is_leased': False, 'num_rdlocks': 0, 'num_wrlocks': 0, 'num_xlocks': 0, 'xlock_by': {}}, 'states': ['auth', 'dirty']}]}], 'is_subvolume': False}] 2024-03-04T10:06:20.021 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:06:20.021 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:06:20.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:06:20.122 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:06:20.122 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:20.122 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:20.137 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:06:20.137 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:06:20.137 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:20.155 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:06:20.155 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:20.155 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:20.194 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:06:20.194 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:06:20.194 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:20.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:06:20.213 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.332+0000 7f8747497640 1 Processor -- start 2024-03-04T10:06:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8747497640 1 -- start start 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8747497640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f87400b2e60 0x7f87400b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8747497640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8747497640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f874015e1e0 con 0x7f874015de00 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8747497640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87400b98c0 con 0x7f87400b2e60 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8747497640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87400b32c0 con 0x7f87400b5470 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8744a32640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f87400b2e60 0x7f87400b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.333+0000 7f8745233640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f874015de00 0x7f87400b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37140/0 (socket says 172.21.15.99:37140) 2024-03-04T10:06:20.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8745233640 1 -- 172.21.15.99:0/3873558399 learned_addr learned my addr 172.21.15.99:0/3873558399 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8745a34640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8744a32640 1 -- 172.21.15.99:0/3873558399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 msgr2=0x7f87400b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8744a32640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8744a32640 1 -- 172.21.15.99:0/3873558399 >> v1:172.21.15.99:6789/0 conn(0x7f874015de00 legacy=0x7f87400b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8744a32640 1 -- 172.21.15.99:0/3873558399 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87400d1c10 con 0x7f87400b2e60 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8737fff640 1 -- 172.21.15.99:0/3873558399 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234126866 0 0) 0x7f874015e1e0 con 0x7f874015de00 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.334+0000 7f8745a34640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8744a32640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87400b2e60 0x7f87400b9380 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f87300097a0 tx=0x7f873002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d661c323ff46b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8737fff640 1 -- 172.21.15.99:0/3873558399 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730035020 con 0x7f87400b2e60 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8737fff640 1 -- 172.21.15.99:0/3873558399 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8730031810 con 0x7f87400b2e60 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8737fff640 1 -- 172.21.15.99:0/3873558399 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730031be0 con 0x7f87400b2e60 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 -- 172.21.15.99:0/3873558399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87400b2e60 msgr2=0x7f87400b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87400b2e60 0x7f87400b9380 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f87300097a0 tx=0x7f873002b8b0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 -- 172.21.15.99:0/3873558399 shutdown_connections 2024-03-04T10:06:20.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 --2- 172.21.15.99:0/3873558399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87400b2e60 0x7f87400b9380 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 -- 172.21.15.99:0/3873558399 >> 172.21.15.99:0/3873558399 conn(0x7f87400d0470 msgr2=0x7f87400d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:20.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 -- 172.21.15.99:0/3873558399 shutdown_connections 2024-03-04T10:06:20.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.335+0000 7f8747497640 1 -- 172.21.15.99:0/3873558399 wait complete. 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.336+0000 7f8747497640 1 Processor -- start 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 -- start start 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87400b2e60 0x7f87400c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 0x7f874014ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400d21d0 con 0x7f874015de00 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400d2820 con 0x7f87400b2e60 2024-03-04T10:06:20.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8747497640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87400d24b0 con 0x7f87400b5470 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8745a34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 0x7f874014ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8745a34640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 0x7f874014ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56714/0 (socket says 172.21.15.99:56714) 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.337+0000 7f8745a34640 1 -- 172.21.15.99:0/1302366012 learned_addr learned my addr 172.21.15.99:0/1302366012 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8745a34640 1 -- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 msgr2=0x7f87400caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8745a34640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8745a34640 1 -- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87400b2e60 msgr2=0x7f87400c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8745a34640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87400b2e60 0x7f87400c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8745a34640 1 -- 172.21.15.99:0/1302366012 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874014d360 con 0x7f874015de00 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8745a34640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 0x7f874014ce20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f873800e7e0 tx=0x7f873800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873800f040 con 0x7f874015de00 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8738004030 con 0x7f874015de00 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87400cd430 con 0x7f874015de00 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87380043e0 con 0x7f874015de00 2024-03-04T10:06:20.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.338+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87400cd9f0 con 0x7f874015de00 2024-03-04T10:06:20.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.339+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87400b6bf0 con 0x7f874015de00 2024-03-04T10:06:20.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.340+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f873800c6d0 con 0x7f874015de00 2024-03-04T10:06:20.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.340+0000 7f8735ffb640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872406b4d0 0x7f872406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.340+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f873808a0f0 con 0x7f874015de00 2024-03-04T10:06:20.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.340+0000 7f8745233640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872406b4d0 0x7f872406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.341+0000 7f8745233640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872406b4d0 0x7f872406d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f873c001560 tx=0x7f873c007ca0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.344+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f87400b6bf0 con 0x7f874015de00 2024-03-04T10:06:20.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.548+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8740074f60 con 0x7f874015de00 2024-03-04T10:06:20.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.548+0000 7f8735ffb640 1 -- 172.21.15.99:0/1302366012 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f874014d360 con 0x7f874015de00 2024-03-04T10:06:20.550 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:20.550 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":34,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:18.760829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:20.550 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 34 2024-03-04T10:06:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872406b4d0 msgr2=0x7f872406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872406b4d0 0x7f872406d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f873c001560 tx=0x7f873c007ca0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 msgr2=0x7f874014ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 0x7f874014ce20 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f873800e7e0 tx=0x7f873800eca0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 shutdown_connections 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f872406b4d0 0x7f872406d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f874015de00 0x7f874014ce20 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.550+0000 7f8747497640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87400b5470 0x7f87400caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.551+0000 7f8747497640 1 --2- 172.21.15.99:0/1302366012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87400b2e60 0x7f87400c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.551+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 >> 172.21.15.99:0/1302366012 conn(0x7f87400d0470 msgr2=0x7f87400b88a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.551+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 shutdown_connections 2024-03-04T10:06:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.551+0000 7f8747497640 1 -- 172.21.15.99:0/1302366012 wait complete. 2024-03-04T10:06:20.564 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:20.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.664+0000 7f727bb74640 1 Processor -- start 2024-03-04T10:06:20.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.664+0000 7f727bb74640 1 -- start start 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727bb74640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f727415de00 0x7f72740b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727bb74640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f72740b9380 0x7f72740b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727bb74640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727415e1e0 con 0x7f72740b70d0 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727bb74640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72740d2820 con 0x7f727415de00 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727bb74640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72740d1c10 con 0x7f72740b9380 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f727415de00 0x7f72740b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f727415de00 0x7f72740b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56718/0 (socket says 172.21.15.99:56718) 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 -- 172.21.15.99:0/2448596889 learned_addr learned my addr 172.21.15.99:0/2448596889 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727910f640 1 --2- 172.21.15.99:0/2448596889 >> v2:172.21.15.162:3300/0 conn(0x7f72740b9380 0x7f72740b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 -- 172.21.15.99:0/2448596889 >> v2:172.21.15.162:3300/0 conn(0x7f72740b9380 msgr2=0x7f72740b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 --2- 172.21.15.99:0/2448596889 >> v2:172.21.15.162:3300/0 conn(0x7f72740b9380 0x7f72740b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 -- 172.21.15.99:0/2448596889 >> v1:172.21.15.162:6789/0 conn(0x7f72740b70d0 legacy=0x7f72740b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f7279910640 1 -- 172.21.15.99:0/2448596889 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72740d1ef0 con 0x7f727415de00 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.665+0000 7f727910f640 1 --2- 172.21.15.99:0/2448596889 >> v2:172.21.15.162:3300/0 conn(0x7f72740b9380 0x7f72740b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.666+0000 7f7279910640 1 --2- 172.21.15.99:0/2448596889 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72740b2e60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7274161830 tx=0x7f727002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec3bf922fa0dd40 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.666+0000 7f727890e640 1 -- 172.21.15.99:0/2448596889 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270035020 con 0x7f727415de00 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.666+0000 7f727890e640 1 -- 172.21.15.99:0/2448596889 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7270031870 con 0x7f727415de00 2024-03-04T10:06:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.666+0000 7f727890e640 1 -- 172.21.15.99:0/2448596889 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270031c40 con 0x7f727415de00 2024-03-04T10:06:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 -- 172.21.15.99:0/2448596889 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 msgr2=0x7f72740b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 --2- 172.21.15.99:0/2448596889 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72740b2e60 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7274161830 tx=0x7f727002b910 comp rx=0 tx=0).stop 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 -- 172.21.15.99:0/2448596889 shutdown_connections 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 --2- 172.21.15.99:0/2448596889 >> v2:172.21.15.162:3300/0 conn(0x7f72740b9380 0x7f72740b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 --2- 172.21.15.99:0/2448596889 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72740b2e60 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 -- 172.21.15.99:0/2448596889 >> 172.21.15.99:0/2448596889 conn(0x7f72740d0470 msgr2=0x7f72740d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 -- 172.21.15.99:0/2448596889 shutdown_connections 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.667+0000 7f727bb74640 1 -- 172.21.15.99:0/2448596889 wait complete. 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 Processor -- start 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 -- start start 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72740b70d0 0x7f72741f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72740b9380 0x7f72741f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72741f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740b97e0 con 0x7f727415de00 2024-03-04T10:06:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72740d24b0 con 0x7f72740b70d0 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.668+0000 7f727bb74640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72740b5b80 con 0x7f72740b9380 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72741f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72741f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56724/0 (socket says 172.21.15.99:56724) 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 -- 172.21.15.99:0/1942444450 learned_addr learned my addr 172.21.15.99:0/1942444450 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727910f640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72740b9380 0x7f72741f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 -- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72740b9380 msgr2=0x7f72741f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72740b9380 0x7f72741f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f7279910640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72740b70d0 0x7f72741f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 -- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72740b70d0 msgr2=0x7f72741f3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72740b70d0 0x7f72741f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 -- 172.21.15.99:0/1942444450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727415fd10 con 0x7f727415de00 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727a111640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72741f50a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f726c00c780 tx=0x7f726c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c017070 con 0x7f727415de00 2024-03-04T10:06:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c003da0 con 0x7f727415de00 2024-03-04T10:06:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727415fff0 con 0x7f727415de00 2024-03-04T10:06:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0052b0 con 0x7f727415de00 2024-03-04T10:06:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.669+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72741602d0 con 0x7f727415de00 2024-03-04T10:06:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.670+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72740b66a0 con 0x7f727415de00 2024-03-04T10:06:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.671+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f726c01a070 con 0x7f727415de00 2024-03-04T10:06:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.671+0000 7f726a7fc640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f725806b500 0x7f725806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.671+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f726c0899b0 con 0x7f727415de00 2024-03-04T10:06:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.671+0000 7f7279910640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f725806b500 0x7f725806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.672+0000 7f7279910640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f725806b500 0x7f725806d9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7270036000 tx=0x7f7270008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.675+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f72740b66a0 con 0x7f727415de00 2024-03-04T10:06:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.847+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7274074f60 con 0x7f727415de00 2024-03-04T10:06:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.848+0000 7f726a7fc640 1 -- 172.21.15.99:0/1942444450 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+17033 (secure 0 0 0) 0x7f726c0586e0 con 0x7f727415de00 2024-03-04T10:06:20.850 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":53,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:17.747087+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6833/370337902":"2024-03-05T10:06:17.747062+0000","172.21.15.99:6832/370337902":"2024-03-05T10:06:17.747062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f725806b500 msgr2=0x7f725806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f725806b500 0x7f725806d9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7270036000 tx=0x7f7270008030 comp rx=0 tx=0).stop 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 msgr2=0x7f72741f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72741f50a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f726c00c780 tx=0x7f726c00cc40 comp rx=0 tx=0).stop 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 shutdown_connections 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f725806b500 0x7f725806d9c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f727415de00 0x7f72741f50a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72740b9380 0x7f72741f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 --2- 172.21.15.99:0/1942444450 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72740b70d0 0x7f72741f3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.850+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 >> 172.21.15.99:0/1942444450 conn(0x7f72740d0470 msgr2=0x7f72740ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.851+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 shutdown_connections 2024-03-04T10:06:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.851+0000 7f727bb74640 1 -- 172.21.15.99:0/1942444450 wait complete. 2024-03-04T10:06:20.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:05:55.899275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:05:56.653255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:06:20.863 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:06:20.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.969+0000 7f0f22de1640 1 Processor -- start 2024-03-04T10:06:20.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.969+0000 7f0f22de1640 1 -- start start 2024-03-04T10:06:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.969+0000 7f0f22de1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 0x7f0f1c0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.969+0000 7f0f22de1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c0d1520 con 0x7f0f1c0d1770 2024-03-04T10:06:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.969+0000 7f0f22de1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1c0cfa30 con 0x7f0f1c15bbf0 2024-03-04T10:06:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.969+0000 7f0f22de1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0cfd10 con 0x7f0f1c0b70d0 2024-03-04T10:06:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f2137e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0f1c0d1770 0x7f0f1c0b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37154/0 (socket says 172.21.15.99:37154) 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f2137e640 1 -- 172.21.15.99:0/1389344328 learned_addr learned my addr 172.21.15.99:0/1389344328 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1bfff640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 0x7f0f1c0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118278080 0 0) 0x7f0f1c0d1520 con 0x7f0f1c0d1770 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c003520 con 0x7f0f1c0d1770 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1111818447 0 0) 0x7f0f0c003520 con 0x7f0f1c0d1770 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 msgr2=0x7f0f1c0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 0x7f0f1c0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 >> v1:172.21.15.162:6789/0 conn(0x7f0f1c15bbf0 legacy=0x7f0f1c0d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0cfff0 con 0x7f0f1c0d1770 2024-03-04T10:06:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f0f140019f0 con 0x7f0f1c0d1770 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 >> v1:172.21.15.99:6789/0 conn(0x7f0f1c0d1770 legacy=0x7f0f1c0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f0c0039e0 0x7f0f0c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 0x7f0f0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c004d40 0x7f0f0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0cfd10 con 0x7f0f0c0039e0 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c00baf0 con 0x7f0f0c004d40 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c00bcf0 con 0x7f0f1c0b70d0 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f0c0039e0 0x7f0f0c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f1bfff640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 0x7f0f0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.970+0000 7f0f2137e640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c004d40 0x7f0f0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f20b7d640 1 -- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 msgr2=0x7f0f0c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f1c0b70d0 0x7f0f0c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f20b7d640 1 -- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c004d40 msgr2=0x7f0f0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c004d40 0x7f0f0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f20b7d640 1 -- 172.21.15.99:0/1389344328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c003520 con 0x7f0f0c0039e0 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f2137e640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f0c004d40 0x7f0f0c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f0c0039e0 0x7f0f0c003e00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0f10001750 tx=0x7f0f0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30aebfc87ea7beb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10001370 con 0x7f0f0c0039e0 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f100052d0 con 0x7f0f0c0039e0 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.971+0000 7f0f1b7fe640 1 -- 172.21.15.99:0/1389344328 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10003830 con 0x7f0f0c0039e0 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f22de1640 1 -- 172.21.15.99:0/1389344328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f0c0039e0 msgr2=0x7f0f0c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1389344328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f0c0039e0 0x7f0f0c003e00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0f10001750 tx=0x7f0f0c005830 comp rx=0 tx=0).stop 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f2137e640 1 -- 172.21.15.99:0/1389344328 reap_dead start 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f22de1640 1 -- 172.21.15.99:0/1389344328 shutdown_connections 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f22de1640 1 -- 172.21.15.99:0/1389344328 >> 172.21.15.99:0/1389344328 conn(0x7f0f1c0b3330 msgr2=0x7f0f1c0b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f22de1640 1 -- 172.21.15.99:0/1389344328 shutdown_connections 2024-03-04T10:06:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.972+0000 7f0f22de1640 1 -- 172.21.15.99:0/1389344328 wait complete. 2024-03-04T10:06:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.973+0000 7f0f22de1640 1 Processor -- start 2024-03-04T10:06:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.973+0000 7f0f22de1640 1 -- start start 2024-03-04T10:06:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f22de1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f0c0039e0 0x7f0f1c0cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f22de1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f1c157de0 0x7f0f1c15c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f22de1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 0x7f0f1c0c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f22de1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c15bfd0 con 0x7f0f1c0c6320 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f22de1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0d05b0 con 0x7f0f1c157de0 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f22de1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0cc840 con 0x7f0f0c0039e0 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f2137e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 0x7f0f1c0c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f2137e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 0x7f0f1c0c67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56732/0 (socket says 172.21.15.99:56732) 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f2137e640 1 -- 172.21.15.99:0/1761484880 learned_addr learned my addr 172.21.15.99:0/1761484880 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f0c0039e0 0x7f0f1c0cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f1bfff640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f1c157de0 0x7f0f1c15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f2137e640 1 -- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f0c0039e0 msgr2=0x7f0f1c0cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f2137e640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f0c0039e0 0x7f0f1c0cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.974+0000 7f0f2137e640 1 -- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f1c157de0 msgr2=0x7f0f1c15c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f2137e640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f1c157de0 0x7f0f1c15c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f2137e640 1 -- 172.21.15.99:0/1761484880 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c158260 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f2137e640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 0x7f0f1c0c67a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0cf850 tx=0x7f0f1400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400ebb0 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1400ed50 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c15c8c0 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f14014670 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.975+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c15ce80 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.976+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c074f60 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.976+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0f14014810 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.977+0000 7f0f197fa640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0006b500 0x7f0f0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.977+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f0f1408a160 con 0x7f0f1c0c6320 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.977+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0006b500 0x7f0f0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.977+0000 7f0f20b7d640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0006b500 0x7f0f0006d9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0f10033000 tx=0x7f0f10009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:20.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:20.981+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0f14058770 con 0x7f0f1c0c6320 2024-03-04T10:06:21.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.150+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f1c15b0f0 con 0x7f0f1c0c6320 2024-03-04T10:06:21.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.151+0000 7f0f197fa640 1 -- 172.21.15.99:0/1761484880 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f1405c810 con 0x7f0f1c0c6320 2024-03-04T10:06:21.152 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:06:21.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0006b500 msgr2=0x7f0f0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0006b500 0x7f0f0006d9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0f10033000 tx=0x7f0f10009030 comp rx=0 tx=0).stop 2024-03-04T10:06:21.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 msgr2=0x7f0f1c0c67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 0x7f0f1c0c67a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0cf850 tx=0x7f0f1400e530 comp rx=0 tx=0).stop 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 shutdown_connections 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0f0006b500 0x7f0f0006d9c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0f1c0c6320 0x7f0f1c0c67a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0f1c157de0 0x7f0f1c15c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 --2- 172.21.15.99:0/1761484880 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0f0c0039e0 0x7f0f1c0cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 >> 172.21.15.99:0/1761484880 conn(0x7f0f1c0b3330 msgr2=0x7f0f1c0b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 shutdown_connections 2024-03-04T10:06:21.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.153+0000 7f0f22de1640 1 -- 172.21.15.99:0/1761484880 wait complete. 2024-03-04T10:06:21.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:06:21.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.266+0000 7f9c627a2640 1 Processor -- start 2024-03-04T10:06:21.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.266+0000 7f9c627a2640 1 -- start start 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c627a2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 0x7f9c5c06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c627a2640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9c5c06b210 0x7f9c5c15f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c627a2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5c160070 con 0x7f9c5c15f8f0 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c627a2640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c5c0cc180 con 0x7f9c5c06b210 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c627a2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c06b670 con 0x7f9c5c15b970 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c61fa1640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9c5c15f8f0 0x7f9c5c0ce120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37158/0 (socket says 172.21.15.99:37158) 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c61fa1640 1 -- 172.21.15.99:0/1710217205 learned_addr learned my addr 172.21.15.99:0/1710217205 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c617a0640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 0x7f9c5c06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1710217205 >> v2:172.21.15.162:3300/0 conn(0x7f9c5c06b210 0x7f9c5c15f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802513265 0 0) 0x7f9c5c160070 con 0x7f9c5c15f8f0 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c48003520 con 0x7f9c5c15f8f0 2024-03-04T10:06:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580266341 0 0) 0x7f9c48003520 con 0x7f9c5c15f8f0 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 msgr2=0x7f9c5c06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 0x7f9c5c06acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 >> v2:172.21.15.162:3300/0 conn(0x7f9c5c06b210 msgr2=0x7f9c5c15f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 --2- 172.21.15.99:0/1710217205 >> v2:172.21.15.162:3300/0 conn(0x7f9c5c06b210 0x7f9c5c15f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0cc3b0 con 0x7f9c5c15f8f0 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f9c540019f0 con 0x7f9c5c15f8f0 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.267+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1710217205 >> v2:172.21.15.162:3300/0 conn(0x7f9c5c06b210 0x7f9c5c15f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 >> v1:172.21.15.99:6789/0 conn(0x7f9c5c15f8f0 legacy=0x7f9c5c0ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c480039b0 0x7f9c48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 0x7f9c480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4800b9c0 con 0x7f9c48004800 2024-03-04T10:06:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4800bb40 con 0x7f9c480039b0 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4800bcc0 con 0x7f9c5c15b970 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c617a0640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c480039b0 0x7f9c48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c61fa1640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 0x7f9c480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c60f9f640 1 -- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 msgr2=0x7f9c480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c15b970 0x7f9c480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c60f9f640 1 -- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c480039b0 msgr2=0x7f9c48003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c480039b0 0x7f9c48003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.268+0000 7f9c60f9f640 1 -- 172.21.15.99:0/1710217205 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48003520 con 0x7f9c48004800 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c48004c80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009cc0 tx=0x7f9c48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e29af43ecbdcc6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c039070 con 0x7f9c48004800 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c02c960 con 0x7f9c48004800 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c53fff640 1 -- 172.21.15.99:0/1710217205 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c009860 con 0x7f9c48004800 2024-03-04T10:06:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c627a2640 1 -- 172.21.15.99:0/1710217205 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 msgr2=0x7f9c48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1710217205 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c48004c80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c009cc0 tx=0x7f9c48005830 comp rx=0 tx=0).stop 2024-03-04T10:06:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c61fa1640 1 -- 172.21.15.99:0/1710217205 reap_dead start 2024-03-04T10:06:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c627a2640 1 -- 172.21.15.99:0/1710217205 shutdown_connections 2024-03-04T10:06:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c627a2640 1 -- 172.21.15.99:0/1710217205 >> 172.21.15.99:0/1710217205 conn(0x7f9c5c099a00 msgr2=0x7f9c5c0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c627a2640 1 -- 172.21.15.99:0/1710217205 shutdown_connections 2024-03-04T10:06:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.269+0000 7f9c627a2640 1 -- 172.21.15.99:0/1710217205 wait complete. 2024-03-04T10:06:21.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 Processor -- start 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 -- start start 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c5c14fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c0c32f0 0x7f9c5c0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c5c0b4220 0x7f9c5c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c0cca00 con 0x7f9c48004800 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c160350 con 0x7f9c5c0b4220 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c627a2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c15fd20 con 0x7f9c5c0c32f0 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c617a0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c5c14fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c617a0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c5c14fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56744/0 (socket says 172.21.15.99:56744) 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.271+0000 7f9c617a0640 1 -- 172.21.15.99:0/1399001786 learned_addr learned my addr 172.21.15.99:0/1399001786 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c617a0640 1 -- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c0c32f0 msgr2=0x7f9c5c0c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c617a0640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c0c32f0 0x7f9c5c0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c617a0640 1 -- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c5c0b4220 msgr2=0x7f9c5c0ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c617a0640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c5c0b4220 0x7f9c5c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c617a0640 1 -- 172.21.15.99:0/1399001786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c14ff90 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c617a0640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c5c14fa50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9c58011940 tx=0x7f9c58011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5800fa70 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5800fc10 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58014440 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c0c9cf0 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.272+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c0c7a30 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.273+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c04f090 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.274+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9c58012040 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.274+0000 7f9c51ffb640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9c4006b500 0x7f9c4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.274+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f9c5808a810 con 0x7f9c48004800 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.274+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9c4006b500 0x7f9c4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.275+0000 7f9c60f9f640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9c4006b500 0x7f9c4006d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c036000 tx=0x7f9c4c02e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.279+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9c5c14ff90 con 0x7f9c48004800 2024-03-04T10:06:21.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.450+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c5c0d0290 con 0x7f9c48004800 2024-03-04T10:06:21.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.451+0000 7f9c51ffb640 1 -- 172.21.15.99:0/1399001786 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c58019030 con 0x7f9c48004800 2024-03-04T10:06:21.452 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:06:21.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9c4006b500 msgr2=0x7f9c4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9c4006b500 0x7f9c4006d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c036000 tx=0x7f9c4c02e000 comp rx=0 tx=0).stop 2024-03-04T10:06:21.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 msgr2=0x7f9c5c14fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c5c14fa50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9c58011940 tx=0x7f9c58011e00 comp rx=0 tx=0).stop 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 shutdown_connections 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9c4006b500 0x7f9c4006d9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9c5c0b4220 0x7f9c5c0ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9c5c0c32f0 0x7f9c5c0c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 --2- 172.21.15.99:0/1399001786 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9c48004800 0x7f9c5c14fa50 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 >> 172.21.15.99:0/1399001786 conn(0x7f9c5c099a00 msgr2=0x7f9c5c0ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 shutdown_connections 2024-03-04T10:06:21.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.453+0000 7f9c627a2640 1 -- 172.21.15.99:0/1399001786 wait complete. 2024-03-04T10:06:21.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:06:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.566+0000 7faacc459640 1 Processor -- start 2024-03-04T10:06:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.566+0000 7faacc459640 1 -- start start 2024-03-04T10:06:21.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faacc459640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac40cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faacc459640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac40cebd0 con 0x7faac41450e0 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faacc459640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac40ceeb0 con 0x7faac40cc340 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faacc459640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac40cf7e0 con 0x7faac40cfc70 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faaca1f5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7faac41450e0 0x7faac415c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37160/0 (socket says 172.21.15.99:37160) 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faaca1f5640 1 -- 172.21.15.99:0/883453228 learned_addr learned my addr 172.21.15.99:0/883453228 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faac99f4640 1 --2- 172.21.15.99:0/883453228 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac40cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faac91f3640 1 -- 172.21.15.99:0/883453228 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559340618 0 0) 0x7faac40cebd0 con 0x7faac41450e0 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.567+0000 7faac91f3640 1 -- 172.21.15.99:0/883453228 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab0003520 con 0x7faac41450e0 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.568+0000 7faac91f3640 1 -- 172.21.15.99:0/883453228 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267872154 0 0) 0x7faac40ceeb0 con 0x7faac40cc340 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.568+0000 7faac99f4640 1 -- 172.21.15.99:0/883453228 >> v1:172.21.15.162:6789/0 conn(0x7faac40cc340 legacy=0x7faac40b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.568+0000 7faac99f4640 1 -- 172.21.15.99:0/883453228 >> v1:172.21.15.99:6789/0 conn(0x7faac41450e0 legacy=0x7faac415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.568+0000 7faac99f4640 1 -- 172.21.15.99:0/883453228 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac40cf190 con 0x7faac40cfc70 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.568+0000 7faac99f4640 1 --2- 172.21.15.99:0/883453228 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac40cbe00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faac40cf960 tx=0x7faab402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76b7030df79f870 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faac91f3640 1 -- 172.21.15.99:0/883453228 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4035020 con 0x7faac40cfc70 2024-03-04T10:06:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faac91f3640 1 -- 172.21.15.99:0/883453228 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab4031890 con 0x7faac40cfc70 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faac91f3640 1 -- 172.21.15.99:0/883453228 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4031c40 con 0x7faac40cfc70 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 -- 172.21.15.99:0/883453228 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 msgr2=0x7faac40cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 --2- 172.21.15.99:0/883453228 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac40cbe00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faac40cf960 tx=0x7faab402b930 comp rx=0 tx=0).stop 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 -- 172.21.15.99:0/883453228 shutdown_connections 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 --2- 172.21.15.99:0/883453228 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac40cbe00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 -- 172.21.15.99:0/883453228 >> 172.21.15.99:0/883453228 conn(0x7faac406acd0 msgr2=0x7faac406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 -- 172.21.15.99:0/883453228 shutdown_connections 2024-03-04T10:06:21.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.569+0000 7faacc459640 1 -- 172.21.15.99:0/883453228 wait complete. 2024-03-04T10:06:21.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.570+0000 7faacc459640 1 Processor -- start 2024-03-04T10:06:21.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.570+0000 7faacc459640 1 -- start start 2024-03-04T10:06:21.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faacc459640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 0x7faac40ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faacc459640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faacc459640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac41450e0 0x7faac413b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faacc459640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac41454c0 con 0x7faac40cc340 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faacc459640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac40cc780 con 0x7faac41450e0 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faacc459640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac40ccd10 con 0x7faac40cfc70 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 0x7faac40ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 0x7faac40ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56748/0 (socket says 172.21.15.99:56748) 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 -- 172.21.15.99:0/1022141039 learned_addr learned my addr 172.21.15.99:0/1022141039 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faac99f4640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca9f6640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac41450e0 0x7faac413b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 -- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 msgr2=0x7faac415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 -- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac41450e0 msgr2=0x7faac413b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac41450e0 0x7faac413b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca1f5640 1 -- 172.21.15.99:0/1022141039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac40ad2c0 con 0x7faac40cc340 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.571+0000 7faaca9f6640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac41450e0 0x7faac413b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.572+0000 7faaca1f5640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 0x7faac40ac870 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7faac000caa0 tx=0x7faac000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.572+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac0017070 con 0x7faac40cc340 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.572+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac000ac00 con 0x7faac40cc340 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.572+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac40ad5a0 con 0x7faac40cc340 2024-03-04T10:06:21.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.572+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac0012400 con 0x7faac40cc340 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.572+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac40adb60 con 0x7faac40cc340 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.573+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac4074f60 con 0x7faac40cc340 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.573+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7faac001a070 con 0x7faac40cc340 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.573+0000 7faabaffd640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaa806b4d0 0x7faaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.573+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6245+0+0 (secure 0 0 0) 0x7faac008a210 con 0x7faac40cc340 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.574+0000 7faac99f4640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaa806b4d0 0x7faaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.574+0000 7faac99f4640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaa806b4d0 0x7faaa806d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faab4036000 tx=0x7faab4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.578+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faac0058e60 con 0x7faac40cc340 2024-03-04T10:06:21.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.746+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faac40cd1c0 con 0x7faac40cc340 2024-03-04T10:06:21.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.747+0000 7faabaffd640 1 -- 172.21.15.99:0/1022141039 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7faac0058760 con 0x7faac40cc340 2024-03-04T10:06:21.748 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaa806b4d0 msgr2=0x7faaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaa806b4d0 0x7faaa806d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faab4036000 tx=0x7faab4008030 comp rx=0 tx=0).stop 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 msgr2=0x7faac40ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 0x7faac40ac870 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7faac000caa0 tx=0x7faac000cf60 comp rx=0 tx=0).stop 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 shutdown_connections 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaa806b4d0 0x7faaa806d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faac41450e0 0x7faac413b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7faac40cfc70 0x7faac415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 --2- 172.21.15.99:0/1022141039 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7faac40cc340 0x7faac40ac870 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 >> 172.21.15.99:0/1022141039 conn(0x7faac406acd0 msgr2=0x7faac4143a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 shutdown_connections 2024-03-04T10:06:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.749+0000 7faacc459640 1 -- 172.21.15.99:0/1022141039 wait complete. 2024-03-04T10:06:21.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:21.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.860+0000 7f2471eea640 1 Processor -- start 2024-03-04T10:06:21.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.860+0000 7f2471eea640 1 -- start start 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f2471eea640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f246c0aeb20 0x7f246c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f2471eea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f2471eea640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c15da40 con 0x7f246c15d660 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f2471eea640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c15cbc0 con 0x7f246c0aeb20 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f2471eea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c0af090 con 0x7f246c071890 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f246c0aeb20 0x7f246c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f246c0aeb20 0x7f246c071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56754/0 (socket says 172.21.15.99:56754) 2024-03-04T10:06:21.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246b7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f246c15d660 0x7f246c15e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37174/0 (socket says 172.21.15.99:37174) 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 -- 172.21.15.99:0/3474159988 learned_addr learned my addr 172.21.15.99:0/3474159988 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246bfff640 1 --2- 172.21.15.99:0/3474159988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 -- 172.21.15.99:0/3474159988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 msgr2=0x7f246c150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 --2- 172.21.15.99:0/3474159988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 -- 172.21.15.99:0/3474159988 >> v1:172.21.15.99:6789/0 conn(0x7f246c15d660 legacy=0x7f246c15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.861+0000 7f246affd640 1 -- 172.21.15.99:0/3474159988 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c15a9f0 con 0x7f246c0aeb20 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.862+0000 7f246affd640 1 --2- 172.21.15.99:0/3474159988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c0aeb20 0x7f246c071350 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2460009870 tx=0x7f246002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290ab22c5644b21d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.862+0000 7f246a7fc640 1 -- 172.21.15.99:0/3474159988 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600317c0 con 0x7f246c0aeb20 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.862+0000 7f246a7fc640 1 -- 172.21.15.99:0/3474159988 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2460037070 con 0x7f246c0aeb20 2024-03-04T10:06:21.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.862+0000 7f246a7fc640 1 -- 172.21.15.99:0/3474159988 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2460031b80 con 0x7f246c0aeb20 2024-03-04T10:06:21.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.862+0000 7f2471eea640 1 -- 172.21.15.99:0/3474159988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c0aeb20 msgr2=0x7f246c071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.862+0000 7f2471eea640 1 --2- 172.21.15.99:0/3474159988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c0aeb20 0x7f246c071350 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2460009870 tx=0x7f246002b860 comp rx=0 tx=0).stop 2024-03-04T10:06:21.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.863+0000 7f2471eea640 1 -- 172.21.15.99:0/3474159988 shutdown_connections 2024-03-04T10:06:21.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.863+0000 7f2471eea640 1 --2- 172.21.15.99:0/3474159988 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.863+0000 7f2471eea640 1 --2- 172.21.15.99:0/3474159988 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c0aeb20 0x7f246c071350 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.863+0000 7f2471eea640 1 -- 172.21.15.99:0/3474159988 >> 172.21.15.99:0/3474159988 conn(0x7f246c06acd0 msgr2=0x7f246c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.863+0000 7f2471eea640 1 -- 172.21.15.99:0/3474159988 shutdown_connections 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.863+0000 7f2471eea640 1 -- 172.21.15.99:0/3474159988 wait complete. 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.864+0000 7f2471eea640 1 Processor -- start 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.864+0000 7f2471eea640 1 -- start start 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f246c0aeb20 0x7f246c1478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 0x7f246c143750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c15a660 con 0x7f246c15d660 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c15afb0 con 0x7f246c0aeb20 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c15b4d0 con 0x7f246c071890 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 0x7f246c143750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 0x7f246c143750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56756/0 (socket says 172.21.15.99:56756) 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 -- 172.21.15.99:0/1194432208 learned_addr learned my addr 172.21.15.99:0/1194432208 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:21.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246b7fe640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246affd640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f246c0aeb20 0x7f246c1478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 -- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 msgr2=0x7f246c154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 -- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f246c0aeb20 msgr2=0x7f246c1478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f246c0aeb20 0x7f246c1478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 -- 172.21.15.99:0/1194432208 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c144510 con 0x7f246c15d660 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246affd640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f246c0aeb20 0x7f246c1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f246bfff640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 0x7f246c143750 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2464007e90 tx=0x7f246400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464014420 con 0x7f246c15d660 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464019070 con 0x7f246c15d660 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c1447f0 con 0x7f246c15d660 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464014800 con 0x7f246c15d660 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.865+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246c144db0 con 0x7f246c15d660 2024-03-04T10:06:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.866+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246c074f60 con 0x7f246c15d660 2024-03-04T10:06:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.867+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f24640149a0 con 0x7f246c15d660 2024-03-04T10:06:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.867+0000 7f2453fff640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f245806b4d0 0x7f245806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.867+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f246408ac70 con 0x7f246c15d660 2024-03-04T10:06:21.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.867+0000 7f246b7fe640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f245806b4d0 0x7f245806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:21.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.868+0000 7f246b7fe640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f245806b4d0 0x7f245806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f245c000f60 tx=0x7f245c007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:21.871+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2464059850 con 0x7f246c15d660 2024-03-04T10:06:22.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.080+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f246c159dd0 con 0x7f246c15d660 2024-03-04T10:06:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.081+0000 7f2453fff640 1 -- 172.21.15.99:0/1194432208 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4939 (secure 0 0 0) 0x7f2464059150 con 0x7f246c15d660 2024-03-04T10:06:22.082 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:22.082 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:20.757897+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 36 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.082+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f245806b4d0 msgr2=0x7f245806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.082+0000 7f2471eea640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f245806b4d0 0x7f245806d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f245c000f60 tx=0x7f245c007cc0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.082+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 msgr2=0x7f246c143750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.082+0000 7f2471eea640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 0x7f246c143750 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f2464007e90 tx=0x7f246400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.082+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 shutdown_connections 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f245806b4d0 0x7f245806d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f246c15d660 0x7f246c143750 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f246c0aeb20 0x7f246c1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 --2- 172.21.15.99:0/1194432208 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f246c071890 0x7f246c154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 >> 172.21.15.99:0/1194432208 conn(0x7f246c06acd0 msgr2=0x7f246c0b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 shutdown_connections 2024-03-04T10:06:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.083+0000 7f2471eea640 1 -- 172.21.15.99:0/1194432208 wait complete. 2024-03-04T10:06:22.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:22.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.194+0000 7f6eb9c4d640 1 Processor -- start 2024-03-04T10:06:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.194+0000 7f6eb9c4d640 1 -- start start 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb9c4d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 0x7f6eb40b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb9c4d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 0x7f6eb40b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb9c4d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb415e1e0 con 0x7f6eb40b70d0 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb9c4d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb40d2820 con 0x7f6eb40b9380 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb9c4d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40d1c10 con 0x7f6eb415de00 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 0x7f6eb40b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.195+0000 7f6eb2ffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 0x7f6eb40b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb3fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6eb40b70d0 0x7f6eb40b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:46688/0 (socket says 172.21.15.99:46688) 2024-03-04T10:06:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb2ffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 0x7f6eb40b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:54644/0 (socket says 172.21.15.99:54644) 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb3fff640 1 -- 172.21.15.99:0/3101442142 learned_addr learned my addr 172.21.15.99:0/3101442142 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb37fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 0x7f6eb40b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:59626/0 (socket says 172.21.15.99:59626) 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb37fe640 1 -- 172.21.15.99:0/3101442142 >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 msgr2=0x7f6eb40b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb37fe640 1 --2- 172.21.15.99:0/3101442142 >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 0x7f6eb40b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb37fe640 1 -- 172.21.15.99:0/3101442142 >> v1:172.21.15.162:6789/0 conn(0x7f6eb40b70d0 legacy=0x7f6eb40b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb37fe640 1 -- 172.21.15.99:0/3101442142 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb40d1ef0 con 0x7f6eb415de00 2024-03-04T10:06:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.196+0000 7f6eb2ffd640 1 --2- 172.21.15.99:0/3101442142 >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 0x7f6eb40b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:22.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.197+0000 7f6eb37fe640 1 --2- 172.21.15.99:0/3101442142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 0x7f6eb40b2e60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40097a0 tx=0x7f6ea402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e409e9b27d77a125 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.197+0000 7f6eb27fc640 1 -- 172.21.15.99:0/3101442142 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4035020 con 0x7f6eb415de00 2024-03-04T10:06:22.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.197+0000 7f6eb27fc640 1 -- 172.21.15.99:0/3101442142 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea4031830 con 0x7f6eb415de00 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.197+0000 7f6eb27fc640 1 -- 172.21.15.99:0/3101442142 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4031c00 con 0x7f6eb415de00 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.197+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3101442142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 msgr2=0x7f6eb40b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.197+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3101442142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 0x7f6eb40b2e60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40097a0 tx=0x7f6ea402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.198+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3101442142 shutdown_connections 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.198+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3101442142 >> v2:172.21.15.162:3300/0 conn(0x7f6eb40b9380 0x7f6eb40b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.198+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3101442142 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb415de00 0x7f6eb40b2e60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.198+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3101442142 >> 172.21.15.99:0/3101442142 conn(0x7f6eb40d0470 msgr2=0x7f6eb40d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.198+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3101442142 shutdown_connections 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.198+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3101442142 wait complete. 2024-03-04T10:06:22.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 Processor -- start 2024-03-04T10:06:22.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 -- start start 2024-03-04T10:06:22.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eb40b70d0 0x7f6eb40c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb40b9380 0x7f6eb415e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 0x7f6eb415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40b97e0 con 0x7f6eb415de00 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40d24b0 con 0x7f6eb40b70d0 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.199+0000 7f6eb9c4d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb40b5b80 con 0x7f6eb40b9380 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 0x7f6eb415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 0x7f6eb415fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56760/0 (socket says 172.21.15.99:56760) 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 -- 172.21.15.99:0/3524201410 learned_addr learned my addr 172.21.15.99:0/3524201410 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb37fe640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eb40b70d0 0x7f6eb40c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb2ffd640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb40b9380 0x7f6eb415e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 -- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb40b9380 msgr2=0x7f6eb415e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb40b9380 0x7f6eb415e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 -- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eb40b70d0 msgr2=0x7f6eb40c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eb40b70d0 0x7f6eb40c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 -- 172.21.15.99:0/3524201410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb415ecb0 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb3fff640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 0x7f6eb415fd50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00cbf0 tx=0x7f6eac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac017070 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac00ad40 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb415ef90 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac012600 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.200+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb415f550 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.201+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb4074f60 con 0x7f6eb415de00 2024-03-04T10:06:22.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.202+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6eac01a070 con 0x7f6eb415de00 2024-03-04T10:06:22.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.202+0000 7f6e9bfff640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ea006b500 0x7f6ea006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.202+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6eac089ff0 con 0x7f6eb415de00 2024-03-04T10:06:22.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.202+0000 7f6eb37fe640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ea006b500 0x7f6ea006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.203+0000 7f6eb37fe640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ea006b500 0x7f6ea006d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4036000 tx=0x7f6ea403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.207+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6eac058690 con 0x7f6eb415de00 2024-03-04T10:06:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.382+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6eb4154490 con 0x7f6eb415de00 2024-03-04T10:06:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.383+0000 7f6e9bfff640 1 -- 172.21.15.99:0/3524201410 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+17072 (secure 0 0 0) 0x7f6eac05c730 con 0x7f6eb415de00 2024-03-04T10:06:22.385 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:22.385 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":55,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:21.944559+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.c","mappings":[{"from":0,"to":6}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6833/370337902":"2024-03-05T10:06:17.747062+0000","172.21.15.99:6832/370337902":"2024-03-05T10:06:17.747062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ea006b500 msgr2=0x7f6ea006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ea006b500 0x7f6ea006d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4036000 tx=0x7f6ea403c040 comp rx=0 tx=0).stop 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 msgr2=0x7f6eb415fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 0x7f6eb415fd50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00cbf0 tx=0x7f6eac00a430 comp rx=0 tx=0).stop 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 shutdown_connections 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6ea006b500 0x7f6ea006d9c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6eb415de00 0x7f6eb415fd50 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6eb40b9380 0x7f6eb415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 --2- 172.21.15.99:0/3524201410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6eb40b70d0 0x7f6eb40c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 >> 172.21.15.99:0/3524201410 conn(0x7f6eb40d0470 msgr2=0x7f6eb40ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.385+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 shutdown_connections 2024-03-04T10:06:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.386+0000 7f6eb9c4d640 1 -- 172.21.15.99:0/3524201410 wait complete. 2024-03-04T10:06:22.399 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:06:22.399 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:22.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.499+0000 7f921f8d1640 1 Processor -- start 2024-03-04T10:06:22.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921f8d1640 1 -- start start 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921f8d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f921815b9d0 0x7f92180af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921f8d1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f921806c010 0x7f92180b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921f8d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92180cc4b0 con 0x7f92180b3af0 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921f8d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92180ade20 con 0x7f921815b9d0 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921f8d1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f921815cdc0 con 0x7f921806c010 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.500+0000 7f921d66d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f921815b9d0 0x7f92180af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.501+0000 7f921d66d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f921815b9d0 0x7f92180af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56764/0 (socket says 172.21.15.99:56764) 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.501+0000 7f921d66d640 1 -- 172.21.15.99:0/1585291608 learned_addr learned my addr 172.21.15.99:0/1585291608 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.501+0000 7f921d66d640 1 -- 172.21.15.99:0/1585291608 >> v2:172.21.15.162:3300/0 conn(0x7f921806c010 msgr2=0x7f92180b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.501+0000 7f921d66d640 1 --2- 172.21.15.99:0/1585291608 >> v2:172.21.15.162:3300/0 conn(0x7f921806c010 0x7f92180b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.501+0000 7f921d66d640 1 -- 172.21.15.99:0/1585291608 >> v1:172.21.15.162:6789/0 conn(0x7f92180b3af0 legacy=0x7f921815f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.501+0000 7f921d66d640 1 -- 172.21.15.99:0/1585291608 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92180ad860 con 0x7f921815b9d0 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921d66d640 1 --2- 172.21.15.99:0/1585291608 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f921815b9d0 0x7f92180af320 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599e88dd07de0968 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f920ffff640 1 -- 172.21.15.99:0/1585291608 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921002e070 con 0x7f921815b9d0 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f920ffff640 1 -- 172.21.15.99:0/1585291608 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9210037070 con 0x7f921815b9d0 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f920ffff640 1 -- 172.21.15.99:0/1585291608 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210031a00 con 0x7f921815b9d0 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 -- 172.21.15.99:0/1585291608 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f921815b9d0 msgr2=0x7f92180af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 --2- 172.21.15.99:0/1585291608 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f921815b9d0 0x7f92180af320 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002b880 comp rx=0 tx=0).stop 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 -- 172.21.15.99:0/1585291608 shutdown_connections 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 --2- 172.21.15.99:0/1585291608 >> v2:172.21.15.162:3300/0 conn(0x7f921806c010 0x7f92180b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 --2- 172.21.15.99:0/1585291608 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f921815b9d0 0x7f92180af320 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 -- 172.21.15.99:0/1585291608 >> 172.21.15.99:0/1585291608 conn(0x7f92180cfe90 msgr2=0x7f92180d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 -- 172.21.15.99:0/1585291608 shutdown_connections 2024-03-04T10:06:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.502+0000 7f921f8d1640 1 -- 172.21.15.99:0/1585291608 wait complete. 2024-03-04T10:06:22.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.503+0000 7f921f8d1640 1 Processor -- start 2024-03-04T10:06:22.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 -- start start 2024-03-04T10:06:22.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f921806c010 0x7f9218149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 0x7f92180cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f921815b9d0 0x7f92180b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921806baf0 con 0x7f92180b3af0 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921806b810 con 0x7f921806c010 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921f8d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92180af860 con 0x7f921815b9d0 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921ce6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 0x7f92180cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921ce6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 0x7f92180cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56768/0 (socket says 172.21.15.99:56768) 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.504+0000 7f921ce6c640 1 -- 172.21.15.99:0/623311593 learned_addr learned my addr 172.21.15.99:0/623311593 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921d66d640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f921806c010 0x7f9218149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921de6e640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f921815b9d0 0x7f92180b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921ce6c640 1 -- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f921815b9d0 msgr2=0x7f92180b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921ce6c640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f921815b9d0 0x7f92180b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921ce6c640 1 -- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f921806c010 msgr2=0x7f9218149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921ce6c640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f921806c010 0x7f9218149ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921ce6c640 1 -- 172.21.15.99:0/623311593 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92180ce3a0 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921ce6c640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 0x7f92180cda30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f920800b440 tx=0x7f920800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9208015020 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9208011890 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9208011c80 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92180ce680 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.505+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92180cec40 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.506+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9218066ef0 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.506+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9208018020 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.507+0000 7f920dffb640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f91fc06b4d0 0x7f91fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.507+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f920808a600 con 0x7f92180b3af0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.507+0000 7f921d66d640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f91fc06b4d0 0x7f91fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.508+0000 7f921d66d640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f91fc06b4d0 0x7f91fc06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9210036000 tx=0x7f921003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.511+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9218066ef0 con 0x7f92180b3af0 2024-03-04T10:06:22.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.716+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9218074f60 con 0x7f92180b3af0 2024-03-04T10:06:22.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.716+0000 7f920dffb640 1 -- 172.21.15.99:0/623311593 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4939 (secure 0 0 0) 0x7f920801f020 con 0x7f92180b3af0 2024-03-04T10:06:22.718 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:22.718 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:20.757897+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:22.718 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 36 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f91fc06b4d0 msgr2=0x7f91fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f91fc06b4d0 0x7f91fc06d990 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9210036000 tx=0x7f921003d040 comp rx=0 tx=0).stop 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 msgr2=0x7f92180cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 0x7f92180cda30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f920800b440 tx=0x7f920800b900 comp rx=0 tx=0).stop 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 shutdown_connections 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f91fc06b4d0 0x7f91fc06d990 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f921815b9d0 0x7f92180b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92180b3af0 0x7f92180cda30 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 --2- 172.21.15.99:0/623311593 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f921806c010 0x7f9218149ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 >> 172.21.15.99:0/623311593 conn(0x7f92180cfe90 msgr2=0x7f92180afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:22.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 shutdown_connections 2024-03-04T10:06:22.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.718+0000 7f921f8d1640 1 -- 172.21.15.99:0/623311593 wait complete. 2024-03-04T10:06:22.730 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:22.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.829+0000 7f9428212640 1 Processor -- start 2024-03-04T10:06:22.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.829+0000 7f9428212640 1 -- start start 2024-03-04T10:06:22.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9428212640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f942015d660 0x7f942015e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9428212640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94200aeb20 0x7f9420071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9428212640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942015da40 con 0x7f9420071890 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9428212640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f942015cbc0 con 0x7f942015d660 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9428212640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942015ebb0 con 0x7f94200aeb20 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9425fae640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f942015d660 0x7f942015e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f94267af640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9420071890 0x7f9420150960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37188/0 (socket says 172.21.15.99:37188) 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f94267af640 1 -- 172.21.15.99:0/2759655502 learned_addr learned my addr 172.21.15.99:0/2759655502 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f94257ad640 1 --2- 172.21.15.99:0/2759655502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94200aeb20 0x7f9420071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9425fae640 1 -- 172.21.15.99:0/2759655502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94200aeb20 msgr2=0x7f9420071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9425fae640 1 --2- 172.21.15.99:0/2759655502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94200aeb20 0x7f9420071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9425fae640 1 -- 172.21.15.99:0/2759655502 >> v1:172.21.15.99:6789/0 conn(0x7f9420071890 legacy=0x7f9420150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9424fac640 1 -- 172.21.15.99:0/2759655502 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816707255 0 0) 0x7f942015da40 con 0x7f9420071890 2024-03-04T10:06:22.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.830+0000 7f9425fae640 1 -- 172.21.15.99:0/2759655502 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200af090 con 0x7f942015d660 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.831+0000 7f9425fae640 1 --2- 172.21.15.99:0/2759655502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f942015e670 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f941c0097a0 tx=0x7f941c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1612bfa2f97c2669 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.831+0000 7f9424fac640 1 -- 172.21.15.99:0/2759655502 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c038070 con 0x7f942015d660 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.831+0000 7f9424fac640 1 -- 172.21.15.99:0/2759655502 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941c031700 con 0x7f942015d660 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.831+0000 7f9424fac640 1 -- 172.21.15.99:0/2759655502 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c031ad0 con 0x7f942015d660 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.831+0000 7f9428212640 1 -- 172.21.15.99:0/2759655502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 msgr2=0x7f942015e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.831+0000 7f9428212640 1 --2- 172.21.15.99:0/2759655502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f942015e670 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f941c0097a0 tx=0x7f941c02b7a0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.832+0000 7f9428212640 1 -- 172.21.15.99:0/2759655502 shutdown_connections 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.832+0000 7f9428212640 1 --2- 172.21.15.99:0/2759655502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94200aeb20 0x7f9420071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.832+0000 7f9428212640 1 --2- 172.21.15.99:0/2759655502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f942015e670 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.832+0000 7f9428212640 1 -- 172.21.15.99:0/2759655502 >> 172.21.15.99:0/2759655502 conn(0x7f942006acd0 msgr2=0x7f942006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:22.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.832+0000 7f9428212640 1 -- 172.21.15.99:0/2759655502 shutdown_connections 2024-03-04T10:06:22.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.832+0000 7f9428212640 1 -- 172.21.15.99:0/2759655502 wait complete. 2024-03-04T10:06:22.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 Processor -- start 2024-03-04T10:06:22.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 -- start start 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9420071890 0x7f9420144310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94200aeb20 0x7f942014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f94200b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942015afb0 con 0x7f942015d660 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942015acd0 con 0x7f94200aeb20 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.833+0000 7f9428212640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942015a660 con 0x7f9420071890 2024-03-04T10:06:22.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f94200b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f94200b0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56788/0 (socket says 172.21.15.99:56788) 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 -- 172.21.15.99:0/2533929647 learned_addr learned my addr 172.21.15.99:0/2533929647 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f9425fae640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9420071890 0x7f9420144310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94257ad640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94200aeb20 0x7f942014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 -- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9420071890 msgr2=0x7f9420144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9420071890 0x7f9420144310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 -- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94200aeb20 msgr2=0x7f942014d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94200aeb20 0x7f942014d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 -- 172.21.15.99:0/2533929647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420167300 con 0x7f942015d660 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94267af640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f94200b0510 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f941800c7d0 tx=0x7f941800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418018070 con 0x7f942015d660 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941800a640 con 0x7f942015d660 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94201675e0 con 0x7f942015d660 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941800aa70 con 0x7f942015d660 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.834+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9420167ba0 con 0x7f942015d660 2024-03-04T10:06:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.835+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420074f60 con 0x7f942015d660 2024-03-04T10:06:22.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.836+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f941801b070 con 0x7f942015d660 2024-03-04T10:06:22.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.836+0000 7f94167fc640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f940406b500 0x7f940406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:22.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.836+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f9418089a10 con 0x7f942015d660 2024-03-04T10:06:22.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.836+0000 7f9425fae640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f940406b500 0x7f940406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:22.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.837+0000 7f9425fae640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f940406b500 0x7f940406d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f941c035000 tx=0x7f941c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:22.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:22.840+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9418058670 con 0x7f942015d660 2024-03-04T10:06:23.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.046+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9420072460 con 0x7f942015d660 2024-03-04T10:06:23.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.047+0000 7f94167fc640 1 -- 172.21.15.99:0/2533929647 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4939 (secure 0 0 0) 0x7f9418057f70 con 0x7f942015d660 2024-03-04T10:06:23.048 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:23.048 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:05:57.681103+0000","modified":"2024-03-04T10:06:20.757897+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6249":{"gid":6249,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":4,"addr":"172.21.15.99:6833/628973925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":628973925},{"type":"v1","addr":"172.21.15.99:6833","nonce":628973925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-04T10:06:23.049 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 36 2024-03-04T10:06:23.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f940406b500 msgr2=0x7f940406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f940406b500 0x7f940406d9c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f941c035000 tx=0x7f941c008030 comp rx=0 tx=0).stop 2024-03-04T10:06:23.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 msgr2=0x7f94200b0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f94200b0510 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f941800c7d0 tx=0x7f941800cc90 comp rx=0 tx=0).stop 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 shutdown_connections 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f940406b500 0x7f940406d9c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f942015d660 0x7f94200b0510 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94200aeb20 0x7f942014d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 --2- 172.21.15.99:0/2533929647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9420071890 0x7f9420144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 >> 172.21.15.99:0/2533929647 conn(0x7f942006acd0 msgr2=0x7f94200b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 shutdown_connections 2024-03-04T10:06:23.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.049+0000 7f9428212640 1 -- 172.21.15.99:0/2533929647 wait complete. 2024-03-04T10:06:23.061 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:23.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 Processor -- start 2024-03-04T10:06:23.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 -- start start 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa6f406d3b0 0x7fa6f40cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6f40ce9c0 0x7fa6f40cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f415cbb0 con 0x7fa6f40ce9c0 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f40d0140 con 0x7fa6f406d3b0 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.160+0000 7fa6fadb3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415c280 con 0x7fa6f40cfd60 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6f40ce9c0 0x7fa6f40cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6f40ce9c0 0x7fa6f40cee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56794/0 (socket says 172.21.15.99:56794) 2024-03-04T10:06:23.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 -- 172.21.15.99:0/3907692664 learned_addr learned my addr 172.21.15.99:0/3907692664 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:23.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f8b4f640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f3fff640 1 --2- 172.21.15.99:0/3907692664 >> v2:172.21.15.162:3300/0 conn(0x7fa6f406d3b0 0x7fa6f40cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 -- 172.21.15.99:0/3907692664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 msgr2=0x7fa6f40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 -- 172.21.15.99:0/3907692664 >> v2:172.21.15.162:3300/0 conn(0x7fa6f406d3b0 msgr2=0x7fa6f40cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 --2- 172.21.15.99:0/3907692664 >> v2:172.21.15.162:3300/0 conn(0x7fa6f406d3b0 0x7fa6f40cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f9350640 1 -- 172.21.15.99:0/3907692664 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f415c560 con 0x7fa6f40ce9c0 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f8b4f640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.161+0000 7fa6f3fff640 1 --2- 172.21.15.99:0/3907692664 >> v2:172.21.15.162:3300/0 conn(0x7fa6f406d3b0 0x7fa6f40cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6f9350640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f40ce9c0 0x7fa6f40cee50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa6f406d280 tx=0x7fa6ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87c6361fad3b159 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6f37fe640 1 -- 172.21.15.99:0/3907692664 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec02ca70 con 0x7fa6f40ce9c0 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6f37fe640 1 -- 172.21.15.99:0/3907692664 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec02cc10 con 0x7fa6f40ce9c0 2024-03-04T10:06:23.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6f37fe640 1 -- 172.21.15.99:0/3907692664 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0343f0 con 0x7fa6f40ce9c0 2024-03-04T10:06:23.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 -- 172.21.15.99:0/3907692664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f40ce9c0 msgr2=0x7fa6f40cee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f40ce9c0 0x7fa6f40cee50 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa6f406d280 tx=0x7fa6ec02bed0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 -- 172.21.15.99:0/3907692664 shutdown_connections 2024-03-04T10:06:23.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f40ce9c0 0x7fa6f40cee50 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/3907692664 >> v2:172.21.15.162:3300/0 conn(0x7fa6f406d3b0 0x7fa6f40cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/3907692664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 -- 172.21.15.99:0/3907692664 >> 172.21.15.99:0/3907692664 conn(0x7fa6f406acd0 msgr2=0x7fa6f406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 -- 172.21.15.99:0/3907692664 shutdown_connections 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.162+0000 7fa6fadb3640 1 -- 172.21.15.99:0/3907692664 wait complete. 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 Processor -- start 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 -- start start 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 0x7fa6f40acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6f40ce9c0 0x7fa6f414a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415bf30 con 0x7fa6f406d3b0 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40cf390 con 0x7fa6f40ce9c0 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.164+0000 7fa6fadb3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4071350 con 0x7fa6f40cfd60 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 0x7fa6f40acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 0x7fa6f40acc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56800/0 (socket says 172.21.15.99:56800) 2024-03-04T10:06:23.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 -- 172.21.15.99:0/2478350231 learned_addr learned my addr 172.21.15.99:0/2478350231 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 -- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 msgr2=0x7fa6f40b2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 -- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6f40ce9c0 msgr2=0x7fa6f414a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6f40ce9c0 0x7fa6f414a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 -- 172.21.15.99:0/2478350231 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40b3620 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f8b4f640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 0x7fa6f40acc30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800d7d0 tx=0x7fa6e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8004030 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e80041d0 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f40b3900 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e80096a0 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.165+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f40b3ec0 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.166+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f4074f60 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.167+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa6e8009840 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.167+0000 7fa6f17fa640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6d806b4d0 0x7fa6d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.167+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa6e808a730 con 0x7fa6f406d3b0 2024-03-04T10:06:23.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.167+0000 7fa6f3fff640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6d806b4d0 0x7fa6d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.168+0000 7fa6f3fff640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6d806b4d0 0x7fa6d806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4155cd0 tx=0x7fa6e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.171+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa6e8059310 con 0x7fa6f406d3b0 2024-03-04T10:06:23.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.342+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6f4068050 con 0x7fa6f406d3b0 2024-03-04T10:06:23.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.343+0000 7fa6f17fa640 1 -- 172.21.15.99:0/2478350231 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+17072 (secure 0 0 0) 0x7fa6e8058c10 con 0x7fa6f406d3b0 2024-03-04T10:06:23.345 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:23.345 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":55,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:21.944559+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:05:55.899275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-04T10:05:56.653255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.c","mappings":[{"from":0,"to":6}]},{"pgid":"8.17","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6833/370337902":"2024-03-05T10:06:17.747062+0000","172.21.15.99:6832/370337902":"2024-03-05T10:06:17.747062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6d806b4d0 msgr2=0x7fa6d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6d806b4d0 0x7fa6d806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4155cd0 tx=0x7fa6e0008040 comp rx=0 tx=0).stop 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 msgr2=0x7fa6f40acc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 0x7fa6f40acc30 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800d7d0 tx=0x7fa6e800dc90 comp rx=0 tx=0).stop 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 shutdown_connections 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6d806b4d0 0x7fa6d806d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6f40cfd60 0x7fa6f40b2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6f40ce9c0 0x7fa6f414a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.345+0000 7fa6fadb3640 1 --2- 172.21.15.99:0/2478350231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6f406d3b0 0x7fa6f40acc30 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.346+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 >> 172.21.15.99:0/2478350231 conn(0x7fa6f406acd0 msgr2=0x7fa6f4159e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:23.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.346+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 shutdown_connections 2024-03-04T10:06:23.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.346+0000 7fa6fadb3640 1 -- 172.21.15.99:0/2478350231 wait complete. 2024-03-04T10:06:23.358 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:06:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.458+0000 7f26d7f5b640 1 Processor -- start 2024-03-04T10:06:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.458+0000 7f26d7f5b640 1 -- start start 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d7f5b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f26d006d3b0 0x7f26d015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d7f5b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f26d015b410 0x7f26d015b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d7f5b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d006d8b0 con 0x7f26d015b030 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d7f5b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d00b2720 con 0x7f26d015b410 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d7f5b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d006db90 con 0x7f26d006d3b0 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d64f8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f26d015b410 0x7f26d015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d5cf7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f26d015b030 0x7f26d015dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47754/0 (socket says 172.21.15.99:47754) 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d5cf7640 1 -- 172.21.15.99:0/1450437208 learned_addr learned my addr 172.21.15.99:0/1450437208 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d4cf5640 1 -- 172.21.15.99:0/1450437208 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445917692 0 0) 0x7f26d006d8b0 con 0x7f26d015b030 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d4cf5640 1 -- 172.21.15.99:0/1450437208 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26bc003520 con 0x7f26d015b030 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d64f8640 1 -- 172.21.15.99:0/1450437208 >> v2:172.21.15.162:3300/0 conn(0x7f26d006d3b0 msgr2=0x7f26d015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d54f6640 1 --2- 172.21.15.99:0/1450437208 >> v2:172.21.15.162:3300/0 conn(0x7f26d006d3b0 0x7f26d015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d64f8640 1 --2- 172.21.15.99:0/1450437208 >> v2:172.21.15.162:3300/0 conn(0x7f26d006d3b0 0x7f26d015c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d64f8640 1 -- 172.21.15.99:0/1450437208 >> v1:172.21.15.99:6789/0 conn(0x7f26d015b030 legacy=0x7f26d015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.459+0000 7f26d64f8640 1 -- 172.21.15.99:0/1450437208 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d006de70 con 0x7f26d015b410 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d64f8640 1 --2- 172.21.15.99:0/1450437208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d015b880 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f26d00b28a0 tx=0x7f26c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1032c2baa8520bad server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d4cf5640 1 -- 172.21.15.99:0/1450437208 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c80315a0 con 0x7f26d015b410 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d4cf5640 1 -- 172.21.15.99:0/1450437208 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c8037070 con 0x7f26d015b410 2024-03-04T10:06:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d4cf5640 1 -- 172.21.15.99:0/1450437208 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8031940 con 0x7f26d015b410 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d7f5b640 1 -- 172.21.15.99:0/1450437208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 msgr2=0x7f26d015b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/1450437208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d015b880 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f26d00b28a0 tx=0x7f26c802b840 comp rx=0 tx=0).stop 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d7f5b640 1 -- 172.21.15.99:0/1450437208 shutdown_connections 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/1450437208 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d015b880 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/1450437208 >> v2:172.21.15.162:3300/0 conn(0x7f26d006d3b0 0x7f26d015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.460+0000 7f26d7f5b640 1 -- 172.21.15.99:0/1450437208 >> 172.21.15.99:0/1450437208 conn(0x7f26d006acd0 msgr2=0x7f26d006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.461+0000 7f26d7f5b640 1 -- 172.21.15.99:0/1450437208 shutdown_connections 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.461+0000 7f26d7f5b640 1 -- 172.21.15.99:0/1450437208 wait complete. 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 Processor -- start 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 -- start start 2024-03-04T10:06:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26d006d3b0 0x7f26d0149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26d015b030 0x7f26d01507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d014df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d006e690 con 0x7f26d015b410 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d015bdc0 con 0x7f26d015b030 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d7f5b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d006e810 con 0x7f26d006d3b0 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d64f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d014df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d64f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d014df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35514/0 (socket says 172.21.15.99:35514) 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.462+0000 7f26d64f8640 1 -- 172.21.15.99:0/2831352410 learned_addr learned my addr 172.21.15.99:0/2831352410 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d5cf7640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26d006d3b0 0x7f26d0149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d64f8640 1 -- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26d006d3b0 msgr2=0x7f26d0149c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d64f8640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26d006d3b0 0x7f26d0149c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d54f6640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26d015b030 0x7f26d01507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d64f8640 1 -- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26d015b030 msgr2=0x7f26d01507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d64f8640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26d015b030 0x7f26d01507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d64f8640 1 -- 172.21.15.99:0/2831352410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d014d1a0 con 0x7f26d015b410 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d64f8640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d014df60 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f26c8034040 tx=0x7f26c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8032630 con 0x7f26d015b410 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c80327d0 con 0x7f26d015b410 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d014d480 con 0x7f26d015b410 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8032b80 con 0x7f26d015b410 2024-03-04T10:06:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.463+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d014a150 con 0x7f26d015b410 2024-03-04T10:06:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.464+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d0074f60 con 0x7f26d015b410 2024-03-04T10:06:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.465+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f26c804c020 con 0x7f26d015b410 2024-03-04T10:06:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.465+0000 7f26c67fc640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26b406b4d0 0x7f26b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.465+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f26c802f020 con 0x7f26d015b410 2024-03-04T10:06:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.465+0000 7f26d5cf7640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26b406b4d0 0x7f26b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.466+0000 7f26d5cf7640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26b406b4d0 0x7f26b406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0012e0 tx=0x7f26cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.469+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f26c8040070 con 0x7f26d015b410 2024-03-04T10:06:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.674+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f26d0149840 con 0x7f26d015b410 2024-03-04T10:06:23.676 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:06:23.675+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:06:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.676+0000 7f26c67fc640 1 -- 172.21.15.99:0/2831352410 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v37) v1 ==== 161+0+0 (secure 0 0 0) 0x7f26c807ad20 con 0x7f26d015b410 2024-03-04T10:06:23.677 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26b406b4d0 msgr2=0x7f26b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26b406b4d0 0x7f26b406d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0012e0 tx=0x7f26cc009040 comp rx=0 tx=0).stop 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 msgr2=0x7f26d014df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d014df60 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f26c8034040 tx=0x7f26c80046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 shutdown_connections 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26b406b4d0 0x7f26b406d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f26d015b410 0x7f26d014df60 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f26d015b030 0x7f26d01507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 --2- 172.21.15.99:0/2831352410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f26d006d3b0 0x7f26d0149c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 >> 172.21.15.99:0/2831352410 conn(0x7f26d006acd0 msgr2=0x7f26d0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 shutdown_connections 2024-03-04T10:06:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.679+0000 7f26d7f5b640 1 -- 172.21.15.99:0/2831352410 wait complete. 2024-03-04T10:06:23.692 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T10:06:23.705 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.793+0000 7f1e1e838640 1 Processor -- start 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.793+0000 7f1e1e838640 1 -- start start 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1e838640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1e180b5700 0x7f1e180d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1e838640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e1806c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1e838640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e180d2400 con 0x7f1e180b9410 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1e838640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e180d26e0 con 0x7f1e180b5700 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1e838640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180d29c0 con 0x7f1e180d34a0 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e17fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1e180b5700 0x7f1e180d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1cdd5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1e180b9410 0x7f1e1806c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47760/0 (socket says 172.21.15.99:47760) 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e17fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1e180b5700 0x7f1e180d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35536/0 (socket says 172.21.15.99:35536) 2024-03-04T10:06:23.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.794+0000 7f1e1cdd5640 1 -- 172.21.15.99:0/3038528554 learned_addr learned my addr 172.21.15.99:0/3038528554 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e177fe640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e1806c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e17fff640 1 -- 172.21.15.99:0/3038528554 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 msgr2=0x7f1e1806c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e17fff640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e1806c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e17fff640 1 -- 172.21.15.99:0/3038528554 >> v1:172.21.15.99:6789/0 conn(0x7f1e180b9410 legacy=0x7f1e1806c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e16ffd640 1 -- 172.21.15.99:0/3038528554 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285147266 0 0) 0x7f1e180d2400 con 0x7f1e180b9410 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e17fff640 1 -- 172.21.15.99:0/3038528554 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180d2ca0 con 0x7f1e180b5700 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e177fe640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e1806c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.795+0000 7f1e17fff640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b5700 0x7f1e180d1b00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1e080097a0 tx=0x7f1e0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5133b09cefa496 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e16ffd640 1 -- 172.21.15.99:0/3038528554 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e080315a0 con 0x7f1e180b5700 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e16ffd640 1 -- 172.21.15.99:0/3038528554 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e08037070 con 0x7f1e180b5700 2024-03-04T10:06:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e16ffd640 1 -- 172.21.15.99:0/3038528554 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e08031960 con 0x7f1e180b5700 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 -- 172.21.15.99:0/3038528554 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b5700 msgr2=0x7f1e180d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b5700 0x7f1e180d1b00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f1e080097a0 tx=0x7f1e0802b840 comp rx=0 tx=0).stop 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 -- 172.21.15.99:0/3038528554 shutdown_connections 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e1806c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 --2- 172.21.15.99:0/3038528554 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b5700 0x7f1e180d1b00 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 -- 172.21.15.99:0/3038528554 >> 172.21.15.99:0/3038528554 conn(0x7f1e180d0490 msgr2=0x7f1e180d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 -- 172.21.15.99:0/3038528554 shutdown_connections 2024-03-04T10:06:23.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.796+0000 7f1e1e838640 1 -- 172.21.15.99:0/3038528554 wait complete. 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.797+0000 7f1e1e838640 1 Processor -- start 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.797+0000 7f1e1e838640 1 -- start start 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1e838640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180b5700 0x7f1e180c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1e838640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 0x7f1e1815eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1e838640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e180aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1e838640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1806ceb0 con 0x7f1e180b9410 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1e838640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1806cd30 con 0x7f1e180b5700 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1e838640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180b5ae0 con 0x7f1e180d34a0 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 0x7f1e1815eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 0x7f1e1815eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35538/0 (socket says 172.21.15.99:35538) 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e177fe640 1 -- 172.21.15.99:0/1524495611 learned_addr learned my addr 172.21.15.99:0/1524495611 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e1cdd5640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e180aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.798+0000 7f1e17fff640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180b5700 0x7f1e180c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e177fe640 1 -- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 msgr2=0x7f1e180aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e177fe640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e180aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e177fe640 1 -- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180b5700 msgr2=0x7f1e180c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e177fe640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180b5700 0x7f1e180c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e177fe640 1 -- 172.21.15.99:0/1524495611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180ca1a0 con 0x7f1e180b9410 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e17fff640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180b5700 0x7f1e180c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e177fe640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 0x7f1e1815eba0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c7f0 tx=0x7f1e0c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c015070 con 0x7f1e180b9410 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c007820 con 0x7f1e180b9410 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c007bd0 con 0x7f1e180b9410 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e180ac5d0 con 0x7f1e180b9410 2024-03-04T10:06:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.799+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e180acb90 con 0x7f1e180b9410 2024-03-04T10:06:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.800+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e18074f60 con 0x7f1e180b9410 2024-03-04T10:06:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.800+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1e0c018070 con 0x7f1e180b9410 2024-03-04T10:06:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.801+0000 7f1e14ff9640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e0406b4d0 0x7f1e0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.801+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f1e0c08a4c0 con 0x7f1e180b9410 2024-03-04T10:06:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.801+0000 7f1e17fff640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e0406b4d0 0x7f1e0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.802+0000 7f1e17fff640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e0406b4d0 0x7f1e0406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1e08036000 tx=0x7f1e08008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:23.805+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e0c058a80 con 0x7f1e180b9410 2024-03-04T10:06:24.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.011+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1e180ced50 con 0x7f1e180b9410 2024-03-04T10:06:24.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.677+0000 7f1e14ff9640 1 -- 172.21.15.99:0/1524495611 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1e0c05cb20 con 0x7f1e180b9410 2024-03-04T10:06:24.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e0406b4d0 msgr2=0x7f1e0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:24.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e0406b4d0 0x7f1e0406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1e08036000 tx=0x7f1e08008030 comp rx=0 tx=0).stop 2024-03-04T10:06:24.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 msgr2=0x7f1e1815eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:24.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 0x7f1e1815eba0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c7f0 tx=0x7f1e0c00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 shutdown_connections 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e0406b4d0 0x7f1e0406d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e180d34a0 0x7f1e180aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e180b9410 0x7f1e1815eba0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 --2- 172.21.15.99:0/1524495611 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e180b5700 0x7f1e180c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 >> 172.21.15.99:0/1524495611 conn(0x7f1e180d0490 msgr2=0x7f1e180ce290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 shutdown_connections 2024-03-04T10:06:24.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.680+0000 7f1e1e838640 1 -- 172.21.15.99:0/1524495611 wait complete. 2024-03-04T10:06:24.693 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T10:06:24.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.839+0000 7f0145292640 1 Processor -- start 2024-03-04T10:06:24.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.839+0000 7f0145292640 1 -- start start 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.839+0000 7f0145292640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f014015db20 0x7f01400b9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.839+0000 7f0145292640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 0x7f01400b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.839+0000 7f0145292640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01400ce120 con 0x7f01400d3c20 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f0145292640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01400ce730 con 0x7f014015db20 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f0145292640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01400b6300 con 0x7f01400b5bc0 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f014015db20 0x7f01400b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 0x7f01400b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f014015db20 0x7f01400b9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:52992/0 (socket says 172.21.15.99:52992) 2024-03-04T10:06:24.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 0x7f01400b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:34006/0 (socket says 172.21.15.99:34006) 2024-03-04T10:06:24.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f0137fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f01400d3c20 0x7f01400b54b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:60466/0 (socket says 172.21.15.99:60466) 2024-03-04T10:06:24.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 -- 172.21.15.99:0/1647812223 learned_addr learned my addr 172.21.15.99:0/1647812223 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 -- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 msgr2=0x7f01400b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 --2- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 0x7f01400b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 -- 172.21.15.99:0/1647812223 >> v1:172.21.15.162:6789/0 conn(0x7f01400d3c20 legacy=0x7f01400b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013e7fc640 1 -- 172.21.15.99:0/1647812223 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048782495 0 0) 0x7f01400ce120 con 0x7f01400d3c20 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013effd640 1 -- 172.21.15.99:0/1647812223 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01400ced90 con 0x7f014015db20 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.840+0000 7f013f7fe640 1 --2- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 0x7f01400b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.841+0000 7f013effd640 1 --2- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f014015db20 0x7f01400b9350 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f01400ce8b0 tx=0x7f013002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb421354510155bb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.841+0000 7f013e7fc640 1 -- 172.21.15.99:0/1647812223 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013002f830 con 0x7f014015db20 2024-03-04T10:06:24.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.841+0000 7f013e7fc640 1 -- 172.21.15.99:0/1647812223 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013002f9d0 con 0x7f014015db20 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.841+0000 7f013e7fc640 1 -- 172.21.15.99:0/1647812223 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013002fda0 con 0x7f014015db20 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 -- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f014015db20 msgr2=0x7f01400b9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 --2- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f014015db20 0x7f01400b9350 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f01400ce8b0 tx=0x7f013002ba10 comp rx=0 tx=0).stop 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 -- 172.21.15.99:0/1647812223 shutdown_connections 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 --2- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400b5bc0 0x7f01400b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 --2- 172.21.15.99:0/1647812223 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f014015db20 0x7f01400b9350 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 -- 172.21.15.99:0/1647812223 >> 172.21.15.99:0/1647812223 conn(0x7f0140072820 msgr2=0x7f0140072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 -- 172.21.15.99:0/1647812223 shutdown_connections 2024-03-04T10:06:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.842+0000 7f0145292640 1 -- 172.21.15.99:0/1647812223 wait complete. 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.843+0000 7f0145292640 1 Processor -- start 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.843+0000 7f0145292640 1 -- start start 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0145292640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01400b5bc0 0x7f0140153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0145292640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400d3c20 0x7f01400c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0145292640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 0x7f01400cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0145292640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400cea20 con 0x7f014015db20 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0145292640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01400b9890 con 0x7f01400b5bc0 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0145292640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014015df00 con 0x7f01400d3c20 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 0x7f01400cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 0x7f01400cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35546/0 (socket says 172.21.15.99:35546) 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 -- 172.21.15.99:0/4077646961 learned_addr learned my addr 172.21.15.99:0/4077646961 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f0137fff640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400d3c20 0x7f01400c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013effd640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01400b5bc0 0x7f0140153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 -- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400d3c20 msgr2=0x7f01400c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400d3c20 0x7f01400c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 -- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01400b5bc0 msgr2=0x7f0140153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01400b5bc0 0x7f0140153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 -- 172.21.15.99:0/4077646961 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01400c7a80 con 0x7f014015db20 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.844+0000 7f013f7fe640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 0x7f01400cc200 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f013800efc0 tx=0x7f0138009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.845+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138018030 con 0x7f014015db20 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.845+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013800cde0 con 0x7f014015db20 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.845+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01400c7d20 con 0x7f014015db20 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.845+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138003bb0 con 0x7f014015db20 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.845+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01400c82e0 con 0x7f014015db20 2024-03-04T10:06:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.846+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01400d2430 con 0x7f014015db20 2024-03-04T10:06:24.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.846+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f013801b070 con 0x7f014015db20 2024-03-04T10:06:24.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.847+0000 7f01377fe640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f012006b4d0 0x7f012006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:24.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.847+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f013808ad20 con 0x7f014015db20 2024-03-04T10:06:24.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.847+0000 7f013effd640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f012006b4d0 0x7f012006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:24.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.847+0000 7f013effd640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f012006b4d0 0x7f012006d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0130036000 tx=0x7f0130008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:24.851+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f01400d2430 con 0x7f014015db20 2024-03-04T10:06:25.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.025+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0140074f60 con 0x7f014015db20 2024-03-04T10:06:25.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.969+0000 7f01377fe640 1 -- 172.21.15.99:0/4077646961 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0138010040 con 0x7f014015db20 2024-03-04T10:06:25.970 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' removed 2024-03-04T10:06:25.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f012006b4d0 msgr2=0x7f012006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:25.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f012006b4d0 0x7f012006d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0130036000 tx=0x7f0130008030 comp rx=0 tx=0).stop 2024-03-04T10:06:25.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 msgr2=0x7f01400cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 0x7f01400cc200 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f013800efc0 tx=0x7f0138009f90 comp rx=0 tx=0).stop 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 shutdown_connections 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f012006b4d0 0x7f012006d990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f014015db20 0x7f01400cc200 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01400d3c20 0x7f01400c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 --2- 172.21.15.99:0/4077646961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01400b5bc0 0x7f0140153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 >> 172.21.15.99:0/4077646961 conn(0x7f0140072820 msgr2=0x7f01400b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 shutdown_connections 2024-03-04T10:06:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:25.973+0000 7f0145292640 1 -- 172.21.15.99:0/4077646961 wait complete. 2024-03-04T10:06:25.988 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T10:06:26.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.096+0000 7f675a419640 1 Processor -- start 2024-03-04T10:06:26.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.096+0000 7f675a419640 1 -- start start 2024-03-04T10:06:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f675a419640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f675415dc80 0x7f675406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f675a419640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675406af20 con 0x7f67540b4ff0 2024-03-04T10:06:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f675a419640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f675406b200 con 0x7f67540671c0 2024-03-04T10:06:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f675a419640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f675406acd0 con 0x7f675415dc80 2024-03-04T10:06:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f67589b6640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f67540b4ff0 0x7f67540ade50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47772/0 (socket says 172.21.15.99:47772) 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f67589b6640 1 -- 172.21.15.99:0/1180044651 learned_addr learned my addr 172.21.15.99:0/1180044651 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.097+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189693186 0 0) 0x7f675406af20 con 0x7f67540b4ff0 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f673c003520 con 0x7f67540b4ff0 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6753fff640 1 --2- 172.21.15.99:0/1180044651 >> v2:172.21.15.162:3300/0 conn(0x7f675415dc80 0x7f675406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1238681103 0 0) 0x7f673c003520 con 0x7f67540b4ff0 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 >> v2:172.21.15.162:3300/0 conn(0x7f675415dc80 msgr2=0x7f675406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 --2- 172.21.15.99:0/1180044651 >> v2:172.21.15.162:3300/0 conn(0x7f675415dc80 0x7f675406bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 >> v1:172.21.15.162:6789/0 conn(0x7f67540671c0 legacy=0x7f67540b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675406b4e0 con 0x7f67540b4ff0 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f674c0019f0 con 0x7f67540b4ff0 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 >> v1:172.21.15.99:6789/0 conn(0x7f67540b4ff0 legacy=0x7f67540ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f673c0039b0 0x7f673c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675406acd0 con 0x7f673c0051c0 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c00c0f0 con 0x7f673c0039b0 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c00c2f0 con 0x7f673c004800 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67589b6640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f6753fff640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f673c0039b0 0x7f673c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67537fe640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67589b6640 1 -- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f673c004800 msgr2=0x7f673c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67589b6640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f673c004800 0x7f673c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67589b6640 1 -- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f673c0039b0 msgr2=0x7f673c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67589b6640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f673c0039b0 0x7f673c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.098+0000 7f67589b6640 1 -- 172.21.15.99:0/1180044651 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c003520 con 0x7f673c0051c0 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.099+0000 7f67589b6640 1 -- 172.21.15.99:0/1180044651 reap_dead start 2024-03-04T10:06:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.099+0000 7f67589b6640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f67540b1c90 tx=0x7f673c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd917fe5c0847a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.099+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c00a480 con 0x7f673c0051c0 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.099+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c034630 con 0x7f673c0051c0 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.099+0000 7f6752ffd640 1 -- 172.21.15.99:0/1180044651 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c02ce60 con 0x7f673c0051c0 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 -- 172.21.15.99:0/1180044651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 msgr2=0x7f673c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f67540b1c90 tx=0x7f673c005830 comp rx=0 tx=0).stop 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 -- 172.21.15.99:0/1180044651 shutdown_connections 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 --2- 172.21.15.99:0/1180044651 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f673c00b9c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 -- 172.21.15.99:0/1180044651 >> 172.21.15.99:0/1180044651 conn(0x7f6754071ca0 msgr2=0x7f6754072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 -- 172.21.15.99:0/1180044651 shutdown_connections 2024-03-04T10:06:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.100+0000 7f675a419640 1 -- 172.21.15.99:0/1180044651 wait complete. 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.101+0000 7f675a419640 1 Processor -- start 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.101+0000 7f675a419640 1 -- start start 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f675a419640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f6754142380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f675a419640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67540b03a0 0x7f6754141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f675a419640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67540b0820 0x7f67541419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f675a419640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675406baa0 con 0x7f673c0051c0 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f675a419640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67540729b0 con 0x7f67540b0820 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f675a419640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67540b2fe0 con 0x7f67540b03a0 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f6754142380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f6754142380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35552/0 (socket says 172.21.15.99:35552) 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 -- 172.21.15.99:0/1981982751 learned_addr learned my addr 172.21.15.99:0/1981982751 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f67537fe640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67540b03a0 0x7f6754141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f67589b6640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67540b0820 0x7f67541419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 -- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67540b03a0 msgr2=0x7f6754141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67540b03a0 0x7f6754141490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 -- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67540b0820 msgr2=0x7f67541419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67540b0820 0x7f67541419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 -- 172.21.15.99:0/1981982751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6754153bd0 con 0x7f673c0051c0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f67589b6640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67540b0820 0x7f67541419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.102+0000 7f6753fff640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f6754142380 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f674400f580 tx=0x7f674400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.103+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744013810 con 0x7f673c0051c0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.103+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674401a070 con 0x7f673c0051c0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.103+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6754153eb0 con 0x7f673c0051c0 2024-03-04T10:06:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.103+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744013bb0 con 0x7f673c0051c0 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.103+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6754154be0 con 0x7f673c0051c0 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.104+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6754068480 con 0x7f673c0051c0 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.104+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6744012070 con 0x7f673c0051c0 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.104+0000 7f6750ff9640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f674006b500 0x7f674006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.104+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5893+0+0 (secure 0 0 0) 0x7f674408a1c0 con 0x7f673c0051c0 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.105+0000 7f67537fe640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f674006b500 0x7f674006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.105+0000 7f67537fe640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f674006b500 0x7f674006d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6748007fd0 tx=0x7f6748009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:26.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.109+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6754068480 con 0x7f673c0051c0 2024-03-04T10:06:26.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.286+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6754160ad0 con 0x7f673c0051c0 2024-03-04T10:06:26.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.970+0000 7f6750ff9640 1 -- 172.21.15.99:0/1981982751 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6744058f90 con 0x7f673c0051c0 2024-03-04T10:06:26.972 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' removed 2024-03-04T10:06:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.974+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f674006b500 msgr2=0x7f674006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.974+0000 7f675a419640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f674006b500 0x7f674006d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6748007fd0 tx=0x7f6748009040 comp rx=0 tx=0).stop 2024-03-04T10:06:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.974+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 msgr2=0x7f6754142380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.974+0000 7f675a419640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f6754142380 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f674400f580 tx=0x7f674400fa40 comp rx=0 tx=0).stop 2024-03-04T10:06:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.974+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 shutdown_connections 2024-03-04T10:06:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.974+0000 7f675a419640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f674006b500 0x7f674006d9c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.975+0000 7f675a419640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67540b0820 0x7f67541419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.975+0000 7f675a419640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67540b03a0 0x7f6754141490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.975+0000 7f675a419640 1 --2- 172.21.15.99:0/1981982751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f673c0051c0 0x7f6754142380 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.975+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 >> 172.21.15.99:0/1981982751 conn(0x7f6754071ca0 msgr2=0x7f67540b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.975+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 shutdown_connections 2024-03-04T10:06:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:26.975+0000 7f675a419640 1 -- 172.21.15.99:0/1981982751 wait complete. 2024-03-04T10:06:26.991 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_abort_on_newly_corrupt_dentry 2024-03-04T10:06:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.096+0000 7f1577745640 1 Processor -- start 2024-03-04T10:06:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.096+0000 7f1577745640 1 -- start start 2024-03-04T10:06:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1577745640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f15700d1e90 0x7f1570166990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1577745640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157015bf50 con 0x7f157015f660 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1577745640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157015fb60 con 0x7f157015bb70 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1577745640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15700cf890 con 0x7f15700d1e90 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1575ce2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f15700d1e90 0x7f1570166990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1574ce0640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f157015f660 0x7f15700d1780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47784/0 (socket says 172.21.15.99:47784) 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.097+0000 7f1574ce0640 1 -- 172.21.15.99:0/3224952413 learned_addr learned my addr 172.21.15.99:0/3224952413 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1575ce2640 1 -- 172.21.15.99:0/3224952413 >> v1:172.21.15.162:6789/0 conn(0x7f157015bb70 legacy=0x7f15700b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1575ce2640 1 -- 172.21.15.99:0/3224952413 >> v1:172.21.15.99:6789/0 conn(0x7f157015f660 legacy=0x7f15700d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1575ce2640 1 -- 172.21.15.99:0/3224952413 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157015fe40 con 0x7f15700d1e90 2024-03-04T10:06:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1575ce2640 1 --2- 172.21.15.99:0/3224952413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15700d1e90 0x7f1570166990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1568009d90 tx=0x7f156802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c2ecaea776b51c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1567fff640 1 -- 172.21.15.99:0/3224952413 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568039070 con 0x7f15700d1e90 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1567fff640 1 -- 172.21.15.99:0/3224952413 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156802cc60 con 0x7f15700d1e90 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.098+0000 7f1567fff640 1 -- 172.21.15.99:0/3224952413 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568034660 con 0x7f15700d1e90 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 -- 172.21.15.99:0/3224952413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15700d1e90 msgr2=0x7f1570166990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 --2- 172.21.15.99:0/3224952413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15700d1e90 0x7f1570166990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1568009d90 tx=0x7f156802beb0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 -- 172.21.15.99:0/3224952413 shutdown_connections 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 --2- 172.21.15.99:0/3224952413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15700d1e90 0x7f1570166990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 -- 172.21.15.99:0/3224952413 >> 172.21.15.99:0/3224952413 conn(0x7f15700b33a0 msgr2=0x7f15700b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 -- 172.21.15.99:0/3224952413 shutdown_connections 2024-03-04T10:06:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.099+0000 7f1577745640 1 -- 172.21.15.99:0/3224952413 wait complete. 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.100+0000 7f1577745640 1 Processor -- start 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.100+0000 7f1577745640 1 -- start start 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1577745640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15700d1e90 0x7f15700a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1577745640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f157015bb70 0x7f1570150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1577745640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 0x7f15700ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1577745640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570160400 con 0x7f157015f660 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1577745640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15700b1150 con 0x7f15700d1e90 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1577745640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15700d0d30 con 0x7f157015bb70 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 0x7f15700ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 0x7f15700ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35576/0 (socket says 172.21.15.99:35576) 2024-03-04T10:06:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 -- 172.21.15.99:0/1191602027 learned_addr learned my addr 172.21.15.99:0/1191602027 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f15754e1640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15700d1e90 0x7f15700a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1574ce0640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f157015bb70 0x7f1570150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 -- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f157015bb70 msgr2=0x7f1570150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f157015bb70 0x7f1570150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 -- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15700d1e90 msgr2=0x7f15700a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15700d1e90 0x7f15700a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.101+0000 7f1575ce2640 1 -- 172.21.15.99:0/1191602027 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157015dc60 con 0x7f157015f660 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.102+0000 7f1575ce2640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 0x7f15700ab130 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1568009eb0 tx=0x7f1568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.102+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568039050 con 0x7f157015f660 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.102+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1568045070 con 0x7f157015f660 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.102+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157015df40 con 0x7f157015f660 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.102+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156803e830 con 0x7f157015f660 2024-03-04T10:06:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.102+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157015e500 con 0x7f157015f660 2024-03-04T10:06:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.103+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570074f60 con 0x7f157015f660 2024-03-04T10:06:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.103+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f156803e9d0 con 0x7f157015f660 2024-03-04T10:06:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.104+0000 7f1565ffb640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155406b4d0 0x7f155406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.104+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f15680acf40 con 0x7f157015f660 2024-03-04T10:06:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.104+0000 7f15754e1640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155406b4d0 0x7f155406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.104+0000 7f15754e1640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155406b4d0 0x7f155406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f156c001780 tx=0x7f156c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.108+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f156807b9b0 con 0x7f157015f660 2024-03-04T10:06:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.282+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_abort_on_newly_corrupt_dentry"} v 0) v1 -- 0x7f15700cbb00 con 0x7f157015f660 2024-03-04T10:06:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.285+0000 7f1565ffb640 1 -- 172.21.15.99:0/1191602027 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_abort_on_newly_corrupt_dentry"}]=0 v16) v1 ==== 116+0+0 (secure 0 0 0) 0x7f156807fa50 con 0x7f157015f660 2024-03-04T10:06:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.287+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155406b4d0 msgr2=0x7f155406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.287+0000 7f1577745640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155406b4d0 0x7f155406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f156c001780 tx=0x7f156c009040 comp rx=0 tx=0).stop 2024-03-04T10:06:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.287+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 msgr2=0x7f15700ab130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.287+0000 7f1577745640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 0x7f15700ab130 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f1568009eb0 tx=0x7f1568004a40 comp rx=0 tx=0).stop 2024-03-04T10:06:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 shutdown_connections 2024-03-04T10:06:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f155406b4d0 0x7f155406d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f157015f660 0x7f15700ab130 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f157015bb70 0x7f1570150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 --2- 172.21.15.99:0/1191602027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15700d1e90 0x7f15700a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 >> 172.21.15.99:0/1191602027 conn(0x7f15700b33a0 msgr2=0x7f15700c7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 shutdown_connections 2024-03-04T10:06:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.288+0000 7f1577745640 1 -- 172.21.15.99:0/1191602027 wait complete. 2024-03-04T10:06:27.299 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T10:06:27.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.400+0000 7f9bc6d89640 1 Processor -- start 2024-03-04T10:06:27.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.400+0000 7f9bc6d89640 1 -- start start 2024-03-04T10:06:27.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bc6d89640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9bc00cf660 0x7f9bc00cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bc6d89640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc01616e0 0x7f9bc0161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bc6d89640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc0162450 con 0x7f9bc015da20 2024-03-04T10:06:27.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bc6d89640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc00d0000 con 0x7f9bc00cf660 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bc6d89640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00cc920 con 0x7f9bc01616e0 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9bc00cf660 0x7f9bc00cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9bc00cf660 0x7f9bc00cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35582/0 (socket says 172.21.15.99:35582) 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 -- 172.21.15.99:0/395396621 learned_addr learned my addr 172.21.15.99:0/395396621 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bc5326640 1 --2- 172.21.15.99:0/395396621 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc01616e0 0x7f9bc0161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 -- 172.21.15.99:0/395396621 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc01616e0 msgr2=0x7f9bc0161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 --2- 172.21.15.99:0/395396621 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc01616e0 0x7f9bc0161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 -- 172.21.15.99:0/395396621 >> v1:172.21.15.162:6789/0 conn(0x7f9bc015da20 legacy=0x7f9bc00ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.401+0000 7f9bbffff640 1 -- 172.21.15.99:0/395396621 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00cc2d0 con 0x7f9bc00cf660 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.402+0000 7f9bbffff640 1 --2- 172.21.15.99:0/395396621 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc00cf660 0x7f9bc00cfac0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009870 tx=0x7f9bb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f1bdbfc3fb2218 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.402+0000 7f9bbf7fe640 1 -- 172.21.15.99:0/395396621 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4035020 con 0x7f9bc00cf660 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.402+0000 7f9bbf7fe640 1 -- 172.21.15.99:0/395396621 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb4031640 con 0x7f9bc00cf660 2024-03-04T10:06:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.402+0000 7f9bbf7fe640 1 -- 172.21.15.99:0/395396621 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb402d040 con 0x7f9bc00cf660 2024-03-04T10:06:27.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 -- 172.21.15.99:0/395396621 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc00cf660 msgr2=0x7f9bc00cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/395396621 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc00cf660 0x7f9bc00cfac0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009870 tx=0x7f9bb402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 -- 172.21.15.99:0/395396621 shutdown_connections 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/395396621 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc01616e0 0x7f9bc0161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/395396621 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc00cf660 0x7f9bc00cfac0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 -- 172.21.15.99:0/395396621 >> 172.21.15.99:0/395396621 conn(0x7f9bc00b33a0 msgr2=0x7f9bc00b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 -- 172.21.15.99:0/395396621 shutdown_connections 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.403+0000 7f9bc6d89640 1 -- 172.21.15.99:0/395396621 wait complete. 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.404+0000 7f9bc6d89640 1 Processor -- start 2024-03-04T10:06:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.404+0000 7f9bc6d89640 1 -- start start 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc6d89640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc00cf660 0x7f9bc0150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc6d89640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc015da20 0x7f9bc00adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc6d89640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 0x7f9bc00aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc6d89640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc015de00 con 0x7f9bc01616e0 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc6d89640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00cbf80 con 0x7f9bc00cf660 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc6d89640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc01620b0 con 0x7f9bc015da20 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 0x7f9bc00aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 0x7f9bc00aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35584/0 (socket says 172.21.15.99:35584) 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 -- 172.21.15.99:0/2015878293 learned_addr learned my addr 172.21.15.99:0/2015878293 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc4b25640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc00cf660 0x7f9bc0150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bbffff640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc015da20 0x7f9bc00adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 -- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc015da20 msgr2=0x7f9bc00adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc015da20 0x7f9bc00adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 -- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc00cf660 msgr2=0x7f9bc0150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc00cf660 0x7f9bc0150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 -- 172.21.15.99:0/2015878293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc015ff50 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.405+0000 7f9bc5326640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 0x7f9bc00aea70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007e40 tx=0x7f9bb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.406+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8014620 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.406+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb8019070 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.406+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc0160230 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.406+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8014a20 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.406+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc01607f0 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.407+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc0074f60 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.407+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9bb8014bc0 con 0x7f9bc01616e0 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.407+0000 7f9bbd7fa640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.408+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9bb808ab00 con 0x7f9bc01616e0 2024-03-04T10:06:27.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.408+0000 7f9bc4b25640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.408+0000 7f9bc4b25640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bac06b4d0 0x7f9bac06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0001640 tx=0x7f9bb0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.412+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9bb8059a10 con 0x7f9bc01616e0 2024-03-04T10:06:27.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.582+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9bc015d070 con 0x7f9bc01616e0 2024-03-04T10:06:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.585+0000 7f9bbd7fa640 1 -- 172.21.15.99:0/2015878293 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9bb8059310 con 0x7f9bc01616e0 2024-03-04T10:06:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.587+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bac06b4d0 msgr2=0x7f9bac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.587+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bac06b4d0 0x7f9bac06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0001640 tx=0x7f9bb0009040 comp rx=0 tx=0).stop 2024-03-04T10:06:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.587+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 msgr2=0x7f9bc00aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.587+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 0x7f9bc00aea70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007e40 tx=0x7f9bb800e4b0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 shutdown_connections 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9bc01616e0 0x7f9bc00aea70 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9bc015da20 0x7f9bc00adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 --2- 172.21.15.99:0/2015878293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9bc00cf660 0x7f9bc0150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 >> 172.21.15.99:0/2015878293 conn(0x7f9bc00b33a0 msgr2=0x7f9bc006c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 shutdown_connections 2024-03-04T10:06:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.588+0000 7f9bc6d89640 1 -- 172.21.15.99:0/2015878293 wait complete. 2024-03-04T10:06:27.600 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-03-04T10:06:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.701+0000 7f2a15cea640 1 Processor -- start 2024-03-04T10:06:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.701+0000 7f2a15cea640 1 -- start start 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a15cea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a15cea640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2a1015f660 0x7f2a100d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a15cea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2a100d1cc0 0x7f2a100ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a15cea640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a100cf890 con 0x7f2a1015f660 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a15cea640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1015bf50 con 0x7f2a100d1cc0 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a15cea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1015fb60 con 0x7f2a1015bb70 2024-03-04T10:06:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2a1015f660 0x7f2a100d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2a1015f660 0x7f2a100d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35596/0 (socket says 172.21.15.99:35596) 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 -- 172.21.15.99:0/253109804 learned_addr learned my addr 172.21.15.99:0/253109804 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0ffff640 1 --2- 172.21.15.99:0/253109804 >> v2:172.21.15.162:3300/0 conn(0x7f2a100d1cc0 0x7f2a100ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0f7fe640 1 --2- 172.21.15.99:0/253109804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 -- 172.21.15.99:0/253109804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 msgr2=0x7f2a100b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 --2- 172.21.15.99:0/253109804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 -- 172.21.15.99:0/253109804 >> v2:172.21.15.162:3300/0 conn(0x7f2a100d1cc0 msgr2=0x7f2a100ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 --2- 172.21.15.99:0/253109804 >> v2:172.21.15.162:3300/0 conn(0x7f2a100d1cc0 0x7f2a100ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0effd640 1 -- 172.21.15.99:0/253109804 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1015fe40 con 0x7f2a1015f660 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.702+0000 7f2a0ffff640 1 --2- 172.21.15.99:0/253109804 >> v2:172.21.15.162:3300/0 conn(0x7f2a100d1cc0 0x7f2a100ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a0effd640 1 --2- 172.21.15.99:0/253109804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100d1780 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2a100b78d0 tx=0x7f2a0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c8a8a792513a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a0e7fc640 1 -- 172.21.15.99:0/253109804 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0402e070 con 0x7f2a1015f660 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a0e7fc640 1 -- 172.21.15.99:0/253109804 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a04037070 con 0x7f2a1015f660 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a0e7fc640 1 -- 172.21.15.99:0/253109804 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04031a00 con 0x7f2a1015f660 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 -- 172.21.15.99:0/253109804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 msgr2=0x7f2a100d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 --2- 172.21.15.99:0/253109804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100d1780 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2a100b78d0 tx=0x7f2a0402b880 comp rx=0 tx=0).stop 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 -- 172.21.15.99:0/253109804 shutdown_connections 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 --2- 172.21.15.99:0/253109804 >> v2:172.21.15.162:3300/0 conn(0x7f2a100d1cc0 0x7f2a100ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 --2- 172.21.15.99:0/253109804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100d1780 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 --2- 172.21.15.99:0/253109804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.703+0000 7f2a15cea640 1 -- 172.21.15.99:0/253109804 >> 172.21.15.99:0/253109804 conn(0x7f2a100b33a0 msgr2=0x7f2a100b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.704+0000 7f2a15cea640 1 -- 172.21.15.99:0/253109804 shutdown_connections 2024-03-04T10:06:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.704+0000 7f2a15cea640 1 -- 172.21.15.99:0/253109804 wait complete. 2024-03-04T10:06:27.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.705+0000 7f2a15cea640 1 Processor -- start 2024-03-04T10:06:27.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.705+0000 7f2a15cea640 1 -- start start 2024-03-04T10:06:27.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a100d1cc0 0x7f2a100c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100b0f80 con 0x7f2a1015f660 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a10160400 con 0x7f2a100d1cc0 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a100b1100 con 0x7f2a1015bb70 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35612/0 (socket says 172.21.15.99:35612) 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 -- 172.21.15.99:0/529432863 learned_addr learned my addr 172.21.15.99:0/529432863 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0effd640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 -- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 msgr2=0x7f2a100c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0f7fe640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a100d1cc0 0x7f2a100c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100c6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 -- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a100d1cc0 msgr2=0x7f2a100c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a100d1cc0 0x7f2a100c7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 -- 172.21.15.99:0/529432863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1015e940 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a0ffff640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100c68b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800cd10 tx=0x7f2a0800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800f680 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0800f840 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800fc30 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1015ebe0 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.706+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1014df00 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.707+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a10074f60 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.708+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2a080213f0 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.708+0000 7f29f7fff640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29fc06b4d0 0x7f29fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.708+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2a0808ade0 con 0x7f2a1015f660 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.709+0000 7f2a0f7fe640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29fc06b4d0 0x7f29fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.709+0000 7f2a0f7fe640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29fc06b4d0 0x7f29fc06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2a00006fd0 tx=0x7f2a00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:27.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.713+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2a08014030 con 0x7f2a1015f660 2024-03-04T10:06:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.882+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) v1 -- 0x7f2a1006e620 con 0x7f2a1015f660 2024-03-04T10:06:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.882+0000 7f29f7fff640 1 -- 172.21.15.99:0/529432863 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v17) v1 ==== 121+0+0 (secure 0 0 0) 0x7f2a080597c0 con 0x7f2a1015f660 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29fc06b4d0 msgr2=0x7f29fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29fc06b4d0 0x7f29fc06d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2a00006fd0 tx=0x7f2a00008040 comp rx=0 tx=0).stop 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 msgr2=0x7f2a100c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100c68b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800cd10 tx=0x7f2a0800a650 comp rx=0 tx=0).stop 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 shutdown_connections 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f29fc06b4d0 0x7f29fc06d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2a1015f660 0x7f2a100c68b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2a1015bb70 0x7f2a100c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 --2- 172.21.15.99:0/529432863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a100d1cc0 0x7f2a100c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 >> 172.21.15.99:0/529432863 conn(0x7f2a100b33a0 msgr2=0x7f2a100b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 shutdown_connections 2024-03-04T10:06:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.884+0000 7f2a15cea640 1 -- 172.21.15.99:0/529432863 wait complete. 2024-03-04T10:06:27.896 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing' 2024-03-04T10:06:27.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.996+0000 7fe3da30a640 1 Processor -- start 2024-03-04T10:06:27.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.996+0000 7fe3da30a640 1 -- start start 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3da30a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d40b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3da30a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3d40b47c0 0x7fe3d40b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3da30a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d40ccad0 con 0x7fe3d415b960 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3da30a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d40b5190 con 0x7fe3d40b47c0 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3da30a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40b10a0 con 0x7fe3d40b0700 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3d40b47c0 0x7fe3d40b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3d40b47c0 0x7fe3d40b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35622/0 (socket says 172.21.15.99:35622) 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 -- 172.21.15.99:0/3034764365 learned_addr learned my addr 172.21.15.99:0/3034764365 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 -- 172.21.15.99:0/3034764365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 msgr2=0x7fe3d40b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 --2- 172.21.15.99:0/3034764365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d40b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 -- 172.21.15.99:0/3034764365 >> v1:172.21.15.162:6789/0 conn(0x7fe3d415b960 legacy=0x7fe3d40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:28.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.997+0000 7fe3d88a7640 1 -- 172.21.15.99:0/3034764365 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40cfab0 con 0x7fe3d40b47c0 2024-03-04T10:06:28.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.998+0000 7fe3d88a7640 1 --2- 172.21.15.99:0/3034764365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40b47c0 0x7fe3d40b4c50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc009e50 tx=0x7fe3cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae17d4b704bfe1aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:28.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.998+0000 7fe3d2ffd640 1 -- 172.21.15.99:0/3034764365 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc02c9e0 con 0x7fe3d40b47c0 2024-03-04T10:06:28.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.998+0000 7fe3d2ffd640 1 -- 172.21.15.99:0/3034764365 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc02cb80 con 0x7fe3d40b47c0 2024-03-04T10:06:28.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.998+0000 7fe3d2ffd640 1 -- 172.21.15.99:0/3034764365 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc0343f0 con 0x7fe3d40b47c0 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 -- 172.21.15.99:0/3034764365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40b47c0 msgr2=0x7fe3d40b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 --2- 172.21.15.99:0/3034764365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40b47c0 0x7fe3d40b4c50 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc009e50 tx=0x7fe3cc02be40 comp rx=0 tx=0).stop 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 -- 172.21.15.99:0/3034764365 shutdown_connections 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 --2- 172.21.15.99:0/3034764365 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40b47c0 0x7fe3d40b4c50 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 --2- 172.21.15.99:0/3034764365 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d40b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 -- 172.21.15.99:0/3034764365 >> 172.21.15.99:0/3034764365 conn(0x7fe3d406f5f0 msgr2=0x7fe3d406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 -- 172.21.15.99:0/3034764365 shutdown_connections 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:27.999+0000 7fe3da30a640 1 -- 172.21.15.99:0/3034764365 wait complete. 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.000+0000 7fe3da30a640 1 Processor -- start 2024-03-04T10:06:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.000+0000 7fe3da30a640 1 -- start start 2024-03-04T10:06:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d40b47c0 0x7fe3d4156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 0x7fe3d41541b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40d0100 con 0x7fe3d415b960 2024-03-04T10:06:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d40cf860 con 0x7fe3d40b47c0 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d415bd40 con 0x7fe3d40b0700 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 0x7fe3d41541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 0x7fe3d41541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35634/0 (socket says 172.21.15.99:35634) 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 -- 172.21.15.99:0/2675286158 learned_addr learned my addr 172.21.15.99:0/2675286158 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d3fff640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d37fe640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d40b47c0 0x7fe3d4156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 -- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 msgr2=0x7fe3d41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 -- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d40b47c0 msgr2=0x7fe3d4156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d40b47c0 0x7fe3d4156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 -- 172.21.15.99:0/2675286158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d415e0d0 con 0x7fe3d415b960 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d88a7640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 0x7fe3d41541b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc036040 tx=0x7fe3cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc043020 con 0x7fe3d415b960 2024-03-04T10:06:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc03e450 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.001+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d415e3b0 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.002+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc009510 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.002+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d4153240 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.002+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d415e580 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.003+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe3cc046020 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.003+0000 7fe3d0ff9640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c006b4d0 0x7fe3c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.003+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe3cc0aca50 con 0x7fe3d415b960 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.004+0000 7fe3d3fff640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c006b4d0 0x7fe3c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.004+0000 7fe3d3fff640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c006b4d0 0x7fe3c006d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4007fd0 tx=0x7fe3c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:28.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.008+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3cc04d030 con 0x7fe3d415b960 2024-03-04T10:06:28.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.250+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]} v 0) v1 -- 0x7fe3d4074f60 con 0x7fe3d415b960 2024-03-04T10:06:28.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.979+0000 7fe3d0ff9640 1 -- 172.21.15.99:0/2675286158 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_existing"]}]=0 v171) v1 ==== 141+0+0 (secure 0 0 0) 0x7fe3cc07b9f0 con 0x7fe3d415b960 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c006b4d0 msgr2=0x7fe3c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c006b4d0 0x7fe3c006d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4007fd0 tx=0x7fe3c4009040 comp rx=0 tx=0).stop 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 msgr2=0x7fe3d41541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 0x7fe3d41541b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc036040 tx=0x7fe3cc004a40 comp rx=0 tx=0).stop 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 shutdown_connections 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3c006b4d0 0x7fe3c006d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d415b960 0x7fe3d41541b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d40b47c0 0x7fe3d4156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 --2- 172.21.15.99:0/2675286158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40b0700 0x7fe3d41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 >> 172.21.15.99:0/2675286158 conn(0x7fe3d406f5f0 msgr2=0x7fe3d40b2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 shutdown_connections 2024-03-04T10:06:28.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:28.982+0000 7fe3da30a640 1 -- 172.21.15.99:0/2675286158 wait complete. 2024-03-04T10:06:28.997 INFO:tasks.cephfs_test_runner:That the MDS won't abort when the dentry is already known to be damaged. ... ok 2024-03-04T10:06:28.997 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-03-04T10:06:28.997 INFO:tasks.cephfs_test_runner:test_dentry_first_precommit (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:06:28.998 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:29.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.102+0000 7f1516537640 1 Processor -- start 2024-03-04T10:06:29.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.102+0000 7f1516537640 1 -- start start 2024-03-04T10:06:29.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1516537640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1510158410 0x7f15100b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1516537640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f151006acd0 0x7f1510161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1516537640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1510161ca0 0x7f151015e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1516537640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15100b3f00 con 0x7f151006acd0 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1516537640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151006b500 con 0x7f1510161ca0 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1516537640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151006b7e0 con 0x7f1510158410 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f151006acd0 0x7f1510161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f151006acd0 0x7f1510161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35636/0 (socket says 172.21.15.99:35636) 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 -- 172.21.15.99:0/1322728161 learned_addr learned my addr 172.21.15.99:0/1322728161 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150ffff640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1510158410 0x7f15100b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1514ad4640 1 --2- 172.21.15.99:0/1322728161 >> v2:172.21.15.162:3300/0 conn(0x7f1510161ca0 0x7f151015e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 -- 172.21.15.99:0/1322728161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1510158410 msgr2=0x7f15100b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1510158410 0x7f15100b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 -- 172.21.15.99:0/1322728161 >> v2:172.21.15.162:3300/0 conn(0x7f1510161ca0 msgr2=0x7f151015e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 --2- 172.21.15.99:0/1322728161 >> v2:172.21.15.162:3300/0 conn(0x7f1510161ca0 0x7f151015e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150f7fe640 1 -- 172.21.15.99:0/1322728161 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151006bac0 con 0x7f151006acd0 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f1514ad4640 1 --2- 172.21.15.99:0/1322728161 >> v2:172.21.15.162:3300/0 conn(0x7f1510161ca0 0x7f151015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.103+0000 7f150ffff640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1510158410 0x7f15100b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.104+0000 7f150f7fe640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f151006acd0 0x7f1510161760 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f151006fc40 tx=0x7f150402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e11b3a6000ca2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.104+0000 7f150effd640 1 -- 172.21.15.99:0/1322728161 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504034070 con 0x7f151006acd0 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.104+0000 7f150effd640 1 -- 172.21.15.99:0/1322728161 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1504030860 con 0x7f151006acd0 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.104+0000 7f150effd640 1 -- 172.21.15.99:0/1322728161 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504030c30 con 0x7f151006acd0 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.104+0000 7f1516537640 1 -- 172.21.15.99:0/1322728161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f151006acd0 msgr2=0x7f1510161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.104+0000 7f1516537640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f151006acd0 0x7f1510161760 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f151006fc40 tx=0x7f150402b9a0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 -- 172.21.15.99:0/1322728161 shutdown_connections 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 --2- 172.21.15.99:0/1322728161 >> v2:172.21.15.162:3300/0 conn(0x7f1510161ca0 0x7f151015e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f151006acd0 0x7f1510161760 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 --2- 172.21.15.99:0/1322728161 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1510158410 0x7f15100b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 -- 172.21.15.99:0/1322728161 >> 172.21.15.99:0/1322728161 conn(0x7f15100cfef0 msgr2=0x7f15100d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 -- 172.21.15.99:0/1322728161 shutdown_connections 2024-03-04T10:06:29.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.105+0000 7f1516537640 1 -- 172.21.15.99:0/1322728161 wait complete. 2024-03-04T10:06:29.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.106+0000 7f1516537640 1 Processor -- start 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.106+0000 7f1516537640 1 -- start start 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f151006acd0 0x7f15100cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1510158410 0x7f1510154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 0x7f15100ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151006c5a0 con 0x7f1510161ca0 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151006c080 con 0x7f1510158410 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151006b130 con 0x7f151006acd0 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 0x7f15100ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 0x7f15100ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35646/0 (socket says 172.21.15.99:35646) 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 -- 172.21.15.99:0/1482567465 learned_addr learned my addr 172.21.15.99:0/1482567465 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f150ffff640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f151006acd0 0x7f15100cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f150f7fe640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1510158410 0x7f1510154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 -- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f151006acd0 msgr2=0x7f15100cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f151006acd0 0x7f15100cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 -- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1510158410 msgr2=0x7f1510154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1510158410 0x7f1510154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 -- 172.21.15.99:0/1482567465 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100ce230 con 0x7f1510161ca0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1514ad4640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 0x7f15100ceff0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f150800cd50 tx=0x7f150800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150800f680 con 0x7f1510161ca0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150800f840 con 0x7f1510161ca0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15100ce510 con 0x7f1510161ca0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150800fc30 con 0x7f1510161ca0 2024-03-04T10:06:29.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.107+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15100cead0 con 0x7f1510161ca0 2024-03-04T10:06:29.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.109+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510074f60 con 0x7f1510161ca0 2024-03-04T10:06:29.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.109+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f15080213f0 con 0x7f1510161ca0 2024-03-04T10:06:29.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.109+0000 7f150cff9640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14fc06b500 0x7f14fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.110+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f150808ac40 con 0x7f1510161ca0 2024-03-04T10:06:29.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.110+0000 7f150ffff640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14fc06b500 0x7f14fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.110+0000 7f150ffff640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14fc06b500 0x7f14fc06d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1510158b10 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.114+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1508014030 con 0x7f1510161ca0 2024-03-04T10:06:29.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.286+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1510066e10 con 0x7f1510161ca0 2024-03-04T10:06:29.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.286+0000 7f150cff9640 1 -- 172.21.15.99:0/1482567465 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13383 (secure 0 0 0) 0x7f1508059620 con 0x7f1510161ca0 2024-03-04T10:06:29.288 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:29.288 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":58,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:26.969671+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6833/628973925":"2024-03-05T10:06:23.675469+0000","172.21.15.99:6832/628973925":"2024-03-05T10:06:23.675469+0000","172.21.15.99:6833/370337902":"2024-03-05T10:06:17.747062+0000","172.21.15.99:6832/370337902":"2024-03-05T10:06:17.747062+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14fc06b500 msgr2=0x7f14fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14fc06b500 0x7f14fc06d9c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1510158b10 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 msgr2=0x7f15100ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 0x7f15100ceff0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f150800cd50 tx=0x7f150800a650 comp rx=0 tx=0).stop 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 shutdown_connections 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14fc06b500 0x7f14fc06d9c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1510161ca0 0x7f15100ceff0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1510158410 0x7f1510154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 --2- 172.21.15.99:0/1482567465 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f151006acd0 0x7f15100cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.288+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 >> 172.21.15.99:0/1482567465 conn(0x7f15100cfef0 msgr2=0x7f15100af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:29.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.289+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 shutdown_connections 2024-03-04T10:06:29.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.289+0000 7f1516537640 1 -- 172.21.15.99:0/1482567465 wait complete. 2024-03-04T10:06:29.301 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:06:29.301 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:06:29.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.403+0000 7fe3f850a640 1 Processor -- start 2024-03-04T10:06:29.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.403+0000 7fe3f850a640 1 -- start start 2024-03-04T10:06:29.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.403+0000 7fe3f850a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 0x7fe3f00cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.403+0000 7fe3f850a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.403+0000 7fe3f850a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f006ae00 con 0x7fe3f00cc900 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.403+0000 7fe3f850a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f00ccde0 con 0x7fe3f00cdd70 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f850a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00d0480 con 0x7fe3f00ccfe0 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f62a6640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 0x7fe3f00cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:34124/0 (socket says 172.21.15.99:34124) 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f62a6640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 0x7fe3f00cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:53124/0 (socket says 172.21.15.99:53124) 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 -- 172.21.15.99:0/623515343 learned_addr learned my addr 172.21.15.99:0/623515343 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 -- 172.21.15.99:0/623515343 >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 msgr2=0x7fe3f00cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 --2- 172.21.15.99:0/623515343 >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 0x7fe3f00cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 -- 172.21.15.99:0/623515343 >> v1:172.21.15.162:6789/0 conn(0x7fe3f00cc900 legacy=0x7fe3f015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f6aa7640 1 -- 172.21.15.99:0/623515343 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f006b0e0 con 0x7fe3f00ccfe0 2024-03-04T10:06:29.406 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.404+0000 7fe3f62a6640 1 --2- 172.21.15.99:0/623515343 >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 0x7fe3f00cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.405+0000 7fe3f6aa7640 1 --2- 172.21.15.99:0/623515343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00cff40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009e50 tx=0x7fe3e002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3a4f127b517a0f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.405+0000 7fe3f52a4640 1 -- 172.21.15.99:0/623515343 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e002ca70 con 0x7fe3f00ccfe0 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.405+0000 7fe3f52a4640 1 -- 172.21.15.99:0/623515343 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e002cc10 con 0x7fe3f00ccfe0 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.405+0000 7fe3f52a4640 1 -- 172.21.15.99:0/623515343 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e00343f0 con 0x7fe3f00ccfe0 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 -- 172.21.15.99:0/623515343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 msgr2=0x7fe3f00cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 --2- 172.21.15.99:0/623515343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00cff40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0009e50 tx=0x7fe3e002bed0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 -- 172.21.15.99:0/623515343 shutdown_connections 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 --2- 172.21.15.99:0/623515343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00cff40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 --2- 172.21.15.99:0/623515343 >> v2:172.21.15.162:3300/0 conn(0x7fe3f00cdd70 0x7fe3f00cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 -- 172.21.15.99:0/623515343 >> 172.21.15.99:0/623515343 conn(0x7fe3f00b22e0 msgr2=0x7fe3f00b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 -- 172.21.15.99:0/623515343 shutdown_connections 2024-03-04T10:06:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.406+0000 7fe3f850a640 1 -- 172.21.15.99:0/623515343 wait complete. 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.407+0000 7fe3f850a640 1 Processor -- start 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.407+0000 7fe3f850a640 1 -- start start 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 0x7fe3f00b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00cdd70 0x7fe3f00cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00b2e90 con 0x7fe3f00cc900 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f006b6a0 con 0x7fe3f00cdd70 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00b3010 con 0x7fe3f00ccfe0 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 0x7fe3f00b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 0x7fe3f00b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35652/0 (socket says 172.21.15.99:35652) 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 -- 172.21.15.99:0/3952687956 learned_addr learned my addr 172.21.15.99:0/3952687956 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:29.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f5aa5640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f6aa7640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00cdd70 0x7fe3f00cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 -- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 msgr2=0x7fe3f00c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 -- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00cdd70 msgr2=0x7fe3f00cb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00cdd70 0x7fe3f00cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 -- 172.21.15.99:0/3952687956 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f00c9780 con 0x7fe3f00cc900 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f62a6640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 0x7fe3f00b5a40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800ebe0 tx=0x7fe3e800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8017070 con 0x7fe3f00cc900 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e800cc30 con 0x7fe3f00cc900 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8013470 con 0x7fe3f00cc900 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.408+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f00c9a60 con 0x7fe3f00cc900 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.409+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f00ca020 con 0x7fe3f00cc900 2024-03-04T10:06:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.410+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f0074f60 con 0x7fe3f00cc900 2024-03-04T10:06:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.410+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe3e801a070 con 0x7fe3f00cc900 2024-03-04T10:06:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.410+0000 7fe3eeffd640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.410+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe3e808a2a0 con 0x7fe3f00cc900 2024-03-04T10:06:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.411+0000 7fe3f5aa5640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.411+0000 7fe3f5aa5640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00c6ca0 tx=0x7fe3e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.415+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3e8058c80 con 0x7fe3f00cc900 2024-03-04T10:06:29.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.584+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3f00ce480 con 0x7fe3f00cc900 2024-03-04T10:06:29.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.584+0000 7fe3eeffd640 1 -- 172.21.15.99:0/3952687956 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3e805cd20 con 0x7fe3f00cc900 2024-03-04T10:06:29.586 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3dc06b4d0 msgr2=0x7fe3dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe3f00c6ca0 tx=0x7fe3e4009040 comp rx=0 tx=0).stop 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 msgr2=0x7fe3f00b5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 0x7fe3f00b5a40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800ebe0 tx=0x7fe3e800c520 comp rx=0 tx=0).stop 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 shutdown_connections 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3dc06b4d0 0x7fe3dc06d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3f00cdd70 0x7fe3f00cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3f00ccfe0 0x7fe3f00c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 --2- 172.21.15.99:0/3952687956 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3f00cc900 0x7fe3f00b5a40 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 >> 172.21.15.99:0/3952687956 conn(0x7fe3f00b22e0 msgr2=0x7fe3f00b0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 shutdown_connections 2024-03-04T10:06:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.586+0000 7fe3f850a640 1 -- 172.21.15.99:0/3952687956 wait complete. 2024-03-04T10:06:29.599 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-03-04T10:06:29.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 Processor -- start 2024-03-04T10:06:29.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 -- start start 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f47700b39c0 0x7f4770161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4770161ca0 0x7f477006de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477006b500 con 0x7f4770158410 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47701620f0 con 0x7f4770161ca0 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.699+0000 7f4776aea640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477006b7e0 con 0x7f47700b39c0 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4775087640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4770161ca0 0x7f477006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4774886640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f4770158410 0x7f477006acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47792/0 (socket says 172.21.15.99:47792) 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4775087640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4770161ca0 0x7f477006de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35666/0 (socket says 172.21.15.99:35666) 2024-03-04T10:06:29.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4774886640 1 -- 172.21.15.99:0/2320189407 learned_addr learned my addr 172.21.15.99:0/2320189407 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f476ffff640 1 --2- 172.21.15.99:0/2320189407 >> v2:172.21.15.162:3300/0 conn(0x7f47700b39c0 0x7f4770161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4775087640 1 -- 172.21.15.99:0/2320189407 >> v2:172.21.15.162:3300/0 conn(0x7f47700b39c0 msgr2=0x7f4770161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4775087640 1 --2- 172.21.15.99:0/2320189407 >> v2:172.21.15.162:3300/0 conn(0x7f47700b39c0 0x7f4770161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4775087640 1 -- 172.21.15.99:0/2320189407 >> v1:172.21.15.99:6789/0 conn(0x7f4770158410 legacy=0x7f477006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f476f7fe640 1 -- 172.21.15.99:0/2320189407 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601728234 0 0) 0x7f477006b500 con 0x7f4770158410 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.700+0000 7f4775087640 1 -- 172.21.15.99:0/2320189407 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477006bac0 con 0x7f4770161ca0 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4775087640 1 --2- 172.21.15.99:0/2320189407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4770161ca0 0x7f477006de50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f477009de80 tx=0x7f476802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf0e1ad8d2075b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f476f7fe640 1 -- 172.21.15.99:0/2320189407 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768039070 con 0x7f4770161ca0 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f476f7fe640 1 -- 172.21.15.99:0/2320189407 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476802cd00 con 0x7f4770161ca0 2024-03-04T10:06:29.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f476f7fe640 1 -- 172.21.15.99:0/2320189407 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768034660 con 0x7f4770161ca0 2024-03-04T10:06:29.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 -- 172.21.15.99:0/2320189407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4770161ca0 msgr2=0x7f477006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 --2- 172.21.15.99:0/2320189407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4770161ca0 0x7f477006de50 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f477009de80 tx=0x7f476802bf50 comp rx=0 tx=0).stop 2024-03-04T10:06:29.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 -- 172.21.15.99:0/2320189407 shutdown_connections 2024-03-04T10:06:29.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 --2- 172.21.15.99:0/2320189407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4770161ca0 0x7f477006de50 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 --2- 172.21.15.99:0/2320189407 >> v2:172.21.15.162:3300/0 conn(0x7f47700b39c0 0x7f4770161760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 -- 172.21.15.99:0/2320189407 >> 172.21.15.99:0/2320189407 conn(0x7f47700cfef0 msgr2=0x7f47700d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 -- 172.21.15.99:0/2320189407 shutdown_connections 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.701+0000 7f4776aea640 1 -- 172.21.15.99:0/2320189407 wait complete. 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.702+0000 7f4776aea640 1 Processor -- start 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 -- start start 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 0x7f4770146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4770158410 0x7f47700cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4770161ca0 0x7f47700cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47700b3e50 con 0x7f47700b39c0 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477006c080 con 0x7f4770158410 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.703+0000 7f4776aea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477006c5a0 con 0x7f4770161ca0 2024-03-04T10:06:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 0x7f4770146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 0x7f4770146820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35674/0 (socket says 172.21.15.99:35674) 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 -- 172.21.15.99:0/1924373307 learned_addr learned my addr 172.21.15.99:0/1924373307 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4775087640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4770161ca0 0x7f47700cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f476ffff640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4770158410 0x7f47700cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 -- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4770161ca0 msgr2=0x7f47700cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4770161ca0 0x7f47700cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 -- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4770158410 msgr2=0x7f47700cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4770158410 0x7f47700cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 -- 172.21.15.99:0/1924373307 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4770152e40 con 0x7f47700b39c0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4774886640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 0x7f4770146820 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f476400c500 tx=0x7f476400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4764018070 con 0x7f47700b39c0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4764007580 con 0x7f47700b39c0 2024-03-04T10:06:29.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4770153120 con 0x7f47700b39c0 2024-03-04T10:06:29.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4764004a20 con 0x7f47700b39c0 2024-03-04T10:06:29.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.704+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47701536e0 con 0x7f47700b39c0 2024-03-04T10:06:29.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.705+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4770074f60 con 0x7f47700b39c0 2024-03-04T10:06:29.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.706+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f476401b070 con 0x7f47700b39c0 2024-03-04T10:06:29.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.706+0000 7f476d7fa640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f475406b4d0 0x7f475406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:29.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.706+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4764089380 con 0x7f47700b39c0 2024-03-04T10:06:29.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.706+0000 7f476ffff640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f475406b4d0 0x7f475406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:29.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.707+0000 7f476ffff640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f475406b4d0 0x7f475406d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f47701f6b70 tx=0x7f475c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:29.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.710+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f47640583b0 con 0x7f47700b39c0 2024-03-04T10:06:29.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.944+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) v1 -- 0x7f477006fa20 con 0x7f47700b39c0 2024-03-04T10:06:29.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.986+0000 7f476d7fa640 1 -- 172.21.15.99:0/1924373307 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v172) v1 ==== 145+0+0 (secure 0 0 0) 0x7f4764057cb0 con 0x7f47700b39c0 2024-03-04T10:06:30.024 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f475406b4d0 msgr2=0x7f475406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f475406b4d0 0x7f475406d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f47701f6b70 tx=0x7f475c008040 comp rx=0 tx=0).stop 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 msgr2=0x7f4770146820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 0x7f4770146820 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f476400c500 tx=0x7f476400c9c0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 shutdown_connections 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f475406b4d0 0x7f475406d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4770161ca0 0x7f47700cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4770158410 0x7f47700cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 --2- 172.21.15.99:0/1924373307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47700b39c0 0x7f4770146820 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 >> 172.21.15.99:0/1924373307 conn(0x7f47700cfef0 msgr2=0x7f4770157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 shutdown_connections 2024-03-04T10:06:30.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:29.988+0000 7f4776aea640 1 -- 172.21.15.99:0/1924373307 wait complete. 2024-03-04T10:06:30.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T10:06:30.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.131+0000 7fafcc861640 1 Processor -- start 2024-03-04T10:06:30.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.131+0000 7fafcc861640 1 -- start start 2024-03-04T10:06:30.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafcc861640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc8161390 0x7fafc806c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafcc861640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafc80cedf0 0x7fafc80cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafcc861640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc80cf740 con 0x7fafc80d20c0 2024-03-04T10:06:30.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafcc861640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc806c5a0 con 0x7fafc80cedf0 2024-03-04T10:06:30.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafcc861640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc806d3e0 con 0x7fafc8161390 2024-03-04T10:06:30.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafc6d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafc80cedf0 0x7fafc80cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafc6d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafc80cedf0 0x7fafc80cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35684/0 (socket says 172.21.15.99:35684) 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafc6d9d640 1 -- 172.21.15.99:0/4527111 learned_addr learned my addr 172.21.15.99:0/4527111 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.132+0000 7fafc659c640 1 --2- 172.21.15.99:0/4527111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc8161390 0x7fafc806c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.133+0000 7fafc6d9d640 1 -- 172.21.15.99:0/4527111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc8161390 msgr2=0x7fafc806c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.133+0000 7fafc6d9d640 1 --2- 172.21.15.99:0/4527111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc8161390 0x7fafc806c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.133+0000 7fafc6d9d640 1 -- 172.21.15.99:0/4527111 >> v1:172.21.15.162:6789/0 conn(0x7fafc80d20c0 legacy=0x7fafc80ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.133+0000 7fafc6d9d640 1 -- 172.21.15.99:0/4527111 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc806c7d0 con 0x7fafc80cedf0 2024-03-04T10:06:30.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.133+0000 7fafc6d9d640 1 --2- 172.21.15.99:0/4527111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc80cc0d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009870 tx=0x7fafc002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abeeff757fd1b604 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:30.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.133+0000 7fafc559a640 1 -- 172.21.15.99:0/4527111 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0034070 con 0x7fafc80cedf0 2024-03-04T10:06:30.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafc559a640 1 -- 172.21.15.99:0/4527111 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0030840 con 0x7fafc80cedf0 2024-03-04T10:06:30.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafc559a640 1 -- 172.21.15.99:0/4527111 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0030c10 con 0x7fafc80cedf0 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 -- 172.21.15.99:0/4527111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 msgr2=0x7fafc80cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 --2- 172.21.15.99:0/4527111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc80cc0d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fafc0009870 tx=0x7fafc002b980 comp rx=0 tx=0).stop 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 -- 172.21.15.99:0/4527111 shutdown_connections 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 --2- 172.21.15.99:0/4527111 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc80cc0d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 --2- 172.21.15.99:0/4527111 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc8161390 0x7fafc806c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 -- 172.21.15.99:0/4527111 >> 172.21.15.99:0/4527111 conn(0x7fafc8066dc0 msgr2=0x7fafc80671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 -- 172.21.15.99:0/4527111 shutdown_connections 2024-03-04T10:06:30.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.134+0000 7fafcc861640 1 -- 172.21.15.99:0/4527111 wait complete. 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.135+0000 7fafcc861640 1 Processor -- start 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 -- start start 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc815c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc80d20c0 0x7fafc81e8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafc8161390 0x7fafc81e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc806d070 con 0x7fafc80cedf0 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc806cd90 con 0x7fafc8161390 2024-03-04T10:06:30.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafcc861640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8161770 con 0x7fafc80d20c0 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafc659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc815c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafc659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc815c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35692/0 (socket says 172.21.15.99:35692) 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafc659c640 1 -- 172.21.15.99:0/1357501179 learned_addr learned my addr 172.21.15.99:0/1357501179 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafc6d9d640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafc8161390 0x7fafc81e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.136+0000 7fafc5d9b640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc80d20c0 0x7fafc81e8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc659c640 1 -- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc80d20c0 msgr2=0x7fafc81e8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc659c640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc80d20c0 0x7fafc81e8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc659c640 1 -- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafc8161390 msgr2=0x7fafc81e8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc659c640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafc8161390 0x7fafc81e8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc659c640 1 -- 172.21.15.99:0/1357501179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80b09c0 con 0x7fafc80cedf0 2024-03-04T10:06:30.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc5d9b640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc80d20c0 0x7fafc81e8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc6d9d640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafc8161390 0x7fafc81e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafc659c640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc815c940 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fafb8007e60 tx=0x7fafb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8014620 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb8019070 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc81e77a0 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8014a20 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.137+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc81e68b0 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.138+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc80ce090 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.139+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fafb8014bc0 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.139+0000 7fafb6ffd640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fafa806b4d0 0x7fafa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.139+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fafb808a9d0 con 0x7fafc80cedf0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.139+0000 7fafc5d9b640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fafa806b4d0 0x7fafa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.140+0000 7fafc5d9b640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fafa806b4d0 0x7fafa806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fafc81f7880 tx=0x7fafbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:30.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.143+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fafc80ce090 con 0x7fafc80cedf0 2024-03-04T10:06:30.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.313+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fafc8074f60 con 0x7fafc80cedf0 2024-03-04T10:06:30.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.318+0000 7fafb6ffd640 1 -- 172.21.15.99:0/1357501179 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fafb8059970 con 0x7fafc80cedf0 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fafa806b4d0 msgr2=0x7fafa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fafa806b4d0 0x7fafa806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fafc81f7880 tx=0x7fafbc009040 comp rx=0 tx=0).stop 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 msgr2=0x7fafc815c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc815c940 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fafb8007e60 tx=0x7fafb800e4b0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 shutdown_connections 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fafa806b4d0 0x7fafa806d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafc8161390 0x7fafc81e8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafc80d20c0 0x7fafc81e8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 --2- 172.21.15.99:0/1357501179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafc80cedf0 0x7fafc815c940 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 >> 172.21.15.99:0/1357501179 conn(0x7fafc8066dc0 msgr2=0x7fafc80cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:30.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 shutdown_connections 2024-03-04T10:06:30.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.320+0000 7fafcc861640 1 -- 172.21.15.99:0/1357501179 wait complete. 2024-03-04T10:06:30.332 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:06:30.332 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:06:30.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:06:30.424 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:06:30.424 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:30.424 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:30.441 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:06:30.441 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:06:30.441 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:30.461 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:06:30.461 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:30.461 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:30.497 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:06:30.497 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:06:30.497 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:30.519 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:06:30.519 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T10:06:30.519 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:30.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.637+0000 7f35bfbd1640 1 Processor -- start 2024-03-04T10:06:30.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.637+0000 7f35bfbd1640 1 -- start start 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bfbd1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f35b8158410 0x7f35b80b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bfbd1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b806b500 con 0x7f35b806acd0 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bfbd1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b806b7e0 con 0x7f35b8161e70 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bfbd1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b80b3f00 con 0x7f35b8158410 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bd96d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f35b8158410 0x7f35b80b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bd16c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f35b806acd0 0x7f35b8161760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47798/0 (socket says 172.21.15.99:47798) 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bd16c640 1 -- 172.21.15.99:0/3461159532 learned_addr learned my addr 172.21.15.99:0/3461159532 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bd96d640 1 -- 172.21.15.99:0/3461159532 >> v1:172.21.15.162:6789/0 conn(0x7f35b8161e70 legacy=0x7f35b806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bd96d640 1 -- 172.21.15.99:0/3461159532 >> v1:172.21.15.99:6789/0 conn(0x7f35b806acd0 legacy=0x7f35b8161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bc96b640 1 -- 172.21.15.99:0/3461159532 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874386936 0 0) 0x7f35b806b500 con 0x7f35b806acd0 2024-03-04T10:06:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.638+0000 7f35bd96d640 1 -- 172.21.15.99:0/3461159532 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b806bac0 con 0x7f35b8158410 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.639+0000 7f35bd96d640 1 --2- 172.21.15.99:0/3461159532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80b39c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f35b40097a0 tx=0x7f35b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1ae5fd19799e6c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.639+0000 7f35bc96b640 1 -- 172.21.15.99:0/3461159532 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4034070 con 0x7f35b8158410 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.639+0000 7f35bc96b640 1 -- 172.21.15.99:0/3461159532 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b4030610 con 0x7f35b8158410 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.639+0000 7f35bc96b640 1 -- 172.21.15.99:0/3461159532 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b40309e0 con 0x7f35b8158410 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 -- 172.21.15.99:0/3461159532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 msgr2=0x7f35b80b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/3461159532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80b39c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f35b40097a0 tx=0x7f35b402b960 comp rx=0 tx=0).stop 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 -- 172.21.15.99:0/3461159532 shutdown_connections 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/3461159532 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80b39c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 -- 172.21.15.99:0/3461159532 >> 172.21.15.99:0/3461159532 conn(0x7f35b80cfef0 msgr2=0x7f35b80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 -- 172.21.15.99:0/3461159532 shutdown_connections 2024-03-04T10:06:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.640+0000 7f35bfbd1640 1 -- 172.21.15.99:0/3461159532 wait complete. 2024-03-04T10:06:30.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.641+0000 7f35bfbd1640 1 Processor -- start 2024-03-04T10:06:30.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.641+0000 7f35bfbd1640 1 -- start start 2024-03-04T10:06:30.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bfbd1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35b806acd0 0x7f35b81587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bfbd1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bfbd1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35b8161e70 0x7f35b80ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bfbd1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b806c5a0 con 0x7f35b8158410 2024-03-04T10:06:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bfbd1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b806c080 con 0x7f35b8161e70 2024-03-04T10:06:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bfbd1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b806b160 con 0x7f35b806acd0 2024-03-04T10:06:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35704/0 (socket says 172.21.15.99:35704) 2024-03-04T10:06:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 -- 172.21.15.99:0/2267813949 learned_addr learned my addr 172.21.15.99:0/2267813949 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd96d640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35b806acd0 0x7f35b81587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35be16e640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35b8161e70 0x7f35b80ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 -- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35b806acd0 msgr2=0x7f35b81587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35b806acd0 0x7f35b81587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 -- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35b8161e70 msgr2=0x7f35b80ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35b8161e70 0x7f35b80ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd16c640 1 -- 172.21.15.99:0/2267813949 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b80cdbc0 con 0x7f35b8158410 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.642+0000 7f35bd96d640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35b806acd0 0x7f35b81587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.643+0000 7f35bd16c640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80cd680 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f35a800c4d0 tx=0x7f35a800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.643+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8018070 con 0x7f35b8158410 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.643+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a8007500 con 0x7f35b8158410 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.643+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b80cf420 con 0x7f35b8158410 2024-03-04T10:06:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.643+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a8004a20 con 0x7f35b8158410 2024-03-04T10:06:30.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.643+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b80ce400 con 0x7f35b8158410 2024-03-04T10:06:30.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.644+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b8074f60 con 0x7f35b8158410 2024-03-04T10:06:30.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.644+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f35a801b070 con 0x7f35b8158410 2024-03-04T10:06:30.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.644+0000 7f35ae7fc640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:30.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.645+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f35a80893c0 con 0x7f35b8158410 2024-03-04T10:06:30.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.645+0000 7f35bd96d640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:30.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.645+0000 7f35bd96d640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f359c06b4d0 0x7f359c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f35b4036000 tx=0x7f35b4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:30.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.649+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f35b80cdbc0 con 0x7f35b8158410 2024-03-04T10:06:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.855+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35b8066e10 con 0x7f35b8158410 2024-03-04T10:06:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.855+0000 7f35ae7fc640 1 -- 172.21.15.99:0/2267813949 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f35a8057c80 con 0x7f35b8158410 2024-03-04T10:06:30.857 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:30.857 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/3902881640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3902881640},{"type":"v1","addr":"172.21.15.99:6835","nonce":3902881640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5482,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1162278829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1162278829},{"type":"v1","addr":"172.21.15.162:6836","nonce":1162278829}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5485,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/456531005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":456531005},{"type":"v1","addr":"172.21.15.162:6837","nonce":456531005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":6387,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/3757026201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3757026201},{"type":"v1","addr":"172.21.15.99:6833","nonce":3757026201}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[]} 2024-03-04T10:06:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 38 2024-03-04T10:06:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f359c06b4d0 msgr2=0x7f359c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f359c06b4d0 0x7f359c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f35b4036000 tx=0x7f35b4008030 comp rx=0 tx=0).stop 2024-03-04T10:06:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 msgr2=0x7f35b80cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80cd680 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f35a800c4d0 tx=0x7f35a800c990 comp rx=0 tx=0).stop 2024-03-04T10:06:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 shutdown_connections 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f359c06b4d0 0x7f359c06d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35b8161e70 0x7f35b80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35b8158410 0x7f35b80cd680 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 --2- 172.21.15.99:0/2267813949 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35b806acd0 0x7f35b81587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.857+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 >> 172.21.15.99:0/2267813949 conn(0x7f35b80cfef0 msgr2=0x7f35b80b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.858+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 shutdown_connections 2024-03-04T10:06:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:30.858+0000 7f35bfbd1640 1 -- 172.21.15.99:0/2267813949 wait complete. 2024-03-04T10:06:30.870 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:06:30.870 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T10:06:30.870 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:06:30.870 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:06:30.870 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:06:30.870 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T10:06:30.871 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:06:30.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:06:30.871 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T10:06:30.871 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T10:06:30.871 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T10:06:30.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:06:30.871 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T10:06:30.872 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T10:06:30.872 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T10:06:30.872 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:06:30.880 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:06:30.880 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:06:30.881 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:06:30.881 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:06:30.883 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T10:06:30.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T10:06:30.884 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T10:06:30.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T10:06:30.922 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:06:30.923 DEBUG:teuthology.parallel:result is None 2024-03-04T10:06:30.923 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:06:30.923 DEBUG:teuthology.parallel:result is None 2024-03-04T10:06:30.926 INFO:tasks.ceph.mds.b:Started 2024-03-04T10:06:30.926 DEBUG:teuthology.parallel:result is None 2024-03-04T10:06:30.926 INFO:tasks.ceph.mds.d:Started 2024-03-04T10:06:30.927 DEBUG:teuthology.parallel:result is None 2024-03-04T10:06:30.927 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T10:06:30.998 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:06:30.999 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:06:31.008 INFO:tasks.ceph.mds.b.smithi162.stdout:starting mds.b at 2024-03-04T10:06:31.009 INFO:tasks.ceph.mds.d.smithi162.stdout:starting mds.d at 2024-03-04T10:06:31.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.031+0000 7f3349d69640 1 Processor -- start 2024-03-04T10:06:31.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.031+0000 7f3349d69640 1 -- start start 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3349d69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 0x7f33440b7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3349d69640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f334415c2f0 0x7f33440cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3349d69640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334415f8c0 con 0x7f334415be90 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3349d69640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334415c7f0 con 0x7f334415c2f0 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3349d69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334415cad0 con 0x7f334415bab0 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3342ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f334415be90 0x7f33440cf790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47808/0 (socket says 172.21.15.99:47808) 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3342ffd640 1 -- 172.21.15.99:0/595018933 learned_addr learned my addr 172.21.15.99:0/595018933 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480415639 0 0) 0x7f334415f8c0 con 0x7f334415be90 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f3343fff640 1 --2- 172.21.15.99:0/595018933 >> v2:172.21.15.162:3300/0 conn(0x7f334415c2f0 0x7f33440cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c003520 con 0x7f334415be90 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f33437fe640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 0x7f33440b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.032+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4070036980 0 0) 0x7f332c003520 con 0x7f334415be90 2024-03-04T10:06:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 msgr2=0x7f33440b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 0x7f33440b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 >> v2:172.21.15.162:3300/0 conn(0x7f334415c2f0 msgr2=0x7f33440cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 --2- 172.21.15.99:0/595018933 >> v2:172.21.15.162:3300/0 conn(0x7f334415c2f0 0x7f33440cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334415cdb0 con 0x7f334415be90 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3338001670 con 0x7f334415be90 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 >> v1:172.21.15.99:6789/0 conn(0x7f334415be90 legacy=0x7f33440cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c0039b0 0x7f332c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f332c004800 0x7f332c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 0x7f332c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334415c7f0 con 0x7f332c004800 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334415cad0 con 0x7f332c0039b0 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00ba70 con 0x7f334415bab0 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3342ffd640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f332c004800 0x7f332c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f33437fe640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c0039b0 0x7f332c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3343fff640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 0x7f332c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3342ffd640 1 -- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 msgr2=0x7f332c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3342ffd640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f334415bab0 0x7f332c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3342ffd640 1 -- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c0039b0 msgr2=0x7f332c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3342ffd640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c0039b0 0x7f332c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.033+0000 7f3342ffd640 1 -- 172.21.15.99:0/595018933 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c003520 con 0x7f332c004800 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3342ffd640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f332c004800 0x7f332c004c80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3338001050 tx=0x7f332c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8079500fb70b624e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338007d50 con 0x7f332c004800 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3338030070 con 0x7f332c004800 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f33427fc640 1 -- 172.21.15.99:0/595018933 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338005d10 con 0x7f332c004800 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3349d69640 1 -- 172.21.15.99:0/595018933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f332c004800 msgr2=0x7f332c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3349d69640 1 --2- 172.21.15.99:0/595018933 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f332c004800 0x7f332c004c80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3338001050 tx=0x7f332c005830 comp rx=0 tx=0).stop 2024-03-04T10:06:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3343fff640 1 -- 172.21.15.99:0/595018933 reap_dead start 2024-03-04T10:06:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3349d69640 1 -- 172.21.15.99:0/595018933 shutdown_connections 2024-03-04T10:06:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3349d69640 1 -- 172.21.15.99:0/595018933 >> 172.21.15.99:0/595018933 conn(0x7f334406f6e0 msgr2=0x7f334406faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3349d69640 1 -- 172.21.15.99:0/595018933 shutdown_connections 2024-03-04T10:06:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.034+0000 7f3349d69640 1 -- 172.21.15.99:0/595018933 wait complete. 2024-03-04T10:06:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.035+0000 7f3349d69640 1 Processor -- start 2024-03-04T10:06:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 -- start start 2024-03-04T10:06:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c004800 0x7f3344154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 0x7f33440c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33440caa30 0x7f33440caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440cc6f0 con 0x7f33440c44a0 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334415d090 con 0x7f332c004800 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3349d69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33440cc870 con 0x7f33440caa30 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.036+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 0x7f33440c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 0x7f33440c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35762/0 (socket says 172.21.15.99:35762) 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 -- 172.21.15.99:0/1725681954 learned_addr learned my addr 172.21.15.99:0/1725681954 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f33437fe640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c004800 0x7f3344154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3343fff640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33440caa30 0x7f33440caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 -- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33440caa30 msgr2=0x7f33440caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33440caa30 0x7f33440caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 -- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c004800 msgr2=0x7f3344154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c004800 0x7f3344154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 -- 172.21.15.99:0/1725681954 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344150570 con 0x7f33440c44a0 2024-03-04T10:06:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3343fff640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33440caa30 0x7f33440caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3342ffd640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 0x7f33440c7f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f333800c2d0 tx=0x7f334415cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338001cb0 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3338004bf0 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3344154970 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338007070 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.037+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3344154f30 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.038+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334406b130 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.038+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3338007210 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.039+0000 7f332bfff640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f333006b4d0 0x7f333006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.039+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f33380ab050 con 0x7f33440c44a0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.039+0000 7f33437fe640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f333006b4d0 0x7f333006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.039+0000 7f33437fe640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f333006b4d0 0x7f333006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3334006fd0 tx=0x7f3334008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.043+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f334406b130 con 0x7f33440c44a0 2024-03-04T10:06:31.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.224+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3344074f60 con 0x7f33440c44a0 2024-03-04T10:06:31.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:31.998+0000 7f332bfff640 1 -- 172.21.15.99:0/1725681954 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v60) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3338079ff0 con 0x7f33440c44a0 2024-03-04T10:06:32.000 INFO:teuthology.orchestra.run.smithi099.stderr: removed all blocklist entries 2024-03-04T10:06:32.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f333006b4d0 msgr2=0x7f333006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f333006b4d0 0x7f333006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3334006fd0 tx=0x7f3334008040 comp rx=0 tx=0).stop 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 msgr2=0x7f33440c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 0x7f33440c7f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f333800c2d0 tx=0x7f334415cf80 comp rx=0 tx=0).stop 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 shutdown_connections 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f333006b4d0 0x7f333006d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33440caa30 0x7f33440caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33440c44a0 0x7f33440c7f20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 --2- 172.21.15.99:0/1725681954 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f332c004800 0x7f3344154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 >> 172.21.15.99:0/1725681954 conn(0x7f334406f6e0 msgr2=0x7f334415d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 shutdown_connections 2024-03-04T10:06:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.002+0000 7f3349d69640 1 -- 172.21.15.99:0/1725681954 wait complete. 2024-03-04T10:06:32.018 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T10:06:32.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.122+0000 7f359cc9a640 1 Processor -- start 2024-03-04T10:06:32.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.122+0000 7f359cc9a640 1 -- start start 2024-03-04T10:06:32.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.123+0000 7f359cc9a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f35980b2e60 0x7f35980b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.123+0000 7f359cc9a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359815e1e0 con 0x7f35980b5470 2024-03-04T10:06:32.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.123+0000 7f359cc9a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35980d1c10 con 0x7f359815de00 2024-03-04T10:06:32.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.123+0000 7f359cc9a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35980b98c0 con 0x7f35980b2e60 2024-03-04T10:06:32.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.123+0000 7f3596d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f35980b5470 0x7f35980d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47816/0 (socket says 172.21.15.99:47816) 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.123+0000 7f3596d9d640 1 -- 172.21.15.99:0/3160601502 learned_addr learned my addr 172.21.15.99:0/3160601502 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 --2- 172.21.15.99:0/3160601502 >> v2:172.21.15.162:3300/0 conn(0x7f35980b2e60 0x7f35980b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159530726 0 0) 0x7f359815e1e0 con 0x7f35980b5470 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580003520 con 0x7f35980b5470 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366337844 0 0) 0x7f3580003520 con 0x7f35980b5470 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 >> v2:172.21.15.162:3300/0 conn(0x7f35980b2e60 msgr2=0x7f35980b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 --2- 172.21.15.99:0/3160601502 >> v2:172.21.15.162:3300/0 conn(0x7f35980b2e60 0x7f35980b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 >> v1:172.21.15.162:6789/0 conn(0x7f359815de00 legacy=0x7f35980b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980d1ef0 con 0x7f35980b5470 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3590001670 con 0x7f35980b5470 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 >> v1:172.21.15.99:6789/0 conn(0x7f35980b5470 legacy=0x7f35980d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35800039b0 0x7f3580003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3580004800 0x7f3580004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35800051c0 0x7f358000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980b98c0 con 0x7f3580004800 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358000c110 con 0x7f35800039b0 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358000c310 con 0x7f35800051c0 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3580004800 0x7f3580004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f359659c640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35800039b0 0x7f3580003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3596d9d640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35800051c0 0x7f358000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 -- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35800051c0 msgr2=0x7f358000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35800051c0 0x7f358000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 -- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35800039b0 msgr2=0x7f3580003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f35800039b0 0x7f3580003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3595d9b640 1 -- 172.21.15.99:0/3160601502 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580003520 con 0x7f3580004800 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.124+0000 7f3596d9d640 1 -- 172.21.15.99:0/3160601502 reap_dead start 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.125+0000 7f3595d9b640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3580004800 0x7f3580004c80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f358c00ba30 tx=0x7f3580005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db9ad8d9b7d3bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.125+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c036020 con 0x7f3580004800 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.125+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c032640 con 0x7f3580004800 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.125+0000 7f359559a640 1 -- 172.21.15.99:0/3160601502 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c00b600 con 0x7f3580004800 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 -- 172.21.15.99:0/3160601502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3580004800 msgr2=0x7f3580004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3580004800 0x7f3580004c80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f358c00ba30 tx=0x7f3580005830 comp rx=0 tx=0).stop 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 -- 172.21.15.99:0/3160601502 shutdown_connections 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 --2- 172.21.15.99:0/3160601502 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3580004800 0x7f3580004c80 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 -- 172.21.15.99:0/3160601502 >> 172.21.15.99:0/3160601502 conn(0x7f35980d0470 msgr2=0x7f35980d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 -- 172.21.15.99:0/3160601502 shutdown_connections 2024-03-04T10:06:32.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.126+0000 7f359cc9a640 1 -- 172.21.15.99:0/3160601502 wait complete. 2024-03-04T10:06:32.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.127+0000 7f359cc9a640 1 Processor -- start 2024-03-04T10:06:32.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.127+0000 7f359cc9a640 1 -- start start 2024-03-04T10:06:32.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f359cc9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3580004800 0x7f3598154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f359cc9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35980caac0 0x7f35981604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f359cc9a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 0x7f3598160a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f359cc9a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980d2820 con 0x7f35980caf40 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f359cc9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35980d24b0 con 0x7f3580004800 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f359cc9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35980b9380 con 0x7f35980caac0 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 0x7f3598160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 0x7f3598160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35778/0 (socket says 172.21.15.99:35778) 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 -- 172.21.15.99:0/2108828919 learned_addr learned my addr 172.21.15.99:0/2108828919 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 -- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35980caac0 msgr2=0x7f35981604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35980caac0 0x7f35981604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 -- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3580004800 msgr2=0x7f3598154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3580004800 0x7f3598154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.128+0000 7f3596d9d640 1 -- 172.21.15.99:0/2108828919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35980cb4a0 con 0x7f35980caf40 2024-03-04T10:06:32.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.129+0000 7f3596d9d640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 0x7f3598160a10 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f35980ce210 tx=0x7f359000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.129+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590010510 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.129+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35900106f0 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.129+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35980cb780 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.129+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590010ae0 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.129+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35980cbd40 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.130+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3598074f60 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.130+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3590010c80 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.130+0000 7f3586ffd640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f357806b4d0 0x7f357806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.131+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3590089670 con 0x7f35980caf40 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.131+0000 7f359659c640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f357806b4d0 0x7f357806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.131+0000 7f359659c640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f357806b4d0 0x7f357806d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3588000f80 tx=0x7f3588009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.135+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3590058740 con 0x7f35980caf40 2024-03-04T10:06:32.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.372+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f359815c6b0 con 0x7f35980caf40 2024-03-04T10:06:32.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.373+0000 7f3586ffd640 1 -- 172.21.15.99:0/2108828919 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3590058040 con 0x7f35980caf40 2024-03-04T10:06:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:32.374 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_dump": [ 2024-03-04T10:06:32.374 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.a", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg==", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.b", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw==", 2024-03-04T10:06:32.375 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.c", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg==", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:06:32.376 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.d", 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw==", 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.377 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.0", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg==", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.1", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ==", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.378 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.2", 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw==", 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.3", 2024-03-04T10:06:32.379 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA==", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.4", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw==", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.380 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.5", 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg==", 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.6", 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg==", 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.381 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.7", 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA==", 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.382 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.0", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ==", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow r", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.1", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA==", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:06:32.383 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.2", 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg==", 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.384 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.3", 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ==", 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.385 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.admin", 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBAnOVlV1MhNhAAJTJsIc9qGmeQBnHKjrkPvw==", 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.386 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.x", 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw==", 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.y", 2024-03-04T10:06:32.387 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ==", 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-03-04T10:06:32.388 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f357806b4d0 msgr2=0x7f357806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f357806b4d0 0x7f357806d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f3588000f80 tx=0x7f3588009040 comp rx=0 tx=0).stop 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 msgr2=0x7f3598160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 0x7f3598160a10 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f35980ce210 tx=0x7f359000e910 comp rx=0 tx=0).stop 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 shutdown_connections 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f357806b4d0 0x7f357806d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f35980caf40 0x7f3598160a10 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f35980caac0 0x7f35981604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 --2- 172.21.15.99:0/2108828919 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3580004800 0x7f3598154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 >> 172.21.15.99:0/2108828919 conn(0x7f35980d0470 msgr2=0x7f35980b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 shutdown_connections 2024-03-04T10:06:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.375+0000 7f359cc9a640 1 -- 172.21.15.99:0/2108828919 wait complete. 2024-03-04T10:06:32.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.491+0000 7f12133d4640 1 Processor -- start 2024-03-04T10:06:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.491+0000 7f12133d4640 1 -- start start 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f12133d4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f120c15b030 0x7f120c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f12133d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f120c15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f12133d4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c15e2d0 con 0x7f120c15b410 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f12133d4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120c0b2720 con 0x7f120c15b030 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f12133d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c06d8b0 con 0x7f120c06d3b0 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1211971640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f120c15b410 0x7f120c15b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47818/0 (socket says 172.21.15.99:47818) 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1211971640 1 -- 172.21.15.99:0/2908942965 learned_addr learned my addr 172.21.15.99:0/2908942965 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793364969 0 0) 0x7f120c15e2d0 con 0x7f120c15b410 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f8003520 con 0x7f120c15b410 2024-03-04T10:06:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f121096f640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f120c15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1211170640 1 --2- 172.21.15.99:0/2908942965 >> v2:172.21.15.162:3300/0 conn(0x7f120c15b030 0x7f120c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184018858 0 0) 0x7f11f8003520 con 0x7f120c15b410 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 msgr2=0x7f120c15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f120c15dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 >> v2:172.21.15.162:3300/0 conn(0x7f120c15b030 msgr2=0x7f120c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 --2- 172.21.15.99:0/2908942965 >> v2:172.21.15.162:3300/0 conn(0x7f120c15b030 0x7f120c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c06db90 con 0x7f120c15b410 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f12080019f0 con 0x7f120c15b410 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 >> v1:172.21.15.99:6789/0 conn(0x7f120c15b410 legacy=0x7f120c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f80039b0 0x7f11f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11f8004800 0x7f11f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f11f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0b2720 con 0x7f11f8004800 2024-03-04T10:06:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c06d8b0 con 0x7f11f80039b0 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f800b9c0 con 0x7f120c06d3b0 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.492+0000 7f121096f640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11f8004800 0x7f11f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f1211170640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f80039b0 0x7f11f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f1211971640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f11f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f121096f640 1 -- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 msgr2=0x7f11f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f121096f640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f11f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f121096f640 1 -- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f80039b0 msgr2=0x7f11f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f121096f640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f80039b0 0x7f11f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f121096f640 1 -- 172.21.15.99:0/2908942965 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f8003520 con 0x7f11f8004800 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f1211971640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c06d3b0 0x7f11f80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.493+0000 7f121096f640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11f8004800 0x7f11f8004c80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009af0 tx=0x7f11f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdd0bbd2ae7bb83 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc039070 con 0x7f11f8004800 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc02c6b0 con 0x7f11f8004800 2024-03-04T10:06:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f1203fff640 1 -- 172.21.15.99:0/2908942965 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc009690 con 0x7f11f8004800 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f12133d4640 1 -- 172.21.15.99:0/2908942965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11f8004800 msgr2=0x7f11f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f12133d4640 1 --2- 172.21.15.99:0/2908942965 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11f8004800 0x7f11f8004c80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009af0 tx=0x7f11f8005830 comp rx=0 tx=0).stop 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f1211971640 1 -- 172.21.15.99:0/2908942965 reap_dead start 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f12133d4640 1 -- 172.21.15.99:0/2908942965 shutdown_connections 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f12133d4640 1 -- 172.21.15.99:0/2908942965 >> 172.21.15.99:0/2908942965 conn(0x7f120c06acd0 msgr2=0x7f120c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f12133d4640 1 -- 172.21.15.99:0/2908942965 shutdown_connections 2024-03-04T10:06:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.494+0000 7f12133d4640 1 -- 172.21.15.99:0/2908942965 wait complete. 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.495+0000 7f12133d4640 1 Processor -- start 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.495+0000 7f12133d4640 1 -- start start 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f12133d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f8004800 0x7f120c155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f12133d4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 0x7f120c0b39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f12133d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c1f25a0 0x7f120c1f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f12133d4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c06e690 con 0x7f120c0b3550 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f12133d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c06e150 con 0x7f11f8004800 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f12133d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c06e810 con 0x7f120c1f25a0 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 0x7f120c0b39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 0x7f120c0b39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35794/0 (socket says 172.21.15.99:35794) 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 -- 172.21.15.99:0/485578501 learned_addr learned my addr 172.21.15.99:0/485578501 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f1211971640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c1f25a0 0x7f120c1f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f1211170640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f8004800 0x7f120c155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 -- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c1f25a0 msgr2=0x7f120c1f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c1f25a0 0x7f120c1f2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 -- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f8004800 msgr2=0x7f120c155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f8004800 0x7f120c155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.496+0000 7f121096f640 1 -- 172.21.15.99:0/485578501 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c150920 con 0x7f120c0b3550 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f1211170640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f8004800 0x7f120c155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f121096f640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 0x7f120c0b39d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f11fc037040 tx=0x7f11fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc039050 con 0x7f120c0b3550 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11fc045070 con 0x7f120c0b3550 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c1f20f0 con 0x7f120c0b3550 2024-03-04T10:06:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc003a50 con 0x7f120c0b3550 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.497+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c155080 con 0x7f120c0b3550 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.498+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c074f60 con 0x7f120c0b3550 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.498+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f11fc003bf0 con 0x7f120c0b3550 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.499+0000 7f1201ffb640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f11f006b4d0 0x7f11f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.499+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f11fc0aba10 con 0x7f120c0b3550 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.499+0000 7f1211170640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f11f006b4d0 0x7f11f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.499+0000 7f1211170640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f11f006b4d0 0x7f11f006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f120c1eefd0 tx=0x7f1204008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.503+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f11fc04e020 con 0x7f120c0b3550 2024-03-04T10:06:32.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.677+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f120c159920 con 0x7f120c0b3550 2024-03-04T10:06:32.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.678+0000 7f1201ffb640 1 -- 172.21.15.99:0/485578501 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+13128 (secure 0 0 0) 0x7f11fc07ac00 con 0x7f120c0b3550 2024-03-04T10:06:32.679 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:32.679 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":62,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:32.005312+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:32.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f11f006b4d0 msgr2=0x7f11f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f11f006b4d0 0x7f11f006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f120c1eefd0 tx=0x7f1204008040 comp rx=0 tx=0).stop 2024-03-04T10:06:32.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 msgr2=0x7f120c0b39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 0x7f120c0b39d0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f11fc037040 tx=0x7f11fc004730 comp rx=0 tx=0).stop 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 shutdown_connections 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f11f006b4d0 0x7f11f006d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f120c1f25a0 0x7f120c1f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f120c0b3550 0x7f120c0b39d0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 --2- 172.21.15.99:0/485578501 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11f8004800 0x7f120c155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 >> 172.21.15.99:0/485578501 conn(0x7f120c06acd0 msgr2=0x7f120c0b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 shutdown_connections 2024-03-04T10:06:32.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.680+0000 7f12133d4640 1 -- 172.21.15.99:0/485578501 wait complete. 2024-03-04T10:06:32.692 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:06:32.693 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:06:32.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.794+0000 7f5839b90640 1 Processor -- start 2024-03-04T10:06:32.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.794+0000 7f5839b90640 1 -- start start 2024-03-04T10:06:32.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5839b90640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f583415c7f0 0x7f583406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5839b90640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f58340b0150 0x7f583406b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5839b90640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583415cf50 con 0x7f58340c3e20 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5839b90640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58340cc390 con 0x7f58340b0150 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5839b90640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cc670 con 0x7f583415c7f0 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f58340b0150 0x7f583406b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f58340b0150 0x7f583406b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35806/0 (socket says 172.21.15.99:35806) 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 -- 172.21.15.99:0/3744478711 learned_addr learned my addr 172.21.15.99:0/3744478711 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f58337fe640 1 --2- 172.21.15.99:0/3744478711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f583415c7f0 0x7f583406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 -- 172.21.15.99:0/3744478711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f583415c7f0 msgr2=0x7f583406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 --2- 172.21.15.99:0/3744478711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f583415c7f0 0x7f583406acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 -- 172.21.15.99:0/3744478711 >> v1:172.21.15.162:6789/0 conn(0x7f58340c3e20 legacy=0x7f583415f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:32.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.795+0000 7f5832ffd640 1 -- 172.21.15.99:0/3744478711 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340cc950 con 0x7f58340b0150 2024-03-04T10:06:32.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.796+0000 7f5832ffd640 1 --2- 172.21.15.99:0/3744478711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340b0150 0x7f583406b210 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5828009870 tx=0x7f582802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=956f9f4350c5f8d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.796+0000 7f58327fc640 1 -- 172.21.15.99:0/3744478711 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828038070 con 0x7f58340b0150 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.796+0000 7f58327fc640 1 -- 172.21.15.99:0/3744478711 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828031740 con 0x7f58340b0150 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.796+0000 7f58327fc640 1 -- 172.21.15.99:0/3744478711 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582802f050 con 0x7f58340b0150 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 -- 172.21.15.99:0/3744478711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340b0150 msgr2=0x7f583406b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 --2- 172.21.15.99:0/3744478711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340b0150 0x7f583406b210 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5828009870 tx=0x7f582802b7e0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 -- 172.21.15.99:0/3744478711 shutdown_connections 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 --2- 172.21.15.99:0/3744478711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340b0150 0x7f583406b210 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 --2- 172.21.15.99:0/3744478711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f583415c7f0 0x7f583406acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 -- 172.21.15.99:0/3744478711 >> 172.21.15.99:0/3744478711 conn(0x7f58340b22e0 msgr2=0x7f58340b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 -- 172.21.15.99:0/3744478711 shutdown_connections 2024-03-04T10:06:32.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.797+0000 7f5839b90640 1 -- 172.21.15.99:0/3744478711 wait complete. 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.798+0000 7f5839b90640 1 Processor -- start 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.798+0000 7f5839b90640 1 -- start start 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5839b90640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340b0150 0x7f58340ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5839b90640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 0x7f58340c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5839b90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58341f7870 0x7f58341f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5839b90640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340b2e00 con 0x7f58340c3e20 2024-03-04T10:06:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5839b90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583415cbe0 con 0x7f58341f7870 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5839b90640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340b2f80 con 0x7f58340b0150 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 0x7f58340c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 0x7f58340c4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35808/0 (socket says 172.21.15.99:35808) 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 -- 172.21.15.99:0/2934293885 learned_addr learned my addr 172.21.15.99:0/2934293885 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 -- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340b0150 msgr2=0x7f58340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340b0150 0x7f58340ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 -- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58341f7870 msgr2=0x7f58341f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58341f7870 0x7f58341f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 -- 172.21.15.99:0/2934293885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340cea60 con 0x7f58340c3e20 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f5832ffd640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 0x7f58340c4d80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5828009870 tx=0x7f58280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828042070 con 0x7f58340c3e20 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.799+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5828031e80 con 0x7f58340c3e20 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.800+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58340ced40 con 0x7f58340c3e20 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.800+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828038040 con 0x7f58340c3e20 2024-03-04T10:06:32.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.800+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58340cf300 con 0x7f58340c3e20 2024-03-04T10:06:32.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.800+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834074f60 con 0x7f58340c3e20 2024-03-04T10:06:32.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.801+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5828045070 con 0x7f58340c3e20 2024-03-04T10:06:32.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.801+0000 7f581bfff640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f582006b4d0 0x7f582006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:32.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.801+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f58280abc60 con 0x7f58340c3e20 2024-03-04T10:06:32.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.802+0000 7f58337fe640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f582006b4d0 0x7f582006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:32.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.802+0000 7f58337fe640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f582006b4d0 0x7f582006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5824006fd0 tx=0x7f5824008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:32.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.806+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f582804c030 con 0x7f58340c3e20 2024-03-04T10:06:32.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.972+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f58340afac0 con 0x7f58340c3e20 2024-03-04T10:06:32.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.973+0000 7f581bfff640 1 -- 172.21.15.99:0/2934293885 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f582807ad30 con 0x7f58340c3e20 2024-03-04T10:06:32.974 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:06:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f582006b4d0 msgr2=0x7f582006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f582006b4d0 0x7f582006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5824006fd0 tx=0x7f5824008040 comp rx=0 tx=0).stop 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 msgr2=0x7f58340c4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 0x7f58340c4d80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5828009870 tx=0x7f58280046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 shutdown_connections 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f582006b4d0 0x7f582006d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58341f7870 0x7f58341f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58340c3e20 0x7f58340c4d80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 --2- 172.21.15.99:0/2934293885 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58340b0150 0x7f58340ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 >> 172.21.15.99:0/2934293885 conn(0x7f58340b22e0 msgr2=0x7f58340cd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 shutdown_connections 2024-03-04T10:06:32.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:32.975+0000 7f5839b90640 1 -- 172.21.15.99:0/2934293885 wait complete. 2024-03-04T10:06:32.987 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T10:06:33.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.086+0000 7f5e61da9640 1 Processor -- start 2024-03-04T10:06:33.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.086+0000 7f5e61da9640 1 -- start start 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e61da9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 0x7f5e5c15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e61da9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5e5c0cf650 0x7f5e5c0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e61da9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0cdaa0 con 0x7f5e5c1616c0 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e61da9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cfff0 con 0x7f5e5c0cf650 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e61da9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c071df0 con 0x7f5e5c154a80 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5bfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5e5c1616c0 0x7f5e5c161b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47830/0 (socket says 172.21.15.99:47830) 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5bfff640 1 -- 172.21.15.99:0/4103697506 learned_addr learned my addr 172.21.15.99:0/4103697506 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5affd640 1 --2- 172.21.15.99:0/4103697506 >> v2:172.21.15.162:3300/0 conn(0x7f5e5c0cf650 0x7f5e5c0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 0x7f5e5c15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359728835 0 0) 0x7f5e5c0cdaa0 con 0x7f5e5c1616c0 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e44003520 con 0x7f5e5c1616c0 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802334671 0 0) 0x7f5e44003520 con 0x7f5e5c1616c0 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 msgr2=0x7f5e5c15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 0x7f5e5c15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 >> v2:172.21.15.162:3300/0 conn(0x7f5e5c0cf650 msgr2=0x7f5e5c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 --2- 172.21.15.99:0/4103697506 >> v2:172.21.15.162:3300/0 conn(0x7f5e5c0cf650 0x7f5e5c0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0cd010 con 0x7f5e5c1616c0 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.087+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5e540019f0 con 0x7f5e5c1616c0 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 >> v1:172.21.15.99:6789/0 conn(0x7f5e5c1616c0 legacy=0x7f5e5c161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e440039e0 0x7f5e44003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 0x7f5e44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e44004d40 0x7f5e440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c071df0 con 0x7f5e440039e0 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4400b9c0 con 0x7f5e44004d40 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4400bb40 con 0x7f5e5c154a80 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5affd640 1 --2- 172.21.15.99:0/4103697506 >> v2:172.21.15.162:3300/0 conn(0x7f5e5c0cf650 0x7f5e5c0cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e440039e0 0x7f5e44003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5bfff640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e44004d40 0x7f5e440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5affd640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 0x7f5e44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5b7fe640 1 -- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 msgr2=0x7f5e44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c154a80 0x7f5e44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5b7fe640 1 -- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e44004d40 msgr2=0x7f5e440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e44004d40 0x7f5e440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5b7fe640 1 -- 172.21.15.99:0/4103697506 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e44003520 con 0x7f5e440039e0 2024-03-04T10:06:33.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.088+0000 7f5e5bfff640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e44004d40 0x7f5e440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e440039e0 0x7f5e44003e00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009af0 tx=0x7f5e44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9714f3fd67a9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c039070 con 0x7f5e440039e0 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c02c6b0 con 0x7f5e440039e0 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e5a7fc640 1 -- 172.21.15.99:0/4103697506 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c009690 con 0x7f5e440039e0 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e61da9640 1 -- 172.21.15.99:0/4103697506 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e440039e0 msgr2=0x7f5e44003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e61da9640 1 --2- 172.21.15.99:0/4103697506 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e440039e0 0x7f5e44003e00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009af0 tx=0x7f5e44005830 comp rx=0 tx=0).stop 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e5bfff640 1 -- 172.21.15.99:0/4103697506 reap_dead start 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e61da9640 1 -- 172.21.15.99:0/4103697506 shutdown_connections 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e61da9640 1 -- 172.21.15.99:0/4103697506 >> 172.21.15.99:0/4103697506 conn(0x7f5e5c06f5f0 msgr2=0x7f5e5c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e61da9640 1 -- 172.21.15.99:0/4103697506 shutdown_connections 2024-03-04T10:06:33.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.089+0000 7f5e61da9640 1 -- 172.21.15.99:0/4103697506 wait complete. 2024-03-04T10:06:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 Processor -- start 2024-03-04T10:06:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 -- start start 2024-03-04T10:06:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e440039e0 0x7f5e5c0c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c0c7c20 0x7f5e5c1529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 0x7f5e5c1f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0ce340 con 0x7f5e5c152f00 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0ce060 con 0x7f5e440039e0 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.091+0000 7f5e61da9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0ce620 con 0x7f5e5c0c7c20 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 0x7f5e5c1f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 0x7f5e5c1f36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35836/0 (socket says 172.21.15.99:35836) 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 -- 172.21.15.99:0/2029948301 learned_addr learned my addr 172.21.15.99:0/2029948301 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5affd640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c0c7c20 0x7f5e5c1529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e440039e0 0x7f5e5c0c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 -- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c0c7c20 msgr2=0x7f5e5c1529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c0c7c20 0x7f5e5c1529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 -- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e440039e0 msgr2=0x7f5e5c0c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e440039e0 0x7f5e5c0c5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 -- 172.21.15.99:0/2029948301 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0c5940 con 0x7f5e5c152f00 2024-03-04T10:06:33.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5affd640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c0c7c20 0x7f5e5c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e5bfff640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 0x7f5e5c1f36c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c070210 tx=0x7f5e5400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5400edd0 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54014430 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54014820 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c1f4930 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.092+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c1f4ef0 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.093+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c1f50f0 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.094+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5e540149c0 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.094+0000 7f5e43fff640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e4806b500 0x7f5e4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.094+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5e54089e20 con 0x7f5e5c152f00 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.095+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e4806b500 0x7f5e4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.095+0000 7f5e5b7fe640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e4806b500 0x7f5e4806d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c036000 tx=0x7f5e4c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:33.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.099+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5e5c1f50f0 con 0x7f5e5c152f00 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.273+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5e5c074f60 con 0x7f5e5c152f00 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.273+0000 7f5e43fff640 1 -- 172.21.15.99:0/2029948301 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v62) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5e54058e60 con 0x7f5e5c152f00 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:06:33.275 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e4806b500 msgr2=0x7f5e4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e4806b500 0x7f5e4806d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c036000 tx=0x7f5e4c030000 comp rx=0 tx=0).stop 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 msgr2=0x7f5e5c1f36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 0x7f5e5c1f36c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c070210 tx=0x7f5e5400e530 comp rx=0 tx=0).stop 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 shutdown_connections 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e4806b500 0x7f5e4806d9c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e5c152f00 0x7f5e5c1f36c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e5c0c7c20 0x7f5e5c1529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 --2- 172.21.15.99:0/2029948301 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e440039e0 0x7f5e5c0c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.275+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 >> 172.21.15.99:0/2029948301 conn(0x7f5e5c06f5f0 msgr2=0x7f5e5c0b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:33.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.276+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 shutdown_connections 2024-03-04T10:06:33.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.276+0000 7f5e61da9640 1 -- 172.21.15.99:0/2029948301 wait complete. 2024-03-04T10:06:33.287 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T10:06:33.288 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T10:06:33.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.390+0000 7f23d07f1640 1 Processor -- start 2024-03-04T10:06:33.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.390+0000 7f23d07f1640 1 -- start start 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23d07f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23d07f1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f23c806b3e0 0x7f23c80b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23d07f1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80d02b0 con 0x7f23c80cfa60 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23d07f1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23c80d0590 con 0x7f23c806b3e0 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23d07f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80b36c0 con 0x7f23c815bd90 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23ced8e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f23c806b3e0 0x7f23c80b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23cdd8c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f23c80cfa60 0x7f23c806acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:47836/0 (socket says 172.21.15.99:47836) 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.391+0000 7f23cdd8c640 1 -- 172.21.15.99:0/699025415 learned_addr learned my addr 172.21.15.99:0/699025415 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23ce58d640 1 --2- 172.21.15.99:0/699025415 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23ced8e640 1 -- 172.21.15.99:0/699025415 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 msgr2=0x7f23c80cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23ced8e640 1 --2- 172.21.15.99:0/699025415 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23ced8e640 1 -- 172.21.15.99:0/699025415 >> v1:172.21.15.99:6789/0 conn(0x7f23c80cfa60 legacy=0x7f23c806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23ced8e640 1 -- 172.21.15.99:0/699025415 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80b39a0 con 0x7f23c806b3e0 2024-03-04T10:06:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23cd58b640 1 -- 172.21.15.99:0/699025415 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883257644 0 0) 0x7f23c80d02b0 con 0x7f23c80cfa60 2024-03-04T10:06:33.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.392+0000 7f23ced8e640 1 --2- 172.21.15.99:0/699025415 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c80b41a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f23c00097a0 tx=0x7f23c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe115b8fd2e2fe5f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23cd58b640 1 -- 172.21.15.99:0/699025415 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0035020 con 0x7f23c806b3e0 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23cd58b640 1 -- 172.21.15.99:0/699025415 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c0031830 con 0x7f23c806b3e0 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23cd58b640 1 -- 172.21.15.99:0/699025415 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0031c00 con 0x7f23c806b3e0 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 -- 172.21.15.99:0/699025415 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 msgr2=0x7f23c80b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 --2- 172.21.15.99:0/699025415 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c80b41a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f23c00097a0 tx=0x7f23c002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 -- 172.21.15.99:0/699025415 shutdown_connections 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 --2- 172.21.15.99:0/699025415 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c80b41a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 --2- 172.21.15.99:0/699025415 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 -- 172.21.15.99:0/699025415 >> 172.21.15.99:0/699025415 conn(0x7f23c80b22e0 msgr2=0x7f23c80b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 -- 172.21.15.99:0/699025415 shutdown_connections 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.393+0000 7f23d07f1640 1 -- 172.21.15.99:0/699025415 wait complete. 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.394+0000 7f23d07f1640 1 Processor -- start 2024-03-04T10:06:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.394+0000 7f23d07f1640 1 -- start start 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23d07f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23d07f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23c80cfa60 0x7f23c814ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23d07f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23d07f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80cfec0 con 0x7f23c806b3e0 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23d07f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c815c170 con 0x7f23c80cfa60 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23d07f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c80d0040 con 0x7f23c815bd90 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23ce58d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23ce58d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c815e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:35842/0 (socket says 172.21.15.99:35842) 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23ce58d640 1 -- 172.21.15.99:0/789283118 learned_addr learned my addr 172.21.15.99:0/789283118 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23cdd8c640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23c80cfa60 0x7f23c814ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23ced8e640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.395+0000 7f23ce58d640 1 -- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 msgr2=0x7f23c80c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23ce58d640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23ce58d640 1 -- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23c80cfa60 msgr2=0x7f23c814ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23ce58d640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23c80cfa60 0x7f23c814ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23ce58d640 1 -- 172.21.15.99:0/789283118 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80cbf20 con 0x7f23c806b3e0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23cdd8c640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23c80cfa60 0x7f23c814ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23ce58d640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c815e3c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f23c4007e40 tx=0x7f23c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4014620 con 0x7f23c806b3e0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c4019070 con 0x7f23c806b3e0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c80c7de0 con 0x7f23c806b3e0 2024-03-04T10:06:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4014a00 con 0x7f23c806b3e0 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.396+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c80cbc40 con 0x7f23c806b3e0 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.397+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c8074f60 con 0x7f23c806b3e0 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.397+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f23c4014ba0 con 0x7f23c806b3e0 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.398+0000 7f23beffd640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23ac06b4d0 0x7f23ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.398+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f23c408a8e0 con 0x7f23c806b3e0 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.398+0000 7f23cdd8c640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23ac06b4d0 0x7f23ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.398+0000 7f23cdd8c640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23ac06b4d0 0x7f23ac06d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f23c80b6410 tx=0x7f23b8007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:33.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.402+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f23c4059920 con 0x7f23c806b3e0 2024-03-04T10:06:33.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:33.571+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f23c80b31b0 con 0x7f23c806b3e0 2024-03-04T10:06:34.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.004+0000 7f23beffd640 1 -- 172.21.15.99:0/789283118 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v63) v1 ==== 138+0+0 (secure 0 0 0) 0x7f23c4059220 con 0x7f23c806b3e0 2024-03-04T10:06:34.005 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23ac06b4d0 msgr2=0x7f23ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23ac06b4d0 0x7f23ac06d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f23c80b6410 tx=0x7f23b8007c10 comp rx=0 tx=0).stop 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 msgr2=0x7f23c815e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c815e3c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f23c4007e40 tx=0x7f23c400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 shutdown_connections 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f23ac06b4d0 0x7f23ac06d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f23c815bd90 0x7f23c80c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f23c80cfa60 0x7f23c814ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 --2- 172.21.15.99:0/789283118 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f23c806b3e0 0x7f23c815e3c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 >> 172.21.15.99:0/789283118 conn(0x7f23c80b22e0 msgr2=0x7f23c815a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 shutdown_connections 2024-03-04T10:06:34.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.006+0000 7f23d07f1640 1 -- 172.21.15.99:0/789283118 wait complete. 2024-03-04T10:06:34.023 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T10:06:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.131+0000 7f14ea28c640 1 Processor -- start 2024-03-04T10:06:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.131+0000 7f14ea28c640 1 -- start start 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.131+0000 7f14ea28c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f14e415a810 0x7f14e40b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14ea28c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f14e406d3c0 0x7f14e40b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14ea28c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e406d8a0 con 0x7f14e40cbfb0 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14ea28c640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e406db80 con 0x7f14e415a810 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14ea28c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e406de60 con 0x7f14e406d3c0 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f14e415a810 0x7f14e40b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f14e415a810 0x7f14e40b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38004/0 (socket says 172.21.15.99:38004) 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 -- 172.21.15.99:0/2399275171 learned_addr learned my addr 172.21.15.99:0/2399275171 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e37fe640 1 --2- 172.21.15.99:0/2399275171 >> v2:172.21.15.162:3300/0 conn(0x7f14e406d3c0 0x7f14e40b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 -- 172.21.15.99:0/2399275171 >> v2:172.21.15.162:3300/0 conn(0x7f14e406d3c0 msgr2=0x7f14e40b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 --2- 172.21.15.99:0/2399275171 >> v2:172.21.15.162:3300/0 conn(0x7f14e406d3c0 0x7f14e40b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 -- 172.21.15.99:0/2399275171 >> v1:172.21.15.162:6789/0 conn(0x7f14e40cbfb0 legacy=0x7f14e415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.132+0000 7f14e3fff640 1 -- 172.21.15.99:0/2399275171 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e406e140 con 0x7f14e415a810 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14e3fff640 1 --2- 172.21.15.99:0/2399275171 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e415a810 0x7f14e40b4190 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f14e40b4d00 tx=0x7f14d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f001b20a405198 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14e2ffd640 1 -- 172.21.15.99:0/2399275171 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4035020 con 0x7f14e415a810 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14e2ffd640 1 -- 172.21.15.99:0/2399275171 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d4031870 con 0x7f14e415a810 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14e2ffd640 1 -- 172.21.15.99:0/2399275171 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4031c40 con 0x7f14e415a810 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 -- 172.21.15.99:0/2399275171 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e415a810 msgr2=0x7f14e40b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 --2- 172.21.15.99:0/2399275171 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e415a810 0x7f14e40b4190 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f14e40b4d00 tx=0x7f14d402b910 comp rx=0 tx=0).stop 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 -- 172.21.15.99:0/2399275171 shutdown_connections 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 --2- 172.21.15.99:0/2399275171 >> v2:172.21.15.162:3300/0 conn(0x7f14e406d3c0 0x7f14e40b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 --2- 172.21.15.99:0/2399275171 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e415a810 0x7f14e40b4190 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 -- 172.21.15.99:0/2399275171 >> 172.21.15.99:0/2399275171 conn(0x7f14e406acd0 msgr2=0x7f14e406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 -- 172.21.15.99:0/2399275171 shutdown_connections 2024-03-04T10:06:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.133+0000 7f14ea28c640 1 -- 172.21.15.99:0/2399275171 wait complete. 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 Processor -- start 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 -- start start 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 0x7f14e4140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14e40cbfb0 0x7f14e40cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14e415a810 0x7f14e40cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e415e3c0 con 0x7f14e406d3c0 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e40b3c50 con 0x7f14e415a810 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.135+0000 7f14ea28c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e415e540 con 0x7f14e40cbfb0 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 0x7f14e4140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 0x7f14e4140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38014/0 (socket says 172.21.15.99:38014) 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 -- 172.21.15.99:0/84876997 learned_addr learned my addr 172.21.15.99:0/84876997 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e8829640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14e415a810 0x7f14e40cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e37fe640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14e40cbfb0 0x7f14e40cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 -- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14e40cbfb0 msgr2=0x7f14e40cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14e40cbfb0 0x7f14e40cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 -- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14e415a810 msgr2=0x7f14e40cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14e415a810 0x7f14e40cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 -- 172.21.15.99:0/84876997 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e41405b0 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e8829640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14e415a810 0x7f14e40cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e3fff640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 0x7f14e4140070 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000c00 tx=0x7f14d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4040070 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d4030d00 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e41529b0 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4035040 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.136+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e4156b20 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.137+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e4074f60 con 0x7f14e406d3c0 2024-03-04T10:06:34.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.138+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f14d404b060 con 0x7f14e406d3c0 2024-03-04T10:06:34.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.138+0000 7f14e0ff9640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14d006b500 0x7f14d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:34.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.138+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f14d40abe70 con 0x7f14e406d3c0 2024-03-04T10:06:34.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.138+0000 7f14e37fe640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14d006b500 0x7f14d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:34.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.139+0000 7f14e37fe640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14d006b500 0x7f14d006d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14e4152f60 tx=0x7f14d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:34.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.143+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f14d407add0 con 0x7f14e406d3c0 2024-03-04T10:06:34.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:34.313+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f14e40af840 con 0x7f14e406d3c0 2024-03-04T10:06:35.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.005+0000 7f14e0ff9640 1 -- 172.21.15.99:0/84876997 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v64) v1 ==== 170+0+0 (secure 0 0 0) 0x7f14d407a6d0 con 0x7f14e406d3c0 2024-03-04T10:06:35.008 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2024-03-04T10:06:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14d006b500 msgr2=0x7f14d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14d006b500 0x7f14d006d9c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14e4152f60 tx=0x7f14d8008040 comp rx=0 tx=0).stop 2024-03-04T10:06:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 msgr2=0x7f14e4140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 0x7f14e4140070 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000c00 tx=0x7f14d40046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 shutdown_connections 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f14d006b500 0x7f14d006d9c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f14e415a810 0x7f14e40cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f14e40cbfb0 0x7f14e40cf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 --2- 172.21.15.99:0/84876997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f14e406d3c0 0x7f14e4140070 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 >> 172.21.15.99:0/84876997 conn(0x7f14e406acd0 msgr2=0x7f14e4155400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 shutdown_connections 2024-03-04T10:06:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.008+0000 7f14ea28c640 1 -- 172.21.15.99:0/84876997 wait complete. 2024-03-04T10:06:35.032 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T10:06:35.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.135+0000 7fab121e1640 1 Processor -- start 2024-03-04T10:06:35.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.135+0000 7fab121e1640 1 -- start start 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab121e1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fab0c0cdd70 0x7fab0c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab121e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0cc900 0x7fab0c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab121e1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fab0c0ccfe0 0x7fab0c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab121e1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0c0ccde0 con 0x7fab0c0ccfe0 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab121e1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0c0d0480 con 0x7fab0c0cdd70 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab121e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c06ae00 con 0x7fab0c0cc900 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fab0c0ccfe0 0x7fab0c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fab0c0ccfe0 0x7fab0c15d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38022/0 (socket says 172.21.15.99:38022) 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 -- 172.21.15.99:0/4210770987 learned_addr learned my addr 172.21.15.99:0/4210770987 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0b7fe640 1 --2- 172.21.15.99:0/4210770987 >> v2:172.21.15.162:3300/0 conn(0x7fab0c0cdd70 0x7fab0c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0affd640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0cc900 0x7fab0c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 -- 172.21.15.99:0/4210770987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0cc900 msgr2=0x7fab0c0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0cc900 0x7fab0c0cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 -- 172.21.15.99:0/4210770987 >> v2:172.21.15.162:3300/0 conn(0x7fab0c0cdd70 msgr2=0x7fab0c0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 --2- 172.21.15.99:0/4210770987 >> v2:172.21.15.162:3300/0 conn(0x7fab0c0cdd70 0x7fab0c0cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0bfff640 1 -- 172.21.15.99:0/4210770987 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c06b0e0 con 0x7fab0c0ccfe0 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0affd640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0cc900 0x7fab0c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.136+0000 7fab0b7fe640 1 --2- 172.21.15.99:0/4210770987 >> v2:172.21.15.162:3300/0 conn(0x7fab0c0cdd70 0x7fab0c0cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.137+0000 7fab0bfff640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0ccfe0 0x7fab0c15d410 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fab0c06e420 tx=0x7fab0402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16ebdfa553d91b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.137+0000 7fab0a7fc640 1 -- 172.21.15.99:0/4210770987 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0402c9e0 con 0x7fab0c0ccfe0 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.137+0000 7fab0a7fc640 1 -- 172.21.15.99:0/4210770987 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0402cb80 con 0x7fab0c0ccfe0 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.137+0000 7fab0a7fc640 1 -- 172.21.15.99:0/4210770987 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab040343f0 con 0x7fab0c0ccfe0 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.137+0000 7fab121e1640 1 -- 172.21.15.99:0/4210770987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0ccfe0 msgr2=0x7fab0c15d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.137+0000 7fab121e1640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0ccfe0 0x7fab0c15d410 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fab0c06e420 tx=0x7fab0402be40 comp rx=0 tx=0).stop 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 -- 172.21.15.99:0/4210770987 shutdown_connections 2024-03-04T10:06:35.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0ccfe0 0x7fab0c15d410 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 --2- 172.21.15.99:0/4210770987 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0cc900 0x7fab0c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 --2- 172.21.15.99:0/4210770987 >> v2:172.21.15.162:3300/0 conn(0x7fab0c0cdd70 0x7fab0c0cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 -- 172.21.15.99:0/4210770987 >> 172.21.15.99:0/4210770987 conn(0x7fab0c0b22e0 msgr2=0x7fab0c0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 -- 172.21.15.99:0/4210770987 shutdown_connections 2024-03-04T10:06:35.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.138+0000 7fab121e1640 1 -- 172.21.15.99:0/4210770987 wait complete. 2024-03-04T10:06:35.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.139+0000 7fab121e1640 1 Processor -- start 2024-03-04T10:06:35.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.139+0000 7fab121e1640 1 -- start start 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 0x7fab0c148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0ccfe0 0x7fab0c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab0c0cdd70 0x7fab0c0b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c0b2e90 con 0x7fab0c0cc900 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c06b6a0 con 0x7fab0c0cdd70 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0c0b3010 con 0x7fab0c0ccfe0 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 0x7fab0c148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 0x7fab0c148530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38028/0 (socket says 172.21.15.99:38028) 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 -- 172.21.15.99:0/642552419 learned_addr learned my addr 172.21.15.99:0/642552419 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0affd640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0ccfe0 0x7fab0c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0bfff640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab0c0cdd70 0x7fab0c0b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 -- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0ccfe0 msgr2=0x7fab0c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0ccfe0 0x7fab0c0c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 -- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab0c0cdd70 msgr2=0x7fab0c0b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:35.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab0c0cdd70 0x7fab0c0b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 -- 172.21.15.99:0/642552419 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c0c5b60 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab0b7fe640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 0x7fab0c148530 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7faafc00c7f0 tx=0x7faafc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc015070 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc007a30 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0c0c5e40 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc007e00 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.140+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0c0c6400 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.141+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0c074f60 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.142+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7faafc018070 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.142+0000 7faaf3fff640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaf806b4d0 0x7faaf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.142+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6001+0+0 (secure 0 0 0) 0x7faafc08a2a0 con 0x7fab0c0cc900 2024-03-04T10:06:35.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.143+0000 7fab0affd640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaf806b4d0 0x7faaf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:35.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.143+0000 7fab0affd640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaf806b4d0 0x7faaf806d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cc2c0 tx=0x7fab00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:35.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.147+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faafc059080 con 0x7fab0c0cc900 2024-03-04T10:06:35.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:35.351+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fab0c06ee00 con 0x7fab0c0cc900 2024-03-04T10:06:36.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.029+0000 7faaf3fff640 1 -- 172.21.15.99:0/642552419 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v41) v1 ==== 173+0+0 (secure 0 0 0) 0x7faafc058980 con 0x7fab0c0cc900 2024-03-04T10:06:36.037 INFO:teuthology.orchestra.run.smithi099.stderr:new fs with metadata pool 9 and data pool 10 2024-03-04T10:06:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.043+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaf806b4d0 msgr2=0x7faaf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.043+0000 7fab121e1640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaf806b4d0 0x7faaf806d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fab0c0cc2c0 tx=0x7fab00008040 comp rx=0 tx=0).stop 2024-03-04T10:06:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 msgr2=0x7fab0c148530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 0x7fab0c148530 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7faafc00c7f0 tx=0x7faafc00cbf0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 shutdown_connections 2024-03-04T10:06:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaf806b4d0 0x7faaf806d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab0c0cdd70 0x7fab0c0b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fab0c0ccfe0 0x7fab0c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 --2- 172.21.15.99:0/642552419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fab0c0cc900 0x7fab0c148530 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 >> 172.21.15.99:0/642552419 conn(0x7fab0c0b22e0 msgr2=0x7fab0c06f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 shutdown_connections 2024-03-04T10:06:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.044+0000 7fab121e1640 1 -- 172.21.15.99:0/642552419 wait complete. 2024-03-04T10:06:36.060 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.163+0000 7f58eb227640 1 Processor -- start 2024-03-04T10:06:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.163+0000 7f58eb227640 1 -- start start 2024-03-04T10:06:36.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58eb227640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f58e40b0700 0x7f58e40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58eb227640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e40b47c0 0x7f58e40b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58eb227640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40ccad0 con 0x7f58e415b960 2024-03-04T10:06:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58eb227640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e40b5190 con 0x7f58e40b0700 2024-03-04T10:06:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58eb227640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40cf860 con 0x7f58e40b47c0 2024-03-04T10:06:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f58e40b0700 0x7f58e40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f58e40b0700 0x7f58e40b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38042/0 (socket says 172.21.15.99:38042) 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 -- 172.21.15.99:0/117822809 learned_addr learned my addr 172.21.15.99:0/117822809 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e97c4640 1 --2- 172.21.15.99:0/117822809 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e40b47c0 0x7f58e40b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 -- 172.21.15.99:0/117822809 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e40b47c0 msgr2=0x7f58e40b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 --2- 172.21.15.99:0/117822809 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e40b47c0 0x7f58e40b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 -- 172.21.15.99:0/117822809 >> v1:172.21.15.162:6789/0 conn(0x7f58e415b960 legacy=0x7f58e40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.164+0000 7f58e3fff640 1 -- 172.21.15.99:0/117822809 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40cfab0 con 0x7f58e40b0700 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.165+0000 7f58e3fff640 1 --2- 172.21.15.99:0/117822809 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40b0b90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009870 tx=0x7f58d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f918da25bac1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.165+0000 7f58e37fe640 1 -- 172.21.15.99:0/117822809 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d0035020 con 0x7f58e40b0700 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.165+0000 7f58e37fe640 1 -- 172.21.15.99:0/117822809 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d0031640 con 0x7f58e40b0700 2024-03-04T10:06:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.165+0000 7f58e37fe640 1 -- 172.21.15.99:0/117822809 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d002d040 con 0x7f58e40b0700 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 -- 172.21.15.99:0/117822809 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 msgr2=0x7f58e40b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 --2- 172.21.15.99:0/117822809 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40b0b90 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f58d0009870 tx=0x7f58d002b8f0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 -- 172.21.15.99:0/117822809 shutdown_connections 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 --2- 172.21.15.99:0/117822809 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e40b47c0 0x7f58e40b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 --2- 172.21.15.99:0/117822809 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40b0b90 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 -- 172.21.15.99:0/117822809 >> 172.21.15.99:0/117822809 conn(0x7f58e406f5f0 msgr2=0x7f58e406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 -- 172.21.15.99:0/117822809 shutdown_connections 2024-03-04T10:06:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.166+0000 7f58eb227640 1 -- 172.21.15.99:0/117822809 wait complete. 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.167+0000 7f58eb227640 1 Processor -- start 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.167+0000 7f58eb227640 1 -- start start 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58eb227640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58eb227640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e40b47c0 0x7f58e4158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58eb227640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e415b960 0x7f58e40aa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58eb227640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e415bd40 con 0x7f58e40b0700 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58eb227640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e40d0100 con 0x7f58e40b47c0 2024-03-04T10:06:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58eb227640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e415bec0 con 0x7f58e415b960 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38052/0 (socket says 172.21.15.99:38052) 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 -- 172.21.15.99:0/96976533 learned_addr learned my addr 172.21.15.99:0/96976533 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e3fff640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e40b47c0 0x7f58e4158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e97c4640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e415b960 0x7f58e40aa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 -- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e415b960 msgr2=0x7f58e40aa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e415b960 0x7f58e40aa6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 -- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e40b47c0 msgr2=0x7f58e4158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e40b47c0 0x7f58e4158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 -- 172.21.15.99:0/96976533 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40c9fa0 con 0x7f58e40b0700 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.168+0000 7f58e8fc3640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40c9980 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f58d800c3c0 tx=0x7f58d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.169+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d80052d0 con 0x7f58e40b0700 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.169+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d8005490 con 0x7f58e40b0700 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.169+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8007260 con 0x7f58e40b0700 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.169+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e40ca280 con 0x7f58e40b0700 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.169+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e40ca840 con 0x7f58e40b0700 2024-03-04T10:06:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.170+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e4074f60 con 0x7f58e40b0700 2024-03-04T10:06:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.170+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f58d8007400 con 0x7f58e40b0700 2024-03-04T10:06:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.170+0000 7f58e17fa640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58c806b500 0x7f58c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.171+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f58d80896a0 con 0x7f58e40b0700 2024-03-04T10:06:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.171+0000 7f58e3fff640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58c806b500 0x7f58c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.171+0000 7f58e3fff640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58c806b500 0x7f58c806d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f58d0036000 tx=0x7f58d003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.175+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f58d8057dd0 con 0x7f58e40b0700 2024-03-04T10:06:36.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.346+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e415aae0 con 0x7f58e40b0700 2024-03-04T10:06:36.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.347+0000 7f58e17fa640 1 -- 172.21.15.99:0/96976533 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16861 (secure 0 0 0) 0x7f58d805be70 con 0x7f58e40b0700 2024-03-04T10:06:36.350 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:36.350 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":65,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:35.353962+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58c806b500 msgr2=0x7f58c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58c806b500 0x7f58c806d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f58d0036000 tx=0x7f58d003c040 comp rx=0 tx=0).stop 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 msgr2=0x7f58e40c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40c9980 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f58d800c3c0 tx=0x7f58d800c880 comp rx=0 tx=0).stop 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 shutdown_connections 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58c806b500 0x7f58c806d9c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f58e415b960 0x7f58e40aa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58e40b47c0 0x7f58e4158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 --2- 172.21.15.99:0/96976533 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58e40b0700 0x7f58e40c9980 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 >> 172.21.15.99:0/96976533 conn(0x7f58e406f5f0 msgr2=0x7f58e40b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 shutdown_connections 2024-03-04T10:06:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.349+0000 7f58eb227640 1 -- 172.21.15.99:0/96976533 wait complete. 2024-03-04T10:06:36.363 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:36.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.461+0000 7fe3d8f0a640 1 Processor -- start 2024-03-04T10:06:36.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.461+0000 7fe3d8f0a640 1 -- start start 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d8f0a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe3d415dea0 0x7fe3d40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d8f0a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3d40cc8d0 0x7fe3d406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d8f0a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d406fd20 con 0x7fe3d40d1200 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d8f0a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d406fad0 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d8f0a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3d4070000 con 0x7fe3d415dea0 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3d40cc8d0 0x7fe3d406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe3d40cc8d0 0x7fe3d406acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38062/0 (socket says 172.21.15.99:38062) 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 -- 172.21.15.99:0/2912340523 learned_addr learned my addr 172.21.15.99:0/2912340523 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d259c640 1 --2- 172.21.15.99:0/2912340523 >> v2:172.21.15.162:3300/0 conn(0x7fe3d415dea0 0x7fe3d40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 -- 172.21.15.99:0/2912340523 >> v2:172.21.15.162:3300/0 conn(0x7fe3d415dea0 msgr2=0x7fe3d40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 --2- 172.21.15.99:0/2912340523 >> v2:172.21.15.162:3300/0 conn(0x7fe3d415dea0 0x7fe3d40cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 -- 172.21.15.99:0/2912340523 >> v1:172.21.15.162:6789/0 conn(0x7fe3d40d1200 legacy=0x7fe3d40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.462+0000 7fe3d1d9b640 1 -- 172.21.15.99:0/2912340523 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d40702e0 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.463+0000 7fe3d1d9b640 1 --2- 172.21.15.99:0/2912340523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d406acd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40ce200 tx=0x7fe3c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a09b5b33eca1f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.463+0000 7fe3d159a640 1 -- 172.21.15.99:0/2912340523 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8035020 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.463+0000 7fe3d159a640 1 -- 172.21.15.99:0/2912340523 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c8031870 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.463+0000 7fe3d159a640 1 -- 172.21.15.99:0/2912340523 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8031c40 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/2912340523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 msgr2=0x7fe3d406acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/2912340523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d406acd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe3d40ce200 tx=0x7fe3c802b910 comp rx=0 tx=0).stop 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/2912340523 shutdown_connections 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/2912340523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d406acd0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/2912340523 >> v2:172.21.15.162:3300/0 conn(0x7fe3d415dea0 0x7fe3d40cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/2912340523 >> 172.21.15.99:0/2912340523 conn(0x7fe3d40d0710 msgr2=0x7fe3d40d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/2912340523 shutdown_connections 2024-03-04T10:06:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.464+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/2912340523 wait complete. 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.465+0000 7fe3d8f0a640 1 Processor -- start 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.465+0000 7fe3d8f0a640 1 -- start start 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d8f0a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d4153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d8f0a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40d1200 0x7fe3d415af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d8f0a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d415dea0 0x7fe3d40b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d8f0a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d406b390 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d8f0a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d406b210 con 0x7fe3d415dea0 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d8f0a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d406b510 con 0x7fe3d40d1200 2024-03-04T10:06:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d4153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d4153d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38072/0 (socket says 172.21.15.99:38072) 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 -- 172.21.15.99:0/4152866859 learned_addr learned my addr 172.21.15.99:0/4152866859 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 -- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40d1200 msgr2=0x7fe3d415af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40d1200 0x7fe3d415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 -- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d415dea0 msgr2=0x7fe3d40b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d415dea0 0x7fe3d40b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 -- 172.21.15.99:0/4152866859 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d4159610 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3d259c640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d4153d70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400c7f0 tx=0x7fe3c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.466+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4015070 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.467+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c4004240 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.467+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d41598b0 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.467+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4007620 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.467+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d4159e70 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.467+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d4074f60 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.468+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe3c40079d0 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.468+0000 7fe3c2ffd640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.469+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe3c408a7a0 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.469+0000 7fe3d1d9b640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.469+0000 7fe3d1d9b640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3b406b4d0 0x7fe3b406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8036000 tx=0x7fe3c8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.473+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe3c4059490 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.646+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3d40cbfe0 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.647+0000 7fe3c2ffd640 1 -- 172.21.15.99:0/4152866859 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16861 (secure 0 0 0) 0x7fe3c4058d90 con 0x7fe3d40cc8d0 2024-03-04T10:06:36.650 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:36.650 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":65,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:35.353962+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3b406b4d0 msgr2=0x7fe3b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3b406b4d0 0x7fe3b406d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8036000 tx=0x7fe3c8030000 comp rx=0 tx=0).stop 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 msgr2=0x7fe3d4153d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d4153d70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe3c400c7f0 tx=0x7fe3c400ccb0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 shutdown_connections 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe3b406b4d0 0x7fe3b406d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe3d415dea0 0x7fe3d40b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe3d40d1200 0x7fe3d415af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 --2- 172.21.15.99:0/4152866859 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe3d40cc8d0 0x7fe3d4153d70 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 >> 172.21.15.99:0/4152866859 conn(0x7fe3d40d0710 msgr2=0x7fe3d40afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 shutdown_connections 2024-03-04T10:06:36.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.649+0000 7fe3d8f0a640 1 -- 172.21.15.99:0/4152866859 wait complete. 2024-03-04T10:06:36.663 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T10:06:36.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.765+0000 7f033d44b640 1 Processor -- start 2024-03-04T10:06:36.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.765+0000 7f033d44b640 1 -- start start 2024-03-04T10:06:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.765+0000 7f033d44b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f033d44b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f03380b9410 0x7f033806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f033d44b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03380d2400 con 0x7f03380d34a0 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f033d44b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03380d26e0 con 0x7f03380b9410 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f033d44b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03380d29c0 con 0x7f03380b5700 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f03367fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f03380d34a0 0x7f033806c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37336/0 (socket says 172.21.15.99:37336) 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f03367fc640 1 -- 172.21.15.99:0/406362630 learned_addr learned my addr 172.21.15.99:0/406362630 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0336ffd640 1 --2- 172.21.15.99:0/406362630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f03377fe640 1 --2- 172.21.15.99:0/406362630 >> v2:172.21.15.162:3300/0 conn(0x7f03380b9410 0x7f033806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0335ffb640 1 -- 172.21.15.99:0/406362630 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160337393 0 0) 0x7f03380d2400 con 0x7f03380d34a0 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0335ffb640 1 -- 172.21.15.99:0/406362630 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0320003520 con 0x7f03380d34a0 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0336ffd640 1 -- 172.21.15.99:0/406362630 >> v2:172.21.15.162:3300/0 conn(0x7f03380b9410 msgr2=0x7f033806c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0335ffb640 1 -- 172.21.15.99:0/406362630 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439929641 0 0) 0x7f0320003520 con 0x7f03380d34a0 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0336ffd640 1 --2- 172.21.15.99:0/406362630 >> v2:172.21.15.162:3300/0 conn(0x7f03380b9410 0x7f033806c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0336ffd640 1 -- 172.21.15.99:0/406362630 >> v1:172.21.15.99:6789/0 conn(0x7f03380d34a0 legacy=0x7f033806c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.766+0000 7f0336ffd640 1 -- 172.21.15.99:0/406362630 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03380d2ca0 con 0x7f03380b5700 2024-03-04T10:06:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.767+0000 7f0336ffd640 1 --2- 172.21.15.99:0/406362630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380d1b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f03280097a0 tx=0x7f032802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d9f3a522b042f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.767+0000 7f0335ffb640 1 -- 172.21.15.99:0/406362630 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0328037070 con 0x7f03380b5700 2024-03-04T10:06:36.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.767+0000 7f0335ffb640 1 -- 172.21.15.99:0/406362630 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0328030510 con 0x7f03380b5700 2024-03-04T10:06:36.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.767+0000 7f0335ffb640 1 -- 172.21.15.99:0/406362630 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c001670 con 0x7f03380b5700 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 -- 172.21.15.99:0/406362630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 msgr2=0x7f03380d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 --2- 172.21.15.99:0/406362630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380d1b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f03280097a0 tx=0x7f032802b710 comp rx=0 tx=0).stop 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 -- 172.21.15.99:0/406362630 shutdown_connections 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 --2- 172.21.15.99:0/406362630 >> v2:172.21.15.162:3300/0 conn(0x7f03380b9410 0x7f033806c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 --2- 172.21.15.99:0/406362630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380d1b00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 -- 172.21.15.99:0/406362630 >> 172.21.15.99:0/406362630 conn(0x7f03380d0490 msgr2=0x7f03380d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 -- 172.21.15.99:0/406362630 shutdown_connections 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.768+0000 7f033d44b640 1 -- 172.21.15.99:0/406362630 wait complete. 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.769+0000 7f033d44b640 1 Processor -- start 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.769+0000 7f033d44b640 1 -- start start 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f033d44b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f033d44b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03380b9410 0x7f033815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f033d44b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 0x7f03381f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f033d44b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033806ceb0 con 0x7f03380d34a0 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f033d44b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033806cd30 con 0x7f03380b9410 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f033d44b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03380b5ae0 con 0x7f03380b5700 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 0x7f03381f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 0x7f03381f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38082/0 (socket says 172.21.15.99:38082) 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 -- 172.21.15.99:0/3665988676 learned_addr learned my addr 172.21.15.99:0/3665988676 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f0336ffd640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03367fc640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03380b9410 0x7f033815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 -- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 msgr2=0x7f03380c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 -- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03380b9410 msgr2=0x7f033815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03380b9410 0x7f033815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 -- 172.21.15.99:0/3665988676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033815dc60 con 0x7f03380d34a0 2024-03-04T10:06:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.770+0000 7f03377fe640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 0x7f03381f2f30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0330009dd0 tx=0x7f033000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.771+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330018020 con 0x7f03380d34a0 2024-03-04T10:06:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.771+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0330014480 con 0x7f03380d34a0 2024-03-04T10:06:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.771+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330014870 con 0x7f03380d34a0 2024-03-04T10:06:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.771+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033815cbc0 con 0x7f03380d34a0 2024-03-04T10:06:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.771+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033815d180 con 0x7f03380d34a0 2024-03-04T10:06:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.772+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03380d16c0 con 0x7f03380d34a0 2024-03-04T10:06:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.772+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0330005d10 con 0x7f03380d34a0 2024-03-04T10:06:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.772+0000 7f031f7fe640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f032406b4d0 0x7f032406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.773+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f033008ad50 con 0x7f03380d34a0 2024-03-04T10:06:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.773+0000 7f0336ffd640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f032406b4d0 0x7f032406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.773+0000 7f0336ffd640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f032406b4d0 0x7f032406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0328034000 tx=0x7f0328008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:36.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.777+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f03380d16c0 con 0x7f03380d34a0 2024-03-04T10:06:36.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:36.986+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0338074f60 con 0x7f03380d34a0 2024-03-04T10:06:37.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.025+0000 7f031f7fe640 1 -- 172.21.15.99:0/3665988676 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v42) v1 ==== 118+0+0 (secure 0 0 0) 0x7f033008e030 con 0x7f03380d34a0 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.027+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f032406b4d0 msgr2=0x7f032406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.027+0000 7f033d44b640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f032406b4d0 0x7f032406d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0328034000 tx=0x7f0328008030 comp rx=0 tx=0).stop 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 msgr2=0x7f03381f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 0x7f03381f2f30 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0330009dd0 tx=0x7f033000c490 comp rx=0 tx=0).stop 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 shutdown_connections 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f032406b4d0 0x7f032406d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f03380d34a0 0x7f03381f2f30 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f03380b9410 0x7f033815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 --2- 172.21.15.99:0/3665988676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f03380b5700 0x7f03380c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 >> 172.21.15.99:0/3665988676 conn(0x7f03380d0490 msgr2=0x7f0338160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:37.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 shutdown_connections 2024-03-04T10:06:37.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.028+0000 7f033d44b640 1 -- 172.21.15.99:0/3665988676 wait complete. 2024-03-04T10:06:37.042 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T10:06:37.042 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T10:06:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.143+0000 7f44af359640 1 Processor -- start 2024-03-04T10:06:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.144+0000 7f44af359640 1 -- start start 2024-03-04T10:06:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.144+0000 7f44af359640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 0x7f44a815d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.144+0000 7f44af359640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a806ae00 con 0x7f44a80cdd70 2024-03-04T10:06:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.144+0000 7f44af359640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a806b0e0 con 0x7f44a80ccfe0 2024-03-04T10:06:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.144+0000 7f44af359640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80ccde0 con 0x7f44a80cc900 2024-03-04T10:06:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f44ad0f5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f44a80cdd70 0x7f44a80cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37346/0 (socket says 172.21.15.99:37346) 2024-03-04T10:06:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f44ad0f5640 1 -- 172.21.15.99:0/703686591 learned_addr learned my addr 172.21.15.99:0/703686591 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208157508 0 0) 0x7f44a806ae00 con 0x7f44a80cdd70 2024-03-04T10:06:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4494003520 con 0x7f44a80cdd70 2024-03-04T10:06:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 0x7f44a815d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917098999 0 0) 0x7f4494003520 con 0x7f44a80cdd70 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 msgr2=0x7f44a815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 0x7f44a815d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 >> v1:172.21.15.162:6789/0 conn(0x7f44a80ccfe0 legacy=0x7f44a815d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a806b3c0 con 0x7f44a80cdd70 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f4498001670 con 0x7f44a80cdd70 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 >> v1:172.21.15.99:6789/0 conn(0x7f44a80cdd70 legacy=0x7f44a80cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 0x7f44940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4494004060 0x7f4494004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4494004d40 0x7f44940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80ccde0 con 0x7f4494004060 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449400bca0 con 0x7f4494004d40 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449400bea0 con 0x7f44a80cc900 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4494004060 0x7f4494004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f44ad0f5640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 0x7f44940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.145+0000 7f44ad8f6640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4494004d40 0x7f44940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ac8f4640 1 -- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 msgr2=0x7f44940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80cc900 0x7f44940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ac8f4640 1 -- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4494004d40 msgr2=0x7f44940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4494004d40 0x7f44940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ac8f4640 1 -- 172.21.15.99:0/703686591 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494003520 con 0x7f4494004060 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ad8f6640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4494004d40 0x7f44940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4494004060 0x7f4494004800 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f44a400ba10 tx=0x7f4494005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70702e7de870c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.146+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a4036020 con 0x7f4494004060 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a4032660 con 0x7f4494004060 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f449ffff640 1 -- 172.21.15.99:0/703686591 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a400b5e0 con 0x7f4494004060 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44af359640 1 -- 172.21.15.99:0/703686591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4494004060 msgr2=0x7f4494004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44af359640 1 --2- 172.21.15.99:0/703686591 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4494004060 0x7f4494004800 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f44a400ba10 tx=0x7f4494005830 comp rx=0 tx=0).stop 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44ad8f6640 1 -- 172.21.15.99:0/703686591 reap_dead start 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44af359640 1 -- 172.21.15.99:0/703686591 shutdown_connections 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44af359640 1 -- 172.21.15.99:0/703686591 >> 172.21.15.99:0/703686591 conn(0x7f44a80b22e0 msgr2=0x7f44a80b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44af359640 1 -- 172.21.15.99:0/703686591 shutdown_connections 2024-03-04T10:06:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.147+0000 7f44af359640 1 -- 172.21.15.99:0/703686591 wait complete. 2024-03-04T10:06:37.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.148+0000 7f44af359640 1 Processor -- start 2024-03-04T10:06:37.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.148+0000 7f44af359640 1 -- start start 2024-03-04T10:06:37.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44af359640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 0x7f44a80c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44af359640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80c7fb0 0x7f44a80ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44af359640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44a80aec10 0x7f44a81f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44af359640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a815e1e0 con 0x7f44a8152fc0 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44af359640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a815e060 con 0x7f44a80aec10 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44af359640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80b2e90 con 0x7f44a80c7fb0 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 0x7f44a80c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 0x7f44a80c7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38106/0 (socket says 172.21.15.99:38106) 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 -- 172.21.15.99:0/3829495148 learned_addr learned my addr 172.21.15.99:0/3829495148 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80c7fb0 0x7f44a80ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad8f6640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44a80aec10 0x7f44a81f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 -- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80c7fb0 msgr2=0x7f44a80ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80c7fb0 0x7f44a80ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 -- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44a80aec10 msgr2=0x7f44a81f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44a80aec10 0x7f44a81f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.149+0000 7f44ad0f5640 1 -- 172.21.15.99:0/3829495148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80c84c0 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.150+0000 7f44ad0f5640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 0x7f44a80c7a70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f44980072c0 tx=0x7f4498007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.150+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449800f960 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.150+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4498018070 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.150+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449800fd40 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.150+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a80c87a0 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.150+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a80c8d60 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.151+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a80c8fa0 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.151+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f449801d460 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.151+0000 7f449dffb640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f448c06b500 0x7f448c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.152+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f449808a410 con 0x7f44a8152fc0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.152+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f448c06b500 0x7f448c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:37.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.152+0000 7f44ac8f4640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f448c06b500 0x7f448c06d9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f44a400b9e0 tx=0x7f44a403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:37.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.156+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4498059100 con 0x7f44a8152fc0 2024-03-04T10:06:37.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:37.363+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f44a80ce480 con 0x7f44a8152fc0 2024-03-04T10:06:38.005 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:06:38.003+0000 7f1ebe01b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.037+0000 7f449dffb640 1 -- 172.21.15.99:0/3829495148 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4498058a00 con 0x7f44a8152fc0 2024-03-04T10:06:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.039+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f448c06b500 msgr2=0x7f448c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.039+0000 7f44af359640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f448c06b500 0x7f448c06d9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f44a400b9e0 tx=0x7f44a403d040 comp rx=0 tx=0).stop 2024-03-04T10:06:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.039+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 msgr2=0x7f44a80c7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.039+0000 7f44af359640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 0x7f44a80c7a70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f44980072c0 tx=0x7f4498007780 comp rx=0 tx=0).stop 2024-03-04T10:06:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 shutdown_connections 2024-03-04T10:06:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f448c06b500 0x7f448c06d9c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44a80aec10 0x7f44a81f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44a80c7fb0 0x7f44a80ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 --2- 172.21.15.99:0/3829495148 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44a8152fc0 0x7f44a80c7a70 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 >> 172.21.15.99:0/3829495148 conn(0x7f44a80b22e0 msgr2=0x7f44a806eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 shutdown_connections 2024-03-04T10:06:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.040+0000 7f44af359640 1 -- 172.21.15.99:0/3829495148 wait complete. 2024-03-04T10:06:38.054 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T10:06:38.054 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:38.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.154+0000 7f595e90a640 1 Processor -- start 2024-03-04T10:06:38.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f595e90a640 1 -- start start 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f595e90a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f595815bb20 0x7f595815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f595e90a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595806d8b0 con 0x7f595806d3b0 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f595e90a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595806db90 con 0x7f595815b030 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f595e90a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59580b2720 con 0x7f595815bb20 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f59577fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f595806d3b0 0x7f595815b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37354/0 (socket says 172.21.15.99:37354) 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.155+0000 7f59577fe640 1 -- 172.21.15.99:0/2734734805 learned_addr learned my addr 172.21.15.99:0/2734734805 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369740379 0 0) 0x7f595806d8b0 con 0x7f595806d3b0 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5940003520 con 0x7f595806d3b0 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 --2- 172.21.15.99:0/2734734805 >> v2:172.21.15.162:3300/0 conn(0x7f595815bb20 0x7f595815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945216169 0 0) 0x7f5940003520 con 0x7f595806d3b0 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 >> v2:172.21.15.162:3300/0 conn(0x7f595815bb20 msgr2=0x7f595815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 --2- 172.21.15.99:0/2734734805 >> v2:172.21.15.162:3300/0 conn(0x7f595815bb20 0x7f595815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 >> v1:172.21.15.162:6789/0 conn(0x7f595815b030 legacy=0x7f595815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595806de70 con 0x7f595806d3b0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5948001670 con 0x7f595806d3b0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 >> v1:172.21.15.99:6789/0 conn(0x7f595806d3b0 legacy=0x7f595815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59400039b0 0x7f5940003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5940004800 0x7f5940004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f594000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59580b2720 con 0x7f59400053c0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594000c250 con 0x7f5940004800 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594000c450 con 0x7f59400039b0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f594000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f59577fe640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5940004800 0x7f5940004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f5957fff640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59400039b0 0x7f5940003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 -- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59400039b0 msgr2=0x7f5940003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59400039b0 0x7f5940003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 -- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5940004800 msgr2=0x7f5940004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5940004800 0x7f5940004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.156+0000 7f595cea7640 1 -- 172.21.15.99:0/2734734805 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5940003520 con 0x7f59400053c0 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.157+0000 7f595cea7640 1 -- 172.21.15.99:0/2734734805 reap_dead start 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.157+0000 7f595cea7640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f594000b9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5950007360 tx=0x7f5940005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b67b460dd4a2194 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.157+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950009be0 con 0x7f59400053c0 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.157+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595002ccf0 con 0x7f59400053c0 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.157+0000 7f5956ffd640 1 -- 172.21.15.99:0/2734734805 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950003bd0 con 0x7f59400053c0 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 -- 172.21.15.99:0/2734734805 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 msgr2=0x7f594000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f594000b9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5950007360 tx=0x7f5940005830 comp rx=0 tx=0).stop 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 -- 172.21.15.99:0/2734734805 shutdown_connections 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 --2- 172.21.15.99:0/2734734805 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f594000b9c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 -- 172.21.15.99:0/2734734805 >> 172.21.15.99:0/2734734805 conn(0x7f595806acd0 msgr2=0x7f595806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 -- 172.21.15.99:0/2734734805 shutdown_connections 2024-03-04T10:06:38.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.158+0000 7f595e90a640 1 -- 172.21.15.99:0/2734734805 wait complete. 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.159+0000 7f595e90a640 1 Processor -- start 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.159+0000 7f595e90a640 1 -- start start 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595e90a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f5958157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595e90a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59581f5410 0x7f5958140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595e90a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5958141380 0x7f5958141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595e90a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595806e810 con 0x7f59400053c0 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595e90a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595806e690 con 0x7f5958141380 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595e90a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595815c120 con 0x7f59581f5410 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f5958157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f5958157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38126/0 (socket says 172.21.15.99:38126) 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 -- 172.21.15.99:0/1871578943 learned_addr learned my addr 172.21.15.99:0/1871578943 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f59577fe640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59581f5410 0x7f5958140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595cea7640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5958141380 0x7f5958141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 -- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59581f5410 msgr2=0x7f5958140e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59581f5410 0x7f5958140e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 -- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5958141380 msgr2=0x7f5958141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5958141380 0x7f5958141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 -- 172.21.15.99:0/1871578943 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59581f5890 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f595cea7640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5958141380 0x7f5958141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.160+0000 7f5957fff640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f5958157560 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f594c00f650 tx=0x7f594c00fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.161+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c0050d0 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.161+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594c018070 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.161+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594c005490 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.161+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5958157c90 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.161+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5958158250 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.162+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5958074f60 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.162+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f594c00d6d0 con 0x7f59400053c0 2024-03-04T10:06:38.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.162+0000 7f5954ff9640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f594406b4d0 0x7f594406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.162+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f594c08a0d0 con 0x7f59400053c0 2024-03-04T10:06:38.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.163+0000 7f59577fe640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f594406b4d0 0x7f594406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.163+0000 7f59577fe640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f594406b4d0 0x7f594406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f59480010e0 tx=0x7f5948009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.167+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f594c058dc0 con 0x7f59400053c0 2024-03-04T10:06:38.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.374+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59580cbeb0 con 0x7f59400053c0 2024-03-04T10:06:38.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.375+0000 7f5954ff9640 1 -- 172.21.15.99:0/1871578943 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f594c0586c0 con 0x7f59400053c0 2024-03-04T10:06:38.377 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:38.377 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":43,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:37.365114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6474},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6474":{"gid":6474,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/2687500379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":2687500379},{"type":"v1","addr":"172.21.15.99:6835","nonce":2687500379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:38.378 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 43 2024-03-04T10:06:38.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f594406b4d0 msgr2=0x7f594406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f594406b4d0 0x7f594406d990 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f59480010e0 tx=0x7f5948009040 comp rx=0 tx=0).stop 2024-03-04T10:06:38.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 msgr2=0x7f5958157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f5958157560 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f594c00f650 tx=0x7f594c00fb10 comp rx=0 tx=0).stop 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 shutdown_connections 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f594406b4d0 0x7f594406d990 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5958141380 0x7f5958141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f59581f5410 0x7f5958140e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 --2- 172.21.15.99:0/1871578943 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59400053c0 0x7f5958157560 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 >> 172.21.15.99:0/1871578943 conn(0x7f595806acd0 msgr2=0x7f59580b4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 shutdown_connections 2024-03-04T10:06:38.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.377+0000 7f595e90a640 1 -- 172.21.15.99:0/1871578943 wait complete. 2024-03-04T10:06:38.390 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:38.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.488+0000 7f6e88af7640 1 Processor -- start 2024-03-04T10:06:38.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.489+0000 7f6e88af7640 1 -- start start 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.489+0000 7f6e88af7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6e840cfd60 0x7f6e840cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.489+0000 7f6e88af7640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6e840cf0d0 0x7f6e840cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.489+0000 7f6e88af7640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840d0140 con 0x7f6e8406d3b0 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.489+0000 7f6e88af7640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8415cbb0 con 0x7f6e840cfd60 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.489+0000 7f6e88af7640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e8415c280 con 0x7f6e840cf0d0 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e8259c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6e840cfd60 0x7f6e840cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e81d9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6e8406d3b0 0x7f6e840ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37358/0 (socket says 172.21.15.99:37358) 2024-03-04T10:06:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e81d9b640 1 -- 172.21.15.99:0/1885257422 learned_addr learned my addr 172.21.15.99:0/1885257422 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e8259c640 1 -- 172.21.15.99:0/1885257422 >> v2:172.21.15.162:3300/0 conn(0x7f6e840cf0d0 msgr2=0x7f6e840cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e8259c640 1 --2- 172.21.15.99:0/1885257422 >> v2:172.21.15.162:3300/0 conn(0x7f6e840cf0d0 0x7f6e840cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e8259c640 1 -- 172.21.15.99:0/1885257422 >> v1:172.21.15.99:6789/0 conn(0x7f6e8406d3b0 legacy=0x7f6e840ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e8159a640 1 -- 172.21.15.99:0/1885257422 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136221284 0 0) 0x7f6e840d0140 con 0x7f6e8406d3b0 2024-03-04T10:06:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.490+0000 7f6e8259c640 1 -- 172.21.15.99:0/1885257422 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8415c560 con 0x7f6e840cfd60 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e8259c640 1 --2- 172.21.15.99:0/1885257422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cfd60 0x7f6e840cbdf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6e8415cd30 tx=0x7f6e6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3abc4814535012f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e8159a640 1 -- 172.21.15.99:0/1885257422 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c035020 con 0x7f6e840cfd60 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e8159a640 1 -- 172.21.15.99:0/1885257422 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c031830 con 0x7f6e840cfd60 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e8159a640 1 -- 172.21.15.99:0/1885257422 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c031c00 con 0x7f6e840cfd60 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 -- 172.21.15.99:0/1885257422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cfd60 msgr2=0x7f6e840cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 --2- 172.21.15.99:0/1885257422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cfd60 0x7f6e840cbdf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6e8415cd30 tx=0x7f6e6c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 -- 172.21.15.99:0/1885257422 shutdown_connections 2024-03-04T10:06:38.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 --2- 172.21.15.99:0/1885257422 >> v2:172.21.15.162:3300/0 conn(0x7f6e840cf0d0 0x7f6e840cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 --2- 172.21.15.99:0/1885257422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cfd60 0x7f6e840cbdf0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 -- 172.21.15.99:0/1885257422 >> 172.21.15.99:0/1885257422 conn(0x7f6e8406acd0 msgr2=0x7f6e8406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 -- 172.21.15.99:0/1885257422 shutdown_connections 2024-03-04T10:06:38.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.491+0000 7f6e88af7640 1 -- 172.21.15.99:0/1885257422 wait complete. 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 Processor -- start 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 -- start start 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e8406d3b0 0x7f6e840afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 0x7f6e8414dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e840cfd60 0x7f6e840b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8415f520 con 0x7f6e840cf0d0 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8415bf30 con 0x7f6e8406d3b0 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.493+0000 7f6e88af7640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8415f6a0 con 0x7f6e840cfd60 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 0x7f6e8414dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 0x7f6e8414dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38152/0 (socket says 172.21.15.99:38152) 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 -- 172.21.15.99:0/938037816 learned_addr learned my addr 172.21.15.99:0/938037816 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:38.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 -- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e840cfd60 msgr2=0x7f6e840b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e840cfd60 0x7f6e840b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 -- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e8406d3b0 msgr2=0x7f6e840afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e8406d3b0 0x7f6e840afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 -- 172.21.15.99:0/938037816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e841f17f0 con 0x7f6e840cf0d0 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e81d9b640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 0x7f6e8414dff0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400c490 tx=0x7f6e7400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74018070 con 0x7f6e840cf0d0 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e740074e0 con 0x7f6e840cf0d0 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e841f1ad0 con 0x7f6e840cf0d0 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e74004a20 con 0x7f6e840cf0d0 2024-03-04T10:06:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.494+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e841f2090 con 0x7f6e840cf0d0 2024-03-04T10:06:38.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.495+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8414ec80 con 0x7f6e840cf0d0 2024-03-04T10:06:38.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.496+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6e7401b070 con 0x7f6e840cf0d0 2024-03-04T10:06:38.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.496+0000 7f6e7affd640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e6406b4d0 0x7f6e6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.496+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e74089550 con 0x7f6e840cf0d0 2024-03-04T10:06:38.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.496+0000 7f6e8259c640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e6406b4d0 0x7f6e6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.497+0000 7f6e8259c640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e6406b4d0 0x7f6e6406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c036000 tx=0x7f6e6c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.500+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6e740582d0 con 0x7f6e840cf0d0 2024-03-04T10:06:38.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.674+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e8415e090 con 0x7f6e840cf0d0 2024-03-04T10:06:38.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.675+0000 7f6e7affd640 1 -- 172.21.15.99:0/938037816 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16848 (secure 0 0 0) 0x7f6e74057bd0 con 0x7f6e840cf0d0 2024-03-04T10:06:38.678 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:38.678 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":66,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:37.015810+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e6406b4d0 msgr2=0x7f6e6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e6406b4d0 0x7f6e6406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c036000 tx=0x7f6e6c030000 comp rx=0 tx=0).stop 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 msgr2=0x7f6e8414dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 0x7f6e8414dff0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400c490 tx=0x7f6e7400c950 comp rx=0 tx=0).stop 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 shutdown_connections 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e6406b4d0 0x7f6e6406d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e840cfd60 0x7f6e840b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e840cf0d0 0x7f6e8414dff0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 --2- 172.21.15.99:0/938037816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e8406d3b0 0x7f6e840afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.677+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 >> 172.21.15.99:0/938037816 conn(0x7f6e8406acd0 msgr2=0x7f6e84159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.678+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 shutdown_connections 2024-03-04T10:06:38.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.678+0000 7f6e88af7640 1 -- 172.21.15.99:0/938037816 wait complete. 2024-03-04T10:06:38.691 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T10:06:38.691 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:38.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.789+0000 7f8511868640 1 Processor -- start 2024-03-04T10:06:38.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.789+0000 7f8511868640 1 -- start start 2024-03-04T10:06:38.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f8511868640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f850c15de00 0x7f850c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f8511868640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f8511868640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c15e1e0 con 0x7f850c0b70d0 2024-03-04T10:06:38.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f8511868640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c0d2820 con 0x7f850c15de00 2024-03-04T10:06:38.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f8511868640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0d1c10 con 0x7f850c0b9380 2024-03-04T10:06:38.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f850b7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f850c0b70d0 0x7f850c0b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37370/0 (socket says 172.21.15.99:37370) 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f850b7fe640 1 -- 172.21.15.99:0/2705343992 learned_addr learned my addr 172.21.15.99:0/2705343992 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f85027fc640 1 --2- 172.21.15.99:0/2705343992 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.790+0000 7f850affd640 1 --2- 172.21.15.99:0/2705343992 >> v2:172.21.15.162:3300/0 conn(0x7f850c15de00 0x7f850c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.791+0000 7f850a7fc640 1 -- 172.21.15.99:0/2705343992 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737285022 0 0) 0x7f850c15e1e0 con 0x7f850c0b70d0 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.791+0000 7f850a7fc640 1 -- 172.21.15.99:0/2705343992 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f8003520 con 0x7f850c0b70d0 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.791+0000 7f85027fc640 1 -- 172.21.15.99:0/2705343992 >> v2:172.21.15.162:3300/0 conn(0x7f850c15de00 msgr2=0x7f850c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.791+0000 7f85027fc640 1 --2- 172.21.15.99:0/2705343992 >> v2:172.21.15.162:3300/0 conn(0x7f850c15de00 0x7f850c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.791+0000 7f85027fc640 1 -- 172.21.15.99:0/2705343992 >> v1:172.21.15.99:6789/0 conn(0x7f850c0b70d0 legacy=0x7f850c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.791+0000 7f85027fc640 1 -- 172.21.15.99:0/2705343992 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0d1ef0 con 0x7f850c0b9380 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f85027fc640 1 --2- 172.21.15.99:0/2705343992 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0b6b90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84f40097a0 tx=0x7f84f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee7d0446e95923e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f850a7fc640 1 -- 172.21.15.99:0/2705343992 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4035020 con 0x7f850c0b9380 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f850a7fc640 1 -- 172.21.15.99:0/2705343992 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4031830 con 0x7f850c0b9380 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f850a7fc640 1 -- 172.21.15.99:0/2705343992 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4031be0 con 0x7f850c0b9380 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 -- 172.21.15.99:0/2705343992 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 msgr2=0x7f850c0b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 --2- 172.21.15.99:0/2705343992 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0b6b90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84f40097a0 tx=0x7f84f402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 -- 172.21.15.99:0/2705343992 shutdown_connections 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 --2- 172.21.15.99:0/2705343992 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0b6b90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 --2- 172.21.15.99:0/2705343992 >> v2:172.21.15.162:3300/0 conn(0x7f850c15de00 0x7f850c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 -- 172.21.15.99:0/2705343992 >> 172.21.15.99:0/2705343992 conn(0x7f850c0d0470 msgr2=0x7f850c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 -- 172.21.15.99:0/2705343992 shutdown_connections 2024-03-04T10:06:38.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.792+0000 7f8511868640 1 -- 172.21.15.99:0/2705343992 wait complete. 2024-03-04T10:06:38.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.793+0000 7f8511868640 1 Processor -- start 2024-03-04T10:06:38.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 -- start start 2024-03-04T10:06:38.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 0x7f850c0cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f850c15de00 0x7f850c0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0b97e0 con 0x7f850c0b70d0 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0d24b0 con 0x7f850c15de00 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f8511868640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0b5b80 con 0x7f850c0b9380 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f850affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 0x7f850c0cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f850affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 0x7f850c0cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38166/0 (socket says 172.21.15.99:38166) 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.794+0000 7f850affd640 1 -- 172.21.15.99:0/359712482 learned_addr learned my addr 172.21.15.99:0/359712482 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850b7fe640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f850c15de00 0x7f850c0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f85027fc640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850affd640 1 -- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 msgr2=0x7f850c0cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850affd640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850affd640 1 -- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f850c15de00 msgr2=0x7f850c0cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850affd640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f850c15de00 0x7f850c0cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850affd640 1 -- 172.21.15.99:0/359712482 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c0b4960 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f850affd640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 0x7f850c0cb3d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00e8a0 tx=0x7f84fc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc017070 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84fc00c940 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c0b4c00 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84fc00cd30 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.795+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c0b51c0 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.796+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c074f60 con 0x7f850c0b70d0 2024-03-04T10:06:38.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.796+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f84fc01a070 con 0x7f850c0b70d0 2024-03-04T10:06:38.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.797+0000 7f8508ff9640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:38.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.797+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f84fc08a5d0 con 0x7f850c0b70d0 2024-03-04T10:06:38.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.797+0000 7f85027fc640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:38.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.798+0000 7f85027fc640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f84ec06b4d0 0x7f84ec06d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f84f4036000 tx=0x7f84f4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:38.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.801+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f84fc0592d0 con 0x7f850c0b70d0 2024-03-04T10:06:38.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.967+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f850c0cdab0 con 0x7f84ec06b4d0 2024-03-04T10:06:38.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.974+0000 7f8508ff9640 1 -- 172.21.15.99:0/359712482 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252004 (secure 0 0 0) 0x7f850c0cdab0 con 0x7f84ec06b4d0 2024-03-04T10:06:38.979 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:38.981 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f84ec06b4d0 msgr2=0x7f84ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f84ec06b4d0 0x7f84ec06d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f84f4036000 tx=0x7f84f4008030 comp rx=0 tx=0).stop 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 msgr2=0x7f850c0cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 0x7f850c0cb3d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00e8a0 tx=0x7f84fc00ed60 comp rx=0 tx=0).stop 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 shutdown_connections 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f84ec06b4d0 0x7f84ec06d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f850c15de00 0x7f850c0cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f850c0b9380 0x7f850c0cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 --2- 172.21.15.99:0/359712482 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f850c0b70d0 0x7f850c0cb3d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 >> 172.21.15.99:0/359712482 conn(0x7f850c0d0470 msgr2=0x7f850c0cf0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 shutdown_connections 2024-03-04T10:06:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:38.977+0000 7f8511868640 1 -- 172.21.15.99:0/359712482 wait complete. 2024-03-04T10:06:38.990 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-04T10:06:37.480325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223580,"kb_used_data":7380,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751140,"statfs":{"total":773094113280,"available":772865167360,"internally_reserved":0,"allocated":7557120,"data_stored":4058120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000888"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.042546+0000","last_change":"2024-03-04T10:06:36.036268+0000","last_active":"2024-03-04T10:06:36.042546+0000","last_peered":"2024-03-04T10:06:36.042546+0000","last_clean":"2024-03-04T10:06:36.042546+0000","last_became_active":"2024-03-04T10:06:36.035997+0000","last_became_peered":"2024-03-04T10:06:36.035997+0000","last_unstale":"2024-03-04T10:06:36.042546+0000","last_undegraded":"2024-03-04T10:06:36.042546+0000","last_fullsized":"2024-03-04T10:06:36.042546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:37.097508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.037160+0000","last_change":"2024-03-04T10:06:36.033385+0000","last_active":"2024-03-04T10:06:36.037160+0000","last_peered":"2024-03-04T10:06:36.037160+0000","last_clean":"2024-03-04T10:06:36.037160+0000","last_became_active":"2024-03-04T10:06:36.033125+0000","last_became_peered":"2024-03-04T10:06:36.033125+0000","last_unstale":"2024-03-04T10:06:36.037160+0000","last_undegraded":"2024-03-04T10:06:36.037160+0000","last_fullsized":"2024-03-04T10:06:36.037160+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:24.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021598+0000","last_change":"2024-03-04T10:06:36.031779+0000","last_active":"2024-03-04T10:06:37.021598+0000","last_peered":"2024-03-04T10:06:37.021598+0000","last_clean":"2024-03-04T10:06:37.021598+0000","last_became_active":"2024-03-04T10:06:36.031538+0000","last_became_peered":"2024-03-04T10:06:36.031538+0000","last_unstale":"2024-03-04T10:06:37.021598+0000","last_undegraded":"2024-03-04T10:06:37.021598+0000","last_fullsized":"2024-03-04T10:06:37.021598+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:59.436908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021535+0000","last_change":"2024-03-04T10:06:36.027369+0000","last_active":"2024-03-04T10:06:37.021535+0000","last_peered":"2024-03-04T10:06:37.021535+0000","last_clean":"2024-03-04T10:06:37.021535+0000","last_became_active":"2024-03-04T10:06:36.027094+0000","last_became_peered":"2024-03-04T10:06:36.027094+0000","last_unstale":"2024-03-04T10:06:37.021535+0000","last_undegraded":"2024-03-04T10:06:37.021535+0000","last_fullsized":"2024-03-04T10:06:37.021535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:26:17.271066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120664+0000","last_change":"2024-03-04T10:06:36.036692+0000","last_active":"2024-03-04T10:06:37.120664+0000","last_peered":"2024-03-04T10:06:37.120664+0000","last_clean":"2024-03-04T10:06:37.120664+0000","last_became_active":"2024-03-04T10:06:36.036399+0000","last_became_peered":"2024-03-04T10:06:36.036399+0000","last_unstale":"2024-03-04T10:06:37.120664+0000","last_undegraded":"2024-03-04T10:06:37.120664+0000","last_fullsized":"2024-03-04T10:06:37.120664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:01.971149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019803+0000","last_change":"2024-03-04T10:06:36.043253+0000","last_active":"2024-03-04T10:06:37.019803+0000","last_peered":"2024-03-04T10:06:37.019803+0000","last_clean":"2024-03-04T10:06:37.019803+0000","last_became_active":"2024-03-04T10:06:36.038732+0000","last_became_peered":"2024-03-04T10:06:36.038732+0000","last_unstale":"2024-03-04T10:06:37.019803+0000","last_undegraded":"2024-03-04T10:06:37.019803+0000","last_fullsized":"2024-03-04T10:06:37.019803+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:10.646857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035170+0000","last_change":"2024-03-04T10:06:36.045118+0000","last_active":"2024-03-04T10:06:37.035170+0000","last_peered":"2024-03-04T10:06:37.035170+0000","last_clean":"2024-03-04T10:06:37.035170+0000","last_became_active":"2024-03-04T10:06:36.044807+0000","last_became_peered":"2024-03-04T10:06:36.044807+0000","last_unstale":"2024-03-04T10:06:37.035170+0000","last_undegraded":"2024-03-04T10:06:37.035170+0000","last_fullsized":"2024-03-04T10:06:37.035170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:49.322018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.032870+0000","last_change":"2024-03-04T10:06:36.040694+0000","last_active":"2024-03-04T10:06:37.032870+0000","last_peered":"2024-03-04T10:06:37.032870+0000","last_clean":"2024-03-04T10:06:37.032870+0000","last_became_active":"2024-03-04T10:06:36.040382+0000","last_became_peered":"2024-03-04T10:06:36.040382+0000","last_unstale":"2024-03-04T10:06:37.032870+0000","last_undegraded":"2024-03-04T10:06:37.032870+0000","last_fullsized":"2024-03-04T10:06:37.032870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:51.700460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022417+0000","last_change":"2024-03-04T10:06:36.037253+0000","last_active":"2024-03-04T10:06:37.022417+0000","last_peered":"2024-03-04T10:06:37.022417+0000","last_clean":"2024-03-04T10:06:37.022417+0000","last_became_active":"2024-03-04T10:06:36.036744+0000","last_became_peered":"2024-03-04T10:06:36.036744+0000","last_unstale":"2024-03-04T10:06:37.022417+0000","last_undegraded":"2024-03-04T10:06:37.022417+0000","last_fullsized":"2024-03-04T10:06:37.022417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:56.906275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021730+0000","last_change":"2024-03-04T10:06:36.035909+0000","last_active":"2024-03-04T10:06:37.021730+0000","last_peered":"2024-03-04T10:06:37.021730+0000","last_clean":"2024-03-04T10:06:37.021730+0000","last_became_active":"2024-03-04T10:06:36.035646+0000","last_became_peered":"2024-03-04T10:06:36.035646+0000","last_unstale":"2024-03-04T10:06:37.021730+0000","last_undegraded":"2024-03-04T10:06:37.021730+0000","last_fullsized":"2024-03-04T10:06:37.021730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:47.538586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.020905+0000","last_change":"2024-03-04T10:06:36.029432+0000","last_active":"2024-03-04T10:06:37.020905+0000","last_peered":"2024-03-04T10:06:37.020905+0000","last_clean":"2024-03-04T10:06:37.020905+0000","last_became_active":"2024-03-04T10:06:36.029159+0000","last_became_peered":"2024-03-04T10:06:36.029159+0000","last_unstale":"2024-03-04T10:06:37.020905+0000","last_undegraded":"2024-03-04T10:06:37.020905+0000","last_fullsized":"2024-03-04T10:06:37.020905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:25.251674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033055+0000","last_change":"2024-03-04T10:06:36.039821+0000","last_active":"2024-03-04T10:06:37.033055+0000","last_peered":"2024-03-04T10:06:37.033055+0000","last_clean":"2024-03-04T10:06:37.033055+0000","last_became_active":"2024-03-04T10:06:36.039510+0000","last_became_peered":"2024-03-04T10:06:36.039510+0000","last_unstale":"2024-03-04T10:06:37.033055+0000","last_undegraded":"2024-03-04T10:06:37.033055+0000","last_fullsized":"2024-03-04T10:06:37.033055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:30.326871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033252+0000","last_change":"2024-03-04T10:06:36.032567+0000","last_active":"2024-03-04T10:06:37.033252+0000","last_peered":"2024-03-04T10:06:37.033252+0000","last_clean":"2024-03-04T10:06:37.033252+0000","last_became_active":"2024-03-04T10:06:36.032321+0000","last_became_peered":"2024-03-04T10:06:36.032321+0000","last_unstale":"2024-03-04T10:06:37.033252+0000","last_undegraded":"2024-03-04T10:06:37.033252+0000","last_fullsized":"2024-03-04T10:06:37.033252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:45.497167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021546+0000","last_change":"2024-03-04T10:06:36.034741+0000","last_active":"2024-03-04T10:06:37.021546+0000","last_peered":"2024-03-04T10:06:37.021546+0000","last_clean":"2024-03-04T10:06:37.021546+0000","last_became_active":"2024-03-04T10:06:36.034324+0000","last_became_peered":"2024-03-04T10:06:36.034324+0000","last_unstale":"2024-03-04T10:06:37.021546+0000","last_undegraded":"2024-03-04T10:06:37.021546+0000","last_fullsized":"2024-03-04T10:06:37.021546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:27.353573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021376+0000","last_change":"2024-03-04T10:06:36.039836+0000","last_active":"2024-03-04T10:06:37.021376+0000","last_peered":"2024-03-04T10:06:37.021376+0000","last_clean":"2024-03-04T10:06:37.021376+0000","last_became_active":"2024-03-04T10:06:36.039475+0000","last_became_peered":"2024-03-04T10:06:36.039475+0000","last_unstale":"2024-03-04T10:06:37.021376+0000","last_undegraded":"2024-03-04T10:06:37.021376+0000","last_fullsized":"2024-03-04T10:06:37.021376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:45.497167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035032+0000","last_change":"2024-03-04T10:06:36.032778+0000","last_active":"2024-03-04T10:06:37.035032+0000","last_peered":"2024-03-04T10:06:37.035032+0000","last_clean":"2024-03-04T10:06:37.035032+0000","last_became_active":"2024-03-04T10:06:36.032515+0000","last_became_peered":"2024-03-04T10:06:36.032515+0000","last_unstale":"2024-03-04T10:06:37.035032+0000","last_undegraded":"2024-03-04T10:06:37.035032+0000","last_fullsized":"2024-03-04T10:06:37.035032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:31.861841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.042518+0000","last_change":"2024-03-04T10:06:36.035173+0000","last_active":"2024-03-04T10:06:36.042518+0000","last_peered":"2024-03-04T10:06:36.042518+0000","last_clean":"2024-03-04T10:06:36.042518+0000","last_became_active":"2024-03-04T10:06:36.034740+0000","last_became_peered":"2024-03-04T10:06:36.034740+0000","last_unstale":"2024-03-04T10:06:36.042518+0000","last_undegraded":"2024-03-04T10:06:36.042518+0000","last_fullsized":"2024-03-04T10:06:36.042518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:07:12.791348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021242+0000","last_change":"2024-03-04T10:06:36.028100+0000","last_active":"2024-03-04T10:06:37.021242+0000","last_peered":"2024-03-04T10:06:37.021242+0000","last_clean":"2024-03-04T10:06:37.021242+0000","last_became_active":"2024-03-04T10:06:36.027819+0000","last_became_peered":"2024-03-04T10:06:36.027819+0000","last_unstale":"2024-03-04T10:06:37.021242+0000","last_undegraded":"2024-03-04T10:06:37.021242+0000","last_fullsized":"2024-03-04T10:06:37.021242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:03:37.046386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120511+0000","last_change":"2024-03-04T10:06:36.036128+0000","last_active":"2024-03-04T10:06:37.120511+0000","last_peered":"2024-03-04T10:06:37.120511+0000","last_clean":"2024-03-04T10:06:37.120511+0000","last_became_active":"2024-03-04T10:06:36.035846+0000","last_became_peered":"2024-03-04T10:06:36.035846+0000","last_unstale":"2024-03-04T10:06:37.120511+0000","last_undegraded":"2024-03-04T10:06:37.120511+0000","last_fullsized":"2024-03-04T10:06:37.120511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:58.051198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.033732+0000","last_change":"2024-03-04T10:06:36.030708+0000","last_active":"2024-03-04T10:06:36.033732+0000","last_peered":"2024-03-04T10:06:36.033732+0000","last_clean":"2024-03-04T10:06:36.033732+0000","last_became_active":"2024-03-04T10:06:36.030446+0000","last_became_peered":"2024-03-04T10:06:36.030446+0000","last_unstale":"2024-03-04T10:06:36.033732+0000","last_undegraded":"2024-03-04T10:06:36.033732+0000","last_fullsized":"2024-03-04T10:06:36.033732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:02:41.703028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033418+0000","last_change":"2024-03-04T10:06:36.042625+0000","last_active":"2024-03-04T10:06:37.033418+0000","last_peered":"2024-03-04T10:06:37.033418+0000","last_clean":"2024-03-04T10:06:37.033418+0000","last_became_active":"2024-03-04T10:06:36.041205+0000","last_became_peered":"2024-03-04T10:06:36.041205+0000","last_unstale":"2024-03-04T10:06:37.033418+0000","last_undegraded":"2024-03-04T10:06:37.033418+0000","last_fullsized":"2024-03-04T10:06:37.033418+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:59.436908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.041220+0000","last_change":"2024-03-04T10:06:36.038934+0000","last_active":"2024-03-04T10:06:36.041220+0000","last_peered":"2024-03-04T10:06:36.041220+0000","last_clean":"2024-03-04T10:06:36.041220+0000","last_became_active":"2024-03-04T10:06:36.038651+0000","last_became_peered":"2024-03-04T10:06:36.038651+0000","last_unstale":"2024-03-04T10:06:36.041220+0000","last_undegraded":"2024-03-04T10:06:36.041220+0000","last_fullsized":"2024-03-04T10:06:36.041220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:56.906275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.043972+0000","last_change":"2024-03-04T10:06:36.043184+0000","last_active":"2024-03-04T10:06:36.043972+0000","last_peered":"2024-03-04T10:06:36.043972+0000","last_clean":"2024-03-04T10:06:36.043972+0000","last_became_active":"2024-03-04T10:06:36.042871+0000","last_became_peered":"2024-03-04T10:06:36.042871+0000","last_unstale":"2024-03-04T10:06:36.043972+0000","last_undegraded":"2024-03-04T10:06:36.043972+0000","last_fullsized":"2024-03-04T10:06:36.043972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:31.059597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021950+0000","last_change":"2024-03-04T10:06:36.037669+0000","last_active":"2024-03-04T10:06:37.021950+0000","last_peered":"2024-03-04T10:06:37.021950+0000","last_clean":"2024-03-04T10:06:37.021950+0000","last_became_active":"2024-03-04T10:06:36.037361+0000","last_became_peered":"2024-03-04T10:06:36.037361+0000","last_unstale":"2024-03-04T10:06:37.021950+0000","last_undegraded":"2024-03-04T10:06:37.021950+0000","last_fullsized":"2024-03-04T10:06:37.021950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:04:43.724815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033090+0000","last_change":"2024-03-04T10:06:36.037306+0000","last_active":"2024-03-04T10:06:37.033090+0000","last_peered":"2024-03-04T10:06:37.033090+0000","last_clean":"2024-03-04T10:06:37.033090+0000","last_became_active":"2024-03-04T10:06:36.036798+0000","last_became_peered":"2024-03-04T10:06:36.036798+0000","last_unstale":"2024-03-04T10:06:37.033090+0000","last_undegraded":"2024-03-04T10:06:37.033090+0000","last_fullsized":"2024-03-04T10:06:37.033090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:40.312621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033321+0000","last_change":"2024-03-04T10:06:36.034622+0000","last_active":"2024-03-04T10:06:37.033321+0000","last_peered":"2024-03-04T10:06:37.033321+0000","last_clean":"2024-03-04T10:06:37.033321+0000","last_became_active":"2024-03-04T10:06:36.034314+0000","last_became_peered":"2024-03-04T10:06:36.034314+0000","last_unstale":"2024-03-04T10:06:37.033321+0000","last_undegraded":"2024-03-04T10:06:37.033321+0000","last_fullsized":"2024-03-04T10:06:37.033321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:27.353573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021693+0000","last_change":"2024-03-04T10:06:36.027163+0000","last_active":"2024-03-04T10:06:37.021693+0000","last_peered":"2024-03-04T10:06:37.021693+0000","last_clean":"2024-03-04T10:06:37.021693+0000","last_became_active":"2024-03-04T10:06:36.026865+0000","last_became_peered":"2024-03-04T10:06:36.026865+0000","last_unstale":"2024-03-04T10:06:37.021693+0000","last_undegraded":"2024-03-04T10:06:37.021693+0000","last_fullsized":"2024-03-04T10:06:37.021693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:48.186426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.045615+0000","last_change":"2024-03-04T10:06:36.043676+0000","last_active":"2024-03-04T10:06:36.045615+0000","last_peered":"2024-03-04T10:06:36.045615+0000","last_clean":"2024-03-04T10:06:36.045615+0000","last_became_active":"2024-03-04T10:06:36.042755+0000","last_became_peered":"2024-03-04T10:06:36.042755+0000","last_unstale":"2024-03-04T10:06:36.045615+0000","last_undegraded":"2024-03-04T10:06:36.045615+0000","last_fullsized":"2024-03-04T10:06:36.045615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:58.051198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022236+0000","last_change":"2024-03-04T10:06:36.034774+0000","last_active":"2024-03-04T10:06:37.022236+0000","last_peered":"2024-03-04T10:06:37.022236+0000","last_clean":"2024-03-04T10:06:37.022236+0000","last_became_active":"2024-03-04T10:06:36.034434+0000","last_became_peered":"2024-03-04T10:06:36.034434+0000","last_unstale":"2024-03-04T10:06:37.022236+0000","last_undegraded":"2024-03-04T10:06:37.022236+0000","last_fullsized":"2024-03-04T10:06:37.022236+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:02:41.703028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.032929+0000","last_change":"2024-03-04T10:06:36.038973+0000","last_active":"2024-03-04T10:06:37.032929+0000","last_peered":"2024-03-04T10:06:37.032929+0000","last_clean":"2024-03-04T10:06:37.032929+0000","last_became_active":"2024-03-04T10:06:36.038701+0000","last_became_peered":"2024-03-04T10:06:36.038701+0000","last_unstale":"2024-03-04T10:06:37.032929+0000","last_undegraded":"2024-03-04T10:06:37.032929+0000","last_fullsized":"2024-03-04T10:06:37.032929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:57.834882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021858+0000","last_change":"2024-03-04T10:06:36.028912+0000","last_active":"2024-03-04T10:06:37.021858+0000","last_peered":"2024-03-04T10:06:37.021858+0000","last_clean":"2024-03-04T10:06:37.021858+0000","last_became_active":"2024-03-04T10:06:36.028639+0000","last_became_peered":"2024-03-04T10:06:36.028639+0000","last_unstale":"2024-03-04T10:06:37.021858+0000","last_undegraded":"2024-03-04T10:06:37.021858+0000","last_fullsized":"2024-03-04T10:06:37.021858+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:24.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019459+0000","last_change":"2024-03-04T10:06:36.032489+0000","last_active":"2024-03-04T10:06:37.019459+0000","last_peered":"2024-03-04T10:06:37.019459+0000","last_clean":"2024-03-04T10:06:37.019459+0000","last_became_active":"2024-03-04T10:06:36.031925+0000","last_became_peered":"2024-03-04T10:06:36.031925+0000","last_unstale":"2024-03-04T10:06:37.019459+0000","last_undegraded":"2024-03-04T10:06:37.019459+0000","last_fullsized":"2024-03-04T10:06:37.019459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:26:21.380677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120496+0000","last_change":"2024-03-04T10:06:36.038811+0000","last_active":"2024-03-04T10:06:37.120496+0000","last_peered":"2024-03-04T10:06:37.120496+0000","last_clean":"2024-03-04T10:06:37.120496+0000","last_became_active":"2024-03-04T10:06:36.038541+0000","last_became_peered":"2024-03-04T10:06:36.038541+0000","last_unstale":"2024-03-04T10:06:37.120496+0000","last_undegraded":"2024-03-04T10:06:37.120496+0000","last_fullsized":"2024-03-04T10:06:37.120496+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:37.097508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"65'4","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.048498+0000","last_change":"2024-03-04T10:06:35.027878+0000","last_active":"2024-03-04T10:06:36.048498+0000","last_peered":"2024-03-04T10:06:36.048498+0000","last_clean":"2024-03-04T10:06:36.048498+0000","last_became_active":"2024-03-04T10:06:35.027509+0000","last_became_peered":"2024-03-04T10:06:35.027509+0000","last_unstale":"2024-03-04T10:06:36.048498+0000","last_undegraded":"2024-03-04T10:06:36.048498+0000","last_fullsized":"2024-03-04T10:06:36.048498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:50.371941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021880+0000","last_change":"2024-03-04T10:06:36.040950+0000","last_active":"2024-03-04T10:06:37.021880+0000","last_peered":"2024-03-04T10:06:37.021880+0000","last_clean":"2024-03-04T10:06:37.021880+0000","last_became_active":"2024-03-04T10:06:36.040241+0000","last_became_peered":"2024-03-04T10:06:36.040241+0000","last_unstale":"2024-03-04T10:06:37.021880+0000","last_undegraded":"2024-03-04T10:06:37.021880+0000","last_fullsized":"2024-03-04T10:06:37.021880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:07:12.791348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019930+0000","last_change":"2024-03-04T10:06:35.027865+0000","last_active":"2024-03-04T10:06:37.019930+0000","last_peered":"2024-03-04T10:06:37.019930+0000","last_clean":"2024-03-04T10:06:37.019930+0000","last_became_active":"2024-03-04T10:06:35.027582+0000","last_became_peered":"2024-03-04T10:06:35.027582+0000","last_unstale":"2024-03-04T10:06:37.019930+0000","last_undegraded":"2024-03-04T10:06:37.019930+0000","last_fullsized":"2024-03-04T10:06:37.019930+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:02:40.698762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019882+0000","last_change":"2024-03-04T10:06:36.029392+0000","last_active":"2024-03-04T10:06:37.019882+0000","last_peered":"2024-03-04T10:06:37.019882+0000","last_clean":"2024-03-04T10:06:37.019882+0000","last_became_active":"2024-03-04T10:06:36.029107+0000","last_became_peered":"2024-03-04T10:06:36.029107+0000","last_unstale":"2024-03-04T10:06:37.019882+0000","last_undegraded":"2024-03-04T10:06:37.019882+0000","last_fullsized":"2024-03-04T10:06:37.019882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:43.094873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019862+0000","last_change":"2024-03-04T10:06:35.025520+0000","last_active":"2024-03-04T10:06:37.019862+0000","last_peered":"2024-03-04T10:06:37.019862+0000","last_clean":"2024-03-04T10:06:37.019862+0000","last_became_active":"2024-03-04T10:06:35.025291+0000","last_became_peered":"2024-03-04T10:06:35.025291+0000","last_unstale":"2024-03-04T10:06:37.019862+0000","last_undegraded":"2024-03-04T10:06:37.019862+0000","last_fullsized":"2024-03-04T10:06:37.019862+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:46.534320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022304+0000","last_change":"2024-03-04T10:06:36.037453+0000","last_active":"2024-03-04T10:06:37.022304+0000","last_peered":"2024-03-04T10:06:37.022304+0000","last_clean":"2024-03-04T10:06:37.022304+0000","last_became_active":"2024-03-04T10:06:36.036396+0000","last_became_peered":"2024-03-04T10:06:36.036396+0000","last_unstale":"2024-03-04T10:06:37.022304+0000","last_undegraded":"2024-03-04T10:06:37.022304+0000","last_fullsized":"2024-03-04T10:06:37.022304+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:03.897058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019709+0000","last_change":"2024-03-04T10:06:35.021094+0000","last_active":"2024-03-04T10:06:37.019709+0000","last_peered":"2024-03-04T10:06:37.019709+0000","last_clean":"2024-03-04T10:06:37.019709+0000","last_became_active":"2024-03-04T10:06:35.020827+0000","last_became_peered":"2024-03-04T10:06:35.020827+0000","last_unstale":"2024-03-04T10:06:37.019709+0000","last_undegraded":"2024-03-04T10:06:37.019709+0000","last_fullsized":"2024-03-04T10:06:37.019709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:04:42.720549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035027+0000","last_change":"2024-03-04T10:06:36.038360+0000","last_active":"2024-03-04T10:06:37.035027+0000","last_peered":"2024-03-04T10:06:37.035027+0000","last_clean":"2024-03-04T10:06:37.035027+0000","last_became_active":"2024-03-04T10:06:36.038091+0000","last_became_peered":"2024-03-04T10:06:36.038091+0000","last_unstale":"2024-03-04T10:06:37.035027+0000","last_undegraded":"2024-03-04T10:06:37.035027+0000","last_fullsized":"2024-03-04T10:06:37.035027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:01.129479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"65'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.044882+0000","last_change":"2024-03-04T10:06:35.021148+0000","last_active":"2024-03-04T10:06:36.044882+0000","last_peered":"2024-03-04T10:06:36.044882+0000","last_clean":"2024-03-04T10:06:36.044882+0000","last_became_active":"2024-03-04T10:06:35.020874+0000","last_became_peered":"2024-03-04T10:06:35.020874+0000","last_unstale":"2024-03-04T10:06:36.044882+0000","last_undegraded":"2024-03-04T10:06:36.044882+0000","last_fullsized":"2024-03-04T10:06:36.044882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:44.492900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.037668+0000","last_change":"2024-03-04T10:06:36.031669+0000","last_active":"2024-03-04T10:06:36.037668+0000","last_peered":"2024-03-04T10:06:36.037668+0000","last_clean":"2024-03-04T10:06:36.037668+0000","last_became_active":"2024-03-04T10:06:36.031389+0000","last_became_peered":"2024-03-04T10:06:36.031389+0000","last_unstale":"2024-03-04T10:06:36.037668+0000","last_undegraded":"2024-03-04T10:06:36.037668+0000","last_fullsized":"2024-03-04T10:06:36.037668+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:48.186426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"65'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.040716+0000","last_change":"2024-03-04T10:06:35.024941+0000","last_active":"2024-03-04T10:06:36.040716+0000","last_peered":"2024-03-04T10:06:36.040716+0000","last_clean":"2024-03-04T10:06:36.040716+0000","last_became_active":"2024-03-04T10:06:35.023439+0000","last_became_peered":"2024-03-04T10:06:35.023439+0000","last_unstale":"2024-03-04T10:06:36.040716+0000","last_undegraded":"2024-03-04T10:06:36.040716+0000","last_fullsized":"2024-03-04T10:06:36.040716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:58.432642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120267+0000","last_change":"2024-03-04T10:06:36.034298+0000","last_active":"2024-03-04T10:06:37.120267+0000","last_peered":"2024-03-04T10:06:37.120267+0000","last_clean":"2024-03-04T10:06:37.120267+0000","last_became_active":"2024-03-04T10:06:36.033748+0000","last_became_peered":"2024-03-04T10:06:36.033748+0000","last_unstale":"2024-03-04T10:06:37.120267+0000","last_undegraded":"2024-03-04T10:06:37.120267+0000","last_fullsized":"2024-03-04T10:06:37.120267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:07:12.791348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021014+0000","last_change":"2024-03-04T10:06:35.024981+0000","last_active":"2024-03-04T10:06:36.021014+0000","last_peered":"2024-03-04T10:06:36.021014+0000","last_clean":"2024-03-04T10:06:36.021014+0000","last_became_active":"2024-03-04T10:06:35.024688+0000","last_became_peered":"2024-03-04T10:06:35.024688+0000","last_unstale":"2024-03-04T10:06:36.021014+0000","last_undegraded":"2024-03-04T10:06:36.021014+0000","last_fullsized":"2024-03-04T10:06:36.021014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:36.102599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.033699+0000","last_change":"2024-03-04T10:06:36.026169+0000","last_active":"2024-03-04T10:06:36.033699+0000","last_peered":"2024-03-04T10:06:36.033699+0000","last_clean":"2024-03-04T10:06:36.033699+0000","last_became_active":"2024-03-04T10:06:36.025900+0000","last_became_peered":"2024-03-04T10:06:36.025900+0000","last_unstale":"2024-03-04T10:06:36.033699+0000","last_undegraded":"2024-03-04T10:06:36.033699+0000","last_fullsized":"2024-03-04T10:06:36.033699+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:04:43.724815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120243+0000","last_change":"2024-03-04T10:06:35.026897+0000","last_active":"2024-03-04T10:06:37.120243+0000","last_peered":"2024-03-04T10:06:37.120243+0000","last_clean":"2024-03-04T10:06:37.120243+0000","last_became_active":"2024-03-04T10:06:35.026612+0000","last_became_peered":"2024-03-04T10:06:35.026612+0000","last_unstale":"2024-03-04T10:06:37.120243+0000","last_undegraded":"2024-03-04T10:06:37.120243+0000","last_fullsized":"2024-03-04T10:06:37.120243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:45:00.599578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.036912+0000","last_change":"2024-03-04T10:06:36.028989+0000","last_active":"2024-03-04T10:06:36.036912+0000","last_peered":"2024-03-04T10:06:36.036912+0000","last_clean":"2024-03-04T10:06:36.036912+0000","last_became_active":"2024-03-04T10:06:36.028715+0000","last_became_peered":"2024-03-04T10:06:36.028715+0000","last_unstale":"2024-03-04T10:06:36.036912+0000","last_undegraded":"2024-03-04T10:06:36.036912+0000","last_fullsized":"2024-03-04T10:06:36.036912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:26:17.271066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022390+0000","last_change":"2024-03-04T10:06:35.026616+0000","last_active":"2024-03-04T10:06:37.022390+0000","last_peered":"2024-03-04T10:06:37.022390+0000","last_clean":"2024-03-04T10:06:37.022390+0000","last_became_active":"2024-03-04T10:06:35.026318+0000","last_became_peered":"2024-03-04T10:06:35.026318+0000","last_unstale":"2024-03-04T10:06:37.022390+0000","last_undegraded":"2024-03-04T10:06:37.022390+0000","last_fullsized":"2024-03-04T10:06:37.022390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:58.622316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.033661+0000","last_change":"2024-03-04T10:06:36.031415+0000","last_active":"2024-03-04T10:06:36.033661+0000","last_peered":"2024-03-04T10:06:36.033661+0000","last_clean":"2024-03-04T10:06:36.033661+0000","last_became_active":"2024-03-04T10:06:36.031139+0000","last_became_peered":"2024-03-04T10:06:36.031139+0000","last_unstale":"2024-03-04T10:06:36.033661+0000","last_undegraded":"2024-03-04T10:06:36.033661+0000","last_fullsized":"2024-03-04T10:06:36.033661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:03.897058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021745+0000","last_change":"2024-03-04T10:06:35.024831+0000","last_active":"2024-03-04T10:06:37.021745+0000","last_peered":"2024-03-04T10:06:37.021745+0000","last_clean":"2024-03-04T10:06:37.021745+0000","last_became_active":"2024-03-04T10:06:35.024497+0000","last_became_peered":"2024-03-04T10:06:35.024497+0000","last_unstale":"2024-03-04T10:06:37.021745+0000","last_undegraded":"2024-03-04T10:06:37.021745+0000","last_fullsized":"2024-03-04T10:06:37.021745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:02:33.360463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120558+0000","last_change":"2024-03-04T10:06:36.031607+0000","last_active":"2024-03-04T10:06:37.120558+0000","last_peered":"2024-03-04T10:06:37.120558+0000","last_clean":"2024-03-04T10:06:37.120558+0000","last_became_active":"2024-03-04T10:06:36.031346+0000","last_became_peered":"2024-03-04T10:06:36.031346+0000","last_unstale":"2024-03-04T10:06:37.120558+0000","last_undegraded":"2024-03-04T10:06:37.120558+0000","last_fullsized":"2024-03-04T10:06:37.120558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:18.529274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"65'3","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021691+0000","last_change":"2024-03-04T10:06:35.027996+0000","last_active":"2024-03-04T10:06:37.021691+0000","last_peered":"2024-03-04T10:06:37.021691+0000","last_clean":"2024-03-04T10:06:37.021691+0000","last_became_active":"2024-03-04T10:06:35.027167+0000","last_became_peered":"2024-03-04T10:06:35.027167+0000","last_unstale":"2024-03-04T10:06:37.021691+0000","last_undegraded":"2024-03-04T10:06:37.021691+0000","last_fullsized":"2024-03-04T10:06:37.021691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:35:07.631737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021362+0000","last_change":"2024-03-04T10:06:36.034517+0000","last_active":"2024-03-04T10:06:37.021362+0000","last_peered":"2024-03-04T10:06:37.021362+0000","last_clean":"2024-03-04T10:06:37.021362+0000","last_became_active":"2024-03-04T10:06:36.034240+0000","last_became_peered":"2024-03-04T10:06:36.034240+0000","last_unstale":"2024-03-04T10:06:37.021362+0000","last_undegraded":"2024-03-04T10:06:37.021362+0000","last_fullsized":"2024-03-04T10:06:37.021362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:51.376207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"65'6","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.050794+0000","last_change":"2024-03-04T10:06:35.026764+0000","last_active":"2024-03-04T10:06:36.050794+0000","last_peered":"2024-03-04T10:06:36.050794+0000","last_clean":"2024-03-04T10:06:36.050794+0000","last_became_active":"2024-03-04T10:06:35.026565+0000","last_became_peered":"2024-03-04T10:06:35.026565+0000","last_unstale":"2024-03-04T10:06:36.050794+0000","last_undegraded":"2024-03-04T10:06:36.050794+0000","last_fullsized":"2024-03-04T10:06:36.050794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:38:04.206144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021933+0000","last_change":"2024-03-04T10:06:36.033313+0000","last_active":"2024-03-04T10:06:37.021933+0000","last_peered":"2024-03-04T10:06:37.021933+0000","last_clean":"2024-03-04T10:06:37.021933+0000","last_became_active":"2024-03-04T10:06:36.032995+0000","last_became_peered":"2024-03-04T10:06:36.032995+0000","last_unstale":"2024-03-04T10:06:37.021933+0000","last_undegraded":"2024-03-04T10:06:37.021933+0000","last_fullsized":"2024-03-04T10:06:37.021933+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:31.584231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022217+0000","last_change":"2024-03-04T10:06:35.027520+0000","last_active":"2024-03-04T10:06:37.022217+0000","last_peered":"2024-03-04T10:06:37.022217+0000","last_clean":"2024-03-04T10:06:37.022217+0000","last_became_active":"2024-03-04T10:06:35.026481+0000","last_became_peered":"2024-03-04T10:06:35.026481+0000","last_unstale":"2024-03-04T10:06:37.022217+0000","last_undegraded":"2024-03-04T10:06:37.022217+0000","last_fullsized":"2024-03-04T10:06:37.022217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:00.966883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033288+0000","last_change":"2024-03-04T10:06:36.036080+0000","last_active":"2024-03-04T10:06:37.033288+0000","last_peered":"2024-03-04T10:06:37.033288+0000","last_clean":"2024-03-04T10:06:37.033288+0000","last_became_active":"2024-03-04T10:06:36.035750+0000","last_became_peered":"2024-03-04T10:06:36.035750+0000","last_unstale":"2024-03-04T10:06:37.033288+0000","last_undegraded":"2024-03-04T10:06:37.033288+0000","last_fullsized":"2024-03-04T10:06:37.033288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:37.106865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033116+0000","last_change":"2024-03-04T10:06:35.024719+0000","last_active":"2024-03-04T10:06:37.033116+0000","last_peered":"2024-03-04T10:06:37.033116+0000","last_clean":"2024-03-04T10:06:37.033116+0000","last_became_active":"2024-03-04T10:06:35.024397+0000","last_became_peered":"2024-03-04T10:06:35.024397+0000","last_unstale":"2024-03-04T10:06:37.033116+0000","last_undegraded":"2024-03-04T10:06:37.033116+0000","last_fullsized":"2024-03-04T10:06:37.033116+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:30.579965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120274+0000","last_change":"2024-03-04T10:06:36.035411+0000","last_active":"2024-03-04T10:06:37.120274+0000","last_peered":"2024-03-04T10:06:37.120274+0000","last_clean":"2024-03-04T10:06:37.120274+0000","last_became_active":"2024-03-04T10:06:36.035119+0000","last_became_peered":"2024-03-04T10:06:36.035119+0000","last_unstale":"2024-03-04T10:06:37.120274+0000","last_undegraded":"2024-03-04T10:06:37.120274+0000","last_fullsized":"2024-03-04T10:06:37.120274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:31.584231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021479+0000","last_change":"2024-03-04T10:06:35.026894+0000","last_active":"2024-03-04T10:06:37.021479+0000","last_peered":"2024-03-04T10:06:37.021479+0000","last_clean":"2024-03-04T10:06:37.021479+0000","last_became_active":"2024-03-04T10:06:35.026599+0000","last_became_peered":"2024-03-04T10:06:35.026599+0000","last_unstale":"2024-03-04T10:06:37.021479+0000","last_undegraded":"2024-03-04T10:06:37.021479+0000","last_fullsized":"2024-03-04T10:06:37.021479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:23.266860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.039838+0000","last_change":"2024-03-04T10:06:36.036066+0000","last_active":"2024-03-04T10:06:36.039838+0000","last_peered":"2024-03-04T10:06:36.039838+0000","last_clean":"2024-03-04T10:06:36.039838+0000","last_became_active":"2024-03-04T10:06:36.035794+0000","last_became_peered":"2024-03-04T10:06:36.035794+0000","last_unstale":"2024-03-04T10:06:36.039838+0000","last_undegraded":"2024-03-04T10:06:36.039838+0000","last_fullsized":"2024-03-04T10:06:36.039838+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:31.584231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"65'8","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022045+0000","last_change":"2024-03-04T10:06:35.027077+0000","last_active":"2024-03-04T10:06:37.022045+0000","last_peered":"2024-03-04T10:06:37.022045+0000","last_clean":"2024-03-04T10:06:37.022045+0000","last_became_active":"2024-03-04T10:06:35.026795+0000","last_became_peered":"2024-03-04T10:06:35.026795+0000","last_unstale":"2024-03-04T10:06:37.022045+0000","last_undegraded":"2024-03-04T10:06:37.022045+0000","last_fullsized":"2024-03-04T10:06:37.022045+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:11.700722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021081+0000","last_change":"2024-03-04T10:06:36.031067+0000","last_active":"2024-03-04T10:06:37.021081+0000","last_peered":"2024-03-04T10:06:37.021081+0000","last_clean":"2024-03-04T10:06:37.021081+0000","last_became_active":"2024-03-04T10:06:36.030780+0000","last_became_peered":"2024-03-04T10:06:36.030780+0000","last_unstale":"2024-03-04T10:06:37.021081+0000","last_undegraded":"2024-03-04T10:06:37.021081+0000","last_fullsized":"2024-03-04T10:06:37.021081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:11:29.231443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033229+0000","last_change":"2024-03-04T10:06:35.025447+0000","last_active":"2024-03-04T10:06:37.033229+0000","last_peered":"2024-03-04T10:06:37.033229+0000","last_clean":"2024-03-04T10:06:37.033229+0000","last_became_active":"2024-03-04T10:06:35.024728+0000","last_became_peered":"2024-03-04T10:06:35.024728+0000","last_unstale":"2024-03-04T10:06:37.033229+0000","last_undegraded":"2024-03-04T10:06:37.033229+0000","last_fullsized":"2024-03-04T10:06:37.033229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:23.266860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120589+0000","last_change":"2024-03-04T10:06:36.036751+0000","last_active":"2024-03-04T10:06:37.120589+0000","last_peered":"2024-03-04T10:06:37.120589+0000","last_clean":"2024-03-04T10:06:37.120589+0000","last_became_active":"2024-03-04T10:06:36.036484+0000","last_became_peered":"2024-03-04T10:06:36.036484+0000","last_unstale":"2024-03-04T10:06:37.120589+0000","last_undegraded":"2024-03-04T10:06:37.120589+0000","last_fullsized":"2024-03-04T10:06:37.120589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:24.271126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022184+0000","last_change":"2024-03-04T10:06:35.026636+0000","last_active":"2024-03-04T10:06:37.022184+0000","last_peered":"2024-03-04T10:06:37.022184+0000","last_clean":"2024-03-04T10:06:37.022184+0000","last_became_active":"2024-03-04T10:06:35.026298+0000","last_became_peered":"2024-03-04T10:06:35.026298+0000","last_unstale":"2024-03-04T10:06:37.022184+0000","last_undegraded":"2024-03-04T10:06:37.022184+0000","last_fullsized":"2024-03-04T10:06:37.022184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.312358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021636+0000","last_change":"2024-03-04T10:06:36.041137+0000","last_active":"2024-03-04T10:06:37.021636+0000","last_peered":"2024-03-04T10:06:37.021636+0000","last_clean":"2024-03-04T10:06:37.021636+0000","last_became_active":"2024-03-04T10:06:36.040283+0000","last_became_peered":"2024-03-04T10:06:36.040283+0000","last_unstale":"2024-03-04T10:06:37.021636+0000","last_undegraded":"2024-03-04T10:06:37.021636+0000","last_fullsized":"2024-03-04T10:06:37.021636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:51.363438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021792+0000","last_change":"2024-03-04T10:06:35.027753+0000","last_active":"2024-03-04T10:06:37.021792+0000","last_peered":"2024-03-04T10:06:37.021792+0000","last_clean":"2024-03-04T10:06:37.021792+0000","last_became_active":"2024-03-04T10:06:35.027464+0000","last_became_peered":"2024-03-04T10:06:35.027464+0000","last_unstale":"2024-03-04T10:06:37.021792+0000","last_undegraded":"2024-03-04T10:06:37.021792+0000","last_fullsized":"2024-03-04T10:06:37.021792+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:01.895797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120547+0000","last_change":"2024-03-04T10:06:36.038584+0000","last_active":"2024-03-04T10:06:37.120547+0000","last_peered":"2024-03-04T10:06:37.120547+0000","last_clean":"2024-03-04T10:06:37.120547+0000","last_became_active":"2024-03-04T10:06:36.038300+0000","last_became_peered":"2024-03-04T10:06:36.038300+0000","last_unstale":"2024-03-04T10:06:37.120547+0000","last_undegraded":"2024-03-04T10:06:37.120547+0000","last_fullsized":"2024-03-04T10:06:37.120547+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:02.900063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021741+0000","last_change":"2024-03-04T10:06:35.022355+0000","last_active":"2024-03-04T10:06:36.021741+0000","last_peered":"2024-03-04T10:06:36.021741+0000","last_clean":"2024-03-04T10:06:36.021741+0000","last_became_active":"2024-03-04T10:06:35.022117+0000","last_became_peered":"2024-03-04T10:06:35.022117+0000","last_unstale":"2024-03-04T10:06:36.021741+0000","last_undegraded":"2024-03-04T10:06:36.021741+0000","last_fullsized":"2024-03-04T10:06:36.021741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:26.349307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019514+0000","last_change":"2024-03-04T10:06:36.032226+0000","last_active":"2024-03-04T10:06:37.019514+0000","last_peered":"2024-03-04T10:06:37.019514+0000","last_clean":"2024-03-04T10:06:37.019514+0000","last_became_active":"2024-03-04T10:06:36.030080+0000","last_became_peered":"2024-03-04T10:06:36.030080+0000","last_unstale":"2024-03-04T10:06:37.019514+0000","last_undegraded":"2024-03-04T10:06:37.019514+0000","last_fullsized":"2024-03-04T10:06:37.019514+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:09.463816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019770+0000","last_change":"2024-03-04T10:06:35.024457+0000","last_active":"2024-03-04T10:06:37.019770+0000","last_peered":"2024-03-04T10:06:37.019770+0000","last_clean":"2024-03-04T10:06:37.019770+0000","last_became_active":"2024-03-04T10:06:35.024204+0000","last_became_peered":"2024-03-04T10:06:35.024204+0000","last_unstale":"2024-03-04T10:06:37.019770+0000","last_undegraded":"2024-03-04T10:06:37.019770+0000","last_fullsized":"2024-03-04T10:06:37.019770+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:02.892792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033452+0000","last_change":"2024-03-04T10:06:36.028762+0000","last_active":"2024-03-04T10:06:37.033452+0000","last_peered":"2024-03-04T10:06:37.033452+0000","last_clean":"2024-03-04T10:06:37.033452+0000","last_became_active":"2024-03-04T10:06:36.028441+0000","last_became_peered":"2024-03-04T10:06:36.028441+0000","last_unstale":"2024-03-04T10:06:37.033452+0000","last_undegraded":"2024-03-04T10:06:37.033452+0000","last_fullsized":"2024-03-04T10:06:37.033452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:47.538586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033042+0000","last_change":"2024-03-04T10:06:35.024601+0000","last_active":"2024-03-04T10:06:37.033042+0000","last_peered":"2024-03-04T10:06:37.033042+0000","last_clean":"2024-03-04T10:06:37.033042+0000","last_became_active":"2024-03-04T10:06:35.024294+0000","last_became_peered":"2024-03-04T10:06:35.024294+0000","last_unstale":"2024-03-04T10:06:37.033042+0000","last_undegraded":"2024-03-04T10:06:37.033042+0000","last_fullsized":"2024-03-04T10:06:37.033042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:36.093242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120553+0000","last_change":"2024-03-04T10:06:36.034033+0000","last_active":"2024-03-04T10:06:37.120553+0000","last_peered":"2024-03-04T10:06:37.120553+0000","last_clean":"2024-03-04T10:06:37.120553+0000","last_became_active":"2024-03-04T10:06:36.033392+0000","last_became_peered":"2024-03-04T10:06:36.033392+0000","last_unstale":"2024-03-04T10:06:37.120553+0000","last_undegraded":"2024-03-04T10:06:37.120553+0000","last_fullsized":"2024-03-04T10:06:37.120553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:11.316624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019515+0000","last_change":"2024-03-04T10:06:35.027805+0000","last_active":"2024-03-04T10:06:37.019515+0000","last_peered":"2024-03-04T10:06:37.019515+0000","last_clean":"2024-03-04T10:06:37.019515+0000","last_became_active":"2024-03-04T10:06:35.027514+0000","last_became_peered":"2024-03-04T10:06:35.027514+0000","last_unstale":"2024-03-04T10:06:37.019515+0000","last_undegraded":"2024-03-04T10:06:37.019515+0000","last_fullsized":"2024-03-04T10:06:37.019515+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:58.432642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019738+0000","last_change":"2024-03-04T10:06:36.023611+0000","last_active":"2024-03-04T10:06:37.019738+0000","last_peered":"2024-03-04T10:06:37.019738+0000","last_clean":"2024-03-04T10:06:37.019738+0000","last_became_active":"2024-03-04T10:06:36.023345+0000","last_became_peered":"2024-03-04T10:06:36.023345+0000","last_unstale":"2024-03-04T10:06:37.019738+0000","last_undegraded":"2024-03-04T10:06:37.019738+0000","last_fullsized":"2024-03-04T10:06:37.019738+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:48.254040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033444+0000","last_change":"2024-03-04T10:06:35.027195+0000","last_active":"2024-03-04T10:06:37.033444+0000","last_peered":"2024-03-04T10:06:37.033444+0000","last_clean":"2024-03-04T10:06:37.033444+0000","last_became_active":"2024-03-04T10:06:35.026880+0000","last_became_peered":"2024-03-04T10:06:35.026880+0000","last_unstale":"2024-03-04T10:06:37.033444+0000","last_undegraded":"2024-03-04T10:06:37.033444+0000","last_fullsized":"2024-03-04T10:06:37.033444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.312358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035276+0000","last_change":"2024-03-04T10:06:36.026853+0000","last_active":"2024-03-04T10:06:37.035276+0000","last_peered":"2024-03-04T10:06:37.035276+0000","last_clean":"2024-03-04T10:06:37.035276+0000","last_became_active":"2024-03-04T10:06:36.026566+0000","last_became_peered":"2024-03-04T10:06:36.026566+0000","last_unstale":"2024-03-04T10:06:37.035276+0000","last_undegraded":"2024-03-04T10:06:37.035276+0000","last_fullsized":"2024-03-04T10:06:37.035276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:31.815185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"66'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.034928+0000","last_change":"2024-03-04T10:06:35.025158+0000","last_active":"2024-03-04T10:06:37.034928+0000","last_peered":"2024-03-04T10:06:37.034928+0000","last_clean":"2024-03-04T10:06:37.034928+0000","last_became_active":"2024-03-04T10:06:35.024610+0000","last_became_peered":"2024-03-04T10:06:35.024610+0000","last_unstale":"2024-03-04T10:06:37.034928+0000","last_undegraded":"2024-03-04T10:06:37.034928+0000","last_fullsized":"2024-03-04T10:06:37.034928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:57.046932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033168+0000","last_change":"2024-03-04T10:06:36.036054+0000","last_active":"2024-03-04T10:06:37.033168+0000","last_peered":"2024-03-04T10:06:37.033168+0000","last_clean":"2024-03-04T10:06:37.033168+0000","last_became_active":"2024-03-04T10:06:36.035767+0000","last_became_peered":"2024-03-04T10:06:36.035767+0000","last_unstale":"2024-03-04T10:06:37.033168+0000","last_undegraded":"2024-03-04T10:06:37.033168+0000","last_fullsized":"2024-03-04T10:06:37.033168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:11:29.231443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019806+0000","last_change":"2024-03-04T10:06:35.024127+0000","last_active":"2024-03-04T10:06:37.019806+0000","last_peered":"2024-03-04T10:06:37.019806+0000","last_clean":"2024-03-04T10:06:37.019806+0000","last_became_active":"2024-03-04T10:06:35.023856+0000","last_became_peered":"2024-03-04T10:06:35.023856+0000","last_unstale":"2024-03-04T10:06:37.019806+0000","last_undegraded":"2024-03-04T10:06:37.019806+0000","last_fullsized":"2024-03-04T10:06:37.019806+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:55.902008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120654+0000","last_change":"2024-03-04T10:06:36.037857+0000","last_active":"2024-03-04T10:06:37.120654+0000","last_peered":"2024-03-04T10:06:37.120654+0000","last_clean":"2024-03-04T10:06:37.120654+0000","last_became_active":"2024-03-04T10:06:36.037619+0000","last_became_peered":"2024-03-04T10:06:36.037619+0000","last_unstale":"2024-03-04T10:06:37.120654+0000","last_undegraded":"2024-03-04T10:06:37.120654+0000","last_fullsized":"2024-03-04T10:06:37.120654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.626582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.024006+0000","last_change":"2024-03-04T10:06:35.025226+0000","last_active":"2024-03-04T10:06:36.024006+0000","last_peered":"2024-03-04T10:06:36.024006+0000","last_clean":"2024-03-04T10:06:36.024006+0000","last_became_active":"2024-03-04T10:06:35.024979+0000","last_became_peered":"2024-03-04T10:06:35.024979+0000","last_unstale":"2024-03-04T10:06:36.024006+0000","last_undegraded":"2024-03-04T10:06:36.024006+0000","last_fullsized":"2024-03-04T10:06:36.024006+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:03:36.042120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120220+0000","last_change":"2024-03-04T10:06:36.038811+0000","last_active":"2024-03-04T10:06:37.120220+0000","last_peered":"2024-03-04T10:06:37.120220+0000","last_clean":"2024-03-04T10:06:37.120220+0000","last_became_active":"2024-03-04T10:06:36.038523+0000","last_became_peered":"2024-03-04T10:06:36.038523+0000","last_unstale":"2024-03-04T10:06:37.120220+0000","last_undegraded":"2024-03-04T10:06:37.120220+0000","last_fullsized":"2024-03-04T10:06:37.120220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:24.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022296+0000","last_change":"2024-03-04T10:06:35.028067+0000","last_active":"2024-03-04T10:06:37.022296+0000","last_peered":"2024-03-04T10:06:37.022296+0000","last_clean":"2024-03-04T10:06:37.022296+0000","last_became_active":"2024-03-04T10:06:35.027770+0000","last_became_peered":"2024-03-04T10:06:35.027770+0000","last_unstale":"2024-03-04T10:06:37.022296+0000","last_undegraded":"2024-03-04T10:06:37.022296+0000","last_fullsized":"2024-03-04T10:06:37.022296+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:17.525008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.020921+0000","last_change":"2024-03-04T10:06:36.044794+0000","last_active":"2024-03-04T10:06:37.020921+0000","last_peered":"2024-03-04T10:06:37.020921+0000","last_clean":"2024-03-04T10:06:37.020921+0000","last_became_active":"2024-03-04T10:06:36.043521+0000","last_became_peered":"2024-03-04T10:06:36.043521+0000","last_unstale":"2024-03-04T10:06:37.020921+0000","last_undegraded":"2024-03-04T10:06:37.020921+0000","last_fullsized":"2024-03-04T10:06:37.020921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:38:05.210411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021964+0000","last_change":"2024-03-04T10:06:35.021230+0000","last_active":"2024-03-04T10:06:36.021964+0000","last_peered":"2024-03-04T10:06:36.021964+0000","last_clean":"2024-03-04T10:06:36.021964+0000","last_became_active":"2024-03-04T10:06:35.020981+0000","last_became_peered":"2024-03-04T10:06:35.020981+0000","last_unstale":"2024-03-04T10:06:36.021964+0000","last_undegraded":"2024-03-04T10:06:36.021964+0000","last_fullsized":"2024-03-04T10:06:36.021964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:46.534320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.037113+0000","last_change":"2024-03-04T10:06:36.031011+0000","last_active":"2024-03-04T10:06:36.037113+0000","last_peered":"2024-03-04T10:06:36.037113+0000","last_clean":"2024-03-04T10:06:36.037113+0000","last_became_active":"2024-03-04T10:06:36.030704+0000","last_became_peered":"2024-03-04T10:06:36.030704+0000","last_unstale":"2024-03-04T10:06:36.037113+0000","last_undegraded":"2024-03-04T10:06:36.037113+0000","last_fullsized":"2024-03-04T10:06:36.037113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:51.363438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021147+0000","last_change":"2024-03-04T10:06:35.024535+0000","last_active":"2024-03-04T10:06:36.021147+0000","last_peered":"2024-03-04T10:06:36.021147+0000","last_clean":"2024-03-04T10:06:36.021147+0000","last_became_active":"2024-03-04T10:06:35.024266+0000","last_became_peered":"2024-03-04T10:06:35.024266+0000","last_unstale":"2024-03-04T10:06:36.021147+0000","last_undegraded":"2024-03-04T10:06:36.021147+0000","last_fullsized":"2024-03-04T10:06:36.021147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:11:28.227177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035139+0000","last_change":"2024-03-04T10:06:36.032224+0000","last_active":"2024-03-04T10:06:37.035139+0000","last_peered":"2024-03-04T10:06:37.035139+0000","last_clean":"2024-03-04T10:06:37.035139+0000","last_became_active":"2024-03-04T10:06:36.031948+0000","last_became_peered":"2024-03-04T10:06:36.031948+0000","last_unstale":"2024-03-04T10:06:37.035139+0000","last_undegraded":"2024-03-04T10:06:37.035139+0000","last_fullsized":"2024-03-04T10:06:37.035139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:47.065602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.022464+0000","last_change":"2024-03-04T10:06:35.024809+0000","last_active":"2024-03-04T10:06:36.022464+0000","last_peered":"2024-03-04T10:06:36.022464+0000","last_clean":"2024-03-04T10:06:36.022464+0000","last_became_active":"2024-03-04T10:06:35.024441+0000","last_became_peered":"2024-03-04T10:06:35.024441+0000","last_unstale":"2024-03-04T10:06:36.022464+0000","last_undegraded":"2024-03-04T10:06:36.022464+0000","last_fullsized":"2024-03-04T10:06:36.022464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:24.247408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120615+0000","last_change":"2024-03-04T10:06:36.039508+0000","last_active":"2024-03-04T10:06:37.120615+0000","last_peered":"2024-03-04T10:06:37.120615+0000","last_clean":"2024-03-04T10:06:37.120615+0000","last_became_active":"2024-03-04T10:06:36.039236+0000","last_became_peered":"2024-03-04T10:06:36.039236+0000","last_unstale":"2024-03-04T10:06:37.120615+0000","last_undegraded":"2024-03-04T10:06:37.120615+0000","last_fullsized":"2024-03-04T10:06:37.120615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:35:08.636003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.020744+0000","last_change":"2024-03-04T10:06:35.019446+0000","last_active":"2024-03-04T10:06:36.020744+0000","last_peered":"2024-03-04T10:06:36.020744+0000","last_clean":"2024-03-04T10:06:36.020744+0000","last_became_active":"2024-03-04T10:06:35.019179+0000","last_became_peered":"2024-03-04T10:06:35.019179+0000","last_unstale":"2024-03-04T10:06:36.020744+0000","last_undegraded":"2024-03-04T10:06:36.020744+0000","last_fullsized":"2024-03-04T10:06:36.020744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:39.308355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":263,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022069+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:06:37.022069+0000","last_peered":"2024-03-04T10:06:37.022069+0000","last_clean":"2024-03-04T10:06:37.022069+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:06:37.022069+0000","last_undegraded":"2024-03-04T10:06:37.022069+0000","last_fullsized":"2024-03-04T10:06:37.022069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28392,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343448,"statfs":{"total":96636764160,"available":96607690752,"internally_reserved":0,"allocated":1396736,"data_stored":949378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":358268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":358256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":359846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":358234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":364926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":360610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28372,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343468,"statfs":{"total":96636764160,"available":96607711232,"internally_reserved":0,"allocated":1380352,"data_stored":948602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:38.995 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:06:39.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.128+0000 7f504b324640 1 Processor -- start 2024-03-04T10:06:39.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.128+0000 7f504b324640 1 -- start start 2024-03-04T10:06:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f504b324640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 0x7f504415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f504b324640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f504415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f504b324640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504415be00 con 0x7f504415b030 2024-03-04T10:06:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f504b324640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50440b2720 con 0x7f504406d3b0 2024-03-04T10:06:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f504b324640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504406d8b0 con 0x7f504415b410 2024-03-04T10:06:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f50488bf640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 0x7f504415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f50498c1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f504415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f50490c0640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f504415b030 0x7f504415dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:45854/0 (socket says 172.21.15.99:45854) 2024-03-04T10:06:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f50488bf640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 0x7f504415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:54656/0 (socket says 172.21.15.99:54656) 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.129+0000 7f50488bf640 1 -- 172.21.15.99:0/2163839583 learned_addr learned my addr 172.21.15.99:0/2163839583 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.130+0000 7f50498c1640 1 -- 172.21.15.99:0/2163839583 >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 msgr2=0x7f504415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.130+0000 7f50498c1640 1 --2- 172.21.15.99:0/2163839583 >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 0x7f504415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.130+0000 7f50498c1640 1 -- 172.21.15.99:0/2163839583 >> v1:172.21.15.162:6789/0 conn(0x7f504415b030 legacy=0x7f504415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.130+0000 7f50498c1640 1 -- 172.21.15.99:0/2163839583 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504406db90 con 0x7f504415b410 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.130+0000 7f50488bf640 1 --2- 172.21.15.99:0/2163839583 >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 0x7f504415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.130+0000 7f50498c1640 1 --2- 172.21.15.99:0/2163839583 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f504415b8c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50440b28a0 tx=0x7f503c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dbe3c9391fdaf5b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f503bfff640 1 -- 172.21.15.99:0/2163839583 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c031770 con 0x7f504415b410 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f503bfff640 1 -- 172.21.15.99:0/2163839583 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c037070 con 0x7f504415b410 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f503bfff640 1 -- 172.21.15.99:0/2163839583 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c031b30 con 0x7f504415b410 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 -- 172.21.15.99:0/2163839583 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 msgr2=0x7f504415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 --2- 172.21.15.99:0/2163839583 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f504415b8c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f50440b28a0 tx=0x7f503c02b810 comp rx=0 tx=0).stop 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 -- 172.21.15.99:0/2163839583 shutdown_connections 2024-03-04T10:06:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 --2- 172.21.15.99:0/2163839583 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f504415b8c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 --2- 172.21.15.99:0/2163839583 >> v2:172.21.15.162:3300/0 conn(0x7f504406d3b0 0x7f504415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 -- 172.21.15.99:0/2163839583 >> 172.21.15.99:0/2163839583 conn(0x7f504406acd0 msgr2=0x7f504406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 -- 172.21.15.99:0/2163839583 shutdown_connections 2024-03-04T10:06:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.131+0000 7f504b324640 1 -- 172.21.15.99:0/2163839583 wait complete. 2024-03-04T10:06:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.132+0000 7f504b324640 1 Processor -- start 2024-03-04T10:06:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.132+0000 7f504b324640 1 -- start start 2024-03-04T10:06:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f504b324640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 0x7f50441f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f504b324640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f504415b030 0x7f50440cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f504b324640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f5044157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f504b324640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504406e690 con 0x7f504406d3b0 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f504b324640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504406e150 con 0x7f504415b030 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f504b324640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504406e810 con 0x7f504415b410 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 0x7f50441f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 0x7f50441f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38172/0 (socket says 172.21.15.99:38172) 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 -- 172.21.15.99:0/1674759582 learned_addr learned my addr 172.21.15.99:0/1674759582 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50498c1640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f5044157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50488bf640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f504415b030 0x7f50440cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 -- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 msgr2=0x7f5044157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f5044157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 -- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f504415b030 msgr2=0x7f50440cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f504415b030 0x7f50440cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.133+0000 7f50490c0640 1 -- 172.21.15.99:0/1674759582 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50440cfc10 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.134+0000 7f50490c0640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 0x7f50441f3910 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f504000ea60 tx=0x7f504000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.134+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040015070 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.134+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5040001c00 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.134+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50441f3ef0 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.134+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040001fd0 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.134+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50441f44b0 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.135+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5044074f60 con 0x7f504406d3b0 2024-03-04T10:06:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.135+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5040018070 con 0x7f504406d3b0 2024-03-04T10:06:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.136+0000 7f5039ffb640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f502806b4d0 0x7f502806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.136+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f504008c840 con 0x7f504406d3b0 2024-03-04T10:06:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.136+0000 7f50488bf640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f502806b4d0 0x7f502806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.136+0000 7f50488bf640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f502806b4d0 0x7f502806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5044147e80 tx=0x7f5034009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.140+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f504005ae50 con 0x7f504406d3b0 2024-03-04T10:06:39.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.307+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5044154dc0 con 0x7f502806b4d0 2024-03-04T10:06:39.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.314+0000 7f5039ffb640 1 -- 172.21.15.99:0/1674759582 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252004 (secure 0 0 0) 0x7f5044154dc0 con 0x7f502806b4d0 2024-03-04T10:06:39.318 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:39.320 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:06:39.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f502806b4d0 msgr2=0x7f502806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f502806b4d0 0x7f502806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5044147e80 tx=0x7f5034009240 comp rx=0 tx=0).stop 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 msgr2=0x7f50441f3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 0x7f50441f3910 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f504000ea60 tx=0x7f504000ef20 comp rx=0 tx=0).stop 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 shutdown_connections 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f502806b4d0 0x7f502806d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f504415b410 0x7f5044157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f504415b030 0x7f50440cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 --2- 172.21.15.99:0/1674759582 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f504406d3b0 0x7f50441f3910 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.316+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 >> 172.21.15.99:0/1674759582 conn(0x7f504406acd0 msgr2=0x7f50440af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.317+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 shutdown_connections 2024-03-04T10:06:39.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.317+0000 7f504b324640 1 -- 172.21.15.99:0/1674759582 wait complete. 2024-03-04T10:06:39.329 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-04T10:06:37.480325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223580,"kb_used_data":7380,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751140,"statfs":{"total":773094113280,"available":772865167360,"internally_reserved":0,"allocated":7557120,"data_stored":4058120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000888"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.042546+0000","last_change":"2024-03-04T10:06:36.036268+0000","last_active":"2024-03-04T10:06:36.042546+0000","last_peered":"2024-03-04T10:06:36.042546+0000","last_clean":"2024-03-04T10:06:36.042546+0000","last_became_active":"2024-03-04T10:06:36.035997+0000","last_became_peered":"2024-03-04T10:06:36.035997+0000","last_unstale":"2024-03-04T10:06:36.042546+0000","last_undegraded":"2024-03-04T10:06:36.042546+0000","last_fullsized":"2024-03-04T10:06:36.042546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:37.097508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.037160+0000","last_change":"2024-03-04T10:06:36.033385+0000","last_active":"2024-03-04T10:06:36.037160+0000","last_peered":"2024-03-04T10:06:36.037160+0000","last_clean":"2024-03-04T10:06:36.037160+0000","last_became_active":"2024-03-04T10:06:36.033125+0000","last_became_peered":"2024-03-04T10:06:36.033125+0000","last_unstale":"2024-03-04T10:06:36.037160+0000","last_undegraded":"2024-03-04T10:06:36.037160+0000","last_fullsized":"2024-03-04T10:06:36.037160+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:24.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021598+0000","last_change":"2024-03-04T10:06:36.031779+0000","last_active":"2024-03-04T10:06:37.021598+0000","last_peered":"2024-03-04T10:06:37.021598+0000","last_clean":"2024-03-04T10:06:37.021598+0000","last_became_active":"2024-03-04T10:06:36.031538+0000","last_became_peered":"2024-03-04T10:06:36.031538+0000","last_unstale":"2024-03-04T10:06:37.021598+0000","last_undegraded":"2024-03-04T10:06:37.021598+0000","last_fullsized":"2024-03-04T10:06:37.021598+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:59.436908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021535+0000","last_change":"2024-03-04T10:06:36.027369+0000","last_active":"2024-03-04T10:06:37.021535+0000","last_peered":"2024-03-04T10:06:37.021535+0000","last_clean":"2024-03-04T10:06:37.021535+0000","last_became_active":"2024-03-04T10:06:36.027094+0000","last_became_peered":"2024-03-04T10:06:36.027094+0000","last_unstale":"2024-03-04T10:06:37.021535+0000","last_undegraded":"2024-03-04T10:06:37.021535+0000","last_fullsized":"2024-03-04T10:06:37.021535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:26:17.271066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120664+0000","last_change":"2024-03-04T10:06:36.036692+0000","last_active":"2024-03-04T10:06:37.120664+0000","last_peered":"2024-03-04T10:06:37.120664+0000","last_clean":"2024-03-04T10:06:37.120664+0000","last_became_active":"2024-03-04T10:06:36.036399+0000","last_became_peered":"2024-03-04T10:06:36.036399+0000","last_unstale":"2024-03-04T10:06:37.120664+0000","last_undegraded":"2024-03-04T10:06:37.120664+0000","last_fullsized":"2024-03-04T10:06:37.120664+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:01.971149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019803+0000","last_change":"2024-03-04T10:06:36.043253+0000","last_active":"2024-03-04T10:06:37.019803+0000","last_peered":"2024-03-04T10:06:37.019803+0000","last_clean":"2024-03-04T10:06:37.019803+0000","last_became_active":"2024-03-04T10:06:36.038732+0000","last_became_peered":"2024-03-04T10:06:36.038732+0000","last_unstale":"2024-03-04T10:06:37.019803+0000","last_undegraded":"2024-03-04T10:06:37.019803+0000","last_fullsized":"2024-03-04T10:06:37.019803+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:34:10.646857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035170+0000","last_change":"2024-03-04T10:06:36.045118+0000","last_active":"2024-03-04T10:06:37.035170+0000","last_peered":"2024-03-04T10:06:37.035170+0000","last_clean":"2024-03-04T10:06:37.035170+0000","last_became_active":"2024-03-04T10:06:36.044807+0000","last_became_peered":"2024-03-04T10:06:36.044807+0000","last_unstale":"2024-03-04T10:06:37.035170+0000","last_undegraded":"2024-03-04T10:06:37.035170+0000","last_fullsized":"2024-03-04T10:06:37.035170+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:49.322018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.032870+0000","last_change":"2024-03-04T10:06:36.040694+0000","last_active":"2024-03-04T10:06:37.032870+0000","last_peered":"2024-03-04T10:06:37.032870+0000","last_clean":"2024-03-04T10:06:37.032870+0000","last_became_active":"2024-03-04T10:06:36.040382+0000","last_became_peered":"2024-03-04T10:06:36.040382+0000","last_unstale":"2024-03-04T10:06:37.032870+0000","last_undegraded":"2024-03-04T10:06:37.032870+0000","last_fullsized":"2024-03-04T10:06:37.032870+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:32:51.700460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022417+0000","last_change":"2024-03-04T10:06:36.037253+0000","last_active":"2024-03-04T10:06:37.022417+0000","last_peered":"2024-03-04T10:06:37.022417+0000","last_clean":"2024-03-04T10:06:37.022417+0000","last_became_active":"2024-03-04T10:06:36.036744+0000","last_became_peered":"2024-03-04T10:06:36.036744+0000","last_unstale":"2024-03-04T10:06:37.022417+0000","last_undegraded":"2024-03-04T10:06:37.022417+0000","last_fullsized":"2024-03-04T10:06:37.022417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:56.906275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021730+0000","last_change":"2024-03-04T10:06:36.035909+0000","last_active":"2024-03-04T10:06:37.021730+0000","last_peered":"2024-03-04T10:06:37.021730+0000","last_clean":"2024-03-04T10:06:37.021730+0000","last_became_active":"2024-03-04T10:06:36.035646+0000","last_became_peered":"2024-03-04T10:06:36.035646+0000","last_unstale":"2024-03-04T10:06:37.021730+0000","last_undegraded":"2024-03-04T10:06:37.021730+0000","last_fullsized":"2024-03-04T10:06:37.021730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:47.538586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.020905+0000","last_change":"2024-03-04T10:06:36.029432+0000","last_active":"2024-03-04T10:06:37.020905+0000","last_peered":"2024-03-04T10:06:37.020905+0000","last_clean":"2024-03-04T10:06:37.020905+0000","last_became_active":"2024-03-04T10:06:36.029159+0000","last_became_peered":"2024-03-04T10:06:36.029159+0000","last_unstale":"2024-03-04T10:06:37.020905+0000","last_undegraded":"2024-03-04T10:06:37.020905+0000","last_fullsized":"2024-03-04T10:06:37.020905+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:25.251674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033055+0000","last_change":"2024-03-04T10:06:36.039821+0000","last_active":"2024-03-04T10:06:37.033055+0000","last_peered":"2024-03-04T10:06:37.033055+0000","last_clean":"2024-03-04T10:06:37.033055+0000","last_became_active":"2024-03-04T10:06:36.039510+0000","last_became_peered":"2024-03-04T10:06:36.039510+0000","last_unstale":"2024-03-04T10:06:37.033055+0000","last_undegraded":"2024-03-04T10:06:37.033055+0000","last_fullsized":"2024-03-04T10:06:37.033055+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:30.326871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033252+0000","last_change":"2024-03-04T10:06:36.032567+0000","last_active":"2024-03-04T10:06:37.033252+0000","last_peered":"2024-03-04T10:06:37.033252+0000","last_clean":"2024-03-04T10:06:37.033252+0000","last_became_active":"2024-03-04T10:06:36.032321+0000","last_became_peered":"2024-03-04T10:06:36.032321+0000","last_unstale":"2024-03-04T10:06:37.033252+0000","last_undegraded":"2024-03-04T10:06:37.033252+0000","last_fullsized":"2024-03-04T10:06:37.033252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:45.497167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021546+0000","last_change":"2024-03-04T10:06:36.034741+0000","last_active":"2024-03-04T10:06:37.021546+0000","last_peered":"2024-03-04T10:06:37.021546+0000","last_clean":"2024-03-04T10:06:37.021546+0000","last_became_active":"2024-03-04T10:06:36.034324+0000","last_became_peered":"2024-03-04T10:06:36.034324+0000","last_unstale":"2024-03-04T10:06:37.021546+0000","last_undegraded":"2024-03-04T10:06:37.021546+0000","last_fullsized":"2024-03-04T10:06:37.021546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:27.353573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021376+0000","last_change":"2024-03-04T10:06:36.039836+0000","last_active":"2024-03-04T10:06:37.021376+0000","last_peered":"2024-03-04T10:06:37.021376+0000","last_clean":"2024-03-04T10:06:37.021376+0000","last_became_active":"2024-03-04T10:06:36.039475+0000","last_became_peered":"2024-03-04T10:06:36.039475+0000","last_unstale":"2024-03-04T10:06:37.021376+0000","last_undegraded":"2024-03-04T10:06:37.021376+0000","last_fullsized":"2024-03-04T10:06:37.021376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:45.497167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035032+0000","last_change":"2024-03-04T10:06:36.032778+0000","last_active":"2024-03-04T10:06:37.035032+0000","last_peered":"2024-03-04T10:06:37.035032+0000","last_clean":"2024-03-04T10:06:37.035032+0000","last_became_active":"2024-03-04T10:06:36.032515+0000","last_became_peered":"2024-03-04T10:06:36.032515+0000","last_unstale":"2024-03-04T10:06:37.035032+0000","last_undegraded":"2024-03-04T10:06:37.035032+0000","last_fullsized":"2024-03-04T10:06:37.035032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:43:31.861841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.042518+0000","last_change":"2024-03-04T10:06:36.035173+0000","last_active":"2024-03-04T10:06:36.042518+0000","last_peered":"2024-03-04T10:06:36.042518+0000","last_clean":"2024-03-04T10:06:36.042518+0000","last_became_active":"2024-03-04T10:06:36.034740+0000","last_became_peered":"2024-03-04T10:06:36.034740+0000","last_unstale":"2024-03-04T10:06:36.042518+0000","last_undegraded":"2024-03-04T10:06:36.042518+0000","last_fullsized":"2024-03-04T10:06:36.042518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:07:12.791348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021242+0000","last_change":"2024-03-04T10:06:36.028100+0000","last_active":"2024-03-04T10:06:37.021242+0000","last_peered":"2024-03-04T10:06:37.021242+0000","last_clean":"2024-03-04T10:06:37.021242+0000","last_became_active":"2024-03-04T10:06:36.027819+0000","last_became_peered":"2024-03-04T10:06:36.027819+0000","last_unstale":"2024-03-04T10:06:37.021242+0000","last_undegraded":"2024-03-04T10:06:37.021242+0000","last_fullsized":"2024-03-04T10:06:37.021242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:03:37.046386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120511+0000","last_change":"2024-03-04T10:06:36.036128+0000","last_active":"2024-03-04T10:06:37.120511+0000","last_peered":"2024-03-04T10:06:37.120511+0000","last_clean":"2024-03-04T10:06:37.120511+0000","last_became_active":"2024-03-04T10:06:36.035846+0000","last_became_peered":"2024-03-04T10:06:36.035846+0000","last_unstale":"2024-03-04T10:06:37.120511+0000","last_undegraded":"2024-03-04T10:06:37.120511+0000","last_fullsized":"2024-03-04T10:06:37.120511+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:58.051198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.033732+0000","last_change":"2024-03-04T10:06:36.030708+0000","last_active":"2024-03-04T10:06:36.033732+0000","last_peered":"2024-03-04T10:06:36.033732+0000","last_clean":"2024-03-04T10:06:36.033732+0000","last_became_active":"2024-03-04T10:06:36.030446+0000","last_became_peered":"2024-03-04T10:06:36.030446+0000","last_unstale":"2024-03-04T10:06:36.033732+0000","last_undegraded":"2024-03-04T10:06:36.033732+0000","last_fullsized":"2024-03-04T10:06:36.033732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:02:41.703028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033418+0000","last_change":"2024-03-04T10:06:36.042625+0000","last_active":"2024-03-04T10:06:37.033418+0000","last_peered":"2024-03-04T10:06:37.033418+0000","last_clean":"2024-03-04T10:06:37.033418+0000","last_became_active":"2024-03-04T10:06:36.041205+0000","last_became_peered":"2024-03-04T10:06:36.041205+0000","last_unstale":"2024-03-04T10:06:37.033418+0000","last_undegraded":"2024-03-04T10:06:37.033418+0000","last_fullsized":"2024-03-04T10:06:37.033418+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:59.436908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.041220+0000","last_change":"2024-03-04T10:06:36.038934+0000","last_active":"2024-03-04T10:06:36.041220+0000","last_peered":"2024-03-04T10:06:36.041220+0000","last_clean":"2024-03-04T10:06:36.041220+0000","last_became_active":"2024-03-04T10:06:36.038651+0000","last_became_peered":"2024-03-04T10:06:36.038651+0000","last_unstale":"2024-03-04T10:06:36.041220+0000","last_undegraded":"2024-03-04T10:06:36.041220+0000","last_fullsized":"2024-03-04T10:06:36.041220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:56.906275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.043972+0000","last_change":"2024-03-04T10:06:36.043184+0000","last_active":"2024-03-04T10:06:36.043972+0000","last_peered":"2024-03-04T10:06:36.043972+0000","last_clean":"2024-03-04T10:06:36.043972+0000","last_became_active":"2024-03-04T10:06:36.042871+0000","last_became_peered":"2024-03-04T10:06:36.042871+0000","last_unstale":"2024-03-04T10:06:36.043972+0000","last_undegraded":"2024-03-04T10:06:36.043972+0000","last_fullsized":"2024-03-04T10:06:36.043972+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:35:31.059597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021950+0000","last_change":"2024-03-04T10:06:36.037669+0000","last_active":"2024-03-04T10:06:37.021950+0000","last_peered":"2024-03-04T10:06:37.021950+0000","last_clean":"2024-03-04T10:06:37.021950+0000","last_became_active":"2024-03-04T10:06:36.037361+0000","last_became_peered":"2024-03-04T10:06:36.037361+0000","last_unstale":"2024-03-04T10:06:37.021950+0000","last_undegraded":"2024-03-04T10:06:37.021950+0000","last_fullsized":"2024-03-04T10:06:37.021950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:04:43.724815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033090+0000","last_change":"2024-03-04T10:06:36.037306+0000","last_active":"2024-03-04T10:06:37.033090+0000","last_peered":"2024-03-04T10:06:37.033090+0000","last_clean":"2024-03-04T10:06:37.033090+0000","last_became_active":"2024-03-04T10:06:36.036798+0000","last_became_peered":"2024-03-04T10:06:36.036798+0000","last_unstale":"2024-03-04T10:06:37.033090+0000","last_undegraded":"2024-03-04T10:06:37.033090+0000","last_fullsized":"2024-03-04T10:06:37.033090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:40.312621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033321+0000","last_change":"2024-03-04T10:06:36.034622+0000","last_active":"2024-03-04T10:06:37.033321+0000","last_peered":"2024-03-04T10:06:37.033321+0000","last_clean":"2024-03-04T10:06:37.033321+0000","last_became_active":"2024-03-04T10:06:36.034314+0000","last_became_peered":"2024-03-04T10:06:36.034314+0000","last_unstale":"2024-03-04T10:06:37.033321+0000","last_undegraded":"2024-03-04T10:06:37.033321+0000","last_fullsized":"2024-03-04T10:06:37.033321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:27.353573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021693+0000","last_change":"2024-03-04T10:06:36.027163+0000","last_active":"2024-03-04T10:06:37.021693+0000","last_peered":"2024-03-04T10:06:37.021693+0000","last_clean":"2024-03-04T10:06:37.021693+0000","last_became_active":"2024-03-04T10:06:36.026865+0000","last_became_peered":"2024-03-04T10:06:36.026865+0000","last_unstale":"2024-03-04T10:06:37.021693+0000","last_undegraded":"2024-03-04T10:06:37.021693+0000","last_fullsized":"2024-03-04T10:06:37.021693+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:48.186426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.045615+0000","last_change":"2024-03-04T10:06:36.043676+0000","last_active":"2024-03-04T10:06:36.045615+0000","last_peered":"2024-03-04T10:06:36.045615+0000","last_clean":"2024-03-04T10:06:36.045615+0000","last_became_active":"2024-03-04T10:06:36.042755+0000","last_became_peered":"2024-03-04T10:06:36.042755+0000","last_unstale":"2024-03-04T10:06:36.045615+0000","last_undegraded":"2024-03-04T10:06:36.045615+0000","last_fullsized":"2024-03-04T10:06:36.045615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:58.051198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022236+0000","last_change":"2024-03-04T10:06:36.034774+0000","last_active":"2024-03-04T10:06:37.022236+0000","last_peered":"2024-03-04T10:06:37.022236+0000","last_clean":"2024-03-04T10:06:37.022236+0000","last_became_active":"2024-03-04T10:06:36.034434+0000","last_became_peered":"2024-03-04T10:06:36.034434+0000","last_unstale":"2024-03-04T10:06:37.022236+0000","last_undegraded":"2024-03-04T10:06:37.022236+0000","last_fullsized":"2024-03-04T10:06:37.022236+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:02:41.703028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.032929+0000","last_change":"2024-03-04T10:06:36.038973+0000","last_active":"2024-03-04T10:06:37.032929+0000","last_peered":"2024-03-04T10:06:37.032929+0000","last_clean":"2024-03-04T10:06:37.032929+0000","last_became_active":"2024-03-04T10:06:36.038701+0000","last_became_peered":"2024-03-04T10:06:36.038701+0000","last_unstale":"2024-03-04T10:06:37.032929+0000","last_undegraded":"2024-03-04T10:06:37.032929+0000","last_fullsized":"2024-03-04T10:06:37.032929+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:54:57.834882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021858+0000","last_change":"2024-03-04T10:06:36.028912+0000","last_active":"2024-03-04T10:06:37.021858+0000","last_peered":"2024-03-04T10:06:37.021858+0000","last_clean":"2024-03-04T10:06:37.021858+0000","last_became_active":"2024-03-04T10:06:36.028639+0000","last_became_peered":"2024-03-04T10:06:36.028639+0000","last_unstale":"2024-03-04T10:06:37.021858+0000","last_undegraded":"2024-03-04T10:06:37.021858+0000","last_fullsized":"2024-03-04T10:06:37.021858+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:24.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019459+0000","last_change":"2024-03-04T10:06:36.032489+0000","last_active":"2024-03-04T10:06:37.019459+0000","last_peered":"2024-03-04T10:06:37.019459+0000","last_clean":"2024-03-04T10:06:37.019459+0000","last_became_active":"2024-03-04T10:06:36.031925+0000","last_became_peered":"2024-03-04T10:06:36.031925+0000","last_unstale":"2024-03-04T10:06:37.019459+0000","last_undegraded":"2024-03-04T10:06:37.019459+0000","last_fullsized":"2024-03-04T10:06:37.019459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:26:21.380677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120496+0000","last_change":"2024-03-04T10:06:36.038811+0000","last_active":"2024-03-04T10:06:37.120496+0000","last_peered":"2024-03-04T10:06:37.120496+0000","last_clean":"2024-03-04T10:06:37.120496+0000","last_became_active":"2024-03-04T10:06:36.038541+0000","last_became_peered":"2024-03-04T10:06:36.038541+0000","last_unstale":"2024-03-04T10:06:37.120496+0000","last_undegraded":"2024-03-04T10:06:37.120496+0000","last_fullsized":"2024-03-04T10:06:37.120496+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:37.097508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"65'4","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.048498+0000","last_change":"2024-03-04T10:06:35.027878+0000","last_active":"2024-03-04T10:06:36.048498+0000","last_peered":"2024-03-04T10:06:36.048498+0000","last_clean":"2024-03-04T10:06:36.048498+0000","last_became_active":"2024-03-04T10:06:35.027509+0000","last_became_peered":"2024-03-04T10:06:35.027509+0000","last_unstale":"2024-03-04T10:06:36.048498+0000","last_undegraded":"2024-03-04T10:06:36.048498+0000","last_fullsized":"2024-03-04T10:06:36.048498+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:50.371941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021880+0000","last_change":"2024-03-04T10:06:36.040950+0000","last_active":"2024-03-04T10:06:37.021880+0000","last_peered":"2024-03-04T10:06:37.021880+0000","last_clean":"2024-03-04T10:06:37.021880+0000","last_became_active":"2024-03-04T10:06:36.040241+0000","last_became_peered":"2024-03-04T10:06:36.040241+0000","last_unstale":"2024-03-04T10:06:37.021880+0000","last_undegraded":"2024-03-04T10:06:37.021880+0000","last_fullsized":"2024-03-04T10:06:37.021880+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:07:12.791348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019930+0000","last_change":"2024-03-04T10:06:35.027865+0000","last_active":"2024-03-04T10:06:37.019930+0000","last_peered":"2024-03-04T10:06:37.019930+0000","last_clean":"2024-03-04T10:06:37.019930+0000","last_became_active":"2024-03-04T10:06:35.027582+0000","last_became_peered":"2024-03-04T10:06:35.027582+0000","last_unstale":"2024-03-04T10:06:37.019930+0000","last_undegraded":"2024-03-04T10:06:37.019930+0000","last_fullsized":"2024-03-04T10:06:37.019930+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:02:40.698762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019882+0000","last_change":"2024-03-04T10:06:36.029392+0000","last_active":"2024-03-04T10:06:37.019882+0000","last_peered":"2024-03-04T10:06:37.019882+0000","last_clean":"2024-03-04T10:06:37.019882+0000","last_became_active":"2024-03-04T10:06:36.029107+0000","last_became_peered":"2024-03-04T10:06:36.029107+0000","last_unstale":"2024-03-04T10:06:37.019882+0000","last_undegraded":"2024-03-04T10:06:37.019882+0000","last_fullsized":"2024-03-04T10:06:37.019882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:43.094873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019862+0000","last_change":"2024-03-04T10:06:35.025520+0000","last_active":"2024-03-04T10:06:37.019862+0000","last_peered":"2024-03-04T10:06:37.019862+0000","last_clean":"2024-03-04T10:06:37.019862+0000","last_became_active":"2024-03-04T10:06:35.025291+0000","last_became_peered":"2024-03-04T10:06:35.025291+0000","last_unstale":"2024-03-04T10:06:37.019862+0000","last_undegraded":"2024-03-04T10:06:37.019862+0000","last_fullsized":"2024-03-04T10:06:37.019862+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:46.534320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022304+0000","last_change":"2024-03-04T10:06:36.037453+0000","last_active":"2024-03-04T10:06:37.022304+0000","last_peered":"2024-03-04T10:06:37.022304+0000","last_clean":"2024-03-04T10:06:37.022304+0000","last_became_active":"2024-03-04T10:06:36.036396+0000","last_became_peered":"2024-03-04T10:06:36.036396+0000","last_unstale":"2024-03-04T10:06:37.022304+0000","last_undegraded":"2024-03-04T10:06:37.022304+0000","last_fullsized":"2024-03-04T10:06:37.022304+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:03.897058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019709+0000","last_change":"2024-03-04T10:06:35.021094+0000","last_active":"2024-03-04T10:06:37.019709+0000","last_peered":"2024-03-04T10:06:37.019709+0000","last_clean":"2024-03-04T10:06:37.019709+0000","last_became_active":"2024-03-04T10:06:35.020827+0000","last_became_peered":"2024-03-04T10:06:35.020827+0000","last_unstale":"2024-03-04T10:06:37.019709+0000","last_undegraded":"2024-03-04T10:06:37.019709+0000","last_fullsized":"2024-03-04T10:06:37.019709+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:04:42.720549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035027+0000","last_change":"2024-03-04T10:06:36.038360+0000","last_active":"2024-03-04T10:06:37.035027+0000","last_peered":"2024-03-04T10:06:37.035027+0000","last_clean":"2024-03-04T10:06:37.035027+0000","last_became_active":"2024-03-04T10:06:36.038091+0000","last_became_peered":"2024-03-04T10:06:36.038091+0000","last_unstale":"2024-03-04T10:06:37.035027+0000","last_undegraded":"2024-03-04T10:06:37.035027+0000","last_fullsized":"2024-03-04T10:06:37.035027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:06:01.129479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"65'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.044882+0000","last_change":"2024-03-04T10:06:35.021148+0000","last_active":"2024-03-04T10:06:36.044882+0000","last_peered":"2024-03-04T10:06:36.044882+0000","last_clean":"2024-03-04T10:06:36.044882+0000","last_became_active":"2024-03-04T10:06:35.020874+0000","last_became_peered":"2024-03-04T10:06:35.020874+0000","last_unstale":"2024-03-04T10:06:36.044882+0000","last_undegraded":"2024-03-04T10:06:36.044882+0000","last_fullsized":"2024-03-04T10:06:36.044882+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:44.492900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.037668+0000","last_change":"2024-03-04T10:06:36.031669+0000","last_active":"2024-03-04T10:06:36.037668+0000","last_peered":"2024-03-04T10:06:36.037668+0000","last_clean":"2024-03-04T10:06:36.037668+0000","last_became_active":"2024-03-04T10:06:36.031389+0000","last_became_peered":"2024-03-04T10:06:36.031389+0000","last_unstale":"2024-03-04T10:06:36.037668+0000","last_undegraded":"2024-03-04T10:06:36.037668+0000","last_fullsized":"2024-03-04T10:06:36.037668+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:48.186426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"65'1","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.040716+0000","last_change":"2024-03-04T10:06:35.024941+0000","last_active":"2024-03-04T10:06:36.040716+0000","last_peered":"2024-03-04T10:06:36.040716+0000","last_clean":"2024-03-04T10:06:36.040716+0000","last_became_active":"2024-03-04T10:06:35.023439+0000","last_became_peered":"2024-03-04T10:06:35.023439+0000","last_unstale":"2024-03-04T10:06:36.040716+0000","last_undegraded":"2024-03-04T10:06:36.040716+0000","last_fullsized":"2024-03-04T10:06:36.040716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:58.432642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120267+0000","last_change":"2024-03-04T10:06:36.034298+0000","last_active":"2024-03-04T10:06:37.120267+0000","last_peered":"2024-03-04T10:06:37.120267+0000","last_clean":"2024-03-04T10:06:37.120267+0000","last_became_active":"2024-03-04T10:06:36.033748+0000","last_became_peered":"2024-03-04T10:06:36.033748+0000","last_unstale":"2024-03-04T10:06:37.120267+0000","last_undegraded":"2024-03-04T10:06:37.120267+0000","last_fullsized":"2024-03-04T10:06:37.120267+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:07:12.791348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021014+0000","last_change":"2024-03-04T10:06:35.024981+0000","last_active":"2024-03-04T10:06:36.021014+0000","last_peered":"2024-03-04T10:06:36.021014+0000","last_clean":"2024-03-04T10:06:36.021014+0000","last_became_active":"2024-03-04T10:06:35.024688+0000","last_became_peered":"2024-03-04T10:06:35.024688+0000","last_unstale":"2024-03-04T10:06:36.021014+0000","last_undegraded":"2024-03-04T10:06:36.021014+0000","last_fullsized":"2024-03-04T10:06:36.021014+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:36.102599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.033699+0000","last_change":"2024-03-04T10:06:36.026169+0000","last_active":"2024-03-04T10:06:36.033699+0000","last_peered":"2024-03-04T10:06:36.033699+0000","last_clean":"2024-03-04T10:06:36.033699+0000","last_became_active":"2024-03-04T10:06:36.025900+0000","last_became_peered":"2024-03-04T10:06:36.025900+0000","last_unstale":"2024-03-04T10:06:36.033699+0000","last_undegraded":"2024-03-04T10:06:36.033699+0000","last_fullsized":"2024-03-04T10:06:36.033699+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:04:43.724815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120243+0000","last_change":"2024-03-04T10:06:35.026897+0000","last_active":"2024-03-04T10:06:37.120243+0000","last_peered":"2024-03-04T10:06:37.120243+0000","last_clean":"2024-03-04T10:06:37.120243+0000","last_became_active":"2024-03-04T10:06:35.026612+0000","last_became_peered":"2024-03-04T10:06:35.026612+0000","last_unstale":"2024-03-04T10:06:37.120243+0000","last_undegraded":"2024-03-04T10:06:37.120243+0000","last_fullsized":"2024-03-04T10:06:37.120243+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:45:00.599578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.036912+0000","last_change":"2024-03-04T10:06:36.028989+0000","last_active":"2024-03-04T10:06:36.036912+0000","last_peered":"2024-03-04T10:06:36.036912+0000","last_clean":"2024-03-04T10:06:36.036912+0000","last_became_active":"2024-03-04T10:06:36.028715+0000","last_became_peered":"2024-03-04T10:06:36.028715+0000","last_unstale":"2024-03-04T10:06:36.036912+0000","last_undegraded":"2024-03-04T10:06:36.036912+0000","last_fullsized":"2024-03-04T10:06:36.036912+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:26:17.271066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022390+0000","last_change":"2024-03-04T10:06:35.026616+0000","last_active":"2024-03-04T10:06:37.022390+0000","last_peered":"2024-03-04T10:06:37.022390+0000","last_clean":"2024-03-04T10:06:37.022390+0000","last_became_active":"2024-03-04T10:06:35.026318+0000","last_became_peered":"2024-03-04T10:06:35.026318+0000","last_unstale":"2024-03-04T10:06:37.022390+0000","last_undegraded":"2024-03-04T10:06:37.022390+0000","last_fullsized":"2024-03-04T10:06:37.022390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:58.622316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.033661+0000","last_change":"2024-03-04T10:06:36.031415+0000","last_active":"2024-03-04T10:06:36.033661+0000","last_peered":"2024-03-04T10:06:36.033661+0000","last_clean":"2024-03-04T10:06:36.033661+0000","last_became_active":"2024-03-04T10:06:36.031139+0000","last_became_peered":"2024-03-04T10:06:36.031139+0000","last_unstale":"2024-03-04T10:06:36.033661+0000","last_undegraded":"2024-03-04T10:06:36.033661+0000","last_fullsized":"2024-03-04T10:06:36.033661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:03.897058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021745+0000","last_change":"2024-03-04T10:06:35.024831+0000","last_active":"2024-03-04T10:06:37.021745+0000","last_peered":"2024-03-04T10:06:37.021745+0000","last_clean":"2024-03-04T10:06:37.021745+0000","last_became_active":"2024-03-04T10:06:35.024497+0000","last_became_peered":"2024-03-04T10:06:35.024497+0000","last_unstale":"2024-03-04T10:06:37.021745+0000","last_undegraded":"2024-03-04T10:06:37.021745+0000","last_fullsized":"2024-03-04T10:06:37.021745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:02:33.360463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120558+0000","last_change":"2024-03-04T10:06:36.031607+0000","last_active":"2024-03-04T10:06:37.120558+0000","last_peered":"2024-03-04T10:06:37.120558+0000","last_clean":"2024-03-04T10:06:37.120558+0000","last_became_active":"2024-03-04T10:06:36.031346+0000","last_became_peered":"2024-03-04T10:06:36.031346+0000","last_unstale":"2024-03-04T10:06:37.120558+0000","last_undegraded":"2024-03-04T10:06:37.120558+0000","last_fullsized":"2024-03-04T10:06:37.120558+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:18.529274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"65'3","reported_seq":17,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021691+0000","last_change":"2024-03-04T10:06:35.027996+0000","last_active":"2024-03-04T10:06:37.021691+0000","last_peered":"2024-03-04T10:06:37.021691+0000","last_clean":"2024-03-04T10:06:37.021691+0000","last_became_active":"2024-03-04T10:06:35.027167+0000","last_became_peered":"2024-03-04T10:06:35.027167+0000","last_unstale":"2024-03-04T10:06:37.021691+0000","last_undegraded":"2024-03-04T10:06:37.021691+0000","last_fullsized":"2024-03-04T10:06:37.021691+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:35:07.631737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021362+0000","last_change":"2024-03-04T10:06:36.034517+0000","last_active":"2024-03-04T10:06:37.021362+0000","last_peered":"2024-03-04T10:06:37.021362+0000","last_clean":"2024-03-04T10:06:37.021362+0000","last_became_active":"2024-03-04T10:06:36.034240+0000","last_became_peered":"2024-03-04T10:06:36.034240+0000","last_unstale":"2024-03-04T10:06:37.021362+0000","last_undegraded":"2024-03-04T10:06:37.021362+0000","last_fullsized":"2024-03-04T10:06:37.021362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:16:51.376207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"65'6","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.050794+0000","last_change":"2024-03-04T10:06:35.026764+0000","last_active":"2024-03-04T10:06:36.050794+0000","last_peered":"2024-03-04T10:06:36.050794+0000","last_clean":"2024-03-04T10:06:36.050794+0000","last_became_active":"2024-03-04T10:06:35.026565+0000","last_became_peered":"2024-03-04T10:06:35.026565+0000","last_unstale":"2024-03-04T10:06:36.050794+0000","last_undegraded":"2024-03-04T10:06:36.050794+0000","last_fullsized":"2024-03-04T10:06:36.050794+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:38:04.206144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021933+0000","last_change":"2024-03-04T10:06:36.033313+0000","last_active":"2024-03-04T10:06:37.021933+0000","last_peered":"2024-03-04T10:06:37.021933+0000","last_clean":"2024-03-04T10:06:37.021933+0000","last_became_active":"2024-03-04T10:06:36.032995+0000","last_became_peered":"2024-03-04T10:06:36.032995+0000","last_unstale":"2024-03-04T10:06:37.021933+0000","last_undegraded":"2024-03-04T10:06:37.021933+0000","last_fullsized":"2024-03-04T10:06:37.021933+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:31.584231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022217+0000","last_change":"2024-03-04T10:06:35.027520+0000","last_active":"2024-03-04T10:06:37.022217+0000","last_peered":"2024-03-04T10:06:37.022217+0000","last_clean":"2024-03-04T10:06:37.022217+0000","last_became_active":"2024-03-04T10:06:35.026481+0000","last_became_peered":"2024-03-04T10:06:35.026481+0000","last_unstale":"2024-03-04T10:06:37.022217+0000","last_undegraded":"2024-03-04T10:06:37.022217+0000","last_fullsized":"2024-03-04T10:06:37.022217+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:12:00.966883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033288+0000","last_change":"2024-03-04T10:06:36.036080+0000","last_active":"2024-03-04T10:06:37.033288+0000","last_peered":"2024-03-04T10:06:37.033288+0000","last_clean":"2024-03-04T10:06:37.033288+0000","last_became_active":"2024-03-04T10:06:36.035750+0000","last_became_peered":"2024-03-04T10:06:36.035750+0000","last_unstale":"2024-03-04T10:06:37.033288+0000","last_undegraded":"2024-03-04T10:06:37.033288+0000","last_fullsized":"2024-03-04T10:06:37.033288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:43:37.106865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033116+0000","last_change":"2024-03-04T10:06:35.024719+0000","last_active":"2024-03-04T10:06:37.033116+0000","last_peered":"2024-03-04T10:06:37.033116+0000","last_clean":"2024-03-04T10:06:37.033116+0000","last_became_active":"2024-03-04T10:06:35.024397+0000","last_became_peered":"2024-03-04T10:06:35.024397+0000","last_unstale":"2024-03-04T10:06:37.033116+0000","last_undegraded":"2024-03-04T10:06:37.033116+0000","last_fullsized":"2024-03-04T10:06:37.033116+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:30.579965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120274+0000","last_change":"2024-03-04T10:06:36.035411+0000","last_active":"2024-03-04T10:06:37.120274+0000","last_peered":"2024-03-04T10:06:37.120274+0000","last_clean":"2024-03-04T10:06:37.120274+0000","last_became_active":"2024-03-04T10:06:36.035119+0000","last_became_peered":"2024-03-04T10:06:36.035119+0000","last_unstale":"2024-03-04T10:06:37.120274+0000","last_undegraded":"2024-03-04T10:06:37.120274+0000","last_fullsized":"2024-03-04T10:06:37.120274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:31.584231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021479+0000","last_change":"2024-03-04T10:06:35.026894+0000","last_active":"2024-03-04T10:06:37.021479+0000","last_peered":"2024-03-04T10:06:37.021479+0000","last_clean":"2024-03-04T10:06:37.021479+0000","last_became_active":"2024-03-04T10:06:35.026599+0000","last_became_peered":"2024-03-04T10:06:35.026599+0000","last_unstale":"2024-03-04T10:06:37.021479+0000","last_undegraded":"2024-03-04T10:06:37.021479+0000","last_fullsized":"2024-03-04T10:06:37.021479+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:23.266860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.039838+0000","last_change":"2024-03-04T10:06:36.036066+0000","last_active":"2024-03-04T10:06:36.039838+0000","last_peered":"2024-03-04T10:06:36.039838+0000","last_clean":"2024-03-04T10:06:36.039838+0000","last_became_active":"2024-03-04T10:06:36.035794+0000","last_became_peered":"2024-03-04T10:06:36.035794+0000","last_unstale":"2024-03-04T10:06:36.039838+0000","last_undegraded":"2024-03-04T10:06:36.039838+0000","last_fullsized":"2024-03-04T10:06:36.039838+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:31.584231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"65'8","reported_seq":22,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022045+0000","last_change":"2024-03-04T10:06:35.027077+0000","last_active":"2024-03-04T10:06:37.022045+0000","last_peered":"2024-03-04T10:06:37.022045+0000","last_clean":"2024-03-04T10:06:37.022045+0000","last_became_active":"2024-03-04T10:06:35.026795+0000","last_became_peered":"2024-03-04T10:06:35.026795+0000","last_unstale":"2024-03-04T10:06:37.022045+0000","last_undegraded":"2024-03-04T10:06:37.022045+0000","last_fullsized":"2024-03-04T10:06:37.022045+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:11.700722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021081+0000","last_change":"2024-03-04T10:06:36.031067+0000","last_active":"2024-03-04T10:06:37.021081+0000","last_peered":"2024-03-04T10:06:37.021081+0000","last_clean":"2024-03-04T10:06:37.021081+0000","last_became_active":"2024-03-04T10:06:36.030780+0000","last_became_peered":"2024-03-04T10:06:36.030780+0000","last_unstale":"2024-03-04T10:06:37.021081+0000","last_undegraded":"2024-03-04T10:06:37.021081+0000","last_fullsized":"2024-03-04T10:06:37.021081+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:11:29.231443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033229+0000","last_change":"2024-03-04T10:06:35.025447+0000","last_active":"2024-03-04T10:06:37.033229+0000","last_peered":"2024-03-04T10:06:37.033229+0000","last_clean":"2024-03-04T10:06:37.033229+0000","last_became_active":"2024-03-04T10:06:35.024728+0000","last_became_peered":"2024-03-04T10:06:35.024728+0000","last_unstale":"2024-03-04T10:06:37.033229+0000","last_undegraded":"2024-03-04T10:06:37.033229+0000","last_fullsized":"2024-03-04T10:06:37.033229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:23.266860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120589+0000","last_change":"2024-03-04T10:06:36.036751+0000","last_active":"2024-03-04T10:06:37.120589+0000","last_peered":"2024-03-04T10:06:37.120589+0000","last_clean":"2024-03-04T10:06:37.120589+0000","last_became_active":"2024-03-04T10:06:36.036484+0000","last_became_peered":"2024-03-04T10:06:36.036484+0000","last_unstale":"2024-03-04T10:06:37.120589+0000","last_undegraded":"2024-03-04T10:06:37.120589+0000","last_fullsized":"2024-03-04T10:06:37.120589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:24.271126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022184+0000","last_change":"2024-03-04T10:06:35.026636+0000","last_active":"2024-03-04T10:06:37.022184+0000","last_peered":"2024-03-04T10:06:37.022184+0000","last_clean":"2024-03-04T10:06:37.022184+0000","last_became_active":"2024-03-04T10:06:35.026298+0000","last_became_peered":"2024-03-04T10:06:35.026298+0000","last_unstale":"2024-03-04T10:06:37.022184+0000","last_undegraded":"2024-03-04T10:06:37.022184+0000","last_fullsized":"2024-03-04T10:06:37.022184+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.312358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021636+0000","last_change":"2024-03-04T10:06:36.041137+0000","last_active":"2024-03-04T10:06:37.021636+0000","last_peered":"2024-03-04T10:06:37.021636+0000","last_clean":"2024-03-04T10:06:37.021636+0000","last_became_active":"2024-03-04T10:06:36.040283+0000","last_became_peered":"2024-03-04T10:06:36.040283+0000","last_unstale":"2024-03-04T10:06:37.021636+0000","last_undegraded":"2024-03-04T10:06:37.021636+0000","last_fullsized":"2024-03-04T10:06:37.021636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:51.363438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"65'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.021792+0000","last_change":"2024-03-04T10:06:35.027753+0000","last_active":"2024-03-04T10:06:37.021792+0000","last_peered":"2024-03-04T10:06:37.021792+0000","last_clean":"2024-03-04T10:06:37.021792+0000","last_became_active":"2024-03-04T10:06:35.027464+0000","last_became_peered":"2024-03-04T10:06:35.027464+0000","last_unstale":"2024-03-04T10:06:37.021792+0000","last_undegraded":"2024-03-04T10:06:37.021792+0000","last_fullsized":"2024-03-04T10:06:37.021792+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:01.895797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120547+0000","last_change":"2024-03-04T10:06:36.038584+0000","last_active":"2024-03-04T10:06:37.120547+0000","last_peered":"2024-03-04T10:06:37.120547+0000","last_clean":"2024-03-04T10:06:37.120547+0000","last_became_active":"2024-03-04T10:06:36.038300+0000","last_became_peered":"2024-03-04T10:06:36.038300+0000","last_unstale":"2024-03-04T10:06:37.120547+0000","last_undegraded":"2024-03-04T10:06:37.120547+0000","last_fullsized":"2024-03-04T10:06:37.120547+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:02.900063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021741+0000","last_change":"2024-03-04T10:06:35.022355+0000","last_active":"2024-03-04T10:06:36.021741+0000","last_peered":"2024-03-04T10:06:36.021741+0000","last_clean":"2024-03-04T10:06:36.021741+0000","last_became_active":"2024-03-04T10:06:35.022117+0000","last_became_peered":"2024-03-04T10:06:35.022117+0000","last_unstale":"2024-03-04T10:06:36.021741+0000","last_undegraded":"2024-03-04T10:06:36.021741+0000","last_fullsized":"2024-03-04T10:06:36.021741+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:54:26.349307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019514+0000","last_change":"2024-03-04T10:06:36.032226+0000","last_active":"2024-03-04T10:06:37.019514+0000","last_peered":"2024-03-04T10:06:37.019514+0000","last_clean":"2024-03-04T10:06:37.019514+0000","last_became_active":"2024-03-04T10:06:36.030080+0000","last_became_peered":"2024-03-04T10:06:36.030080+0000","last_unstale":"2024-03-04T10:06:37.019514+0000","last_undegraded":"2024-03-04T10:06:37.019514+0000","last_fullsized":"2024-03-04T10:06:37.019514+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:17:09.463816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019770+0000","last_change":"2024-03-04T10:06:35.024457+0000","last_active":"2024-03-04T10:06:37.019770+0000","last_peered":"2024-03-04T10:06:37.019770+0000","last_clean":"2024-03-04T10:06:37.019770+0000","last_became_active":"2024-03-04T10:06:35.024204+0000","last_became_peered":"2024-03-04T10:06:35.024204+0000","last_unstale":"2024-03-04T10:06:37.019770+0000","last_undegraded":"2024-03-04T10:06:37.019770+0000","last_fullsized":"2024-03-04T10:06:37.019770+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:25:02.892792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033452+0000","last_change":"2024-03-04T10:06:36.028762+0000","last_active":"2024-03-04T10:06:37.033452+0000","last_peered":"2024-03-04T10:06:37.033452+0000","last_clean":"2024-03-04T10:06:37.033452+0000","last_became_active":"2024-03-04T10:06:36.028441+0000","last_became_peered":"2024-03-04T10:06:36.028441+0000","last_unstale":"2024-03-04T10:06:37.033452+0000","last_undegraded":"2024-03-04T10:06:37.033452+0000","last_fullsized":"2024-03-04T10:06:37.033452+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:47.538586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033042+0000","last_change":"2024-03-04T10:06:35.024601+0000","last_active":"2024-03-04T10:06:37.033042+0000","last_peered":"2024-03-04T10:06:37.033042+0000","last_clean":"2024-03-04T10:06:37.033042+0000","last_became_active":"2024-03-04T10:06:35.024294+0000","last_became_peered":"2024-03-04T10:06:35.024294+0000","last_unstale":"2024-03-04T10:06:37.033042+0000","last_undegraded":"2024-03-04T10:06:37.033042+0000","last_fullsized":"2024-03-04T10:06:37.033042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:36.093242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120553+0000","last_change":"2024-03-04T10:06:36.034033+0000","last_active":"2024-03-04T10:06:37.120553+0000","last_peered":"2024-03-04T10:06:37.120553+0000","last_clean":"2024-03-04T10:06:37.120553+0000","last_became_active":"2024-03-04T10:06:36.033392+0000","last_became_peered":"2024-03-04T10:06:36.033392+0000","last_unstale":"2024-03-04T10:06:37.120553+0000","last_undegraded":"2024-03-04T10:06:37.120553+0000","last_fullsized":"2024-03-04T10:06:37.120553+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:11.316624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019515+0000","last_change":"2024-03-04T10:06:35.027805+0000","last_active":"2024-03-04T10:06:37.019515+0000","last_peered":"2024-03-04T10:06:37.019515+0000","last_clean":"2024-03-04T10:06:37.019515+0000","last_became_active":"2024-03-04T10:06:35.027514+0000","last_became_peered":"2024-03-04T10:06:35.027514+0000","last_unstale":"2024-03-04T10:06:37.019515+0000","last_undegraded":"2024-03-04T10:06:37.019515+0000","last_fullsized":"2024-03-04T10:06:37.019515+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:42:58.432642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019738+0000","last_change":"2024-03-04T10:06:36.023611+0000","last_active":"2024-03-04T10:06:37.019738+0000","last_peered":"2024-03-04T10:06:37.019738+0000","last_clean":"2024-03-04T10:06:37.019738+0000","last_became_active":"2024-03-04T10:06:36.023345+0000","last_became_peered":"2024-03-04T10:06:36.023345+0000","last_unstale":"2024-03-04T10:06:37.019738+0000","last_undegraded":"2024-03-04T10:06:37.019738+0000","last_fullsized":"2024-03-04T10:06:37.019738+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:48.254040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"65'2","reported_seq":16,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033444+0000","last_change":"2024-03-04T10:06:35.027195+0000","last_active":"2024-03-04T10:06:37.033444+0000","last_peered":"2024-03-04T10:06:37.033444+0000","last_clean":"2024-03-04T10:06:37.033444+0000","last_became_active":"2024-03-04T10:06:35.026880+0000","last_became_peered":"2024-03-04T10:06:35.026880+0000","last_unstale":"2024-03-04T10:06:37.033444+0000","last_undegraded":"2024-03-04T10:06:37.033444+0000","last_fullsized":"2024-03-04T10:06:37.033444+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.312358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035276+0000","last_change":"2024-03-04T10:06:36.026853+0000","last_active":"2024-03-04T10:06:37.035276+0000","last_peered":"2024-03-04T10:06:37.035276+0000","last_clean":"2024-03-04T10:06:37.035276+0000","last_became_active":"2024-03-04T10:06:36.026566+0000","last_became_peered":"2024-03-04T10:06:36.026566+0000","last_unstale":"2024-03-04T10:06:37.035276+0000","last_undegraded":"2024-03-04T10:06:37.035276+0000","last_fullsized":"2024-03-04T10:06:37.035276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:31.815185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"66'1","reported_seq":15,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.034928+0000","last_change":"2024-03-04T10:06:35.025158+0000","last_active":"2024-03-04T10:06:37.034928+0000","last_peered":"2024-03-04T10:06:37.034928+0000","last_clean":"2024-03-04T10:06:37.034928+0000","last_became_active":"2024-03-04T10:06:35.024610+0000","last_became_peered":"2024-03-04T10:06:35.024610+0000","last_unstale":"2024-03-04T10:06:37.034928+0000","last_undegraded":"2024-03-04T10:06:37.034928+0000","last_fullsized":"2024-03-04T10:06:37.034928+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:57.046932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.033168+0000","last_change":"2024-03-04T10:06:36.036054+0000","last_active":"2024-03-04T10:06:37.033168+0000","last_peered":"2024-03-04T10:06:37.033168+0000","last_clean":"2024-03-04T10:06:37.033168+0000","last_became_active":"2024-03-04T10:06:36.035767+0000","last_became_peered":"2024-03-04T10:06:36.035767+0000","last_unstale":"2024-03-04T10:06:37.033168+0000","last_undegraded":"2024-03-04T10:06:37.033168+0000","last_fullsized":"2024-03-04T10:06:37.033168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:11:29.231443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.019806+0000","last_change":"2024-03-04T10:06:35.024127+0000","last_active":"2024-03-04T10:06:37.019806+0000","last_peered":"2024-03-04T10:06:37.019806+0000","last_clean":"2024-03-04T10:06:37.019806+0000","last_became_active":"2024-03-04T10:06:35.023856+0000","last_became_peered":"2024-03-04T10:06:35.023856+0000","last_unstale":"2024-03-04T10:06:37.019806+0000","last_undegraded":"2024-03-04T10:06:37.019806+0000","last_fullsized":"2024-03-04T10:06:37.019806+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:46:55.902008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120654+0000","last_change":"2024-03-04T10:06:36.037857+0000","last_active":"2024-03-04T10:06:37.120654+0000","last_peered":"2024-03-04T10:06:37.120654+0000","last_clean":"2024-03-04T10:06:37.120654+0000","last_became_active":"2024-03-04T10:06:36.037619+0000","last_became_peered":"2024-03-04T10:06:36.037619+0000","last_unstale":"2024-03-04T10:06:37.120654+0000","last_undegraded":"2024-03-04T10:06:37.120654+0000","last_fullsized":"2024-03-04T10:06:37.120654+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.626582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.024006+0000","last_change":"2024-03-04T10:06:35.025226+0000","last_active":"2024-03-04T10:06:36.024006+0000","last_peered":"2024-03-04T10:06:36.024006+0000","last_clean":"2024-03-04T10:06:36.024006+0000","last_became_active":"2024-03-04T10:06:35.024979+0000","last_became_peered":"2024-03-04T10:06:35.024979+0000","last_unstale":"2024-03-04T10:06:36.024006+0000","last_undegraded":"2024-03-04T10:06:36.024006+0000","last_fullsized":"2024-03-04T10:06:36.024006+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:03:36.042120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120220+0000","last_change":"2024-03-04T10:06:36.038811+0000","last_active":"2024-03-04T10:06:37.120220+0000","last_peered":"2024-03-04T10:06:37.120220+0000","last_clean":"2024-03-04T10:06:37.120220+0000","last_became_active":"2024-03-04T10:06:36.038523+0000","last_became_peered":"2024-03-04T10:06:36.038523+0000","last_unstale":"2024-03-04T10:06:37.120220+0000","last_undegraded":"2024-03-04T10:06:37.120220+0000","last_fullsized":"2024-03-04T10:06:37.120220+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:39:24.361515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022296+0000","last_change":"2024-03-04T10:06:35.028067+0000","last_active":"2024-03-04T10:06:37.022296+0000","last_peered":"2024-03-04T10:06:37.022296+0000","last_clean":"2024-03-04T10:06:37.022296+0000","last_became_active":"2024-03-04T10:06:35.027770+0000","last_became_peered":"2024-03-04T10:06:35.027770+0000","last_unstale":"2024-03-04T10:06:37.022296+0000","last_undegraded":"2024-03-04T10:06:37.022296+0000","last_fullsized":"2024-03-04T10:06:37.022296+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:17.525008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.020921+0000","last_change":"2024-03-04T10:06:36.044794+0000","last_active":"2024-03-04T10:06:37.020921+0000","last_peered":"2024-03-04T10:06:37.020921+0000","last_clean":"2024-03-04T10:06:37.020921+0000","last_became_active":"2024-03-04T10:06:36.043521+0000","last_became_peered":"2024-03-04T10:06:36.043521+0000","last_unstale":"2024-03-04T10:06:37.020921+0000","last_undegraded":"2024-03-04T10:06:37.020921+0000","last_fullsized":"2024-03-04T10:06:37.020921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:38:05.210411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021964+0000","last_change":"2024-03-04T10:06:35.021230+0000","last_active":"2024-03-04T10:06:36.021964+0000","last_peered":"2024-03-04T10:06:36.021964+0000","last_clean":"2024-03-04T10:06:36.021964+0000","last_became_active":"2024-03-04T10:06:35.020981+0000","last_became_peered":"2024-03-04T10:06:35.020981+0000","last_unstale":"2024-03-04T10:06:36.021964+0000","last_undegraded":"2024-03-04T10:06:36.021964+0000","last_fullsized":"2024-03-04T10:06:36.021964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:46.534320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.037113+0000","last_change":"2024-03-04T10:06:36.031011+0000","last_active":"2024-03-04T10:06:36.037113+0000","last_peered":"2024-03-04T10:06:36.037113+0000","last_clean":"2024-03-04T10:06:36.037113+0000","last_became_active":"2024-03-04T10:06:36.030704+0000","last_became_peered":"2024-03-04T10:06:36.030704+0000","last_unstale":"2024-03-04T10:06:36.037113+0000","last_undegraded":"2024-03-04T10:06:36.037113+0000","last_fullsized":"2024-03-04T10:06:36.037113+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:21:51.363438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.021147+0000","last_change":"2024-03-04T10:06:35.024535+0000","last_active":"2024-03-04T10:06:36.021147+0000","last_peered":"2024-03-04T10:06:36.021147+0000","last_clean":"2024-03-04T10:06:36.021147+0000","last_became_active":"2024-03-04T10:06:35.024266+0000","last_became_peered":"2024-03-04T10:06:35.024266+0000","last_unstale":"2024-03-04T10:06:36.021147+0000","last_undegraded":"2024-03-04T10:06:36.021147+0000","last_fullsized":"2024-03-04T10:06:36.021147+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:11:28.227177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.035139+0000","last_change":"2024-03-04T10:06:36.032224+0000","last_active":"2024-03-04T10:06:37.035139+0000","last_peered":"2024-03-04T10:06:37.035139+0000","last_clean":"2024-03-04T10:06:37.035139+0000","last_became_active":"2024-03-04T10:06:36.031948+0000","last_became_peered":"2024-03-04T10:06:36.031948+0000","last_unstale":"2024-03-04T10:06:37.035139+0000","last_undegraded":"2024-03-04T10:06:37.035139+0000","last_fullsized":"2024-03-04T10:06:37.035139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:47.065602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.022464+0000","last_change":"2024-03-04T10:06:35.024809+0000","last_active":"2024-03-04T10:06:36.022464+0000","last_peered":"2024-03-04T10:06:36.022464+0000","last_clean":"2024-03-04T10:06:36.022464+0000","last_became_active":"2024-03-04T10:06:35.024441+0000","last_became_peered":"2024-03-04T10:06:35.024441+0000","last_unstale":"2024-03-04T10:06:36.022464+0000","last_undegraded":"2024-03-04T10:06:36.022464+0000","last_fullsized":"2024-03-04T10:06:36.022464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:24.247408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.120615+0000","last_change":"2024-03-04T10:06:36.039508+0000","last_active":"2024-03-04T10:06:37.120615+0000","last_peered":"2024-03-04T10:06:37.120615+0000","last_clean":"2024-03-04T10:06:37.120615+0000","last_became_active":"2024-03-04T10:06:36.039236+0000","last_became_peered":"2024-03-04T10:06:36.039236+0000","last_unstale":"2024-03-04T10:06:37.120615+0000","last_undegraded":"2024-03-04T10:06:37.120615+0000","last_fullsized":"2024-03-04T10:06:37.120615+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:35.004946+0000","last_clean_scrub_stamp":"2024-03-04T10:06:35.004946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:35:08.636003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-04T10:06:36.020744+0000","last_change":"2024-03-04T10:06:35.019446+0000","last_active":"2024-03-04T10:06:36.020744+0000","last_peered":"2024-03-04T10:06:36.020744+0000","last_clean":"2024-03-04T10:06:36.020744+0000","last_became_active":"2024-03-04T10:06:35.019179+0000","last_became_peered":"2024-03-04T10:06:35.019179+0000","last_unstale":"2024-03-04T10:06:36.020744+0000","last_undegraded":"2024-03-04T10:06:36.020744+0000","last_fullsized":"2024-03-04T10:06:36.020744+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:06:34.000680+0000","last_clean_scrub_stamp":"2024-03-04T10:06:34.000680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:39.308355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":263,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-04T10:06:37.022069+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:06:37.022069+0000","last_peered":"2024-03-04T10:06:37.022069+0000","last_clean":"2024-03-04T10:06:37.022069+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:06:37.022069+0000","last_undegraded":"2024-03-04T10:06:37.022069+0000","last_fullsized":"2024-03-04T10:06:37.022069+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28392,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343448,"statfs":{"total":96636764160,"available":96607690752,"internally_reserved":0,"allocated":1396736,"data_stored":949378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":358268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":790528,"data_stored":358256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":359846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":358234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":364926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":815104,"data_stored":360610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28372,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343468,"statfs":{"total":96636764160,"available":96607711232,"internally_reserved":0,"allocated":1380352,"data_stored":948602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:06:39.334 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T10:06:39.334 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T10:06:39.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.475+0000 7face9606640 1 Processor -- start 2024-03-04T10:06:39.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.475+0000 7face9606640 1 -- start start 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face9606640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face415a810 0x7face40b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face9606640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7face406d3c0 0x7face40b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face9606640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face406d8a0 con 0x7face40cbfb0 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face9606640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face406db80 con 0x7face406d3c0 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face9606640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face406de60 con 0x7face415a810 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face2ffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face415a810 0x7face40b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face27fc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7face406d3c0 0x7face40b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face37fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7face40cbfb0 0x7face415abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:45858/0 (socket says 172.21.15.99:45858) 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.476+0000 7face37fe640 1 -- 172.21.15.99:0/2517898095 learned_addr learned my addr 172.21.15.99:0/2517898095 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.477+0000 7face1ffb640 1 -- 172.21.15.99:0/2517898095 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851738992 0 0) 0x7face406d8a0 con 0x7face40cbfb0 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.477+0000 7face2ffd640 1 -- 172.21.15.99:0/2517898095 >> v2:172.21.15.162:3300/0 conn(0x7face406d3c0 msgr2=0x7face40b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.477+0000 7face2ffd640 1 --2- 172.21.15.99:0/2517898095 >> v2:172.21.15.162:3300/0 conn(0x7face406d3c0 0x7face40b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.477+0000 7face2ffd640 1 -- 172.21.15.99:0/2517898095 >> v1:172.21.15.162:6789/0 conn(0x7face40cbfb0 legacy=0x7face415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.477+0000 7face2ffd640 1 -- 172.21.15.99:0/2517898095 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face406e140 con 0x7face415a810 2024-03-04T10:06:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.477+0000 7face2ffd640 1 --2- 172.21.15.99:0/2517898095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face415a810 0x7face40b4190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7facd4009d90 tx=0x7facd402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c067bfb2b5350aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face1ffb640 1 -- 172.21.15.99:0/2517898095 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4039070 con 0x7face415a810 2024-03-04T10:06:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face1ffb640 1 -- 172.21.15.99:0/2517898095 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd402cc50 con 0x7face415a810 2024-03-04T10:06:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face1ffb640 1 -- 172.21.15.99:0/2517898095 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4034440 con 0x7face415a810 2024-03-04T10:06:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 -- 172.21.15.99:0/2517898095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face415a810 msgr2=0x7face40b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 --2- 172.21.15.99:0/2517898095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face415a810 0x7face40b4190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7facd4009d90 tx=0x7facd402bea0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 -- 172.21.15.99:0/2517898095 shutdown_connections 2024-03-04T10:06:39.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 --2- 172.21.15.99:0/2517898095 >> v2:172.21.15.162:3300/0 conn(0x7face406d3c0 0x7face40b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 --2- 172.21.15.99:0/2517898095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face415a810 0x7face40b4190 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 -- 172.21.15.99:0/2517898095 >> 172.21.15.99:0/2517898095 conn(0x7face406acd0 msgr2=0x7face406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:39.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 -- 172.21.15.99:0/2517898095 shutdown_connections 2024-03-04T10:06:39.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.478+0000 7face9606640 1 -- 172.21.15.99:0/2517898095 wait complete. 2024-03-04T10:06:39.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.479+0000 7face9606640 1 Processor -- start 2024-03-04T10:06:39.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 -- start start 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7face406d3c0 0x7face40a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face40cbfb0 0x7face4149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 0x7face40a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face415e3c0 con 0x7face415a810 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face40b3c10 con 0x7face406d3c0 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.480+0000 7face9606640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face415e540 con 0x7face40cbfb0 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 0x7face40a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 0x7face40a8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38186/0 (socket says 172.21.15.99:38186) 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 -- 172.21.15.99:0/4279195215 learned_addr learned my addr 172.21.15.99:0/4279195215 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face27fc640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face40cbfb0 0x7face4149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face2ffd640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7face406d3c0 0x7face40a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 -- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face40cbfb0 msgr2=0x7face4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face40cbfb0 0x7face4149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 -- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7face406d3c0 msgr2=0x7face40a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7face406d3c0 0x7face40a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 -- 172.21.15.99:0/4279195215 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face40a8f20 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face37fe640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 0x7face40a8460 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c450 tx=0x7facdc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc018070 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc007500 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face40a9260 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc004a20 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.481+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face40a9aa0 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.482+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face40a9ca0 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.483+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7facdc01b070 con 0x7face415a810 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.483+0000 7faccb7fe640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd006b4d0 0x7facd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.483+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7facdc0894c0 con 0x7face415a810 2024-03-04T10:06:39.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.483+0000 7face2ffd640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd006b4d0 0x7facd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.484+0000 7face2ffd640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd006b4d0 0x7facd006d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7facd4036000 tx=0x7facd4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.487+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7face40a9ca0 con 0x7face415a810 2024-03-04T10:06:39.651 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:39.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:06:39.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.725+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7face4074f60 con 0x7face415a810 2024-03-04T10:06:39.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.727+0000 7faccb7fe640 1 -- 172.21.15.99:0/4279195215 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7face40a9260 con 0x7face415a810 2024-03-04T10:06:39.730 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-04T10:06:39.730 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:06:39.730 INFO:teuthology.orchestra.run.smithi099.stdout: caps mds = "allow" 2024-03-04T10:06:39.730 INFO:teuthology.orchestra.run.smithi099.stdout: caps mon = "allow r" 2024-03-04T10:06:39.730 INFO:teuthology.orchestra.run.smithi099.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T10:06:39.730 INFO:teuthology.orchestra.run.smithi099.stderr:updated caps for client.0 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd006b4d0 msgr2=0x7facd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd006b4d0 0x7facd006d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7facd4036000 tx=0x7facd4030000 comp rx=0 tx=0).stop 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 msgr2=0x7face40a8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 0x7face40a8460 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c450 tx=0x7facdc00c910 comp rx=0 tx=0).stop 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 shutdown_connections 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd006b4d0 0x7facd006d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.732 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7face415a810 0x7face40a8460 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7face40cbfb0 0x7face4149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 --2- 172.21.15.99:0/4279195215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7face406d3c0 0x7face40a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 >> 172.21.15.99:0/4279195215 conn(0x7face406acd0 msgr2=0x7face40ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:39.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 shutdown_connections 2024-03-04T10:06:39.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.729+0000 7face9606640 1 -- 172.21.15.99:0/4279195215 wait complete. 2024-03-04T10:06:39.743 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:39.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.880+0000 7fb7412c6640 1 Processor -- start 2024-03-04T10:06:39.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb7412c6640 1 -- start start 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb7412c6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb73c15de00 0x7fb73c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb7412c6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c0b5470 0x7fb73c0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb7412c6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c15e1e0 con 0x7fb73c0b9380 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb7412c6640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73c0d2820 con 0x7fb73c15de00 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb7412c6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0d1c10 con 0x7fb73c0b5470 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb73affd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb73c15de00 0x7fb73c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb73affd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb73c15de00 0x7fb73c0b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38200/0 (socket says 172.21.15.99:38200) 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.881+0000 7fb73affd640 1 -- 172.21.15.99:0/4001067887 learned_addr learned my addr 172.21.15.99:0/4001067887 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.882+0000 7fb73affd640 1 -- 172.21.15.99:0/4001067887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c0b5470 msgr2=0x7fb73c0b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.882+0000 7fb73affd640 1 --2- 172.21.15.99:0/4001067887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c0b5470 0x7fb73c0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.882+0000 7fb73affd640 1 -- 172.21.15.99:0/4001067887 >> v1:172.21.15.162:6789/0 conn(0x7fb73c0b9380 legacy=0x7fb73c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.882+0000 7fb73affd640 1 -- 172.21.15.99:0/4001067887 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c0d1ef0 con 0x7fb73c15de00 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.882+0000 7fb73affd640 1 --2- 172.21.15.99:0/4001067887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c15de00 0x7fb73c0b2e60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009870 tx=0x7fb72c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fde74036bbe1c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb739ffb640 1 -- 172.21.15.99:0/4001067887 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c035020 con 0x7fb73c15de00 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb739ffb640 1 -- 172.21.15.99:0/4001067887 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c031640 con 0x7fb73c15de00 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb739ffb640 1 -- 172.21.15.99:0/4001067887 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c02d040 con 0x7fb73c15de00 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 -- 172.21.15.99:0/4001067887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c15de00 msgr2=0x7fb73c0b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 --2- 172.21.15.99:0/4001067887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c15de00 0x7fb73c0b2e60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009870 tx=0x7fb72c02b8f0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 -- 172.21.15.99:0/4001067887 shutdown_connections 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 --2- 172.21.15.99:0/4001067887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c0b5470 0x7fb73c0b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 --2- 172.21.15.99:0/4001067887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c15de00 0x7fb73c0b2e60 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 -- 172.21.15.99:0/4001067887 >> 172.21.15.99:0/4001067887 conn(0x7fb73c0d0470 msgr2=0x7fb73c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 -- 172.21.15.99:0/4001067887 shutdown_connections 2024-03-04T10:06:39.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.883+0000 7fb7412c6640 1 -- 172.21.15.99:0/4001067887 wait complete. 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.884+0000 7fb7412c6640 1 Processor -- start 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.884+0000 7fb7412c6640 1 -- start start 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb7412c6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 0x7fb73c153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb7412c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb73c0b9380 0x7fb73c15f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb7412c6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c15de00 0x7fb73c145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb7412c6640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0b97e0 con 0x7fb73c0b5470 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb7412c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0d24b0 con 0x7fb73c0b9380 2024-03-04T10:06:39.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb7412c6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73c0b72a0 con 0x7fb73c15de00 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb73affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 0x7fb73c153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb73affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 0x7fb73c153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38210/0 (socket says 172.21.15.99:38210) 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb73affd640 1 -- 172.21.15.99:0/1747132422 learned_addr learned my addr 172.21.15.99:0/1747132422 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb73a7fc640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb73c0b9380 0x7fb73c15f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.885+0000 7fb73b7fe640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c15de00 0x7fb73c145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb73affd640 1 -- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c15de00 msgr2=0x7fb73c145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb73affd640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c15de00 0x7fb73c145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb73affd640 1 -- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb73c0b9380 msgr2=0x7fb73c15f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb73affd640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb73c0b9380 0x7fb73c15f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb73affd640 1 -- 172.21.15.99:0/1747132422 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c15fd90 con 0x7fb73c0b5470 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb73affd640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 0x7fb73c153cf0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb72c02b930 tx=0x7fb72c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c03ac60 con 0x7fb73c0b5470 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c03ae20 con 0x7fb73c0b5470 2024-03-04T10:06:39.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c160070 con 0x7fb73c0b5470 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c044630 con 0x7fb73c0b5470 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.886+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c160630 con 0x7fb73c0b5470 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.887+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73c074f60 con 0x7fb73c0b5470 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.887+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb72c04b060 con 0x7fb73c0b5470 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.888+0000 7fb7237fe640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb72806b4d0 0x7fb72806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.888+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb72c0ac1b0 con 0x7fb73c0b5470 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.888+0000 7fb73a7fc640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb72806b4d0 0x7fb72806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:39.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.889+0000 7fb73a7fc640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb72806b4d0 0x7fb72806d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb730007fd0 tx=0x7fb730009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:39.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:39.892+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb72c03f070 con 0x7fb73c0b5470 2024-03-04T10:06:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.099+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb73c0b7550 con 0x7fb73c0b5470 2024-03-04T10:06:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.099+0000 7fb7237fe640 1 -- 172.21.15.99:0/1747132422 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fb72c02f300 con 0x7fb73c0b5470 2024-03-04T10:06:40.102 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:40.102 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":43,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:37.365114+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6474},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6474":{"gid":6474,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/2687500379","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":2687500379},{"type":"v1","addr":"172.21.15.99:6835","nonce":2687500379}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:40.102 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 43 2024-03-04T10:06:40.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb72806b4d0 msgr2=0x7fb72806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb72806b4d0 0x7fb72806d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb730007fd0 tx=0x7fb730009040 comp rx=0 tx=0).stop 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 msgr2=0x7fb73c153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 0x7fb73c153cf0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb72c02b930 tx=0x7fb72c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 shutdown_connections 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb72806b4d0 0x7fb72806d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb73c15de00 0x7fb73c145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb73c0b9380 0x7fb73c15f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 --2- 172.21.15.99:0/1747132422 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb73c0b5470 0x7fb73c153cf0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 >> 172.21.15.99:0/1747132422 conn(0x7fb73c0d0470 msgr2=0x7fb73c0b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 shutdown_connections 2024-03-04T10:06:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.101+0000 7fb7412c6640 1 -- 172.21.15.99:0/1747132422 wait complete. 2024-03-04T10:06:40.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 43, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:06:37.365114+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6474}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6474': {'gid': 6474, 'name': 'c', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6835/2687500379', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 2687500379}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 2687500379}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:06:40.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:06:40.519 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:06:40.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.621+0000 7f45a1dd4640 1 Processor -- start 2024-03-04T10:06:40.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.621+0000 7f45a1dd4640 1 -- start start 2024-03-04T10:06:40.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f45a1dd4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f459c15b960 0x7f459c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f45a1dd4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f459c0d0ad0 0x7f459c0b0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f45a1dd4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0ccad0 con 0x7f459c0b0c40 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f45a1dd4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c0cf860 con 0x7f459c15b960 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f45a1dd4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c0d0100 con 0x7f459c0d0ad0 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f459c15b960 0x7f459c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f459c15b960 0x7f459c0d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38216/0 (socket says 172.21.15.99:38216) 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 -- 172.21.15.99:0/3653009162 learned_addr learned my addr 172.21.15.99:0/3653009162 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459affd640 1 --2- 172.21.15.99:0/3653009162 >> v2:172.21.15.162:3300/0 conn(0x7f459c0d0ad0 0x7f459c0b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 -- 172.21.15.99:0/3653009162 >> v2:172.21.15.162:3300/0 conn(0x7f459c0d0ad0 msgr2=0x7f459c0b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3653009162 >> v2:172.21.15.162:3300/0 conn(0x7f459c0d0ad0 0x7f459c0b0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 -- 172.21.15.99:0/3653009162 >> v1:172.21.15.162:6789/0 conn(0x7f459c0b0c40 legacy=0x7f459c0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.622+0000 7f459b7fe640 1 -- 172.21.15.99:0/3653009162 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0cfab0 con 0x7f459c15b960 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.623+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3653009162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c15b960 0x7f459c0d0590 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f459c0b73c0 tx=0x7f458c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e858f4e62f0c3cf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.623+0000 7f459a7fc640 1 -- 172.21.15.99:0/3653009162 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c02e070 con 0x7f459c15b960 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.623+0000 7f459a7fc640 1 -- 172.21.15.99:0/3653009162 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c037070 con 0x7f459c15b960 2024-03-04T10:06:40.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.623+0000 7f459a7fc640 1 -- 172.21.15.99:0/3653009162 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c031a00 con 0x7f459c15b960 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.623+0000 7f45a1dd4640 1 -- 172.21.15.99:0/3653009162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c15b960 msgr2=0x7f459c0d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.623+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/3653009162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c15b960 0x7f459c0d0590 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f459c0b73c0 tx=0x7f458c02b880 comp rx=0 tx=0).stop 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.624+0000 7f45a1dd4640 1 -- 172.21.15.99:0/3653009162 shutdown_connections 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.624+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/3653009162 >> v2:172.21.15.162:3300/0 conn(0x7f459c0d0ad0 0x7f459c0b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.624+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/3653009162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c15b960 0x7f459c0d0590 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.624+0000 7f45a1dd4640 1 -- 172.21.15.99:0/3653009162 >> 172.21.15.99:0/3653009162 conn(0x7f459c06f5f0 msgr2=0x7f459c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.624+0000 7f45a1dd4640 1 -- 172.21.15.99:0/3653009162 shutdown_connections 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.624+0000 7f45a1dd4640 1 -- 172.21.15.99:0/3653009162 wait complete. 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.625+0000 7f45a1dd4640 1 Processor -- start 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.625+0000 7f45a1dd4640 1 -- start start 2024-03-04T10:06:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.625+0000 7f45a1dd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f459c0b0c40 0x7f459c14b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 0x7f459c14aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f459c15b960 0x7f459c0a8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c15bec0 con 0x7f459c0d0ad0 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c15bd40 con 0x7f459c15b960 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0b4ed0 con 0x7f459c0b0c40 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 0x7f459c14aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 0x7f459c14aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38220/0 (socket says 172.21.15.99:38220) 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 -- 172.21.15.99:0/1218916799 learned_addr learned my addr 172.21.15.99:0/1218916799 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 -- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f459c0b0c40 msgr2=0x7f459c14b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f459c0b0c40 0x7f459c14b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 -- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f459c15b960 msgr2=0x7f459c0a8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f459c15b960 0x7f459c0a8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 -- 172.21.15.99:0/1218916799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c14bf90 con 0x7f459c0d0ad0 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f459affd640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 0x7f459c14aec0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f459000d830 tx=0x7f459000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f4583fff640 1 -- 172.21.15.99:0/1218916799 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000b6c0 con 0x7f459c0d0ad0 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f4583fff640 1 -- 172.21.15.99:0/1218916799 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4590019070 con 0x7f459c0d0ad0 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c14c270 con 0x7f459c0d0ad0 2024-03-04T10:06:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f4583fff640 1 -- 172.21.15.99:0/1218916799 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459000ba80 con 0x7f459c0d0ad0 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.626+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c14cb10 con 0x7f459c0d0ad0 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.628+0000 7f4583fff640 1 -- 172.21.15.99:0/1218916799 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f459001e460 con 0x7f459c0d0ad0 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.628+0000 7f4583fff640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f458806b4d0 0x7f458806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.628+0000 7f4583fff640 1 -- 172.21.15.99:0/1218916799 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f459008aa40 con 0x7f459c0d0ad0 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.628+0000 7f459b7fe640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f458806b4d0 0x7f458806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.629+0000 7f459b7fe640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f458806b4d0 0x7f458806d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f458c036000 tx=0x7f458c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.629+0000 7f45a49bc740 1 Processor -- start 2024-03-04T10:06:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.629+0000 7f45a49bc740 1 -- start start 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2ed8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f08c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2ed9380 0x5609c2f09170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f45a49bc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c2e437c0 con 0x5609c2f05ea0 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c2decb50 con 0x5609c2ed9380 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609c2e3f860 con 0x5609c2f06280 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2ed8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2ed8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38222/0 (socket says 172.21.15.99:38222) 2024-03-04T10:06:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 -- 172.21.15.99:0/1493820238 learned_addr learned my addr 172.21.15.99:0/1493820238 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459affd640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f08c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459bfff640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2ed9380 0x5609c2f09170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 -- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 msgr2=0x5609c2f08c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f08c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 -- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2ed9380 msgr2=0x5609c2f09170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2ed9380 0x5609c2f09170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459b7fe640 1 -- 172.21.15.99:0/1493820238 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609c2e009a0 con 0x5609c2f05ea0 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459affd640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f08c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.630+0000 7f459bfff640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2ed9380 0x5609c2f09170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.631+0000 7f459b7fe640 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2ed8e40 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f458c02b8c0 tx=0x7f458c031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7318cfb4f9b510 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.631+0000 7f45817fa640 1 -- 172.21.15.99:0/1493820238 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c041070 con 0x5609c2f05ea0 2024-03-04T10:06:40.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.631+0000 7f45817fa640 1 -- 172.21.15.99:0/1493820238 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c037040 con 0x5609c2f05ea0 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.631+0000 7f45817fa640 1 -- 172.21.15.99:0/1493820238 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c032970 con 0x5609c2f05ea0 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.631+0000 7f45a49bc740 1 -- 172.21.15.99:0/1493820238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 msgr2=0x5609c2ed8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.631+0000 7f45a49bc740 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2ed8e40 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f458c02b8c0 tx=0x7f458c031000 comp rx=0 tx=0).stop 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 -- 172.21.15.99:0/1493820238 shutdown_connections 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2ed9380 0x5609c2f09170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f08c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 --2- 172.21.15.99:0/1493820238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2ed8e40 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 -- 172.21.15.99:0/1493820238 >> 172.21.15.99:0/1493820238 conn(0x5609c2edb0b0 msgr2=0x5609c2f23c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 -- 172.21.15.99:0/1493820238 shutdown_connections 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 -- 172.21.15.99:0/1493820238 wait complete. 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 -- start start 2024-03-04T10:06:40.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.632+0000 7f45a49bc740 1 Processor -- start 2024-03-04T10:06:40.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f0f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f0df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f36af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c2edd750 con 0x5609c2f05ea0 2024-03-04T10:06:40.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c2dfff90 con 0x5609c2f06280 2024-03-04T10:06:40.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609c2edff90 con 0x5609c2f36670 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f45a49bc740 10 client.0 _ll_register_callbacks cb 0x5609c2e21d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f0f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f0f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38236/0 (socket says 172.21.15.99:38236) 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 -- 172.21.15.99:0/4083163972 learned_addr learned my addr 172.21.15.99:0/4083163972 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459bfff640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f36af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459affd640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f0df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 -- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 msgr2=0x5609c2f36af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f36af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 -- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 msgr2=0x5609c2f0df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f0df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.633+0000 7f459b7fe640 1 -- 172.21.15.99:0/4083163972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609c2b582d0 con 0x5609c2f05ea0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f459b7fe640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f0f9b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f458c034040 tx=0x7f458c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc89b7fdf8f9fe99 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 1 -- 172.21.15.99:0/4083163972 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c059030 con 0x5609c2f05ea0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 1 -- 172.21.15.99:0/4083163972 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f458c004030 con 0x5609c2f05ea0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f458c037040 con 0x5609c2f05ea0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 1 -- 172.21.15.99:0/4083163972 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f458c062050 con 0x5609c2f05ea0 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 10 client.6603 fetch_fsmap learned FSMap version 43 2024-03-04T10:06:40.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x5609c2c226b0 con 0x5609c2f05ea0 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 1 -- 172.21.15.99:0/4083163972 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c037040 con 0x5609c2f05ea0 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 1 -- 172.21.15.99:0/4083163972 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 43) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f458c065020 con 0x5609c2f05ea0 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 10 client.6603 fetch_fsmap finished waiting for FSMap version 43 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 10 client.6603 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 20 client.6603 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f37370 0x5609c2f37750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 4 client.6603 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 --> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609c2e50d00 con 0x5609c2f37370 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f459affd640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f37370 0x5609c2f37750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f457a7fc640 10 client.6603 ms_handle_connect on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.634+0000 7f459affd640 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f37370 0x5609c2f37750 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=580be0f8078f5099 server_cookie=7a34dd7d60a10460 in_seq=0 out_seq=0 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f457a7fc640 1 -- 172.21.15.99:0/4083163972 <== mds.0 v2:172.21.15.99:6834/2687500379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5609c2e50d00 con 0x5609c2f37370 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f457a7fc640 10 client.6603 handle_command_reply: tid=0 2024-03-04T10:06:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 client.6603 shutdown 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 msgr2=0x5609c2f0f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f0f9b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f458c034040 tx=0x7f458c004730 comp rx=0 tx=0).stop 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 shutdown_connections 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f37370 0x5609c2f37750 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f36af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f0df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 --2- 172.21.15.99:0/4083163972 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f0f9b0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f457a7fc640 0 client.6603 ms_handle_reset on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 >> 172.21.15.99:0/4083163972 conn(0x5609c2ed9380 msgr2=0x5609c2edcf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 shutdown_connections 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 1 -- 172.21.15.99:0/4083163972 wait complete. 2024-03-04T10:06:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.635+0000 7f45a49bc740 20 client.6603 trim_cache size 0 max 16384 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.651+0000 7f45a49bc740 1 Processor -- start 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.651+0000 7f45a49bc740 1 -- start start 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f33550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f23cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f24200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f45a49bc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1fd3d0 con 0x5609c2f05ea0 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1fc7d0 con 0x5609c2f06280 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1fcdd0 con 0x5609c2f36670 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f33550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f33550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38242/0 (socket says 172.21.15.99:38242) 2024-03-04T10:06:40.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 -- 172.21.15.99:0/3325435289 learned_addr learned my addr 172.21.15.99:0/3325435289 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459bfff640 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f24200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 -- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 msgr2=0x5609c2f24200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f24200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459affd640 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f23cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 -- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 msgr2=0x5609c2f23cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f23cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 -- 172.21.15.99:0/3325435289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609c2edf980 con 0x5609c2f05ea0 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459affd640 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f23cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f33550 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f458c006b10 tx=0x7f458c054040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbab641173c2af40 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f4580ff9640 1 -- 172.21.15.99:0/3325435289 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c059030 con 0x5609c2f05ea0 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f4580ff9640 1 -- 172.21.15.99:0/3325435289 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c004280 con 0x5609c2f05ea0 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.652+0000 7f4580ff9640 1 -- 172.21.15.99:0/3325435289 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c032e60 con 0x5609c2f05ea0 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 -- 172.21.15.99:0/3325435289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 msgr2=0x5609c2f33550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f33550 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f458c006b10 tx=0x7f458c054040 comp rx=0 tx=0).stop 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 -- 172.21.15.99:0/3325435289 shutdown_connections 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f36670 0x5609c2f24200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f06280 0x5609c2f23cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3325435289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f05ea0 0x5609c2f33550 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 -- 172.21.15.99:0/3325435289 >> 172.21.15.99:0/3325435289 conn(0x5609c2ed9380 msgr2=0x5609c2ed7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 -- 172.21.15.99:0/3325435289 shutdown_connections 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.653+0000 7f45a49bc740 1 -- 172.21.15.99:0/3325435289 wait complete. 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 -- start start 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 Processor -- start 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f05ea0 0x5609c2f8b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f42920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 0x5609c2f23c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c2f33f60 con 0x5609c2f36670 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609c2f0a250 con 0x5609c2f05ea0 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609c2f0af00 con 0x5609c2f06280 2024-03-04T10:06:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.654+0000 7f45a49bc740 10 client.0 _ll_register_callbacks cb 0x7f458c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 0x5609c2f23c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 0x5609c2f23c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38248/0 (socket says 172.21.15.99:38248) 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 -- 172.21.15.99:0/3113736414 learned_addr learned my addr 172.21.15.99:0/3113736414 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459affd640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f42920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f05ea0 0x5609c2f8b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 -- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 msgr2=0x5609c2f42920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f42920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 -- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f05ea0 msgr2=0x5609c2f8b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f05ea0 0x5609c2f8b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 -- 172.21.15.99:0/3113736414 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609c2f8e0b0 con 0x5609c2f36670 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f05ea0 0x5609c2f8b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f459bfff640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 0x5609c2f23c60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4594011530 tx=0x7f45940119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cba82305ebdb6b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f457a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f457a7fc640 1 -- 172.21.15.99:0/3113736414 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4594019070 con 0x5609c2f36670 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f457a7fc640 1 -- 172.21.15.99:0/3113736414 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4594015640 con 0x5609c2f36670 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f458c004280 con 0x5609c2f36670 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f457a7fc640 1 -- 172.21.15.99:0/3113736414 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4594023050 con 0x5609c2f36670 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f45a49bc740 10 client.6609 fetch_fsmap learned FSMap version 43 2024-03-04T10:06:40.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x5609c2f0b140 con 0x5609c2f36670 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f457a7fc640 1 -- 172.21.15.99:0/3113736414 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c004280 con 0x5609c2f36670 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f457a7fc640 1 -- 172.21.15.99:0/3113736414 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 43) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4594026020 con 0x5609c2f36670 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f45a49bc740 10 client.6609 fetch_fsmap finished waiting for FSMap version 43 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.655+0000 7f45a49bc740 10 client.6609 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 20 client.6609 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f27620 0x5609c2f99740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 4 client.6609 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 --> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5609c2f11510 con 0x5609c2f27620 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f27620 0x5609c2f99740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f457a7fc640 10 client.6609 ms_handle_connect on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f459b7fe640 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f27620 0x5609c2f99740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4b29fbb0361932f server_cookie=bfca881714322f37 in_seq=0 out_seq=0 2024-03-04T10:06:40.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f457a7fc640 1 -- 172.21.15.99:0/3113736414 <== mds.0 v2:172.21.15.99:6834/2687500379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5609c2f11510 con 0x5609c2f27620 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f457a7fc640 10 client.6609 handle_command_reply: tid=0 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 client.6609 shutdown 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 msgr2=0x5609c2f23c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 0x5609c2f23c60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4594011530 tx=0x7f45940119f0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 shutdown_connections 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5609c2f36670 0x5609c2f23c60 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x5609c2f27620 0x5609c2f99740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5609c2f06280 0x5609c2f42920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 --2- 172.21.15.99:0/3113736414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5609c2f05ea0 0x5609c2f8b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f457a7fc640 0 client.6609 ms_handle_reset on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 >> 172.21.15.99:0/3113736414 conn(0x5609c2ed9380 msgr2=0x5609c2f33550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 shutdown_connections 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 1 -- 172.21.15.99:0/3113736414 wait complete. 2024-03-04T10:06:40.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.656+0000 7f45a49bc740 20 client.6609 trim_cache size 0 max 16384 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 6474, 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:06:40.674 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.6310123010000002, 2024-03-04T10:06:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 43, 2024-03-04T10:06:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 66, 2024-03-04T10:06:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 66, 2024-03-04T10:06:40.675 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 9.6569200550000005 2024-03-04T10:06:40.675 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:40.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f458806b4d0 msgr2=0x7f458806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f458806b4d0 0x7f458806d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f458c036000 tx=0x7f458c030000 comp rx=0 tx=0).stop 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 msgr2=0x7f459c14aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 0x7f459c14aec0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f459000d830 tx=0x7f459000dcf0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 shutdown_connections 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f458806b4d0 0x7f458806d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f459c15b960 0x7f459c0a8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f459c0d0ad0 0x7f459c14aec0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 --2- 172.21.15.99:0/1218916799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f459c0b0c40 0x7f459c14b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 >> 172.21.15.99:0/1218916799 conn(0x7f459c06f5f0 msgr2=0x7f459c15e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 shutdown_connections 2024-03-04T10:06:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.657+0000 7f45a1dd4640 1 -- 172.21.15.99:0/1218916799 wait complete. 2024-03-04T10:06:40.678 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:06:40.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.775+0000 7f51d8fa9640 1 Processor -- start 2024-03-04T10:06:40.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.775+0000 7f51d8fa9640 1 -- start start 2024-03-04T10:06:40.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d8fa9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51d40af7b0 0x7f51d40d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d8fa9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f51d40cf080 0x7f51d40ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d8fa9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d40cc760 con 0x7f51d415ac40 2024-03-04T10:06:40.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d8fa9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d415b020 con 0x7f51d40cf080 2024-03-04T10:06:40.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d8fa9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40ce720 con 0x7f51d40af7b0 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d259c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f51d415ac40 0x7f51d40ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37384/0 (socket says 172.21.15.99:37384) 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.776+0000 7f51d259c640 1 -- 172.21.15.99:0/669650767 learned_addr learned my addr 172.21.15.99:0/669650767 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431142488 0 0) 0x7f51d40cc760 con 0x7f51d415ac40 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51bc003520 con 0x7f51d415ac40 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d1d9b640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51d40af7b0 0x7f51d40d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d2d9d640 1 --2- 172.21.15.99:0/669650767 >> v2:172.21.15.162:3300/0 conn(0x7f51d40cf080 0x7f51d40ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609163415 0 0) 0x7f51bc003520 con 0x7f51d415ac40 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51d40af7b0 msgr2=0x7f51d40d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51d40af7b0 0x7f51d40d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 >> v2:172.21.15.162:3300/0 conn(0x7f51d40cf080 msgr2=0x7f51d40ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 --2- 172.21.15.99:0/669650767 >> v2:172.21.15.162:3300/0 conn(0x7f51d40cf080 0x7f51d40ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d40cbe30 con 0x7f51d415ac40 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f51c40019f0 con 0x7f51d415ac40 2024-03-04T10:06:40.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 >> v1:172.21.15.99:6789/0 conn(0x7f51d415ac40 legacy=0x7f51d40ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51bc0039b0 0x7f51bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc004800 0x7f51bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0051c0 0x7f51bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d415b020 con 0x7f51bc0039b0 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40ce720 con 0x7f51bc004800 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51bc00bf00 con 0x7f51bc0051c0 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d259c640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51bc0039b0 0x7f51bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d1d9b640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc004800 0x7f51bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d2d9d640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0051c0 0x7f51bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d259c640 1 -- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0051c0 msgr2=0x7f51bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d259c640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0051c0 0x7f51bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d259c640 1 -- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc004800 msgr2=0x7f51bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d259c640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc004800 0x7f51bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d259c640 1 -- 172.21.15.99:0/669650767 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51bc003520 con 0x7f51bc0039b0 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d1d9b640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51bc004800 0x7f51bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.777+0000 7f51d2d9d640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0051c0 0x7f51bc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:40.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.778+0000 7f51d259c640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51bc0039b0 0x7f51bc003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f51c40013d0 tx=0x7f51bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21bcea2b6c9a65b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.778+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c400a410 con 0x7f51bc0039b0 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.778+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c4034630 con 0x7f51bc0039b0 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.778+0000 7f51d159a640 1 -- 172.21.15.99:0/669650767 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4034a20 con 0x7f51bc0039b0 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d8fa9640 1 -- 172.21.15.99:0/669650767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51bc0039b0 msgr2=0x7f51bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/669650767 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51bc0039b0 0x7f51bc003dd0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f51c40013d0 tx=0x7f51bc005830 comp rx=0 tx=0).stop 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d2d9d640 1 -- 172.21.15.99:0/669650767 reap_dead start 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d8fa9640 1 -- 172.21.15.99:0/669650767 shutdown_connections 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d8fa9640 1 -- 172.21.15.99:0/669650767 >> 172.21.15.99:0/669650767 conn(0x7f51d40d0770 msgr2=0x7f51d40d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d8fa9640 1 -- 172.21.15.99:0/669650767 shutdown_connections 2024-03-04T10:06:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.779+0000 7f51d8fa9640 1 -- 172.21.15.99:0/669650767 wait complete. 2024-03-04T10:06:40.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.780+0000 7f51d8fa9640 1 Processor -- start 2024-03-04T10:06:40.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.780+0000 7f51d8fa9640 1 -- start start 2024-03-04T10:06:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d8fa9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0039b0 0x7f51d40ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d8fa9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 0x7f51d409dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d8fa9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51d409e240 0x7f51d409af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d8fa9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40ce340 con 0x7f51d40c7840 2024-03-04T10:06:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d8fa9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40cc3f0 con 0x7f51d409e240 2024-03-04T10:06:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d8fa9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d40ce4c0 con 0x7f51bc0039b0 2024-03-04T10:06:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 0x7f51d409dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 0x7f51d409dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38278/0 (socket says 172.21.15.99:38278) 2024-03-04T10:06:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 -- 172.21.15.99:0/3772566625 learned_addr learned my addr 172.21.15.99:0/3772566625 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d259c640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0039b0 0x7f51d40ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d2d9d640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51d409e240 0x7f51d409af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 -- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0039b0 msgr2=0x7f51d40ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0039b0 0x7f51d40ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 -- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51d409e240 msgr2=0x7f51d409af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51d409e240 0x7f51d409af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 -- 172.21.15.99:0/3772566625 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d41f13e0 con 0x7f51d40c7840 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d259c640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0039b0 0x7f51d40ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.781+0000 7f51d1d9b640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 0x7f51d409dd00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f51c8011540 tx=0x7f51c8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.782+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8015870 con 0x7f51d40c7840 2024-03-04T10:06:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.782+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c8015a10 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.782+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c8015dc0 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.782+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d41f16c0 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.782+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d41f1c80 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.783+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d406cd70 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.783+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f51c801d400 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.784+0000 7f51c2ffd640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f51b406b500 0x7f51b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.784+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f51c808aba0 con 0x7f51d40c7840 2024-03-04T10:06:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.784+0000 7f51d259c640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f51b406b500 0x7f51b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.784+0000 7f51d259c640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f51b406b500 0x7f51b406d9c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51c4034000 tx=0x7f51c4009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.788+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f51c8019030 con 0x7f51d40c7840 2024-03-04T10:06:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.993+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f51d4074f60 con 0x7f51d40c7840 2024-03-04T10:06:41.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.993+0000 7f51c2ffd640 1 -- 172.21.15.99:0/3772566625 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7f51c8059890 con 0x7f51d40c7840 2024-03-04T10:06:41.002 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f51b406b500 msgr2=0x7f51b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f51b406b500 0x7f51b406d9c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51c4034000 tx=0x7f51c4009030 comp rx=0 tx=0).stop 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 msgr2=0x7f51d409dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 0x7f51d409dd00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f51c8011540 tx=0x7f51c8011a00 comp rx=0 tx=0).stop 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 shutdown_connections 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f51b406b500 0x7f51b406d9c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51d409e240 0x7f51d409af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f51d40c7840 0x7f51d409dd00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 --2- 172.21.15.99:0/3772566625 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f51bc0039b0 0x7f51d40ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.995+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 >> 172.21.15.99:0/3772566625 conn(0x7f51d40d0770 msgr2=0x7f51d406eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.996+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 shutdown_connections 2024-03-04T10:06:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:40.996+0000 7f51d8fa9640 1 -- 172.21.15.99:0/3772566625 wait complete. 2024-03-04T10:06:41.011 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:06:41.012 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:41.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.013 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:41.032 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42584sec preferred_lft 42584sec 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:06:41.033 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:06:41.034 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:41.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.035 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:06:41.101 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:06:41.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.103 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:06:41.163 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:06:41.163 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:06:41.163 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:06:41.163 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:06:41.163 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:06:41.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.164 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:06:41.164 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:06:41.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:06:41.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:06:41.164 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:06:41.164 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:06:41.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.262 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:06:41.279 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:06:41.279 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:06:41.280 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:41.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.280 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:06:41.348 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:06:41.348 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:41.405 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:41.405 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:06:41.405 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:41.461 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:06:41.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.462 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:41.545 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:41.546 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:06:41.553 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:06:41.554 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:06:41.554 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:06:41.554 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:06:41.555 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:06:41.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.556 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:41.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.623 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:06:41.683 INFO:teuthology.orchestra.run.smithi099.stdout:client6618 2024-03-04T10:06:41.684 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:41.684 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:41.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:41.739 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mkdir -p a/{b,c}/d; sync .') 2024-03-04T10:06:41.804 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:41.804 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:41.859 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:41.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.996+0000 7fbc44668640 1 Processor -- start 2024-03-04T10:06:41.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.996+0000 7fbc44668640 1 -- start start 2024-03-04T10:06:42.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.997+0000 7fbc44668640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbc3c0b2e60 0x7fbc3c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.997+0000 7fbc44668640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbc3c0b5470 0x7fbc3c0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.997+0000 7fbc44668640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c15e1e0 con 0x7fbc3c15de00 2024-03-04T10:06:42.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.997+0000 7fbc44668640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3c0b98c0 con 0x7fbc3c0b2e60 2024-03-04T10:06:42.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.997+0000 7fbc44668640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3c0b32c0 con 0x7fbc3c0b5470 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc41c03640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbc3c0b2e60 0x7fbc3c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc42404640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbc3c15de00 0x7fbc3c0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37390/0 (socket says 172.21.15.99:37390) 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc42404640 1 -- 172.21.15.99:0/1144421124 learned_addr learned my addr 172.21.15.99:0/1144421124 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc42c05640 1 --2- 172.21.15.99:0/1144421124 >> v2:172.21.15.162:3300/0 conn(0x7fbc3c0b5470 0x7fbc3c0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc41c03640 1 -- 172.21.15.99:0/1144421124 >> v2:172.21.15.162:3300/0 conn(0x7fbc3c0b5470 msgr2=0x7fbc3c0b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc41c03640 1 --2- 172.21.15.99:0/1144421124 >> v2:172.21.15.162:3300/0 conn(0x7fbc3c0b5470 0x7fbc3c0b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc41c03640 1 -- 172.21.15.99:0/1144421124 >> v1:172.21.15.99:6789/0 conn(0x7fbc3c15de00 legacy=0x7fbc3c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc41402640 1 -- 172.21.15.99:0/1144421124 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297355158 0 0) 0x7fbc3c15e1e0 con 0x7fbc3c15de00 2024-03-04T10:06:42.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.998+0000 7fbc41c03640 1 -- 172.21.15.99:0/1144421124 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c0d1c10 con 0x7fbc3c0b2e60 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.999+0000 7fbc41c03640 1 --2- 172.21.15.99:0/1144421124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0b9380 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0b2d40 tx=0x7fbc2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d542897ab56792 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.999+0000 7fbc41402640 1 -- 172.21.15.99:0/1144421124 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c035020 con 0x7fbc3c0b2e60 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.999+0000 7fbc41402640 1 -- 172.21.15.99:0/1144421124 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c031830 con 0x7fbc3c0b2e60 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.999+0000 7fbc41402640 1 -- 172.21.15.99:0/1144421124 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c031c00 con 0x7fbc3c0b2e60 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.999+0000 7fbc44668640 1 -- 172.21.15.99:0/1144421124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c0b2e60 msgr2=0x7fbc3c0b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:41.999+0000 7fbc44668640 1 --2- 172.21.15.99:0/1144421124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0b9380 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0b2d40 tx=0x7fbc2c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.000+0000 7fbc44668640 1 -- 172.21.15.99:0/1144421124 shutdown_connections 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.000+0000 7fbc44668640 1 --2- 172.21.15.99:0/1144421124 >> v2:172.21.15.162:3300/0 conn(0x7fbc3c0b5470 0x7fbc3c0b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.000+0000 7fbc44668640 1 --2- 172.21.15.99:0/1144421124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0b9380 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.000+0000 7fbc44668640 1 -- 172.21.15.99:0/1144421124 >> 172.21.15.99:0/1144421124 conn(0x7fbc3c0d0470 msgr2=0x7fbc3c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.000+0000 7fbc44668640 1 -- 172.21.15.99:0/1144421124 shutdown_connections 2024-03-04T10:06:42.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.000+0000 7fbc44668640 1 -- 172.21.15.99:0/1144421124 wait complete. 2024-03-04T10:06:42.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 Processor -- start 2024-03-04T10:06:42.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 -- start start 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0b5470 0x7fbc3c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 0x7fbc3c0b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0d21d0 con 0x7fbc3c15de00 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0d2820 con 0x7fbc3c0b2e60 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.001+0000 7fbc44668640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c0d24b0 con 0x7fbc3c0b5470 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 0x7fbc3c0b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 0x7fbc3c0b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38288/0 (socket says 172.21.15.99:38288) 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 -- 172.21.15.99:0/3373200433 learned_addr learned my addr 172.21.15.99:0/3373200433 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42404640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc41c03640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0b5470 0x7fbc3c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 -- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0b5470 msgr2=0x7fbc3c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0b5470 0x7fbc3c0ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 -- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0b2e60 msgr2=0x7fbc3c0cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 -- 172.21.15.99:0/3373200433 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c0aad40 con 0x7fbc3c15de00 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42404640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc41c03640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0b5470 0x7fbc3c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc42c05640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 0x7fbc3c0b7380 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbc34007e40 tx=0x7fbc3400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34014620 con 0x7fbc3c15de00 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34019070 con 0x7fbc3c15de00 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3c0aa3d0 con 0x7fbc3c15de00 2024-03-04T10:06:42.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc34014a20 con 0x7fbc3c15de00 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.002+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3c0ab000 con 0x7fbc3c15de00 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.003+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3c074f60 con 0x7fbc3c15de00 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.004+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbc34014bc0 con 0x7fbc3c15de00 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.004+0000 7fbc32ffd640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.004+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbc3408acb0 con 0x7fbc3c15de00 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.004+0000 7fbc42404640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.005+0000 7fbc42404640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbc38000f60 tx=0x7fbc38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.008+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbc340599a0 con 0x7fbc3c15de00 2024-03-04T10:06:42.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.181+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc3c068050 con 0x7fbc3c15de00 2024-03-04T10:06:42.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.182+0000 7fbc32ffd640 1 -- 172.21.15.99:0/3373200433 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16848 (secure 0 0 0) 0x7fbc340592a0 con 0x7fbc3c15de00 2024-03-04T10:06:42.185 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:42.185 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":66,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:37.015810+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:42.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.184+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc2006b4d0 msgr2=0x7fbc2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.184+0000 7fbc44668640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc2006b4d0 0x7fbc2006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbc38000f60 tx=0x7fbc38009040 comp rx=0 tx=0).stop 2024-03-04T10:06:42.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.184+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 msgr2=0x7fbc3c0b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.184+0000 7fbc44668640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 0x7fbc3c0b7380 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbc34007e40 tx=0x7fbc3400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.184+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 shutdown_connections 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.184+0000 7fbc44668640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc2006b4d0 0x7fbc2006d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.185+0000 7fbc44668640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbc3c15de00 0x7fbc3c0b7380 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.185+0000 7fbc44668640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbc3c0b5470 0x7fbc3c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.185+0000 7fbc44668640 1 --2- 172.21.15.99:0/3373200433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc3c0b2e60 0x7fbc3c0cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.185+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 >> 172.21.15.99:0/3373200433 conn(0x7fbc3c0d0470 msgr2=0x7fbc3c0c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.185+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 shutdown_connections 2024-03-04T10:06:42.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.185+0000 7fbc44668640 1 -- 172.21.15.99:0/3373200433 wait complete. 2024-03-04T10:06:42.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:06:33.589849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:06:34.331851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:06:42.198 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:06:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.298+0000 7f9f76a85640 1 Processor -- start 2024-03-04T10:06:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.298+0000 7f9f76a85640 1 -- start start 2024-03-04T10:06:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f76a85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b5700 0x7f9f700d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f76a85640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9f700d34a0 0x7f9f7006c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f76a85640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9f700b9410 0x7f9f7006c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f76a85640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700d2400 con 0x7f9f700b9410 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f76a85640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700d26e0 con 0x7f9f700d34a0 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f76a85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700d29c0 con 0x7f9f700b5700 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f75022640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9f700b9410 0x7f9f7006c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f75022640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9f700b9410 0x7f9f7006c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38296/0 (socket says 172.21.15.99:38296) 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.299+0000 7f9f75022640 1 -- 172.21.15.99:0/746228170 learned_addr learned my addr 172.21.15.99:0/746228170 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f74821640 1 --2- 172.21.15.99:0/746228170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b5700 0x7f9f700d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f6ffff640 1 --2- 172.21.15.99:0/746228170 >> v2:172.21.15.162:3300/0 conn(0x7f9f700d34a0 0x7f9f7006c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f75022640 1 -- 172.21.15.99:0/746228170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b5700 msgr2=0x7f9f700d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f75022640 1 --2- 172.21.15.99:0/746228170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b5700 0x7f9f700d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f75022640 1 -- 172.21.15.99:0/746228170 >> v2:172.21.15.162:3300/0 conn(0x7f9f700d34a0 msgr2=0x7f9f7006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f75022640 1 --2- 172.21.15.99:0/746228170 >> v2:172.21.15.162:3300/0 conn(0x7f9f700d34a0 0x7f9f7006c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f75022640 1 -- 172.21.15.99:0/746228170 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700d2ca0 con 0x7f9f700b9410 2024-03-04T10:06:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f6ffff640 1 --2- 172.21.15.99:0/746228170 >> v2:172.21.15.162:3300/0 conn(0x7f9f700d34a0 0x7f9f7006c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f75022640 1 --2- 172.21.15.99:0/746228170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700b9410 0x7f9f7006c620 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f9f700d4a70 tx=0x7f9f6802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f501b786b36ebb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.300+0000 7f9f6f7fe640 1 -- 172.21.15.99:0/746228170 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6802c9e0 con 0x7f9f700b9410 2024-03-04T10:06:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f6f7fe640 1 -- 172.21.15.99:0/746228170 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6802cb80 con 0x7f9f700b9410 2024-03-04T10:06:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f6f7fe640 1 -- 172.21.15.99:0/746228170 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f680343f0 con 0x7f9f700b9410 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 -- 172.21.15.99:0/746228170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700b9410 msgr2=0x7f9f7006c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 --2- 172.21.15.99:0/746228170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700b9410 0x7f9f7006c620 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f9f700d4a70 tx=0x7f9f6802be40 comp rx=0 tx=0).stop 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 -- 172.21.15.99:0/746228170 shutdown_connections 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 --2- 172.21.15.99:0/746228170 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700b9410 0x7f9f7006c620 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 --2- 172.21.15.99:0/746228170 >> v2:172.21.15.162:3300/0 conn(0x7f9f700d34a0 0x7f9f7006c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 --2- 172.21.15.99:0/746228170 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b5700 0x7f9f700d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 -- 172.21.15.99:0/746228170 >> 172.21.15.99:0/746228170 conn(0x7f9f700d0490 msgr2=0x7f9f700d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 -- 172.21.15.99:0/746228170 shutdown_connections 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.301+0000 7f9f76a85640 1 -- 172.21.15.99:0/746228170 wait complete. 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.302+0000 7f9f76a85640 1 Processor -- start 2024-03-04T10:06:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.302+0000 7f9f76a85640 1 -- start start 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f76a85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f700b5700 0x7f9f701f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f76a85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b9410 0x7f9f701f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f76a85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 0x7f9f701f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f76a85640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b5c60 con 0x7f9f700d34a0 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f76a85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b5ae0 con 0x7f9f700b5700 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f76a85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f700b5de0 con 0x7f9f700b9410 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 0x7f9f701f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 0x7f9f701f2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38308/0 (socket says 172.21.15.99:38308) 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 -- 172.21.15.99:0/575110747 learned_addr learned my addr 172.21.15.99:0/575110747 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f74821640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f700b5700 0x7f9f701f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f6ffff640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b9410 0x7f9f701f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 -- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b9410 msgr2=0x7f9f701f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b9410 0x7f9f701f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 -- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f700b5700 msgr2=0x7f9f701f6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f700b5700 0x7f9f701f6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.303+0000 7f9f75022640 1 -- 172.21.15.99:0/575110747 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700cb910 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f6ffff640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b9410 0x7f9f701f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f75022640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 0x7f9f701f2b90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009f40 tx=0x7f9f68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68043020 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6803e450 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f700cbbf0 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f68009510 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.304+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f701f7630 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.305+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f700c7400 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.305+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9f68046020 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.305+0000 7f9f6d7fa640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f5c06b4d0 0x7f9f5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.306+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9f680acb40 con 0x7f9f700d34a0 2024-03-04T10:06:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.306+0000 7f9f74821640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f5c06b4d0 0x7f9f5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.306+0000 7f9f74821640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f5c06b4d0 0x7f9f5c06d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9f60003610 tx=0x7f9f60008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.310+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f6804d030 con 0x7f9f700d34a0 2024-03-04T10:06:42.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.478+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f700cbdc0 con 0x7f9f700d34a0 2024-03-04T10:06:42.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.479+0000 7f9f6d7fa640 1 -- 172.21.15.99:0/575110747 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f6807b830 con 0x7f9f700d34a0 2024-03-04T10:06:42.481 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:06:42.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f5c06b4d0 msgr2=0x7f9f5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f5c06b4d0 0x7f9f5c06d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9f60003610 tx=0x7f9f60008040 comp rx=0 tx=0).stop 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 msgr2=0x7f9f701f2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 0x7f9f701f2b90 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009f40 tx=0x7f9f68004a40 comp rx=0 tx=0).stop 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 shutdown_connections 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f5c06b4d0 0x7f9f5c06d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f700d34a0 0x7f9f701f2b90 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f700b9410 0x7f9f701f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 --2- 172.21.15.99:0/575110747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f700b5700 0x7f9f701f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 >> 172.21.15.99:0/575110747 conn(0x7f9f700d0490 msgr2=0x7f9f700b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 shutdown_connections 2024-03-04T10:06:42.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.481+0000 7f9f76a85640 1 -- 172.21.15.99:0/575110747 wait complete. 2024-03-04T10:06:42.494 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:06:42.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.594+0000 7f2cc2c75640 1 Processor -- start 2024-03-04T10:06:42.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.594+0000 7f2cc2c75640 1 -- start start 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc2c75640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2cbc0adc40 0x7f2cbc0cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc2c75640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cbc0cc7a0 con 0x7f2cbc15bc50 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc2c75640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cbc0cca80 con 0x7f2cbc06acd0 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc2c75640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cbc0ccd60 con 0x7f2cbc0adc40 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc0a11640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2cbc15bc50 0x7f2cbc0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37392/0 (socket says 172.21.15.99:37392) 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc0a11640 1 -- 172.21.15.99:0/3516537885 learned_addr learned my addr 172.21.15.99:0/3516537885 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.595+0000 7f2cc1212640 1 --2- 172.21.15.99:0/3516537885 >> v2:172.21.15.99:3300/0 conn(0x7f2cbc0adc40 0x7f2cbc0cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.596+0000 7f2cc1212640 1 -- 172.21.15.99:0/3516537885 >> v1:172.21.15.162:6789/0 conn(0x7f2cbc06acd0 legacy=0x7f2cbc0ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.596+0000 7f2cc1212640 1 -- 172.21.15.99:0/3516537885 >> v1:172.21.15.99:6789/0 conn(0x7f2cbc15bc50 legacy=0x7f2cbc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.596+0000 7f2cc1212640 1 -- 172.21.15.99:0/3516537885 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc0cd040 con 0x7f2cbc0adc40 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.596+0000 7f2cc1212640 1 --2- 172.21.15.99:0/3516537885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc0adc40 0x7f2cbc0cf580 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4009d90 tx=0x7f2cb402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4d1acb2301469a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.596+0000 7f2cbb7fe640 1 -- 172.21.15.99:0/3516537885 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4039070 con 0x7f2cbc0adc40 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cbb7fe640 1 -- 172.21.15.99:0/3516537885 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb402cc60 con 0x7f2cbc0adc40 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cbb7fe640 1 -- 172.21.15.99:0/3516537885 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4034660 con 0x7f2cbc0adc40 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 -- 172.21.15.99:0/3516537885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc0adc40 msgr2=0x7f2cbc0cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/3516537885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc0adc40 0x7f2cbc0cf580 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4009d90 tx=0x7f2cb402beb0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 -- 172.21.15.99:0/3516537885 shutdown_connections 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/3516537885 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc0adc40 0x7f2cbc0cf580 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 -- 172.21.15.99:0/3516537885 >> 172.21.15.99:0/3516537885 conn(0x7f2cbc0b2250 msgr2=0x7f2cbc0b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 -- 172.21.15.99:0/3516537885 shutdown_connections 2024-03-04T10:06:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.597+0000 7f2cc2c75640 1 -- 172.21.15.99:0/3516537885 wait complete. 2024-03-04T10:06:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.598+0000 7f2cc2c75640 1 Processor -- start 2024-03-04T10:06:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.598+0000 7f2cc2c75640 1 -- start start 2024-03-04T10:06:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc2c75640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cbc06acd0 0x7f2cbc148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc2c75640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cbc0adc40 0x7f2cbc1f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc2c75640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 0x7f2cbc0c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc2c75640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0cc330 con 0x7f2cbc15bc50 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc2c75640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0cdf50 con 0x7f2cbc06acd0 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc2c75640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc0cc4b0 con 0x7f2cbc0adc40 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc1212640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 0x7f2cbc0c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc1212640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 0x7f2cbc0c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38336/0 (socket says 172.21.15.99:38336) 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc1212640 1 -- 172.21.15.99:0/2732427850 learned_addr learned my addr 172.21.15.99:0/2732427850 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cc0a11640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cbc06acd0 0x7f2cbc148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.599+0000 7f2cbbfff640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cbc0adc40 0x7f2cbc1f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc1212640 1 -- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cbc0adc40 msgr2=0x7f2cbc1f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc1212640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cbc0adc40 0x7f2cbc1f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc1212640 1 -- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cbc06acd0 msgr2=0x7f2cbc148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc1212640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cbc06acd0 0x7f2cbc148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc1212640 1 -- 172.21.15.99:0/2732427850 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc1e50d0 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc0a11640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cbc06acd0 0x7f2cbc148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc1212640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 0x7f2cbc0c9240 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4037040 tx=0x7f2cb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4039050 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb403a040 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc1e53b0 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb403e610 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.600+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc1e5970 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.601+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc1e3860 con 0x7f2cbc15bc50 2024-03-04T10:06:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.601+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2cb4047020 con 0x7f2cbc15bc50 2024-03-04T10:06:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.602+0000 7f2cb97fa640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ca806b4d0 0x7f2ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.602+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2cb40ac760 con 0x7f2cbc15bc50 2024-03-04T10:06:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.602+0000 7f2cc0a11640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ca806b4d0 0x7f2ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.602+0000 7f2cc0a11640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ca806b4d0 0x7f2ca806d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0c9ff0 tx=0x7f2cac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.606+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2cbc1e3860 con 0x7f2cbc15bc50 2024-03-04T10:06:42.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.774+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2cbc074f60 con 0x7f2cbc15bc50 2024-03-04T10:06:42.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.775+0000 7f2cb97fa640 1 -- 172.21.15.99:0/2732427850 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2cb404e020 con 0x7f2cbc15bc50 2024-03-04T10:06:42.777 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:06:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.776+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ca806b4d0 msgr2=0x7f2ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.776+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ca806b4d0 0x7f2ca806d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc0c9ff0 tx=0x7f2cac009040 comp rx=0 tx=0).stop 2024-03-04T10:06:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.776+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 msgr2=0x7f2cbc0c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.776+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 0x7f2cbc0c9240 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4037040 tx=0x7f2cb4004a40 comp rx=0 tx=0).stop 2024-03-04T10:06:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 shutdown_connections 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ca806b4d0 0x7f2ca806d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cbc15bc50 0x7f2cbc0c9240 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2cbc0adc40 0x7f2cbc1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 --2- 172.21.15.99:0/2732427850 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cbc06acd0 0x7f2cbc148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 >> 172.21.15.99:0/2732427850 conn(0x7f2cbc0b2250 msgr2=0x7f2cbc0ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 shutdown_connections 2024-03-04T10:06:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.777+0000 7f2cc2c75640 1 -- 172.21.15.99:0/2732427850 wait complete. 2024-03-04T10:06:42.790 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:06:42.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.890+0000 7f481dca4640 1 Processor -- start 2024-03-04T10:06:42.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.890+0000 7f481dca4640 1 -- start start 2024-03-04T10:06:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f481dca4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f48181450e0 0x7f48180cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f481dca4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f48180b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f481dca4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48180cebd0 con 0x7f481815c9b0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f481dca4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48180cf7e0 con 0x7f48181450e0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f481dca4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180ceeb0 con 0x7f48180cc510 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f4816ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f481815c9b0 0x7f48180cbe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37394/0 (socket says 172.21.15.99:37394) 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f4816ffd640 1 -- 172.21.15.99:0/2568640428 learned_addr learned my addr 172.21.15.99:0/2568640428 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48177fe640 1 --2- 172.21.15.99:0/2568640428 >> v2:172.21.15.162:3300/0 conn(0x7f48181450e0 0x7f48180cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043599198 0 0) 0x7f48180cebd0 con 0x7f481815c9b0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f4817fff640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f48180b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4800003520 con 0x7f481815c9b0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 901192631 0 0) 0x7f4800003520 con 0x7f481815c9b0 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 msgr2=0x7f48180b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f48180b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 >> v2:172.21.15.162:3300/0 conn(0x7f48181450e0 msgr2=0x7f48180cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 --2- 172.21.15.99:0/2568640428 >> v2:172.21.15.162:3300/0 conn(0x7f48181450e0 0x7f48180cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48180cf190 con 0x7f481815c9b0 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f480c001670 con 0x7f481815c9b0 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 >> v1:172.21.15.99:6789/0 conn(0x7f481815c9b0 legacy=0x7f48180cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.891+0000 7f48167fc640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4800003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48167fc640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f4800004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48167fc640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4800004d40 0x7f48000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180cf7e0 con 0x7f48000039b0 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180ceeb0 con 0x7f4800004d40 2024-03-04T10:06:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480000b9c0 con 0x7f48180cc510 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48177fe640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4800003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f4816ffd640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f4800004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f4817fff640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4800004d40 0x7f48000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48177fe640 1 -- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 msgr2=0x7f4800004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48177fe640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f4800004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48177fe640 1 -- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4800004d40 msgr2=0x7f48000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48177fe640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4800004d40 0x7f48000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f48177fe640 1 -- 172.21.15.99:0/2568640428 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4800003520 con 0x7f48000039b0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.892+0000 7f4816ffd640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f48180cc510 0x7f4800004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f48177fe640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4800003dd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4808009af0 tx=0x7f4800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69670ffd34faca7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808039070 con 0x7f48000039b0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480802c710 con 0x7f48000039b0 2024-03-04T10:06:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f48167fc640 1 -- 172.21.15.99:0/2568640428 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808009690 con 0x7f48000039b0 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f481dca4640 1 -- 172.21.15.99:0/2568640428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 msgr2=0x7f4800003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f481dca4640 1 --2- 172.21.15.99:0/2568640428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4800003dd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f4808009af0 tx=0x7f4800005830 comp rx=0 tx=0).stop 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f4817fff640 1 -- 172.21.15.99:0/2568640428 reap_dead start 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f481dca4640 1 -- 172.21.15.99:0/2568640428 shutdown_connections 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f481dca4640 1 -- 172.21.15.99:0/2568640428 >> 172.21.15.99:0/2568640428 conn(0x7f481806acd0 msgr2=0x7f481806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f481dca4640 1 -- 172.21.15.99:0/2568640428 shutdown_connections 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.893+0000 7f481dca4640 1 -- 172.21.15.99:0/2568640428 wait complete. 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.894+0000 7f481dca4640 1 Processor -- start 2024-03-04T10:06:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 -- start start 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4818139120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f481813e0e0 0x7f481814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f481813a460 0x7f48180b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48181454c0 con 0x7f48000039b0 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815ce60 con 0x7f481813e0e0 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f481dca4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180ccd10 con 0x7f481813a460 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f48177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4818139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f48177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4818139120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38352/0 (socket says 172.21.15.99:38352) 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.895+0000 7f48177fe640 1 -- 172.21.15.99:0/3735039709 learned_addr learned my addr 172.21.15.99:0/3735039709 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f48177fe640 1 -- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f481813a460 msgr2=0x7f48180b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f48177fe640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f481813a460 0x7f48180b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f48177fe640 1 -- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f481813e0e0 msgr2=0x7f481814e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f4816ffd640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f481813e0e0 0x7f481814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f48177fe640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f481813e0e0 0x7f481814e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f48177fe640 1 -- 172.21.15.99:0/3735039709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4818139660 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f48177fe640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4818139120 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4808037040 tx=0x7f4808004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808039050 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4808045070 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481813e610 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48080044e0 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.896+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481813ebd0 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.897+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4818074f60 con 0x7f48000039b0 2024-03-04T10:06:42.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.898+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4808003a50 con 0x7f48000039b0 2024-03-04T10:06:42.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.898+0000 7f47fffff640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f480406b4d0 0x7f480406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:42.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.898+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48080ac4c0 con 0x7f48000039b0 2024-03-04T10:06:42.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.898+0000 7f4816ffd640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f480406b4d0 0x7f480406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:42.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.899+0000 7f4816ffd640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f480406b4d0 0x7f480406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f480c001eb0 tx=0x7f480c000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:42.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:42.902+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4818139660 con 0x7f48000039b0 2024-03-04T10:06:43.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.077+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48180cb950 con 0x7f48000039b0 2024-03-04T10:06:43.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.077+0000 7f47fffff640 1 -- 172.21.15.99:0/3735039709 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f480807b1b0 con 0x7f48000039b0 2024-03-04T10:06:43.080 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:06:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f480406b4d0 msgr2=0x7f480406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f480406b4d0 0x7f480406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f480c001eb0 tx=0x7f480c000ed0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 msgr2=0x7f4818139120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4818139120 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f4808037040 tx=0x7f4808004730 comp rx=0 tx=0).stop 2024-03-04T10:06:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 shutdown_connections 2024-03-04T10:06:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f480406b4d0 0x7f480406d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f481813a460 0x7f48180b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f481813e0e0 0x7f481814e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 --2- 172.21.15.99:0/3735039709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f48000039b0 0x7f4818139120 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 >> 172.21.15.99:0/3735039709 conn(0x7f481806acd0 msgr2=0x7f48180b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 shutdown_connections 2024-03-04T10:06:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.079+0000 7f481dca4640 1 -- 172.21.15.99:0/3735039709 wait complete. 2024-03-04T10:06:43.092 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:06:43.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.191+0000 7f296c105640 1 Processor -- start 2024-03-04T10:06:43.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.191+0000 7f296c105640 1 -- start start 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f296c105640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f29640aeb20 0x7f2964071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f296c105640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296415da40 con 0x7f296415d660 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f296c105640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29640af090 con 0x7f2964071890 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f296c105640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2964071ce0 con 0x7f29640aeb20 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f29696a0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f29640aeb20 0x7f2964071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f29696a0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f29640aeb20 0x7f2964071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38356/0 (socket says 172.21.15.99:38356) 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f29696a0640 1 -- 172.21.15.99:0/3196640611 learned_addr learned my addr 172.21.15.99:0/3196640611 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.192+0000 7f2969ea1640 1 --1- 172.21.15.99:0/3196640611 >> v1:172.21.15.99:6789/0 conn(0x7f296415d660 0x7f296415e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:37408/0 (socket says 172.21.15.99:37408) 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f29696a0640 1 -- 172.21.15.99:0/3196640611 >> v1:172.21.15.162:6789/0 conn(0x7f2964071890 legacy=0x7f2964150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f29696a0640 1 -- 172.21.15.99:0/3196640611 >> v1:172.21.15.99:6789/0 conn(0x7f296415d660 legacy=0x7f296415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f29696a0640 1 -- 172.21.15.99:0/3196640611 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296415a9f0 con 0x7f29640aeb20 2024-03-04T10:06:43.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f2968e9f640 1 -- 172.21.15.99:0/3196640611 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948785884 0 0) 0x7f296415da40 con 0x7f296415d660 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f29696a0640 1 --2- 172.21.15.99:0/3196640611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29640aeb20 0x7f2964071350 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f295400bb00 tx=0x7f295402deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3adba4b968fe5f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f2968e9f640 1 -- 172.21.15.99:0/3196640611 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954034070 con 0x7f29640aeb20 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f2968e9f640 1 -- 172.21.15.99:0/3196640611 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295402ec60 con 0x7f29640aeb20 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.193+0000 7f2968e9f640 1 -- 172.21.15.99:0/3196640611 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954038620 con 0x7f29640aeb20 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 -- 172.21.15.99:0/3196640611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29640aeb20 msgr2=0x7f2964071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 --2- 172.21.15.99:0/3196640611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29640aeb20 0x7f2964071350 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f295400bb00 tx=0x7f295402deb0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 -- 172.21.15.99:0/3196640611 shutdown_connections 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 --2- 172.21.15.99:0/3196640611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f29640aeb20 0x7f2964071350 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 -- 172.21.15.99:0/3196640611 >> 172.21.15.99:0/3196640611 conn(0x7f296406acd0 msgr2=0x7f296406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 -- 172.21.15.99:0/3196640611 shutdown_connections 2024-03-04T10:06:43.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.194+0000 7f296c105640 1 -- 172.21.15.99:0/3196640611 wait complete. 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.195+0000 7f296c105640 1 Processor -- start 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.195+0000 7f296c105640 1 -- start start 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296c105640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 0x7f29641f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296c105640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29640aeb20 0x7f29640b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296c105640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f296415d660 0x7f2964142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296c105640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296415afb0 con 0x7f2964071890 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296c105640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296415cbc0 con 0x7f296415d660 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296c105640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296415a660 con 0x7f29640aeb20 2024-03-04T10:06:43.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 0x7f29641f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 0x7f29641f3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:38366/0 (socket says 172.21.15.99:38366) 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 -- 172.21.15.99:0/510904330 learned_addr learned my addr 172.21.15.99:0/510904330 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f29696a0640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29640aeb20 0x7f29640b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296a6a2640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f296415d660 0x7f2964142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 -- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29640aeb20 msgr2=0x7f29640b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29640aeb20 0x7f29640b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 -- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f296415d660 msgr2=0x7f2964142960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f296415d660 0x7f2964142960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 -- 172.21.15.99:0/510904330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29641f3cd0 con 0x7f2964071890 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f296a6a2640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f296415d660 0x7f2964142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.196+0000 7f2969ea1640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 0x7f29641f3790 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f296000c780 tx=0x7f296000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.197+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296000a640 con 0x7f2964071890 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.197+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296000a7e0 con 0x7f2964071890 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.197+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2964143460 con 0x7f2964071890 2024-03-04T10:06:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.197+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296000abb0 con 0x7f2964071890 2024-03-04T10:06:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.197+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29641f2f90 con 0x7f2964071890 2024-03-04T10:06:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.198+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29641f2d80 con 0x7f2964071890 2024-03-04T10:06:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.198+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2960007070 con 0x7f2964071890 2024-03-04T10:06:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.198+0000 7f295a7fc640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f294806b4d0 0x7f294806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.199+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f296008a1f0 con 0x7f2964071890 2024-03-04T10:06:43.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.199+0000 7f29696a0640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f294806b4d0 0x7f294806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.199+0000 7f29696a0640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f294806b4d0 0x7f294806d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2954039000 tx=0x7f2954034000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.203+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f29641f3cd0 con 0x7f2964071890 2024-03-04T10:06:43.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.383+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f29640ae900 con 0x7f2964071890 2024-03-04T10:06:43.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.383+0000 7f295a7fc640 1 -- 172.21.15.99:0/510904330 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v66) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2960058f70 con 0x7f2964071890 2024-03-04T10:06:43.386 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:06:43.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f294806b4d0 msgr2=0x7f294806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f294806b4d0 0x7f294806d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2954039000 tx=0x7f2954034000 comp rx=0 tx=0).stop 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 msgr2=0x7f29641f3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 0x7f29641f3790 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f296000c780 tx=0x7f296000cc40 comp rx=0 tx=0).stop 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 shutdown_connections 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f294806b4d0 0x7f294806d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f296415d660 0x7f2964142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f29640aeb20 0x7f29640b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 --2- 172.21.15.99:0/510904330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2964071890 0x7f29641f3790 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 >> 172.21.15.99:0/510904330 conn(0x7f296406acd0 msgr2=0x7f29640b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 shutdown_connections 2024-03-04T10:06:43.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.385+0000 7f296c105640 1 -- 172.21.15.99:0/510904330 wait complete. 2024-03-04T10:06:43.398 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:06:43.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:43.398 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:43.431 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:06:43.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:43.432 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:06:43.487 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T10:06:43.487 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-03-04T10:06:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.627+0000 7fc92b880640 1 Processor -- start 2024-03-04T10:06:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.627+0000 7fc92b880640 1 -- start start 2024-03-04T10:06:43.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc92b880640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc92406d3c0 0x7fc9240b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc92b880640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92406d8a0 con 0x7fc92415a810 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc92b880640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92406db80 con 0x7fc9240b48a0 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc92b880640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc92406de60 con 0x7fc92406d3c0 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc92961c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc92415a810 0x7fc9240b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58718/0 (socket says 172.21.15.99:58718) 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc92961c640 1 -- 172.21.15.99:0/8235342 learned_addr learned my addr 172.21.15.99:0/8235342 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc929e1d640 1 --2- 172.21.15.99:0/8235342 >> v2:172.21.15.99:3300/0 conn(0x7fc92406d3c0 0x7fc9240b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc91bfff640 1 -- 172.21.15.99:0/8235342 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786896420 0 0) 0x7fc92406d8a0 con 0x7fc92415a810 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc91bfff640 1 -- 172.21.15.99:0/8235342 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc910003520 con 0x7fc92415a810 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc929e1d640 1 -- 172.21.15.99:0/8235342 >> v1:172.21.15.162:6789/0 conn(0x7fc9240b48a0 legacy=0x7fc9240b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc929e1d640 1 -- 172.21.15.99:0/8235342 >> v1:172.21.15.99:6789/0 conn(0x7fc92415a810 legacy=0x7fc9240b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.628+0000 7fc929e1d640 1 -- 172.21.15.99:0/8235342 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92406e140 con 0x7fc92406d3c0 2024-03-04T10:06:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.629+0000 7fc929e1d640 1 --2- 172.21.15.99:0/8235342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc9240b3100 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009d90 tx=0x7fc91c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d7aa567a8191c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.629+0000 7fc91bfff640 1 -- 172.21.15.99:0/8235342 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c039070 con 0x7fc92406d3c0 2024-03-04T10:06:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.629+0000 7fc91bfff640 1 -- 172.21.15.99:0/8235342 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c02cd00 con 0x7fc92406d3c0 2024-03-04T10:06:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.629+0000 7fc91bfff640 1 -- 172.21.15.99:0/8235342 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c034660 con 0x7fc92406d3c0 2024-03-04T10:06:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 -- 172.21.15.99:0/8235342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 msgr2=0x7fc9240b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 --2- 172.21.15.99:0/8235342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc9240b3100 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009d90 tx=0x7fc91c02bf50 comp rx=0 tx=0).stop 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 -- 172.21.15.99:0/8235342 shutdown_connections 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 --2- 172.21.15.99:0/8235342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc9240b3100 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 -- 172.21.15.99:0/8235342 >> 172.21.15.99:0/8235342 conn(0x7fc92406acd0 msgr2=0x7fc92406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 -- 172.21.15.99:0/8235342 shutdown_connections 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.630+0000 7fc92b880640 1 -- 172.21.15.99:0/8235342 wait complete. 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.631+0000 7fc92b880640 1 Processor -- start 2024-03-04T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.631+0000 7fc92b880640 1 -- start start 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc924146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9240b48a0 0x7fc92414e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc92415a810 0x7fc92415d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240cc130 con 0x7fc92406d3c0 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240cbfb0 con 0x7fc9240b48a0 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241f15a0 con 0x7fc92415a810 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc924146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc924146a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58816/0 (socket says 172.21.15.99:58816) 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 -- 172.21.15.99:0/1431988876 learned_addr learned my addr 172.21.15.99:0/1431988876 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 -- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc92415a810 msgr2=0x7fc92415d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc92415a810 0x7fc92415d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 -- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9240b48a0 msgr2=0x7fc92414e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9240b48a0 0x7fc92414e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 -- 172.21.15.99:0/1431988876 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92414c010 con 0x7fc92406d3c0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92961c640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc924146a80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc92000c440 tx=0x7fc92000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc919ffb640 1 -- 172.21.15.99:0/1431988876 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920017070 con 0x7fc92406d3c0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc919ffb640 1 -- 172.21.15.99:0/1431988876 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920005ce0 con 0x7fc92406d3c0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc919ffb640 1 -- 172.21.15.99:0/1431988876 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9200072b0 con 0x7fc92406d3c0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.632+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92414c2f0 con 0x7fc92406d3c0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.633+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc92414c8b0 con 0x7fc92406d3c0 2024-03-04T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.634+0000 7fc919ffb640 1 -- 172.21.15.99:0/1431988876 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc92001a070 con 0x7fc92406d3c0 2024-03-04T10:06:43.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.634+0000 7fc919ffb640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc90806b4d0 0x7fc90806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.634+0000 7fc919ffb640 1 -- 172.21.15.99:0/1431988876 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc920089650 con 0x7fc92406d3c0 2024-03-04T10:06:43.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.635+0000 7fc928e1b640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc90806b4d0 0x7fc90806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.635+0000 7fc928e1b640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc90806b4d0 0x7fc90806d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc914000f60 tx=0x7fc9140079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.635+0000 7fc92e45c740 1 Processor -- start 2024-03-04T10:06:43.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.635+0000 7fc92e45c740 1 -- start start 2024-03-04T10:06:43.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a96993c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96993cd40 0x55a96996cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92e45c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9698a5e00 con 0x55a969969860 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a96984eb10 con 0x55a969969c40 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9698a1ea0 con 0x55a96993cd40 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a96993c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a96993c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58822/0 (socket says 172.21.15.99:58822) 2024-03-04T10:06:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 -- 172.21.15.99:0/324583347 learned_addr learned my addr 172.21.15.99:0/324583347 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc928e1b640 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc929e1d640 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96993cd40 0x55a96996cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 -- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96993cd40 msgr2=0x55a96996cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96993cd40 0x55a96996cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 -- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 msgr2=0x55a96996c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.636+0000 7fc92961c640 1 -- 172.21.15.99:0/324583347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9696847d0 con 0x55a969969860 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92961c640 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a96993c800 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc920087a20 tx=0x7fc920057e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a962933767d6af7b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc906ffd640 1 -- 172.21.15.99:0/324583347 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920055e30 con 0x55a969969860 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc906ffd640 1 -- 172.21.15.99:0/324583347 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920056090 con 0x55a969969860 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc906ffd640 1 -- 172.21.15.99:0/324583347 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920054400 con 0x55a969969860 2024-03-04T10:06:43.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 -- 172.21.15.99:0/324583347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 msgr2=0x55a96993c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a96993c800 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fc920087a20 tx=0x7fc920057e50 comp rx=0 tx=0).stop 2024-03-04T10:06:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 -- 172.21.15.99:0/324583347 shutdown_connections 2024-03-04T10:06:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96993cd40 0x55a96996cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 --2- 172.21.15.99:0/324583347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a96993c800 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 -- 172.21.15.99:0/324583347 >> 172.21.15.99:0/324583347 conn(0x55a96993ea70 msgr2=0x55a9699875f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 -- 172.21.15.99:0/324583347 shutdown_connections 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.637+0000 7fc92e45c740 1 -- 172.21.15.99:0/324583347 wait complete. 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.638+0000 7fc92e45c740 1 -- start start 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.638+0000 7fc92e45c740 1 Processor -- start 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969860 0x55a969968720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 0x55a96997e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a96997fdf0 0x55a969980c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a969941010 con 0x55a969969c40 2024-03-04T10:06:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9698620a0 con 0x55a96997fdf0 2024-03-04T10:06:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a969940400 con 0x55a969969860 2024-03-04T10:06:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92e45c740 10 client.0 _ll_register_callbacks cb 0x55a969861a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:43.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 0x55a96997e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 0x55a96997e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58824/0 (socket says 172.21.15.99:58824) 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 -- 172.21.15.99:0/4102988192 learned_addr learned my addr 172.21.15.99:0/4102988192 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc92961c640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969860 0x55a969968720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc929e1d640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a96997fdf0 0x55a969980c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 -- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969860 msgr2=0x55a969968720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969860 0x55a969968720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 -- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a96997fdf0 msgr2=0x55a969980c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a96997fdf0 0x55a969980c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 -- 172.21.15.99:0/4102988192 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a9695ba3f0 con 0x55a969969c40 2024-03-04T10:06:43.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc929e1d640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a96997fdf0 0x55a969980c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:43.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc928e1b640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 0x55a96997e470 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fc9140136c0 tx=0x7fc914001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a2a0f8c762a82a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc8fffff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.639+0000 7fc8fffff640 1 -- 172.21.15.99:0/4102988192 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914007050 con 0x55a969969c40 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc8fffff640 1 -- 172.21.15.99:0/4102988192 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc914024c80 con 0x55a969969c40 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc920056090 con 0x55a969969c40 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc8fffff640 1 -- 172.21.15.99:0/4102988192 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9140071f0 con 0x55a969969c40 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 10 client.6672 fetch_fsmap learned FSMap version 43 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55a96957a0f0 con 0x55a969969c40 2024-03-04T10:06:43.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc8fffff640 1 -- 172.21.15.99:0/4102988192 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920056090 con 0x55a969969c40 2024-03-04T10:06:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc8fffff640 1 -- 172.21.15.99:0/4102988192 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 43) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc914021020 con 0x55a969969c40 2024-03-04T10:06:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 10 client.6672 fetch_fsmap finished waiting for FSMap version 43 2024-03-04T10:06:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 10 client.6672 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-04T10:06:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 20 client.6672 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969968d50 0x55a969996e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 4 client.6672 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:43.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 --> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9698b3340 con 0x55a969968d50 2024-03-04T10:06:43.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92961c640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969968d50 0x55a969996e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc8fffff640 10 client.6672 ms_handle_connect on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:43.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.640+0000 7fc92961c640 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969968d50 0x55a969996e90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60df53d1c8741aab server_cookie=eac55389d6cd86d8 in_seq=0 out_seq=0 2024-03-04T10:06:43.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc8fffff640 1 -- 172.21.15.99:0/4102988192 <== mds.0 v2:172.21.15.99:6834/2687500379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a9698b3340 con 0x55a969968d50 2024-03-04T10:06:43.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc8fffff640 10 client.6672 handle_command_reply: tid=0 2024-03-04T10:06:43.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 client.6672 shutdown 2024-03-04T10:06:43.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 msgr2=0x55a96997e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 0x55a96997e470 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fc9140136c0 tx=0x7fc914001120 comp rx=0 tx=0).stop 2024-03-04T10:06:43.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 shutdown_connections 2024-03-04T10:06:43.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a96997fdf0 0x55a969980c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969c40 0x55a96997e470 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969968d50 0x55a969996e90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 --2- 172.21.15.99:0/4102988192 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969860 0x55a969968720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc8fffff640 0 client.6672 ms_handle_reset on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 >> 172.21.15.99:0/4102988192 conn(0x55a96993cd40 msgr2=0x55a96998ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 shutdown_connections 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 1 -- 172.21.15.99:0/4102988192 wait complete. 2024-03-04T10:06:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.641+0000 7fc92e45c740 20 client.6672 trim_cache size 0 max 16384 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.654+0000 7fc92e45c740 1 Processor -- start 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.654+0000 7fc92e45c740 1 -- start start 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a9699405f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96997fdf0 0x55a96996c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92e45c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241fe150 con 0x55a969969860 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241fd350 con 0x55a969969c40 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241f0460 con 0x55a96997fdf0 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a9699405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a9699405f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58834/0 (socket says 172.21.15.99:58834) 2024-03-04T10:06:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 -- 172.21.15.99:0/3198332445 learned_addr learned my addr 172.21.15.99:0/3198332445 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 -- 172.21.15.99:0/3198332445 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96997fdf0 msgr2=0x55a96996c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96997fdf0 0x55a96996c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 -- 172.21.15.99:0/3198332445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 msgr2=0x55a96996c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.655+0000 7fc92961c640 1 -- 172.21.15.99:0/3198332445 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a969982030 con 0x55a969969860 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92961c640 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a9699405f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc9200011e0 tx=0x7fc92005e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980a0ffd9919bf46 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc9067fc640 1 -- 172.21.15.99:0/3198332445 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920074070 con 0x55a969969860 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc9067fc640 1 -- 172.21.15.99:0/3198332445 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920056090 con 0x55a969969860 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc9067fc640 1 -- 172.21.15.99:0/3198332445 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92005e0b0 con 0x55a969969860 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 -- 172.21.15.99:0/3198332445 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 msgr2=0x55a9699405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a9699405f0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc9200011e0 tx=0x7fc92005e040 comp rx=0 tx=0).stop 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 -- 172.21.15.99:0/3198332445 shutdown_connections 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a96997fdf0 0x55a96996c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969c40 0x55a96996c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 --2- 172.21.15.99:0/3198332445 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a969969860 0x55a9699405f0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 -- 172.21.15.99:0/3198332445 >> 172.21.15.99:0/3198332445 conn(0x55a96993cd40 msgr2=0x55a969943820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 -- 172.21.15.99:0/3198332445 shutdown_connections 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.656+0000 7fc92e45c740 1 -- 172.21.15.99:0/3198332445 wait complete. 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.657+0000 7fc92e45c740 1 -- start start 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.657+0000 7fc92e45c740 1 Processor -- start 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969860 0x55a96993b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969c40 0x55a9699f2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 0x55a969966150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a969980ed0 con 0x55a96997fdf0 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241fce80 con 0x55a969969860 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a969967650 con 0x55a969969c40 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 10 client.0 _ll_register_callbacks cb 0x7fc914014d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:43.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 0x55a969966150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 0x55a969966150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58850/0 (socket says 172.21.15.99:58850) 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 -- 172.21.15.99:0/1392965698 learned_addr learned my addr 172.21.15.99:0/1392965698 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc928e1b640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969c40 0x55a9699f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92961c640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969860 0x55a96993b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 -- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969c40 msgr2=0x55a9699f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969c40 0x55a9699f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 -- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969860 msgr2=0x55a96993b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969860 0x55a96993b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 -- 172.21.15.99:0/1392965698 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a969987110 con 0x55a96997fdf0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92961c640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969860 0x55a96993b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc929e1d640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 0x55a969966150 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc91c02c560 tx=0x7fc91c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343dc3afb4fe59c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc8fffff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc8fffff640 1 -- 172.21.15.99:0/1392965698 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c045030 con 0x55a96997fdf0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc8fffff640 1 -- 172.21.15.99:0/1392965698 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc91c0349f0 con 0x55a96997fdf0 2024-03-04T10:06:43.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.658+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc920056090 con 0x55a96997fdf0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc8fffff640 1 -- 172.21.15.99:0/1392965698 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc91c050050 con 0x55a96997fdf0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 10 client.6678 fetch_fsmap learned FSMap version 43 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=2+}) v3 -- 0x55a96993bd80 con 0x55a96997fdf0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc8fffff640 1 -- 172.21.15.99:0/1392965698 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920056090 con 0x55a96997fdf0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc8fffff640 1 -- 172.21.15.99:0/1392965698 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 43) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc91c053020 con 0x55a96997fdf0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 10 client.6678 fetch_fsmap finished waiting for FSMap version 43 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 10 client.6678 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 20 client.6678 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969942ca0 0x55a969966690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 4 client.6678 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 --> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a9698ad690 con 0x55a969942ca0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92961c640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969942ca0 0x55a969966690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc8fffff640 10 client.6678 ms_handle_connect on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.659+0000 7fc92961c640 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969942ca0 0x55a969966690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6e0f485da2abe0c server_cookie=d261a06e455453bb in_seq=0 out_seq=0 2024-03-04T10:06:43.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.673+0000 7fc8fffff640 1 -- 172.21.15.99:0/1392965698 <== mds.0 v2:172.21.15.99:6834/2687500379 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a9698ad690 con 0x55a969942ca0 2024-03-04T10:06:43.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.673+0000 7fc8fffff640 10 client.6678 handle_command_reply: tid=0 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.673+0000 7fc92e45c740 1 client.6678 shutdown 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 msgr2=0x55a969966150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 0x55a969966150 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc91c02c560 tx=0x7fc91c034670 comp rx=0 tx=0).stop 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 shutdown_connections 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55a96997fdf0 0x55a969966150 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.99:6834/2687500379,v1:172.21.15.99:6835/2687500379] conn(0x55a969942ca0 0x55a969966690 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55a969969c40 0x55a9699f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 --2- 172.21.15.99:0/1392965698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a969969860 0x55a96993b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc8fffff640 0 client.6678 ms_handle_reset on v2:172.21.15.99:6834/2687500379 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 >> 172.21.15.99:0/1392965698 conn(0x55a96993cd40 msgr2=0x55a96996f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 shutdown_connections 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 1 -- 172.21.15.99:0/1392965698 wait complete. 2024-03-04T10:06:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92e45c740 20 client.6678 trim_cache size 0 max 16384 2024-03-04T10:06:43.678 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:43.678 INFO:teuthology.orchestra.run.smithi099.stdout: "message": "", 2024-03-04T10:06:43.678 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0 2024-03-04T10:06:43.678 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:43.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc90806b4d0 msgr2=0x7fc90806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92b880640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc90806b4d0 0x7fc90806d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc914000f60 tx=0x7fc9140079d0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 msgr2=0x7fc924146a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.674+0000 7fc92b880640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc924146a80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fc92000c440 tx=0x7fc92000c900 comp rx=0 tx=0).stop 2024-03-04T10:06:43.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 shutdown_connections 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc90806b4d0 0x7fc90806d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc92415a810 0x7fc92415d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc9240b48a0 0x7fc92414e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 --2- 172.21.15.99:0/1431988876 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc92406d3c0 0x7fc924146a80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 >> 172.21.15.99:0/1431988876 conn(0x7fc92406acd0 msgr2=0x7fc9240b34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 shutdown_connections 2024-03-04T10:06:43.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.675+0000 7fc92b880640 1 -- 172.21.15.99:0/1431988876 wait complete. 2024-03-04T10:06:43.687 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:06:43.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.786+0000 7f43f7129640 1 Processor -- start 2024-03-04T10:06:43.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.786+0000 7f43f7129640 1 -- start start 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f7129640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f43f015b960 0x7f43f00d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f7129640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f43f00b0e10 0x7f43f00b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f7129640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f00ccad0 con 0x7f43f00d0ad0 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f7129640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f00cf860 con 0x7f43f00b0e10 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f7129640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f00d0100 con 0x7f43f015b960 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f43f00b0e10 0x7f43f00b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f43f00b0e10 0x7f43f00b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58858/0 (socket says 172.21.15.99:58858) 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 -- 172.21.15.99:0/3279522257 learned_addr learned my addr 172.21.15.99:0/3279522257 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f4ec5640 1 --2- 172.21.15.99:0/3279522257 >> v2:172.21.15.162:3300/0 conn(0x7f43f015b960 0x7f43f00d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 -- 172.21.15.99:0/3279522257 >> v2:172.21.15.162:3300/0 conn(0x7f43f015b960 msgr2=0x7f43f00d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 --2- 172.21.15.99:0/3279522257 >> v2:172.21.15.162:3300/0 conn(0x7f43f015b960 0x7f43f00d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 -- 172.21.15.99:0/3279522257 >> v1:172.21.15.162:6789/0 conn(0x7f43f00d0ad0 legacy=0x7f43f00b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.787+0000 7f43f56c6640 1 -- 172.21.15.99:0/3279522257 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f00cfab0 con 0x7f43f00b0e10 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.788+0000 7f43f56c6640 1 --2- 172.21.15.99:0/3279522257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f00b47c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f43f006e970 tx=0x7f43e802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7f095dbe842e0e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.788+0000 7f43ef7fe640 1 -- 172.21.15.99:0/3279522257 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e802c9f0 con 0x7f43f00b0e10 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.788+0000 7f43ef7fe640 1 -- 172.21.15.99:0/3279522257 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e802cb90 con 0x7f43f00b0e10 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.788+0000 7f43ef7fe640 1 -- 172.21.15.99:0/3279522257 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80343f0 con 0x7f43f00b0e10 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 -- 172.21.15.99:0/3279522257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 msgr2=0x7f43f00b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 --2- 172.21.15.99:0/3279522257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f00b47c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f43f006e970 tx=0x7f43e802be50 comp rx=0 tx=0).stop 2024-03-04T10:06:43.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 -- 172.21.15.99:0/3279522257 shutdown_connections 2024-03-04T10:06:43.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 --2- 172.21.15.99:0/3279522257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f00b47c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 --2- 172.21.15.99:0/3279522257 >> v2:172.21.15.162:3300/0 conn(0x7f43f015b960 0x7f43f00d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 -- 172.21.15.99:0/3279522257 >> 172.21.15.99:0/3279522257 conn(0x7f43f006f5f0 msgr2=0x7f43f006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:43.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 -- 172.21.15.99:0/3279522257 shutdown_connections 2024-03-04T10:06:43.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.789+0000 7f43f7129640 1 -- 172.21.15.99:0/3279522257 wait complete. 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.790+0000 7f43f7129640 1 Processor -- start 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.790+0000 7f43f7129640 1 -- start start 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f014dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f00d0ad0 0x7f43f01e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43f015b960 0x7f43f01e4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f015bec0 con 0x7f43f00b0e10 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f015bd40 con 0x7f43f00d0ad0 2024-03-04T10:06:43.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f00b4d00 con 0x7f43f015b960 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f014dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f014dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58866/0 (socket says 172.21.15.99:58866) 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 -- 172.21.15.99:0/3390884634 learned_addr learned my addr 172.21.15.99:0/3390884634 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43effff640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f00d0ad0 0x7f43f01e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f56c6640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43f015b960 0x7f43f01e4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 -- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43f015b960 msgr2=0x7f43f01e4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43f015b960 0x7f43f01e4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 -- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f00d0ad0 msgr2=0x7f43f01e49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f00d0ad0 0x7f43f01e49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 -- 172.21.15.99:0/3390884634 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f00ca180 con 0x7f43f00b0e10 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f4ec5640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f014dcd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f43e400d5d0 tx=0x7f43e400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e40125f0 con 0x7f43f00b0e10 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4019070 con 0x7f43f00b0e10 2024-03-04T10:06:43.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f00caac0 con 0x7f43f00b0e10 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.791+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e40129f0 con 0x7f43f00b0e10 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.792+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f00cb080 con 0x7f43f00b0e10 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.792+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f0074f60 con 0x7f43f00b0e10 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.793+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f43e4012b90 con 0x7f43f00b0e10 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.793+0000 7f43ed7fa640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.793+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43e408a7d0 con 0x7f43f00b0e10 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.793+0000 7f43effff640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:43.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.794+0000 7f43effff640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f43d406b4d0 0x7f43d406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f43f00b0c10 tx=0x7f43dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:43.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:43.797+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f43e40594c0 con 0x7f43f00b0e10 2024-03-04T10:06:44.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.003+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f43f00c4f80 con 0x7f43f00b0e10 2024-03-04T10:06:44.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.003+0000 7f43ed7fa640 1 -- 172.21.15.99:0/3390884634 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v43) v1 ==== 53+0+73 (secure 0 0 0) 0x7f43e4058dc0 con 0x7f43f00b0e10 2024-03-04T10:06:44.006 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:06:44.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f43d406b4d0 msgr2=0x7f43d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:44.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f43d406b4d0 0x7f43d406d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f43f00b0c10 tx=0x7f43dc009040 comp rx=0 tx=0).stop 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 msgr2=0x7f43f014dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f014dcd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f43e400d5d0 tx=0x7f43e400da90 comp rx=0 tx=0).stop 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 shutdown_connections 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f43d406b4d0 0x7f43d406d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43f015b960 0x7f43f01e4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43f00d0ad0 0x7f43f01e49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 --2- 172.21.15.99:0/3390884634 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f43f00b0e10 0x7f43f014dcd0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 >> 172.21.15.99:0/3390884634 conn(0x7f43f006f5f0 msgr2=0x7f43f0070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 shutdown_connections 2024-03-04T10:06:44.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.005+0000 7f43f7129640 1 -- 172.21.15.99:0/3390884634 wait complete. 2024-03-04T10:06:44.018 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:06:44.018 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:06:44.018 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:06:44.018 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:06:44.018 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:44.019 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:06:44.019 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:06:44.019 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:06:44.019 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:44.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.019 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:06:44.034 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:44.035 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42581sec preferred_lft 42581sec 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:06:44.036 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:06:44.037 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:06:44.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.038 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:06:44.100 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:06:44.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.102 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:06:44.157 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:06:44.157 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:06:44.157 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:06:44.157 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:06:44.158 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:06:44.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.158 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:06:44.158 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:06:44.158 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:06:44.158 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:06:44.158 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:06:44.158 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:06:44.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.252 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:06:44.267 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:06:44.267 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:06:44.267 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:44.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.267 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:06:44.334 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:06:44.334 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:44.390 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:06:44.390 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:06:44.390 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:44.444 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:06:44.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.444 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:44.528 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:44.528 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:06:44.536 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:06:44.537 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:06:44.537 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:06:44.537 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:06:44.538 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:06:44.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.538 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:44.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.564 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:06:44.621 INFO:teuthology.orchestra.run.smithi099.stdout:client5926 2024-03-04T10:06:44.621 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:44.621 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:44.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:44.676 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'chmod 711 a/b/d; sync .') 2024-03-04T10:06:44.734 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:06:44.734 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:06:44.789 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:06:44.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.926+0000 7fadc5b7e640 1 Processor -- start 2024-03-04T10:06:44.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.926+0000 7fadc5b7e640 1 -- start start 2024-03-04T10:06:44.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadc5b7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fadc006acd0 0x7fadc015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadc5b7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fadc006b0b0 0x7fadc006b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadc5b7e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc00cf700 con 0x7fadc00ce340 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadc5b7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc00d02d0 con 0x7fadc006b0b0 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadc5b7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc006ba50 con 0x7fadc006acd0 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadb6ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fadc006b0b0 0x7fadc006b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadb6ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fadc006b0b0 0x7fadc006b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58876/0 (socket says 172.21.15.99:58876) 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.927+0000 7fadb6ffd640 1 -- 172.21.15.99:0/3760277785 learned_addr learned my addr 172.21.15.99:0/3760277785 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadb6ffd640 1 -- 172.21.15.99:0/3760277785 >> v2:172.21.15.162:3300/0 conn(0x7fadc006acd0 msgr2=0x7fadc015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadbf7fe640 1 --2- 172.21.15.99:0/3760277785 >> v2:172.21.15.162:3300/0 conn(0x7fadc006acd0 0x7fadc015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadb6ffd640 1 --2- 172.21.15.99:0/3760277785 >> v2:172.21.15.162:3300/0 conn(0x7fadc006acd0 0x7fadc015c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadb6ffd640 1 -- 172.21.15.99:0/3760277785 >> v1:172.21.15.162:6789/0 conn(0x7fadc00ce340 legacy=0x7fadc00ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadb6ffd640 1 -- 172.21.15.99:0/3760277785 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00cf9a0 con 0x7fadc006b0b0 2024-03-04T10:06:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadbf7fe640 1 --2- 172.21.15.99:0/3760277785 >> v2:172.21.15.162:3300/0 conn(0x7fadc006acd0 0x7fadc015c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.928+0000 7fadb6ffd640 1 --2- 172.21.15.99:0/3760277785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc006b0b0 0x7fadc006b510 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fadc006cdc0 tx=0x7fada802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436b4fb766d0fba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadbeffd640 1 -- 172.21.15.99:0/3760277785 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada802e070 con 0x7fadc006b0b0 2024-03-04T10:06:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadbeffd640 1 -- 172.21.15.99:0/3760277785 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada8037070 con 0x7fadc006b0b0 2024-03-04T10:06:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadbeffd640 1 -- 172.21.15.99:0/3760277785 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8031a00 con 0x7fadc006b0b0 2024-03-04T10:06:44.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 -- 172.21.15.99:0/3760277785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc006b0b0 msgr2=0x7fadc006b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:44.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/3760277785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc006b0b0 0x7fadc006b510 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fadc006cdc0 tx=0x7fada802b880 comp rx=0 tx=0).stop 2024-03-04T10:06:44.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 -- 172.21.15.99:0/3760277785 shutdown_connections 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/3760277785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc006b0b0 0x7fadc006b510 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/3760277785 >> v2:172.21.15.162:3300/0 conn(0x7fadc006acd0 0x7fadc015c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 -- 172.21.15.99:0/3760277785 >> 172.21.15.99:0/3760277785 conn(0x7fadc00d0770 msgr2=0x7fadc00d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 -- 172.21.15.99:0/3760277785 shutdown_connections 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.929+0000 7fadc5b7e640 1 -- 172.21.15.99:0/3760277785 wait complete. 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.930+0000 7fadc5b7e640 1 Processor -- start 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.930+0000 7fadc5b7e640 1 -- start start 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadc5b7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc006acd0 0x7fadc014caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadc5b7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadc006b0b0 0x7fadc00c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadc5b7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 0x7fadc00b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadc5b7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc00d13e0 con 0x7fadc00ce340 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadc5b7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc00cff60 con 0x7fadc006acd0 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadc5b7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0159850 con 0x7fadc006b0b0 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadbffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 0x7fadc00b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:44.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadbffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 0x7fadc00b5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58892/0 (socket says 172.21.15.99:58892) 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.931+0000 7fadbffff640 1 -- 172.21.15.99:0/2695799782 learned_addr learned my addr 172.21.15.99:0/2695799782 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbf7fe640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc006acd0 0x7fadc014caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadb6ffd640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadc006b0b0 0x7fadc00c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbffff640 1 -- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadc006b0b0 msgr2=0x7fadc00c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbffff640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadc006b0b0 0x7fadc00c5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbffff640 1 -- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc006acd0 msgr2=0x7fadc014caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbffff640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc006acd0 0x7fadc014caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbffff640 1 -- 172.21.15.99:0/2695799782 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00c5140 con 0x7fadc00ce340 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbffff640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 0x7fadc00b5640 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fadb800c780 tx=0x7fadb800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb8004b60 con 0x7fadc00ce340 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb8004d00 con 0x7fadc00ce340 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc00c5420 con 0x7fadc00ce340 2024-03-04T10:06:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb800aa90 con 0x7fadc00ce340 2024-03-04T10:06:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.932+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc00b3580 con 0x7fadc00ce340 2024-03-04T10:06:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.933+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc0074f60 con 0x7fadc00ce340 2024-03-04T10:06:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.933+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fadb800ac30 con 0x7fadc00ce340 2024-03-04T10:06:44.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.934+0000 7fadbcff9640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fada006b4d0 0x7fada006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:44.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.934+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fadb808a670 con 0x7fadc00ce340 2024-03-04T10:06:44.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.934+0000 7fadbf7fe640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fada006b4d0 0x7fada006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:44.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.934+0000 7fadbf7fe640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fada006b4d0 0x7fada006d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fadc01f61a0 tx=0x7fadb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:44.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:44.938+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fadb8020070 con 0x7fadc00ce340 2024-03-04T10:06:45.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.113+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadc015d310 con 0x7fadc00ce340 2024-03-04T10:06:45.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.114+0000 7fadbcff9640 1 -- 172.21.15.99:0/2695799782 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16848 (secure 0 0 0) 0x7fadb8059360 con 0x7fadc00ce340 2024-03-04T10:06:45.116 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:45.117 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":66,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:06:37.015810+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fada006b4d0 msgr2=0x7fada006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fada006b4d0 0x7fada006d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fadc01f61a0 tx=0x7fadb0008040 comp rx=0 tx=0).stop 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 msgr2=0x7fadc00b5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 0x7fadc00b5640 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fadb800c780 tx=0x7fadb800cc40 comp rx=0 tx=0).stop 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 shutdown_connections 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fada006b4d0 0x7fada006d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadc00ce340 0x7fadc00b5640 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadc006b0b0 0x7fadc00c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 --2- 172.21.15.99:0/2695799782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadc006acd0 0x7fadc014caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 >> 172.21.15.99:0/2695799782 conn(0x7fadc00d0770 msgr2=0x7fadc015c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:45.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 shutdown_connections 2024-03-04T10:06:45.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.116+0000 7fadc5b7e640 1 -- 172.21.15.99:0/2695799782 wait complete. 2024-03-04T10:06:45.129 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:06:33.589849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:06:34.331851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:06:45.130 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:06:45.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.231+0000 7ff6f508e640 1 Processor -- start 2024-03-04T10:06:45.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.231+0000 7ff6f508e640 1 -- start start 2024-03-04T10:06:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6f508e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff6f015b700 0x7ff6f00ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6f508e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff6f015e810 0x7ff6f006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6f508e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f015bae0 con 0x7ff6f006bce0 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6f508e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f006c680 con 0x7ff6f015e810 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6f508e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f00adb90 con 0x7ff6f015b700 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6ef59e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff6f015e810 0x7ff6f006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6ef59e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff6f015e810 0x7ff6f006c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58908/0 (socket says 172.21.15.99:58908) 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6ef59e640 1 -- 172.21.15.99:0/3630277098 learned_addr learned my addr 172.21.15.99:0/3630277098 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6eed9d640 1 --2- 172.21.15.99:0/3630277098 >> v2:172.21.15.162:3300/0 conn(0x7ff6f015b700 0x7ff6f00ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6ef59e640 1 -- 172.21.15.99:0/3630277098 >> v2:172.21.15.162:3300/0 conn(0x7ff6f015b700 msgr2=0x7ff6f00ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.232+0000 7ff6ef59e640 1 --2- 172.21.15.99:0/3630277098 >> v2:172.21.15.162:3300/0 conn(0x7ff6f015b700 0x7ff6f00ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.233+0000 7ff6ef59e640 1 -- 172.21.15.99:0/3630277098 >> v1:172.21.15.162:6789/0 conn(0x7ff6f006bce0 legacy=0x7ff6f015e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.233+0000 7ff6ef59e640 1 -- 172.21.15.99:0/3630277098 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f015ee60 con 0x7ff6f015e810 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.233+0000 7ff6ef59e640 1 --2- 172.21.15.99:0/3630277098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f006c140 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0071410 tx=0x7ff6e802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840a05ae4bd00147 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.233+0000 7ff6ee59c640 1 -- 172.21.15.99:0/3630277098 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e802cb20 con 0x7ff6f015e810 2024-03-04T10:06:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.233+0000 7ff6ee59c640 1 -- 172.21.15.99:0/3630277098 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e802ccc0 con 0x7ff6f015e810 2024-03-04T10:06:45.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.233+0000 7ff6ee59c640 1 -- 172.21.15.99:0/3630277098 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8034610 con 0x7ff6f015e810 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 -- 172.21.15.99:0/3630277098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 msgr2=0x7ff6f006c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 --2- 172.21.15.99:0/3630277098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f006c140 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0071410 tx=0x7ff6e802bf80 comp rx=0 tx=0).stop 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 -- 172.21.15.99:0/3630277098 shutdown_connections 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 --2- 172.21.15.99:0/3630277098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f006c140 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 --2- 172.21.15.99:0/3630277098 >> v2:172.21.15.162:3300/0 conn(0x7ff6f015b700 0x7ff6f00ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 -- 172.21.15.99:0/3630277098 >> 172.21.15.99:0/3630277098 conn(0x7ff6f0071bc0 msgr2=0x7ff6f0071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 -- 172.21.15.99:0/3630277098 shutdown_connections 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.234+0000 7ff6f508e640 1 -- 172.21.15.99:0/3630277098 wait complete. 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.235+0000 7ff6f508e640 1 Processor -- start 2024-03-04T10:06:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.235+0000 7ff6f508e640 1 -- start start 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6f508e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f006bce0 0x7ff6f01459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6f508e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f015b700 0x7ff6f01e6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6f508e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f00cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6f508e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00ae560 con 0x7ff6f015e810 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6f508e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00ad820 con 0x7ff6f006bce0 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6f508e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f00ae6e0 con 0x7ff6f015b700 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6ef59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f00cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6ef59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f00cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58910/0 (socket says 172.21.15.99:58910) 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6ef59e640 1 -- 172.21.15.99:0/2056130518 learned_addr learned my addr 172.21.15.99:0/2056130518 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6ef59e640 1 -- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f015b700 msgr2=0x7ff6f01e6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.236+0000 7ff6ef59e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f015b700 0x7ff6f01e6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6ef59e640 1 -- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f006bce0 msgr2=0x7ff6f01459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6ef59e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f006bce0 0x7ff6f01459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6ef59e640 1 -- 172.21.15.99:0/2056130518 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f01e5d70 con 0x7ff6f015e810 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6ef59e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f00cdf40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009f40 tx=0x7ff6e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8043020 con 0x7ff6f015e810 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e803e450 con 0x7ff6f015e810 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8009510 con 0x7ff6f015e810 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f00cd460 con 0x7ff6f015e810 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.237+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f00cda20 con 0x7ff6f015e810 2024-03-04T10:06:45.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.238+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f0074f60 con 0x7ff6f015e810 2024-03-04T10:06:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.238+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff6e8046020 con 0x7ff6f015e810 2024-03-04T10:06:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.239+0000 7ff6e7fff640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6d006b4d0 0x7ff6d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.239+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff6e80ac8f0 con 0x7ff6f015e810 2024-03-04T10:06:45.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.239+0000 7ff6eed9d640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6d006b4d0 0x7ff6d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.240+0000 7ff6eed9d640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6d006b4d0 0x7ff6d006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00098f0 tx=0x7ff6e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.243+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff6e804d070 con 0x7ff6f015e810 2024-03-04T10:06:45.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.413+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6f00cdbf0 con 0x7ff6f015e810 2024-03-04T10:06:45.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.413+0000 7ff6e7fff640 1 -- 172.21.15.99:0/2056130518 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6e807b5e0 con 0x7ff6f015e810 2024-03-04T10:06:45.416 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6d006b4d0 msgr2=0x7ff6d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6d006b4d0 0x7ff6d006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff6e00098f0 tx=0x7ff6e0008040 comp rx=0 tx=0).stop 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 msgr2=0x7ff6f00cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f00cdf40 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8009f40 tx=0x7ff6e8004a40 comp rx=0 tx=0).stop 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 shutdown_connections 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6d006b4d0 0x7ff6d006d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff6f015e810 0x7ff6f00cdf40 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff6f015b700 0x7ff6f01e6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 --2- 172.21.15.99:0/2056130518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6f006bce0 0x7ff6f01459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 >> 172.21.15.99:0/2056130518 conn(0x7ff6f0071bc0 msgr2=0x7ff6f00b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:45.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 shutdown_connections 2024-03-04T10:06:45.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.415+0000 7ff6f508e640 1 -- 172.21.15.99:0/2056130518 wait complete. 2024-03-04T10:06:45.428 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:06:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.529+0000 7f7d783df640 1 Processor -- start 2024-03-04T10:06:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.529+0000 7f7d783df640 1 -- start start 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d783df640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7d700cfc70 0x7f7d700cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d783df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cc340 0x7f7d700b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d783df640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d700cebd0 con 0x7f7d701450e0 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d783df640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d700cc7b0 con 0x7f7d700cfc70 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d783df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700cf7e0 con 0x7f7d700cc340 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7597a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7d700cfc70 0x7f7d700cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7597a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7d700cfc70 0x7f7d700cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58922/0 (socket says 172.21.15.99:58922) 2024-03-04T10:06:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7597a640 1 -- 172.21.15.99:0/2806481448 learned_addr learned my addr 172.21.15.99:0/2806481448 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7697c640 1 --2- 172.21.15.99:0/2806481448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cc340 0x7f7d700b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7597a640 1 -- 172.21.15.99:0/2806481448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cc340 msgr2=0x7f7d700b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7597a640 1 --2- 172.21.15.99:0/2806481448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cc340 0x7f7d700b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.530+0000 7f7d7597a640 1 -- 172.21.15.99:0/2806481448 >> v1:172.21.15.162:6789/0 conn(0x7f7d701450e0 legacy=0x7f7d7015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.531+0000 7f7d7597a640 1 -- 172.21.15.99:0/2806481448 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d700ceeb0 con 0x7f7d700cfc70 2024-03-04T10:06:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.531+0000 7f7d7597a640 1 --2- 172.21.15.99:0/2806481448 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d700cfc70 0x7f7d700cbe00 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009870 tx=0x7f7d6002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e670cff45709b70a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.531+0000 7f7d75179640 1 -- 172.21.15.99:0/2806481448 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60034070 con 0x7f7d700cfc70 2024-03-04T10:06:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.531+0000 7f7d75179640 1 -- 172.21.15.99:0/2806481448 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d60030840 con 0x7f7d700cfc70 2024-03-04T10:06:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.531+0000 7f7d75179640 1 -- 172.21.15.99:0/2806481448 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60030c10 con 0x7f7d700cfc70 2024-03-04T10:06:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 -- 172.21.15.99:0/2806481448 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d700cfc70 msgr2=0x7f7d700cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 --2- 172.21.15.99:0/2806481448 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d700cfc70 0x7f7d700cbe00 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009870 tx=0x7f7d6002b980 comp rx=0 tx=0).stop 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 -- 172.21.15.99:0/2806481448 shutdown_connections 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 --2- 172.21.15.99:0/2806481448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cc340 0x7f7d700b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 --2- 172.21.15.99:0/2806481448 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d700cfc70 0x7f7d700cbe00 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 -- 172.21.15.99:0/2806481448 >> 172.21.15.99:0/2806481448 conn(0x7f7d7006acd0 msgr2=0x7f7d7006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 -- 172.21.15.99:0/2806481448 shutdown_connections 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.532+0000 7f7d783df640 1 -- 172.21.15.99:0/2806481448 wait complete. 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.533+0000 7f7d783df640 1 Processor -- start 2024-03-04T10:06:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.533+0000 7f7d783df640 1 -- start start 2024-03-04T10:06:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d783df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d700cc340 0x7f7d701412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d783df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cfc70 0x7f7d7013bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d783df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 0x7f7d7013e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d783df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d701454c0 con 0x7f7d701450e0 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d783df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700cf470 con 0x7f7d700cc340 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d783df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d700ccd10 con 0x7f7d700cfc70 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 0x7f7d7013e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 0x7f7d7013e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58936/0 (socket says 172.21.15.99:58936) 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 -- 172.21.15.99:0/2564890411 learned_addr learned my addr 172.21.15.99:0/2564890411 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7597a640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cfc70 0x7f7d7013bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7617b640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d700cc340 0x7f7d701412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 -- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cfc70 msgr2=0x7f7d7013bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cfc70 0x7f7d7013bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 -- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d700cc340 msgr2=0x7f7d701412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d700cc340 0x7f7d701412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7697c640 1 -- 172.21.15.99:0/2564890411 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7015d720 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.534+0000 7f7d7617b640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d700cc340 0x7f7d701412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.535+0000 7f7d7697c640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 0x7f7d7013e310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800d800 tx=0x7f7d6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.535+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800b6c0 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.535+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d68019070 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.535+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7015da00 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.535+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800bac0 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.535+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d7015d410 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.536+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d70074f60 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.536+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7d6800bc60 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.536+0000 7f7d66ffd640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d5406b500 0x7f7d5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.537+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d6808a7a0 con 0x7f7d701450e0 2024-03-04T10:06:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.537+0000 7f7d7617b640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d5406b500 0x7f7d5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.537+0000 7f7d7617b640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d5406b500 0x7f7d5406d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c001d60 tx=0x7f7d6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.541+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7d68059490 con 0x7f7d701450e0 2024-03-04T10:06:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.709+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d70141800 con 0x7f7d701450e0 2024-03-04T10:06:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.710+0000 7f7d66ffd640 1 -- 172.21.15.99:0/2564890411 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v66) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d68058d90 con 0x7f7d701450e0 2024-03-04T10:06:45.712 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:06:45.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d5406b500 msgr2=0x7f7d5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d5406b500 0x7f7d5406d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c001d60 tx=0x7f7d6c009040 comp rx=0 tx=0).stop 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 msgr2=0x7f7d7013e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 0x7f7d7013e310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800d800 tx=0x7f7d6800dcc0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 shutdown_connections 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d5406b500 0x7f7d5406d9c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d701450e0 0x7f7d7013e310 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d700cfc70 0x7f7d7013bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 --2- 172.21.15.99:0/2564890411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d700cc340 0x7f7d701412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 >> 172.21.15.99:0/2564890411 conn(0x7f7d7006acd0 msgr2=0x7f7d7006d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 shutdown_connections 2024-03-04T10:06:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.712+0000 7f7d783df640 1 -- 172.21.15.99:0/2564890411 wait complete. 2024-03-04T10:06:45.725 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:06:45.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.826+0000 7f618b3ab640 1 Processor -- start 2024-03-04T10:06:45.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f618b3ab640 1 -- start start 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f618b3ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61840d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f618b3ab640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f61840d34a0 0x7f618406c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f618b3ab640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61840d2400 con 0x7f61840b9410 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f618b3ab640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61840d26e0 con 0x7f61840d34a0 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f618b3ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61840d29c0 con 0x7f61840b5700 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f6189948640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f61840b9410 0x7f618406c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58730/0 (socket says 172.21.15.99:58730) 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.827+0000 7f6189948640 1 -- 172.21.15.99:0/2633984222 learned_addr learned my addr 172.21.15.99:0/2633984222 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061652276 0 0) 0x7f61840d2400 con 0x7f61840b9410 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 --2- 172.21.15.99:0/2633984222 >> v2:172.21.15.162:3300/0 conn(0x7f61840d34a0 0x7f618406c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6170003520 con 0x7f61840b9410 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6189147640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61840d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720455942 0 0) 0x7f6170003520 con 0x7f61840b9410 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 msgr2=0x7f61840d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61840d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 >> v2:172.21.15.162:3300/0 conn(0x7f61840d34a0 msgr2=0x7f618406c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 --2- 172.21.15.99:0/2633984222 >> v2:172.21.15.162:3300/0 conn(0x7f61840d34a0 0x7f618406c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61840d2ca0 con 0x7f61840b9410 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f617c0019f0 con 0x7f61840b9410 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 >> v1:172.21.15.99:6789/0 conn(0x7f61840b9410 legacy=0x7f618406c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6170003f30 0x7f6170004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170004d40 0x7f61700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61840d26e0 con 0x7f6170003f30 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61840d29c0 con 0x7f6170004d40 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617000ba10 con 0x7f61840b5700 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6170003f30 0x7f6170004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6189147640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6189948640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170004d40 0x7f61700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 -- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 msgr2=0x7f61700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 -- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170004d40 msgr2=0x7f61700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6170004d40 0x7f61700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6188946640 1 -- 172.21.15.99:0/2633984222 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6170003520 con 0x7f6170003f30 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.828+0000 7f6189147640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61840b5700 0x7f61700039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.829+0000 7f6188946640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6170003f30 0x7f6170004800 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6174009af0 tx=0x7f6170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d4f23aa7854fe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.829+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174039070 con 0x7f6170003f30 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.829+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f617402c710 con 0x7f6170003f30 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.829+0000 7f617bfff640 1 -- 172.21.15.99:0/2633984222 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174009690 con 0x7f6170003f30 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f618b3ab640 1 -- 172.21.15.99:0/2633984222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6170003f30 msgr2=0x7f6170004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2633984222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6170003f30 0x7f6170004800 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f6174009af0 tx=0x7f6170005830 comp rx=0 tx=0).stop 2024-03-04T10:06:45.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f6189948640 1 -- 172.21.15.99:0/2633984222 reap_dead start 2024-03-04T10:06:45.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f618b3ab640 1 -- 172.21.15.99:0/2633984222 shutdown_connections 2024-03-04T10:06:45.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f618b3ab640 1 -- 172.21.15.99:0/2633984222 >> 172.21.15.99:0/2633984222 conn(0x7f61840d0490 msgr2=0x7f61840d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:45.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f618b3ab640 1 -- 172.21.15.99:0/2633984222 shutdown_connections 2024-03-04T10:06:45.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.830+0000 7f618b3ab640 1 -- 172.21.15.99:0/2633984222 wait complete. 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.831+0000 7f618b3ab640 1 Processor -- start 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.831+0000 7f618b3ab640 1 -- start start 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6170003f30 0x7f6184158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 0x7f61840c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61841569b0 0x7f6184156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61840b5c60 con 0x7f61840caae0 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61840b5ae0 con 0x7f61841569b0 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61840b5de0 con 0x7f6170003f30 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 0x7f61840c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 0x7f61840c8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58952/0 (socket says 172.21.15.99:58952) 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 -- 172.21.15.99:0/2501583750 learned_addr learned my addr 172.21.15.99:0/2501583750 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6189147640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6170003f30 0x7f6184158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6189948640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61841569b0 0x7f6184156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 -- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6170003f30 msgr2=0x7f6184158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6170003f30 0x7f6184158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 -- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61841569b0 msgr2=0x7f6184156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61841569b0 0x7f6184156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 -- 172.21.15.99:0/2501583750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61840cafe0 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6188946640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 0x7f61840c8610 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6174037040 tx=0x7f6174004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174039050 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174045070 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61840cb2c0 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.832+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61740044e0 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.833+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61840cb880 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.833+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6184074f60 con 0x7f61840caae0 2024-03-04T10:06:45.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.834+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6174003a50 con 0x7f61840caae0 2024-03-04T10:06:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.834+0000 7f6179ffb640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f616806b4d0 0x7f616806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.834+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f61740ac0a0 con 0x7f61840caae0 2024-03-04T10:06:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.834+0000 7f6189147640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f616806b4d0 0x7f616806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.835+0000 7f6189147640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f616806b4d0 0x7f616806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f61800099b0 tx=0x7f6180008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:45.839+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f617404e020 con 0x7f61840caae0 2024-03-04T10:06:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.009+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f61840b3cd0 con 0x7f61840caae0 2024-03-04T10:06:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.009+0000 7f6179ffb640 1 -- 172.21.15.99:0/2501583750 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v66) v1 ==== 100+0+11 (secure 0 0 0) 0x7f617407aeb0 con 0x7f61840caae0 2024-03-04T10:06:46.011 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:06:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f616806b4d0 msgr2=0x7f616806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f616806b4d0 0x7f616806d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f61800099b0 tx=0x7f6180008040 comp rx=0 tx=0).stop 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 msgr2=0x7f61840c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 0x7f61840c8610 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f6174037040 tx=0x7f6174004730 comp rx=0 tx=0).stop 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 shutdown_connections 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f616806b4d0 0x7f616806d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61841569b0 0x7f6184156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61840caae0 0x7f61840c8610 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 --2- 172.21.15.99:0/2501583750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6170003f30 0x7f6184158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 >> 172.21.15.99:0/2501583750 conn(0x7f61840d0490 msgr2=0x7f61840b2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 shutdown_connections 2024-03-04T10:06:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.011+0000 7f618b3ab640 1 -- 172.21.15.99:0/2501583750 wait complete. 2024-03-04T10:06:46.024 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:06:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.123+0000 7ffb39681640 1 Processor -- start 2024-03-04T10:06:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.123+0000 7ffb39681640 1 -- start start 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb39681640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb3415de00 0x7ffb340b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb39681640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb340b9380 0x7ffb340b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb39681640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3415e1e0 con 0x7ffb340b70d0 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb39681640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340d2820 con 0x7ffb340b9380 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb39681640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d1c10 con 0x7ffb3415de00 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb340b9380 0x7ffb340b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb340b9380 0x7ffb340b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58958/0 (socket says 172.21.15.99:58958) 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 -- 172.21.15.99:0/3067614597 learned_addr learned my addr 172.21.15.99:0/3067614597 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 -- 172.21.15.99:0/3067614597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb3415de00 msgr2=0x7ffb340b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 --2- 172.21.15.99:0/3067614597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb3415de00 0x7ffb340b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 -- 172.21.15.99:0/3067614597 >> v1:172.21.15.162:6789/0 conn(0x7ffb340b70d0 legacy=0x7ffb340b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.124+0000 7ffb327fc640 1 -- 172.21.15.99:0/3067614597 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340d1ef0 con 0x7ffb340b9380 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.125+0000 7ffb327fc640 1 --2- 172.21.15.99:0/3067614597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340b9380 0x7ffb340b6b90 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009870 tx=0x7ffb2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1396f421e8904d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.125+0000 7ffb31ffb640 1 -- 172.21.15.99:0/3067614597 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28035020 con 0x7ffb340b9380 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.125+0000 7ffb31ffb640 1 -- 172.21.15.99:0/3067614597 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb28031640 con 0x7ffb340b9380 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.125+0000 7ffb31ffb640 1 -- 172.21.15.99:0/3067614597 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2802d040 con 0x7ffb340b9380 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 -- 172.21.15.99:0/3067614597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340b9380 msgr2=0x7ffb340b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 --2- 172.21.15.99:0/3067614597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340b9380 0x7ffb340b6b90 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009870 tx=0x7ffb2802b8f0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 -- 172.21.15.99:0/3067614597 shutdown_connections 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 --2- 172.21.15.99:0/3067614597 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340b9380 0x7ffb340b6b90 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 --2- 172.21.15.99:0/3067614597 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb3415de00 0x7ffb340b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 -- 172.21.15.99:0/3067614597 >> 172.21.15.99:0/3067614597 conn(0x7ffb340d0470 msgr2=0x7ffb340d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:46.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 -- 172.21.15.99:0/3067614597 shutdown_connections 2024-03-04T10:06:46.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.126+0000 7ffb39681640 1 -- 172.21.15.99:0/3067614597 wait complete. 2024-03-04T10:06:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.127+0000 7ffb39681640 1 Processor -- start 2024-03-04T10:06:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.127+0000 7ffb39681640 1 -- start start 2024-03-04T10:06:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340b70d0 0x7ffb3415e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340b9380 0x7ffb340c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 0x7ffb341488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340b97e0 con 0x7ffb3415de00 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340d24b0 con 0x7ffb340b70d0 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340b5b80 con 0x7ffb340b9380 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 0x7ffb341488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 0x7ffb341488b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58972/0 (socket says 172.21.15.99:58972) 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 -- 172.21.15.99:0/4110177888 learned_addr learned my addr 172.21.15.99:0/4110177888 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb327fc640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340b9380 0x7ffb340c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb32ffd640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340b70d0 0x7ffb3415e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 -- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340b9380 msgr2=0x7ffb340c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340b9380 0x7ffb340c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 -- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340b70d0 msgr2=0x7ffb3415e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340b70d0 0x7ffb3415e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 -- 172.21.15.99:0/4110177888 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3415ea60 con 0x7ffb3415de00 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb32ffd640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340b70d0 0x7ffb3415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb327fc640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340b9380 0x7ffb340c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb337fe640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 0x7ffb341488b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00c7b0 tx=0x7ffb2c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c017070 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c003da0 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3415ed40 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.128+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c0052b0 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.129+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3415f300 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.129+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb34074f60 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.130+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffb2c01a070 con 0x7ffb3415de00 2024-03-04T10:06:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.130+0000 7ffb1b7fe640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.131+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb2c0899d0 con 0x7ffb3415de00 2024-03-04T10:06:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.131+0000 7ffb32ffd640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.131+0000 7ffb32ffd640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400a870 tx=0x7ffb24008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.135+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffb2c0586c0 con 0x7ffb3415de00 2024-03-04T10:06:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.315+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffb34152430 con 0x7ffb3415de00 2024-03-04T10:06:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.315+0000 7ffb1b7fe640 1 -- 172.21.15.99:0/4110177888 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v66) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffb2c057fc0 con 0x7ffb3415de00 2024-03-04T10:06:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:06:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.317+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 msgr2=0x7ffb2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.317+0000 7ffb39681640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400a870 tx=0x7ffb24008040 comp rx=0 tx=0).stop 2024-03-04T10:06:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.317+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 msgr2=0x7ffb341488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.317+0000 7ffb39681640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 0x7ffb341488b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00c7b0 tx=0x7ffb2c00cc70 comp rx=0 tx=0).stop 2024-03-04T10:06:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 shutdown_connections 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb3415de00 0x7ffb341488b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340b9380 0x7ffb340c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 --2- 172.21.15.99:0/4110177888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340b70d0 0x7ffb3415e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 >> 172.21.15.99:0/4110177888 conn(0x7ffb340d0470 msgr2=0x7ffb340b87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 shutdown_connections 2024-03-04T10:06:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.318+0000 7ffb39681640 1 -- 172.21.15.99:0/4110177888 wait complete. 2024-03-04T10:06:46.330 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:06:46.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:46.331 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:06:46.360 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:06:46.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:06:46.361 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:06:46.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:06:46.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.553+0000 7f4874f0a640 1 Processor -- start 2024-03-04T10:06:46.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.553+0000 7f4874f0a640 1 -- start start 2024-03-04T10:06:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f4874f0a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f487015b030 0x7f487015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f4874f0a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f487015b410 0x7f487015b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f4874f0a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f487006d8b0 con 0x7f487006d3b0 2024-03-04T10:06:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f4874f0a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48700b2720 con 0x7f487015b410 2024-03-04T10:06:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f4874f0a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f487006db90 con 0x7f487015b030 2024-03-04T10:06:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f486ed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f487015b410 0x7f487015b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f486dd9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f487006d3b0 0x7f487015dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58744/0 (socket says 172.21.15.99:58744) 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f486ed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f487015b410 0x7f487015b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58986/0 (socket says 172.21.15.99:58986) 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.554+0000 7f486dd9b640 1 -- 172.21.15.99:0/506161898 learned_addr learned my addr 172.21.15.99:0/506161898 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486e59c640 1 --2- 172.21.15.99:0/506161898 >> v2:172.21.15.162:3300/0 conn(0x7f487015b030 0x7f487015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486ed9d640 1 -- 172.21.15.99:0/506161898 >> v2:172.21.15.162:3300/0 conn(0x7f487015b030 msgr2=0x7f487015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486ed9d640 1 --2- 172.21.15.99:0/506161898 >> v2:172.21.15.162:3300/0 conn(0x7f487015b030 0x7f487015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486ed9d640 1 -- 172.21.15.99:0/506161898 >> v1:172.21.15.99:6789/0 conn(0x7f487006d3b0 legacy=0x7f487015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486d59a640 1 -- 172.21.15.99:0/506161898 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208815287 0 0) 0x7f487006d8b0 con 0x7f487006d3b0 2024-03-04T10:06:46.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486ed9d640 1 -- 172.21.15.99:0/506161898 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487006de70 con 0x7f487015b410 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.555+0000 7f486ed9d640 1 --2- 172.21.15.99:0/506161898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b410 0x7f487015b880 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f48700b28a0 tx=0x7f486802e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af59cc57a39c57d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f486d59a640 1 -- 172.21.15.99:0/506161898 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486803b070 con 0x7f487015b410 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f486d59a640 1 -- 172.21.15.99:0/506161898 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486802edc0 con 0x7f487015b410 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f486d59a640 1 -- 172.21.15.99:0/506161898 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4868036660 con 0x7f487015b410 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 -- 172.21.15.99:0/506161898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b410 msgr2=0x7f487015b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 --2- 172.21.15.99:0/506161898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b410 0x7f487015b880 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f48700b28a0 tx=0x7f486802e010 comp rx=0 tx=0).stop 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 -- 172.21.15.99:0/506161898 shutdown_connections 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 --2- 172.21.15.99:0/506161898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b410 0x7f487015b880 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 --2- 172.21.15.99:0/506161898 >> v2:172.21.15.162:3300/0 conn(0x7f487015b030 0x7f487015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 -- 172.21.15.99:0/506161898 >> 172.21.15.99:0/506161898 conn(0x7f487006acd0 msgr2=0x7f487006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 -- 172.21.15.99:0/506161898 shutdown_connections 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.556+0000 7f4874f0a640 1 -- 172.21.15.99:0/506161898 wait complete. 2024-03-04T10:06:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 Processor -- start 2024-03-04T10:06:46.561 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 -- start start 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487006d3b0 0x7f4870156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 0x7f48700adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487015b410 0x7f4870156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487006e690 con 0x7f487015b030 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487015bdc0 con 0x7f487015b410 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f4874f0a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487006e810 con 0x7f487006d3b0 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f486dd9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 0x7f48700adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f486dd9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 0x7f48700adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58988/0 (socket says 172.21.15.99:58988) 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.558+0000 7f486dd9b640 1 -- 172.21.15.99:0/1617243751 learned_addr learned my addr 172.21.15.99:0/1617243751 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486e59c640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487006d3b0 0x7f4870156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486ed9d640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487015b410 0x7f4870156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486dd9b640 1 -- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487006d3b0 msgr2=0x7f4870156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486dd9b640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487006d3b0 0x7f4870156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486dd9b640 1 -- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487015b410 msgr2=0x7f4870156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486dd9b640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487015b410 0x7f4870156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486dd9b640 1 -- 172.21.15.99:0/1617243751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700ae2f0 con 0x7f487015b030 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486e59c640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487006d3b0 0x7f4870156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486ed9d640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487015b410 0x7f4870156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f486dd9b640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 0x7f48700adc30 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f486400c4b0 tx=0x7f486400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864018070 con 0x7f487015b030 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4864007500 con 0x7f487015b030 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48700ae5d0 con 0x7f487015b030 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864004a20 con 0x7f487015b030 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.559+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48700aeb90 con 0x7f487015b030 2024-03-04T10:06:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.560+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4870074f60 con 0x7f487015b030 2024-03-04T10:06:46.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.560+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f486401b070 con 0x7f487015b030 2024-03-04T10:06:46.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.561+0000 7f485effd640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485006b4d0 0x7f485006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.561+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4864089660 con 0x7f487015b030 2024-03-04T10:06:46.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.561+0000 7f486e59c640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485006b4d0 0x7f485006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.561+0000 7f486e59c640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485006b4d0 0x7f485006d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4860009b20 tx=0x7f48600039d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.565+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f48640584a0 con 0x7f487015b030 2024-03-04T10:06:46.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.774+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f487015bac0 con 0x7f487015b030 2024-03-04T10:06:46.777 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:06:46.775+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:06:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.776+0000 7f485effd640 1 -- 172.21.15.99:0/1617243751 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v44) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4864087e20 con 0x7f487015b030 2024-03-04T10:06:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.778+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485006b4d0 msgr2=0x7f485006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.778+0000 7f4874f0a640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485006b4d0 0x7f485006d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4860009b20 tx=0x7f48600039d0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.778+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 msgr2=0x7f48700adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.778+0000 7f4874f0a640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 0x7f48700adc30 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f486400c4b0 tx=0x7f486400c970 comp rx=0 tx=0).stop 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.778+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 shutdown_connections 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.778+0000 7f4874f0a640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485006b4d0 0x7f485006d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.779+0000 7f4874f0a640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487015b410 0x7f4870156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.779+0000 7f4874f0a640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487015b030 0x7f48700adc30 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.779+0000 7f4874f0a640 1 --2- 172.21.15.99:0/1617243751 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487006d3b0 0x7f4870156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.779+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 >> 172.21.15.99:0/1617243751 conn(0x7f487006acd0 msgr2=0x7f48701596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.779+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 shutdown_connections 2024-03-04T10:06:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.779+0000 7f4874f0a640 1 -- 172.21.15.99:0/1617243751 wait complete. 2024-03-04T10:06:46.783 INFO:tasks.ceph.mds.c.smithi099.stderr: -4995> 2024-03-04T10:06:38.003+0000 7f1ebe01b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:46.792 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_journal_corrupt_dentry_first 1.0 2024-03-04T10:06:46.812 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:06:46.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.893+0000 7f3ff8169640 1 Processor -- start 2024-03-04T10:06:46.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.893+0000 7f3ff8169640 1 -- start start 2024-03-04T10:06:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff8169640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3ff00cf660 0x7f3ff00cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff8169640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff0162450 con 0x7f3ff015da20 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff8169640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff00cc2d0 con 0x7f3ff01616e0 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff8169640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ff00d0000 con 0x7f3ff00cf660 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff5704640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3ff00cf660 0x7f3ff00cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff5f05640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3ff015da20 0x7f3ff00ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58758/0 (socket says 172.21.15.99:58758) 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff5f05640 1 -- 172.21.15.99:0/3073475641 learned_addr learned my addr 172.21.15.99:0/3073475641 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff5704640 1 -- 172.21.15.99:0/3073475641 >> v1:172.21.15.162:6789/0 conn(0x7f3ff01616e0 legacy=0x7f3ff0161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff5704640 1 -- 172.21.15.99:0/3073475641 >> v1:172.21.15.99:6789/0 conn(0x7f3ff015da20 legacy=0x7f3ff00ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.894+0000 7f3ff5704640 1 -- 172.21.15.99:0/3073475641 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff00cc5b0 con 0x7f3ff00cf660 2024-03-04T10:06:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.895+0000 7f3ff5704640 1 --2- 172.21.15.99:0/3073475641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff00cf660 0x7f3ff00cfac0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00097a0 tx=0x7f3fe002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3564bcc97bba68d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.895+0000 7f3ff4f03640 1 -- 172.21.15.99:0/3073475641 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe00315a0 con 0x7f3ff00cf660 2024-03-04T10:06:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.895+0000 7f3ff4f03640 1 -- 172.21.15.99:0/3073475641 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe0037070 con 0x7f3ff00cf660 2024-03-04T10:06:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.895+0000 7f3ff4f03640 1 -- 172.21.15.99:0/3073475641 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0031960 con 0x7f3ff00cf660 2024-03-04T10:06:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 -- 172.21.15.99:0/3073475641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff00cf660 msgr2=0x7f3ff00cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3073475641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff00cf660 0x7f3ff00cfac0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00097a0 tx=0x7f3fe002b840 comp rx=0 tx=0).stop 2024-03-04T10:06:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 -- 172.21.15.99:0/3073475641 shutdown_connections 2024-03-04T10:06:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3073475641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff00cf660 0x7f3ff00cfac0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 -- 172.21.15.99:0/3073475641 >> 172.21.15.99:0/3073475641 conn(0x7f3ff00b33a0 msgr2=0x7f3ff00b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 -- 172.21.15.99:0/3073475641 shutdown_connections 2024-03-04T10:06:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.896+0000 7f3ff8169640 1 -- 172.21.15.99:0/3073475641 wait complete. 2024-03-04T10:06:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.897+0000 7f3ff8169640 1 Processor -- start 2024-03-04T10:06:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.897+0000 7f3ff8169640 1 -- start start 2024-03-04T10:06:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff00cf660 0x7f3ff01f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 0x7f3ff01f7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff01616e0 0x7f3ff00c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff015de00 con 0x7f3ff015da20 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff00cc920 con 0x7f3ff01616e0 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff006bde0 con 0x7f3ff00cf660 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 0x7f3ff01f7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 0x7f3ff01f7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:59008/0 (socket says 172.21.15.99:59008) 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 -- 172.21.15.99:0/3574986331 learned_addr learned my addr 172.21.15.99:0/3574986331 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff6706640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff01616e0 0x7f3ff00c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5f05640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff00cf660 0x7f3ff01f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 -- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff00cf660 msgr2=0x7f3ff01f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff00cf660 0x7f3ff01f7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 -- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff01616e0 msgr2=0x7f3ff00c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff01616e0 0x7f3ff00c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 -- 172.21.15.99:0/3574986331 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff00cb3a0 con 0x7f3ff015da20 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff6706640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff01616e0 0x7f3ff00c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5f05640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff00cf660 0x7f3ff01f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff5704640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 0x7f3ff01f7950 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00098c0 tx=0x7f3fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0032840 con 0x7f3ff015da20 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe00329e0 con 0x7f3ff015da20 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff00ca820 con 0x7f3ff015da20 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0032dd0 con 0x7f3ff015da20 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.898+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff00cade0 con 0x7f3ff015da20 2024-03-04T10:06:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.899+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff00cfdf0 con 0x7f3ff015da20 2024-03-04T10:06:46.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.900+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3fe004d3f0 con 0x7f3ff015da20 2024-03-04T10:06:46.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.900+0000 7f3fe67fc640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:46.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.900+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3fe002f030 con 0x7f3ff015da20 2024-03-04T10:06:46.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.900+0000 7f3ff5f05640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:46.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.901+0000 7f3ff5f05640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3fec000f60 tx=0x7f3fec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:46.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:46.905+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ff00cfdf0 con 0x7f3ff015da20 2024-03-04T10:06:47.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.076+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}] v 0) v1 -- 0x7f3ff0074f60 con 0x7f3ff015da20 2024-03-04T10:06:47.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.081+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3574986331 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_journal_corrupt_dentry_first}]=0 v19)=0 v19) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3fe0040070 con 0x7f3ff015da20 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 msgr2=0x7f3fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3fec000f60 tx=0x7f3fec009040 comp rx=0 tx=0).stop 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 msgr2=0x7f3ff01f7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 0x7f3ff01f7950 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00098c0 tx=0x7f3fe00046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 shutdown_connections 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff01616e0 0x7f3ff00c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:47.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff015da20 0x7f3ff01f7950 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:47.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 --2- 172.21.15.99:0/3574986331 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff00cf660 0x7f3ff01f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:47.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 >> 172.21.15.99:0/3574986331 conn(0x7f3ff00b33a0 msgr2=0x7f3ff00b63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:47.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 shutdown_connections 2024-03-04T10:06:47.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:47.083+0000 7f3ff8169640 1 -- 172.21.15.99:0/3574986331 wait complete. 2024-03-04T10:06:52.097 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:06:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 Processor -- start 2024-03-04T10:06:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 -- start start 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb64406d3c0 0x7fb6441508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d810 0x7fb644150df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64406dd10 con 0x7fb644071320 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64406dff0 con 0x7fb64406d3c0 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.209+0000 7fb64a524640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64406e2d0 con 0x7fb64406d810 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb643fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb644071320 0x7fb644071700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58770/0 (socket says 172.21.15.99:58770) 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb6437fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb64406d3c0 0x7fb6441508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb643fff640 1 -- 172.21.15.99:0/1398036090 learned_addr learned my addr 172.21.15.99:0/1398036090 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb648ac1640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d810 0x7fb644150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb6437fe640 1 -- 172.21.15.99:0/1398036090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d810 msgr2=0x7fb644150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb6437fe640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d810 0x7fb644150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb6437fe640 1 -- 172.21.15.99:0/1398036090 >> v1:172.21.15.99:6789/0 conn(0x7fb644071320 legacy=0x7fb644071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb6437fe640 1 -- 172.21.15.99:0/1398036090 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440b2820 con 0x7fb64406d3c0 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb642ffd640 1 -- 172.21.15.99:0/1398036090 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284122311 0 0) 0x7fb64406dd10 con 0x7fb644071320 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.210+0000 7fb648ac1640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d810 0x7fb644150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb6437fe640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d3c0 0x7fb6441508b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb6380097a0 tx=0x7fb63802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1263b29257e4bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb642ffd640 1 -- 172.21.15.99:0/1398036090 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638035020 con 0x7fb64406d3c0 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb642ffd640 1 -- 172.21.15.99:0/1398036090 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb638031810 con 0x7fb64406d3c0 2024-03-04T10:06:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb642ffd640 1 -- 172.21.15.99:0/1398036090 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638031be0 con 0x7fb64406d3c0 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 -- 172.21.15.99:0/1398036090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d3c0 msgr2=0x7fb6441508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d3c0 0x7fb6441508b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb6380097a0 tx=0x7fb63802b8b0 comp rx=0 tx=0).stop 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 -- 172.21.15.99:0/1398036090 shutdown_connections 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d810 0x7fb644150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 --2- 172.21.15.99:0/1398036090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d3c0 0x7fb6441508b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 -- 172.21.15.99:0/1398036090 >> 172.21.15.99:0/1398036090 conn(0x7fb64406acd0 msgr2=0x7fb64406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 -- 172.21.15.99:0/1398036090 shutdown_connections 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.211+0000 7fb64a524640 1 -- 172.21.15.99:0/1398036090 wait complete. 2024-03-04T10:06:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 Processor -- start 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 -- start start 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d3c0 0x7fb6440cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 0x7fb644155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb644071320 0x7fb64414fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440b3fd0 con 0x7fb64406d810 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440b3e50 con 0x7fb644071320 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.213+0000 7fb64a524640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cf660 con 0x7fb64406d3c0 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 0x7fb644155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 0x7fb644155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:59020/0 (socket says 172.21.15.99:59020) 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 -- 172.21.15.99:0/1532280759 learned_addr learned my addr 172.21.15.99:0/1532280759 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb643fff640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d3c0 0x7fb6440cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb648ac1640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb644071320 0x7fb64414fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 -- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d3c0 msgr2=0x7fb6440cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d3c0 0x7fb6440cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 -- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb644071320 msgr2=0x7fb64414fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb644071320 0x7fb64414fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 -- 172.21.15.99:0/1532280759 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb644155fc0 con 0x7fb64406d810 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb6437fe640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 0x7fb644155950 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb638000c00 tx=0x7fb6380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638040070 con 0x7fb64406d810 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb638039e70 con 0x7fb64406d810 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638035040 con 0x7fb64406d810 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6441562a0 con 0x7fb64406d810 2024-03-04T10:06:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.214+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb644156860 con 0x7fb64406d810 2024-03-04T10:06:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.215+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb644074f60 con 0x7fb64406d810 2024-03-04T10:06:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.216+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb63804b060 con 0x7fb64406d810 2024-03-04T10:06:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.216+0000 7fb640ff9640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb63006b500 0x7fb63006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.216+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb6380abe20 con 0x7fb64406d810 2024-03-04T10:06:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.216+0000 7fb643fff640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb63006b500 0x7fb63006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.217+0000 7fb643fff640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb63006b500 0x7fb63006d9c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb634000f60 tx=0x7fb634007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.220+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb63807a500 con 0x7fb64406d810 2024-03-04T10:06:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:52.429+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fb6441509c0 con 0x7fb64406d810 2024-03-04T10:06:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.103+0000 7fb640ff9640 1 -- 172.21.15.99:0/1532280759 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v46) v1 ==== 162+0+0 (secure 0 0 0) 0x7fb63807e5a0 con 0x7fb64406d810 2024-03-04T10:06:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:06:53.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb63006b500 msgr2=0x7fb63006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb63006b500 0x7fb63006d9c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb634000f60 tx=0x7fb634007d50 comp rx=0 tx=0).stop 2024-03-04T10:06:53.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 msgr2=0x7fb644155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 0x7fb644155950 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb638000c00 tx=0x7fb6380046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 shutdown_connections 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb63006b500 0x7fb63006d9c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb644071320 0x7fb64414fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb64406d810 0x7fb644155950 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 --2- 172.21.15.99:0/1532280759 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb64406d3c0 0x7fb6440cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 >> 172.21.15.99:0/1532280759 conn(0x7fb64406acd0 msgr2=0x7fb64414d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 shutdown_connections 2024-03-04T10:06:53.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.105+0000 7fb64a524640 1 -- 172.21.15.99:0/1532280759 wait complete. 2024-03-04T10:06:53.120 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:53.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.220+0000 7f54e97b3640 1 Processor -- start 2024-03-04T10:06:53.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.220+0000 7f54e97b3640 1 -- start start 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e97b3640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f54e40cc8e0 0x7f54e40cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e97b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e40cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e97b3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e40d0400 con 0x7f54e40cfae0 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e97b3640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e406ae50 con 0x7f54e40cc8e0 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e97b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e406b130 con 0x7f54e40d1290 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e27fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f54e40cc8e0 0x7f54e40cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e2ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f54e40cfae0 0x7f54e40cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58774/0 (socket says 172.21.15.99:58774) 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.221+0000 7f54e2ffd640 1 -- 172.21.15.99:0/67922584 learned_addr learned my addr 172.21.15.99:0/67922584 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e37fe640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e40cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e27fc640 1 -- 172.21.15.99:0/67922584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 msgr2=0x7f54e40cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e27fc640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e40cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e27fc640 1 -- 172.21.15.99:0/67922584 >> v1:172.21.15.99:6789/0 conn(0x7f54e40cfae0 legacy=0x7f54e40cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e1ffb640 1 -- 172.21.15.99:0/67922584 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425309190 0 0) 0x7f54e40d0400 con 0x7f54e40cfae0 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e27fc640 1 -- 172.21.15.99:0/67922584 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e406b410 con 0x7f54e40cc8e0 2024-03-04T10:06:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e37fe640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e40cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.222+0000 7f54e27fc640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cc8e0 0x7f54e40cfec0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f54d80097a0 tx=0x7f54d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ee2c342b7ac099 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e1ffb640 1 -- 172.21.15.99:0/67922584 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8035020 con 0x7f54e40cc8e0 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e1ffb640 1 -- 172.21.15.99:0/67922584 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d8031830 con 0x7f54e40cc8e0 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e1ffb640 1 -- 172.21.15.99:0/67922584 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8031c00 con 0x7f54e40cc8e0 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 -- 172.21.15.99:0/67922584 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cc8e0 msgr2=0x7f54e40cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cc8e0 0x7f54e40cfec0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f54d80097a0 tx=0x7f54d802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 -- 172.21.15.99:0/67922584 shutdown_connections 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e40cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 --2- 172.21.15.99:0/67922584 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cc8e0 0x7f54e40cfec0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 -- 172.21.15.99:0/67922584 >> 172.21.15.99:0/67922584 conn(0x7f54e40d0770 msgr2=0x7f54e40d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 -- 172.21.15.99:0/67922584 shutdown_connections 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.223+0000 7f54e97b3640 1 -- 172.21.15.99:0/67922584 wait complete. 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.224+0000 7f54e97b3640 1 Processor -- start 2024-03-04T10:06:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.224+0000 7f54e97b3640 1 -- start start 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e97b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54e40cc8e0 0x7f54e41585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e97b3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 0x7f54e4157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e97b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e4155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e97b3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e406b9d0 con 0x7f54e40cfae0 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e97b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40ce440 con 0x7f54e40cc8e0 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e97b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e40cd980 con 0x7f54e40d1290 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e27fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 0x7f54e4157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e27fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 0x7f54e4157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:59034/0 (socket says 172.21.15.99:59034) 2024-03-04T10:06:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e27fc640 1 -- 172.21.15.99:0/932023537 learned_addr learned my addr 172.21.15.99:0/932023537 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e37fe640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e4155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.225+0000 7f54e2ffd640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54e40cc8e0 0x7f54e41585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e27fc640 1 -- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 msgr2=0x7f54e4155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e27fc640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e4155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e27fc640 1 -- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54e40cc8e0 msgr2=0x7f54e41585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e27fc640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54e40cc8e0 0x7f54e41585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e27fc640 1 -- 172.21.15.99:0/932023537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e41578e0 con 0x7f54e40cfae0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e27fc640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 0x7f54e4157140 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009890 tx=0x7f54d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8041070 con 0x7f54e40cfae0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d803aa30 con 0x7f54e40cfae0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e40b3410 con 0x7f54e40cfae0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8035040 con 0x7f54e40cfae0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.226+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e40b39d0 con 0x7f54e40cfae0 2024-03-04T10:06:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.227+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e406cd70 con 0x7f54e40cfae0 2024-03-04T10:06:53.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.227+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f54d804c060 con 0x7f54e40cfae0 2024-03-04T10:06:53.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.228+0000 7f54cb7fe640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f54d006b500 0x7f54d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:53.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.228+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f54d80abe20 con 0x7f54e40cfae0 2024-03-04T10:06:53.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.228+0000 7f54e2ffd640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f54d006b500 0x7f54d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:53.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.228+0000 7f54e2ffd640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f54d006b500 0x7f54d006d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f54d40012e0 tx=0x7f54d4007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:53.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.232+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f54d807aac0 con 0x7f54e40cfae0 2024-03-04T10:06:53.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.435+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54e4074f60 con 0x7f54e40cfae0 2024-03-04T10:06:53.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.436+0000 7f54cb7fe640 1 -- 172.21.15.99:0/932023537 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f54d807a3c0 con 0x7f54e40cfae0 2024-03-04T10:06:53.437 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:53.437 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":47,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:53.105231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:53.438 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 47 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f54d006b500 msgr2=0x7f54d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f54d006b500 0x7f54d006d9c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f54d40012e0 tx=0x7f54d4007ef0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 msgr2=0x7f54e4157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 0x7f54e4157140 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009890 tx=0x7f54d80046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 shutdown_connections 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f54d006b500 0x7f54d006d9c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f54e40d1290 0x7f54e4155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f54e40cfae0 0x7f54e4157140 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 --2- 172.21.15.99:0/932023537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f54e40cc8e0 0x7f54e41585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:53.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 >> 172.21.15.99:0/932023537 conn(0x7f54e40d0770 msgr2=0x7f54e40cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:53.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 shutdown_connections 2024-03-04T10:06:53.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:53.438+0000 7f54e97b3640 1 -- 172.21.15.99:0/932023537 wait complete. 2024-03-04T10:06:53.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:06:53.105231+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 67, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6729': {'gid': 6729, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.99:6835/3022309305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 3022309305}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 3022309305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:06:53.853 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6729:up:replay 2024-03-04T10:06:54.854 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:54.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.965+0000 7f4d2e560640 1 Processor -- start 2024-03-04T10:06:54.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.965+0000 7f4d2e560640 1 -- start start 2024-03-04T10:06:54.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2e560640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4d28071e80 0x7f4d280cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:54.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2e560640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d2806d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:54.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2e560640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d280ad560 con 0x7f4d2815db80 2024-03-04T10:06:54.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2e560640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d280cfd80 con 0x7f4d28071e80 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2e560640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d280ad840 con 0x7f4d280cf930 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4d28071e80 0x7f4d280cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4d28071e80 0x7f4d280cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51932/0 (socket says 172.21.15.99:51932) 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 -- 172.21.15.99:0/3863034701 learned_addr learned my addr 172.21.15.99:0/3863034701 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 -- 172.21.15.99:0/3863034701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 msgr2=0x7f4d2806d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2cafd640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d2806d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d2806d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 -- 172.21.15.99:0/3863034701 >> v1:172.21.15.162:6789/0 conn(0x7f4d2815db80 legacy=0x7f4d280acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d277fe640 1 -- 172.21.15.99:0/3863034701 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280adb20 con 0x7f4d28071e80 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.966+0000 7f4d2cafd640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d2806d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.967+0000 7f4d277fe640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d28071e80 0x7f4d280cf3f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4d2806d250 tx=0x7f4d1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6578738c77f3fc71 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.967+0000 7f4d26ffd640 1 -- 172.21.15.99:0/3863034701 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c035020 con 0x7f4d28071e80 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.967+0000 7f4d26ffd640 1 -- 172.21.15.99:0/3863034701 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c031830 con 0x7f4d28071e80 2024-03-04T10:06:54.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.967+0000 7f4d26ffd640 1 -- 172.21.15.99:0/3863034701 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c031c00 con 0x7f4d28071e80 2024-03-04T10:06:54.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 -- 172.21.15.99:0/3863034701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d28071e80 msgr2=0x7f4d280cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d28071e80 0x7f4d280cf3f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f4d2806d250 tx=0x7f4d1c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 -- 172.21.15.99:0/3863034701 shutdown_connections 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d2806d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 --2- 172.21.15.99:0/3863034701 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d28071e80 0x7f4d280cf3f0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 -- 172.21.15.99:0/3863034701 >> 172.21.15.99:0/3863034701 conn(0x7f4d2806acd0 msgr2=0x7f4d2806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 -- 172.21.15.99:0/3863034701 shutdown_connections 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.968+0000 7f4d2e560640 1 -- 172.21.15.99:0/3863034701 wait complete. 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.969+0000 7f4d2e560640 1 Processor -- start 2024-03-04T10:06:54.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.969+0000 7f4d2e560640 1 -- start start 2024-03-04T10:06:54.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2e560640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d28071e80 0x7f4d280cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:54.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2e560640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d28151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:54.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2e560640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 0x7f4d2814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2e560640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d280722e0 con 0x7f4d2815db80 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2e560640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d280ae0e0 con 0x7f4d28071e80 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2e560640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2815df60 con 0x7f4d280cf930 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 0x7f4d2814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 0x7f4d2814d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51944/0 (socket says 172.21.15.99:51944) 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 -- 172.21.15.99:0/613983383 learned_addr learned my addr 172.21.15.99:0/613983383 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d277fe640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d28151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 -- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 msgr2=0x7f4d28151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d27fff640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d28071e80 0x7f4d280cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d28151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 -- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d28071e80 msgr2=0x7f4d280cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d28071e80 0x7f4d280cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:54.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.970+0000 7f4d2cafd640 1 -- 172.21.15.99:0/613983383 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d28158080 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.971+0000 7f4d2cafd640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 0x7f4d2814d0c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000cf20 tx=0x7f4d2000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.971+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2001a070 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.971+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2000d840 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.971+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d28158360 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.971+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2000dc30 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.971+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d28158920 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.972+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d28074f60 con 0x7f4d2815db80 2024-03-04T10:06:54.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.972+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4d2001d070 con 0x7f4d2815db80 2024-03-04T10:06:54.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.973+0000 7f4d24ff9640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4d1406b4d0 0x7f4d1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:54.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.973+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4d2008ac10 con 0x7f4d2815db80 2024-03-04T10:06:54.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.973+0000 7f4d27fff640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4d1406b4d0 0x7f4d1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:54.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.974+0000 7f4d27fff640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4d1406b4d0 0x7f4d1406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800a750 tx=0x7f4d18008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:54.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:54.977+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4d20059940 con 0x7f4d2815db80 2024-03-04T10:06:55.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.183+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d281f6d50 con 0x7f4d2815db80 2024-03-04T10:06:55.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.183+0000 7f4d24ff9640 1 -- 172.21.15.99:0/613983383 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f4d20059240 con 0x7f4d2815db80 2024-03-04T10:06:55.185 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:55.185 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":49,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:54.110540+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:55.185 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 49 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4d1406b4d0 msgr2=0x7f4d1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4d1406b4d0 0x7f4d1406d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4d1800a750 tx=0x7f4d18008040 comp rx=0 tx=0).stop 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 msgr2=0x7f4d2814d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 0x7f4d2814d0c0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000cf20 tx=0x7f4d2000a7d0 comp rx=0 tx=0).stop 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 shutdown_connections 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4d1406b4d0 0x7f4d1406d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4d2815db80 0x7f4d2814d0c0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4d280cf930 0x7f4d28151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 --2- 172.21.15.99:0/613983383 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4d28071e80 0x7f4d280cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 >> 172.21.15.99:0/613983383 conn(0x7f4d2806acd0 msgr2=0x7f4d280b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 shutdown_connections 2024-03-04T10:06:55.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:55.185+0000 7f4d2e560640 1 -- 172.21.15.99:0/613983383 wait complete. 2024-03-04T10:06:55.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:06:54.110540+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 67, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6729': {'gid': 6729, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.99:6835/3022309305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 3022309305}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 3022309305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:06:55.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6729:up:rejoin 2024-03-04T10:06:56.601 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.712+0000 7f64b5081640 1 Processor -- start 2024-03-04T10:06:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.712+0000 7f64b5081640 1 -- start start 2024-03-04T10:06:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.712+0000 7f64b5081640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64b015dd70 0x7f64b00d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64b5081640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00d2630 con 0x7f64b00cef60 2024-03-04T10:06:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64b5081640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00d2910 con 0x7f64b00b9710 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64b5081640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b015e150 con 0x7f64b015dd70 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64aed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64b015dd70 0x7f64b00d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64af59e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f64b00cef60 0x7f64b00b6bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48832/0 (socket says 172.21.15.99:48832) 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64af59e640 1 -- 172.21.15.99:0/2298063983 learned_addr learned my addr 172.21.15.99:0/2298063983 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64aed9d640 1 -- 172.21.15.99:0/2298063983 >> v1:172.21.15.162:6789/0 conn(0x7f64b00b9710 legacy=0x7f64b00ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64aed9d640 1 -- 172.21.15.99:0/2298063983 >> v1:172.21.15.99:6789/0 conn(0x7f64b00cef60 legacy=0x7f64b00b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64aed9d640 1 -- 172.21.15.99:0/2298063983 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b00d2bf0 con 0x7f64b015dd70 2024-03-04T10:06:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.713+0000 7f64ae59c640 1 -- 172.21.15.99:0/2298063983 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126228608 0 0) 0x7f64b00d2630 con 0x7f64b00cef60 2024-03-04T10:06:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.714+0000 7f64aed9d640 1 --2- 172.21.15.99:0/2298063983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00d1de0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f64a00097a0 tx=0x7f64a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e8d3f461c734a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.714+0000 7f64ae59c640 1 -- 172.21.15.99:0/2298063983 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0034070 con 0x7f64b015dd70 2024-03-04T10:06:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.714+0000 7f64ae59c640 1 -- 172.21.15.99:0/2298063983 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0030610 con 0x7f64b015dd70 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.714+0000 7f64ae59c640 1 -- 172.21.15.99:0/2298063983 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00309e0 con 0x7f64b015dd70 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.714+0000 7f64b5081640 1 -- 172.21.15.99:0/2298063983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 msgr2=0x7f64b00d1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.714+0000 7f64b5081640 1 --2- 172.21.15.99:0/2298063983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00d1de0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f64a00097a0 tx=0x7f64a002b960 comp rx=0 tx=0).stop 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.715+0000 7f64b5081640 1 -- 172.21.15.99:0/2298063983 shutdown_connections 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.715+0000 7f64b5081640 1 --2- 172.21.15.99:0/2298063983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00d1de0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.715+0000 7f64b5081640 1 -- 172.21.15.99:0/2298063983 >> 172.21.15.99:0/2298063983 conn(0x7f64b0072a60 msgr2=0x7f64b00d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.715+0000 7f64b5081640 1 -- 172.21.15.99:0/2298063983 shutdown_connections 2024-03-04T10:06:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.715+0000 7f64b5081640 1 -- 172.21.15.99:0/2298063983 wait complete. 2024-03-04T10:06:56.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.716+0000 7f64b5081640 1 Processor -- start 2024-03-04T10:06:56.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.716+0000 7f64b5081640 1 -- start start 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b00b9710 0x7f64b0160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b00cef60 0x7f64b014af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00d2320 con 0x7f64b015dd70 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00d31b0 con 0x7f64b00b9710 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00d36d0 con 0x7f64b00cef60 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00b54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51964/0 (socket says 172.21.15.99:51964) 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 -- 172.21.15.99:0/3055252453 learned_addr learned my addr 172.21.15.99:0/3055252453 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64aed9d640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b00b9710 0x7f64b0160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64a7fff640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b00cef60 0x7f64b014af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 -- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b00cef60 msgr2=0x7f64b014af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:56.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b00cef60 0x7f64b014af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 -- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b00b9710 msgr2=0x7f64b0160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b00b9710 0x7f64b0160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 -- 172.21.15.99:0/3055252453 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b015ec60 con 0x7f64b015dd70 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64af59e640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00b54f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f64a800c7b0 tx=0x7f64a800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8018070 con 0x7f64b015dd70 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a800a640 con 0x7f64b015dd70 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b015ef40 con 0x7f64b015dd70 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.717+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a800aa70 con 0x7f64b015dd70 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.718+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b015f500 con 0x7f64b015dd70 2024-03-04T10:06:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.719+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0074f60 con 0x7f64b015dd70 2024-03-04T10:06:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.719+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f64a801b070 con 0x7f64b015dd70 2024-03-04T10:06:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.719+0000 7f64a77fe640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.719+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f64a80898d0 con 0x7f64b015dd70 2024-03-04T10:06:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.720+0000 7f64aed9d640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.720+0000 7f64aed9d640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f64a0036000 tx=0x7f64a003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:56.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.723+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f64a8057fb0 con 0x7f64b015dd70 2024-03-04T10:06:56.930 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.929+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64b015c830 con 0x7f64b015dd70 2024-03-04T10:06:56.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.930+0000 7f64a77fe640 1 -- 172.21.15.99:0/3055252453 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f64a805c050 con 0x7f64b015dd70 2024-03-04T10:06:56.931 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:56.932 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:55.115395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:56.932 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 50 2024-03-04T10:06:56.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 msgr2=0x7f649006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:56.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f64a0036000 tx=0x7f64a003b040 comp rx=0 tx=0).stop 2024-03-04T10:06:56.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 msgr2=0x7f64b00b54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:56.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00b54f0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f64a800c7b0 tx=0x7f64a800cc70 comp rx=0 tx=0).stop 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 shutdown_connections 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f649006b4d0 0x7f649006d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64b015dd70 0x7f64b00b54f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64b00cef60 0x7f64b014af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 --2- 172.21.15.99:0/3055252453 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64b00b9710 0x7f64b0160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 >> 172.21.15.99:0/3055252453 conn(0x7f64b0072a60 msgr2=0x7f64b015be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 shutdown_connections 2024-03-04T10:06:56.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:56.932+0000 7f64b5081640 1 -- 172.21.15.99:0/3055252453 wait complete. 2024-03-04T10:06:57.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:06:55.115395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 67, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6729': {'gid': 6729, 'name': 'c', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.99:6835/3022309305', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 3022309305}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 3022309305}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:06:57.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:06:57.347 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:06:57.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.447+0000 7f63cf41f640 1 Processor -- start 2024-03-04T10:06:57.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.448+0000 7f63cf41f640 1 -- start start 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.448+0000 7f63cf41f640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 0x7f63c80b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.448+0000 7f63cf41f640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c815e1e0 con 0x7f63c815de00 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.448+0000 7f63cf41f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c80d1c10 con 0x7f63c80b5470 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.448+0000 7f63cf41f640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80b98c0 con 0x7f63c80b2e60 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.448+0000 7f63cd1bb640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f63c815de00 0x7f63c80b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48844/0 (socket says 172.21.15.99:48844) 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2393875932 learned_addr learned my addr 172.21.15.99:0/2393875932 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 0x7f63c80b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159452413 0 0) 0x7f63c815e1e0 con 0x7f63c815de00 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b0003520 con 0x7f63c815de00 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3217488024 0 0) 0x7f63b0003520 con 0x7f63c815de00 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 msgr2=0x7f63c80b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 0x7f63c80b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 >> v1:172.21.15.162:6789/0 conn(0x7f63c80b5470 legacy=0x7f63c80d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80d1ef0 con 0x7f63c815de00 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f63c0001670 con 0x7f63c815de00 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 >> v1:172.21.15.99:6789/0 conn(0x7f63c815de00 legacy=0x7f63c80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b00039e0 0x7f63b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b0004390 0x7f63b0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 0x7f63b00050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80b98c0 con 0x7f63b00039e0 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b000b9c0 con 0x7f63b0004390 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b000bbc0 con 0x7f63c80b2e60 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b00039e0 0x7f63b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b0004390 0x7f63b0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 0x7f63b00050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 msgr2=0x7f63b00050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c80b2e60 0x7f63b00050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b0004390 msgr2=0x7f63b0004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b0004390 0x7f63b0004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.449+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2393875932 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b0003520 con 0x7f63b00039e0 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.450+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b0004390 0x7f63b0004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.450+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b00039e0 0x7f63b0003dc0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c240 tx=0x7f63b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9832dc3a05ea46 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.450+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0007d10 con 0x7f63b00039e0 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.450+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0030070 con 0x7f63b00039e0 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.450+0000 7f63c7fff640 1 -- 172.21.15.99:0/2393875932 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c002cd60 con 0x7f63b00039e0 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cf41f640 1 -- 172.21.15.99:0/2393875932 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b00039e0 msgr2=0x7f63b0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cf41f640 1 --2- 172.21.15.99:0/2393875932 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b00039e0 0x7f63b0003dc0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c240 tx=0x7f63b0005830 comp rx=0 tx=0).stop 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cd9bc640 1 -- 172.21.15.99:0/2393875932 reap_dead start 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cf41f640 1 -- 172.21.15.99:0/2393875932 shutdown_connections 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cf41f640 1 -- 172.21.15.99:0/2393875932 >> 172.21.15.99:0/2393875932 conn(0x7f63c80d0470 msgr2=0x7f63c80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cf41f640 1 -- 172.21.15.99:0/2393875932 shutdown_connections 2024-03-04T10:06:57.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.451+0000 7f63cf41f640 1 -- 172.21.15.99:0/2393875932 wait complete. 2024-03-04T10:06:57.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.452+0000 7f63cf41f640 1 Processor -- start 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.452+0000 7f63cf41f640 1 -- start start 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00039e0 0x7f63c80cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 0x7f63c80b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c814d890 0x7f63c814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80d2820 con 0x7f63c81551f0 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80d24b0 con 0x7f63b00039e0 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c80b72a0 con 0x7f63c814d890 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 0x7f63c80b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 0x7f63c80b5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51982/0 (socket says 172.21.15.99:51982) 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 -- 172.21.15.99:0/3445762505 learned_addr learned my addr 172.21.15.99:0/3445762505 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 -- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c814d890 msgr2=0x7f63c814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c814d890 0x7f63c814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 -- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00039e0 msgr2=0x7f63c80cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00039e0 0x7f63c80cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 -- 172.21.15.99:0/3445762505 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c80cc9b0 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 0x7f63c80b5b30 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f63bc007fb0 tx=0x7f63bc010650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63c5ffb640 1 -- 172.21.15.99:0/3445762505 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc01a030 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63c5ffb640 1 -- 172.21.15.99:0/3445762505 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc010e80 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.453+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c81559b0 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.454+0000 7f63c5ffb640 1 -- 172.21.15.99:0/3445762505 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc003b90 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.454+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c8155f70 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.455+0000 7f63c5ffb640 1 -- 172.21.15.99:0/3445762505 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f63bc01d070 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.455+0000 7f63c5ffb640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f63b406b4d0 0x7f63b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.455+0000 7f63c5ffb640 1 -- 172.21.15.99:0/3445762505 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f63bc08b160 con 0x7f63c81551f0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.456+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f63b406b4d0 0x7f63b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.456+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f63b406b4d0 0x7f63b406d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c240 tx=0x7f63c0039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.456+0000 7f63d1ffd740 1 Processor -- start 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 -- start start 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca350de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca353dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca350e380 0x562ca353e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ca34787c0 con 0x562ca353aea0 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ca3421b50 con 0x562ca350e380 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ca3474860 con 0x562ca353b280 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63cd1bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca350de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63cd1bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca350de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51984/0 (socket says 172.21.15.99:51984) 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.457+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2974063613 learned_addr learned my addr 172.21.15.99:0/2974063613 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca350e380 0x562ca353e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 msgr2=0x562ca353dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca353dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca353dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca350e380 msgr2=0x562ca353e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca350e380 0x562ca353e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd1bb640 1 -- 172.21.15.99:0/2974063613 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ca34359a0 con 0x562ca353aea0 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca350e380 0x562ca353e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca350de40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f63c002c570 tx=0x7f63c002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6618dd32bc627911 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63aeffd640 1 -- 172.21.15.99:0/2974063613 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c002ce50 con 0x562ca353aea0 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63aeffd640 1 -- 172.21.15.99:0/2974063613 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0030040 con 0x562ca353aea0 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63aeffd640 1 -- 172.21.15.99:0/2974063613 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0007270 con 0x562ca353aea0 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2974063613 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 msgr2=0x562ca350de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.458+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca350de40 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f63c002c570 tx=0x7f63c002e000 comp rx=0 tx=0).stop 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2974063613 shutdown_connections 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca350e380 0x562ca353e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca353dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2974063613 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca350de40 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2974063613 >> 172.21.15.99:0/2974063613 conn(0x562ca35100b0 msgr2=0x562ca3558c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2974063613 shutdown_connections 2024-03-04T10:06:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2974063613 wait complete. 2024-03-04T10:06:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 -- start start 2024-03-04T10:06:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.459+0000 7f63d1ffd740 1 Processor -- start 2024-03-04T10:06:57.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca35697f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3534020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca3558a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ca3512750 con 0x562ca353aea0 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ca3434f90 con 0x562ca35419c0 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ca3514f90 con 0x562ca353b280 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63d1ffd740 10 client.0 _ll_register_callbacks cb 0x562ca3456d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca35697f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca35697f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51994/0 (socket says 172.21.15.99:51994) 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 -- 172.21.15.99:0/563872617 learned_addr learned my addr 172.21.15.99:0/563872617 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 -- 172.21.15.99:0/563872617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 msgr2=0x562ca3534020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3534020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 -- 172.21.15.99:0/563872617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 msgr2=0x562ca3558a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca3558a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.460+0000 7f63cd1bb640 1 -- 172.21.15.99:0/563872617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ca318d2d0 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca35697f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f63c0007f60 tx=0x7f63c80b2d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164db778375da20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63a7fff640 1 -- 172.21.15.99:0/563872617 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0007480 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63a7fff640 1 -- 172.21.15.99:0/563872617 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f63c0007620 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63c0030040 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63a7fff640 1 -- 172.21.15.99:0/563872617 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63c005f050 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 10 client.6783 fetch_fsmap learned FSMap version 50 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x562ca32576b0 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63a7fff640 1 -- 172.21.15.99:0/563872617 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c0030040 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63a7fff640 1 -- 172.21.15.99:0/563872617 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 50) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f63c0062020 con 0x562ca353aea0 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 10 client.6783 fetch_fsmap finished waiting for FSMap version 50 2024-03-04T10:06:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 10 client.6783 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 20 client.6783 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3532850 0x562ca3532c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 4 client.6783 mds_command: new command op to 6729 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 --> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ca3485d00 con 0x562ca3532850 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.461+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3532850 0x562ca3532c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63a7fff640 10 client.6783 ms_handle_connect on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3532850 0x562ca3532c30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9397aa2ec5efa186 server_cookie=ab4a8b5ca8df5b21 in_seq=0 out_seq=0 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63a7fff640 1 -- 172.21.15.99:0/563872617 <== mds.0 v2:172.21.15.99:6834/3022309305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562ca3485d00 con 0x562ca3532850 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63a7fff640 10 client.6783 handle_command_reply: tid=0 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 client.6783 shutdown 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 msgr2=0x562ca35697f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca35697f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f63c0007f60 tx=0x7f63c80b2d40 comp rx=0 tx=0).stop 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 shutdown_connections 2024-03-04T10:06:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3532850 0x562ca3532c30 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca3558a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3534020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63a7fff640 0 client.6783 ms_handle_reset on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/563872617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca35697f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 >> 172.21.15.99:0/563872617 conn(0x562ca350e380 msgr2=0x562ca355ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 shutdown_connections 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.462+0000 7f63d1ffd740 1 -- 172.21.15.99:0/563872617 wait complete. 2024-03-04T10:06:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.463+0000 7f63d1ffd740 20 client.6783 trim_cache size 0 max 16384 2024-03-04T10:06:57.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 Processor -- start 2024-03-04T10:06:57.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 -- start start 2024-03-04T10:06:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca353aea0 0x562ca357cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3559df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 0x562ca3568260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c81ffe50 con 0x562ca35419c0 2024-03-04T10:06:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c81ff250 con 0x562ca353aea0 2024-03-04T10:06:57.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c81ff850 con 0x562ca353b280 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63cd9bc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 0x562ca3568260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63cd9bc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 0x562ca3568260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:51998/0 (socket says 172.21.15.99:51998) 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.479+0000 7f63cd9bc640 1 -- 172.21.15.99:0/2434351014 learned_addr learned my addr 172.21.15.99:0/2434351014 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca353aea0 0x562ca357cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3559df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd9bc640 1 -- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 msgr2=0x562ca3559df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3559df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd9bc640 1 -- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca353aea0 msgr2=0x562ca357cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca353aea0 0x562ca357cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd9bc640 1 -- 172.21.15.99:0/2434351014 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ca355cbf0 con 0x562ca35419c0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 0x562ca3568260 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f63b8007ac0 tx=0x7f63b8001bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c468364f19ed79ca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63ae7fc640 1 -- 172.21.15.99:0/2434351014 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b80052e0 con 0x562ca35419c0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63ae7fc640 1 -- 172.21.15.99:0/2434351014 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8018070 con 0x562ca35419c0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.480+0000 7f63ae7fc640 1 -- 172.21.15.99:0/2434351014 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b800f940 con 0x562ca35419c0 2024-03-04T10:06:57.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2434351014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 msgr2=0x562ca3568260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 0x562ca3568260 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f63b8007ac0 tx=0x7f63b8001bd0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2434351014 shutdown_connections 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca35419c0 0x562ca3568260 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca3559df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/2434351014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca353aea0 0x562ca357cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2434351014 >> 172.21.15.99:0/2434351014 conn(0x562ca350e380 msgr2=0x562ca3559350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2434351014 shutdown_connections 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 -- 172.21.15.99:0/2434351014 wait complete. 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 -- start start 2024-03-04T10:06:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.481+0000 7f63d1ffd740 1 Processor -- start 2024-03-04T10:06:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca355a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca35c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca35c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ca3512750 con 0x562ca353aea0 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ca3434f90 con 0x562ca35419c0 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ca3540530 con 0x562ca353b280 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63d1ffd740 10 client.0 _ll_register_callbacks cb 0x7f63b00036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca355a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca355a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52014/0 (socket says 172.21.15.99:52014) 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 -- 172.21.15.99:0/94148466 learned_addr learned my addr 172.21.15.99:0/94148466 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca35c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca35c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 -- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 msgr2=0x562ca35c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca35c2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 -- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 msgr2=0x562ca35c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca35c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.482+0000 7f63cd1bb640 1 -- 172.21.15.99:0/94148466 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ca35c3560 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63cd9bc640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca35c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63cd1bb640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca355a1f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c210 tx=0x7f63c0046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecdc8d6f545eaa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63a7fff640 1 -- 172.21.15.99:0/94148466 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0035670 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63a7fff640 1 -- 172.21.15.99:0/94148466 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f63c0064c80 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f63b8018070 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63a7fff640 1 -- 172.21.15.99:0/94148466 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63c00076a0 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 10 client.6789 fetch_fsmap learned FSMap version 50 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) v3 -- 0x562ca35325f0 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63a7fff640 1 -- 172.21.15.99:0/94148466 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b8018070 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63a7fff640 1 -- 172.21.15.99:0/94148466 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 50) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f63c0062040 con 0x562ca353aea0 2024-03-04T10:06:57.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 10 client.6789 fetch_fsmap finished waiting for FSMap version 50 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 10 client.6789 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 20 client.6789 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3539930 0x562ca355b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 4 client.6789 mds_command: new command op to 6729 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 --> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562ca3480050 con 0x562ca3539930 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.483+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3539930 0x562ca355b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63a7fff640 10 client.6789 ms_handle_connect on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63cc9ba640 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3539930 0x562ca355b2f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b883c960c63996f server_cookie=1b3e2f36b223287f in_seq=0 out_seq=0 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63a7fff640 1 -- 172.21.15.99:0/94148466 <== mds.0 v2:172.21.15.99:6834/3022309305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562ca3480050 con 0x562ca3539930 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63a7fff640 10 client.6789 handle_command_reply: tid=0 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 client.6789 shutdown 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 msgr2=0x562ca355a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca355a1f0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c210 tx=0x7f63c0046040 comp rx=0 tx=0).stop 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 shutdown_connections 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x562ca3539930 0x562ca355b2f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ca35419c0 0x562ca35c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x562ca353b280 0x562ca35c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 --2- 172.21.15.99:0/94148466 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x562ca353aea0 0x562ca355a1f0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63a7fff640 0 client.6789 ms_handle_reset on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 >> 172.21.15.99:0/94148466 conn(0x562ca350e380 msgr2=0x562ca350e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 shutdown_connections 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 1 -- 172.21.15.99:0/94148466 wait complete. 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.484+0000 7f63d1ffd740 20 client.6789 trim_cache size 0 max 16384 2024-03-04T10:06:57.492 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 6729, 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.3779826530000001, 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 50, 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 67, 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 67, 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 10.671280213999999 2024-03-04T10:06:57.493 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:06:57.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f63b406b4d0 msgr2=0x7f63b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f63b406b4d0 0x7f63b406d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f63c000c240 tx=0x7f63c0039040 comp rx=0 tx=0).stop 2024-03-04T10:06:57.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 msgr2=0x7f63c80b5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 0x7f63c80b5b30 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f63bc007fb0 tx=0x7f63bc010650 comp rx=0 tx=0).stop 2024-03-04T10:06:57.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 shutdown_connections 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f63b406b4d0 0x7f63b406d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63c814d890 0x7f63c814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63c81551f0 0x7f63c80b5b30 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 --2- 172.21.15.99:0/3445762505 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b00039e0 0x7f63c80cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 >> 172.21.15.99:0/3445762505 conn(0x7f63c80d0470 msgr2=0x7f63c80b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 shutdown_connections 2024-03-04T10:06:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.485+0000 7f63cf41f640 1 -- 172.21.15.99:0/3445762505 wait complete. 2024-03-04T10:06:57.497 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:57.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.595+0000 7f9f6fef1640 1 Processor -- start 2024-03-04T10:06:57.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6fef1640 1 -- start start 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6fef1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9f680b2e60 0x7f9f680b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6fef1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f680b5470 0x7f9f680b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6fef1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6815e1e0 con 0x7f9f6815de00 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6fef1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f680b98c0 con 0x7f9f680b2e60 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6fef1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680b32c0 con 0x7f9f680b5470 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6d48c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9f680b2e60 0x7f9f680b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6dc8d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9f6815de00 0x7f9f680b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48846/0 (socket says 172.21.15.99:48846) 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.596+0000 7f9f6dc8d640 1 -- 172.21.15.99:0/3508045374 learned_addr learned my addr 172.21.15.99:0/3508045374 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:57.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.597+0000 7f9f6d48c640 1 -- 172.21.15.99:0/3508045374 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f680b5470 msgr2=0x7f9f680b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.597+0000 7f9f6d48c640 1 --2- 172.21.15.99:0/3508045374 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f680b5470 0x7f9f680b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.597+0000 7f9f6d48c640 1 -- 172.21.15.99:0/3508045374 >> v1:172.21.15.99:6789/0 conn(0x7f9f6815de00 legacy=0x7f9f680b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.597+0000 7f9f6d48c640 1 -- 172.21.15.99:0/3508045374 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f680d1c10 con 0x7f9f680b2e60 2024-03-04T10:06:57.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.597+0000 7f9f6cc8b640 1 -- 172.21.15.99:0/3508045374 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343743418 0 0) 0x7f9f6815e1e0 con 0x7f9f6815de00 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.597+0000 7f9f6d48c640 1 --2- 172.21.15.99:0/3508045374 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680b9380 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9f580097a0 tx=0x7f9f5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad55dc290a9578d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6cc8b640 1 -- 172.21.15.99:0/3508045374 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58035020 con 0x7f9f680b2e60 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6cc8b640 1 -- 172.21.15.99:0/3508045374 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58031830 con 0x7f9f680b2e60 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6cc8b640 1 -- 172.21.15.99:0/3508045374 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58031c00 con 0x7f9f680b2e60 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 -- 172.21.15.99:0/3508045374 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 msgr2=0x7f9f680b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/3508045374 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680b9380 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f9f580097a0 tx=0x7f9f5802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 -- 172.21.15.99:0/3508045374 shutdown_connections 2024-03-04T10:06:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/3508045374 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f680b5470 0x7f9f680b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/3508045374 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680b9380 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 -- 172.21.15.99:0/3508045374 >> 172.21.15.99:0/3508045374 conn(0x7f9f680d0470 msgr2=0x7f9f680d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 -- 172.21.15.99:0/3508045374 shutdown_connections 2024-03-04T10:06:57.601 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.598+0000 7f9f6fef1640 1 -- 172.21.15.99:0/3508045374 wait complete. 2024-03-04T10:06:57.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.599+0000 7f9f6fef1640 1 Processor -- start 2024-03-04T10:06:57.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.599+0000 7f9f6fef1640 1 -- start start 2024-03-04T10:06:57.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6fef1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6fef1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f680b5470 0x7f9f6814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6fef1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6815de00 0x7f9f680b7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6fef1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680d21d0 con 0x7f9f680b2e60 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6fef1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680d2820 con 0x7f9f680b5470 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6fef1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f680d24b0 con 0x7f9f6815de00 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52036/0 (socket says 172.21.15.99:52036) 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 -- 172.21.15.99:0/2305634124 learned_addr learned my addr 172.21.15.99:0/2305634124 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 -- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6815de00 msgr2=0x7f9f680b7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6e48e640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6815de00 0x7f9f680b7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6d48c640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f680b5470 0x7f9f6814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6815de00 0x7f9f680b7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 -- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f680b5470 msgr2=0x7f9f6814dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f680b5470 0x7f9f6814dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6dc8d640 1 -- 172.21.15.99:0/2305634124 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6814e3e0 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6e48e640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6815de00 0x7f9f680b7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.600+0000 7f9f6d48c640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f680b5470 0x7f9f6814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.601+0000 7f9f6dc8d640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680c9fc0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c7c0 tx=0x7f9f6400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.601+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64018070 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.601+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6400a640 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.601+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f680c8f20 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.601+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6400aa30 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.601+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f680c94e0 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.602+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f680cf630 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.602+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9f6401b070 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.603+0000 7f9f5e7fc640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f4c06b4d0 0x7f9f4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.603+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9f640899f0 con 0x7f9f680b2e60 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.603+0000 7f9f6d48c640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f4c06b4d0 0x7f9f4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.603+0000 7f9f6d48c640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f4c06b4d0 0x7f9f4c06d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9f58036000 tx=0x7f9f58008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.607+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9f680cf630 con 0x7f9f680b2e60 2024-03-04T10:06:57.787 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:06:57.785+0000 7f9352863640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:57.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.815+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f68074f60 con 0x7f9f680b2e60 2024-03-04T10:06:57.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.815+0000 7f9f5e7fc640 1 -- 172.21.15.99:0/2305634124 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f9f64058690 con 0x7f9f680b2e60 2024-03-04T10:06:57.817 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:57.817 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:55.115395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:57.817 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 50 2024-03-04T10:06:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f4c06b4d0 msgr2=0x7f9f4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f4c06b4d0 0x7f9f4c06d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9f58036000 tx=0x7f9f58008030 comp rx=0 tx=0).stop 2024-03-04T10:06:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 msgr2=0x7f9f680c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680c9fc0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c7c0 tx=0x7f9f6400cc80 comp rx=0 tx=0).stop 2024-03-04T10:06:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 shutdown_connections 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f4c06b4d0 0x7f9f4c06d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f6815de00 0x7f9f680b7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f680b5470 0x7f9f6814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 --2- 172.21.15.99:0/2305634124 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9f680b2e60 0x7f9f680c9fc0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 >> 172.21.15.99:0/2305634124 conn(0x7f9f680d0470 msgr2=0x7f9f6815cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.817+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 shutdown_connections 2024-03-04T10:06:57.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.818+0000 7f9f6fef1640 1 -- 172.21.15.99:0/2305634124 wait complete. 2024-03-04T10:06:57.830 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 true 2024-03-04T10:06:57.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.930+0000 7f24e677e640 1 Processor -- start 2024-03-04T10:06:57.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.930+0000 7f24e677e640 1 -- start start 2024-03-04T10:06:57.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e677e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f24e0158230 0x7f24e00cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e677e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00cd110 0x7f24e00cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e677e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e00cc350 con 0x7f24e00d1770 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e677e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e00cc630 con 0x7f24e0158230 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e677e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00cc910 con 0x7f24e00cd110 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24dffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f24e0158230 0x7f24e00cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e4d1b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f24e00d1770 0x7f24e00d1b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48854/0 (socket says 172.21.15.99:48854) 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.931+0000 7f24e4d1b640 1 -- 172.21.15.99:0/582602950 learned_addr learned my addr 172.21.15.99:0/582602950 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24df7fe640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00cd110 0x7f24e00cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24dffff640 1 -- 172.21.15.99:0/582602950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00cd110 msgr2=0x7f24e00cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24dffff640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00cd110 0x7f24e00cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24dffff640 1 -- 172.21.15.99:0/582602950 >> v1:172.21.15.99:6789/0 conn(0x7f24e00d1770 legacy=0x7f24e00d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24deffd640 1 -- 172.21.15.99:0/582602950 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014846370 0 0) 0x7f24e00cc350 con 0x7f24e00d1770 2024-03-04T10:06:57.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24dffff640 1 -- 172.21.15.99:0/582602950 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00ccbf0 con 0x7f24e0158230 2024-03-04T10:06:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24df7fe640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00cd110 0x7f24e00cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24dffff640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e0158230 0x7f24e00cf6e0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f24d00097a0 tx=0x7f24d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fd9ee4a3a140f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.932+0000 7f24deffd640 1 -- 172.21.15.99:0/582602950 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0035020 con 0x7f24e0158230 2024-03-04T10:06:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24deffd640 1 -- 172.21.15.99:0/582602950 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d0031830 con 0x7f24e0158230 2024-03-04T10:06:57.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24deffd640 1 -- 172.21.15.99:0/582602950 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0031c00 con 0x7f24e0158230 2024-03-04T10:06:57.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 -- 172.21.15.99:0/582602950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e0158230 msgr2=0x7f24e00cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e0158230 0x7f24e00cf6e0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f24d00097a0 tx=0x7f24d002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 -- 172.21.15.99:0/582602950 shutdown_connections 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00cd110 0x7f24e00cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 --2- 172.21.15.99:0/582602950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e0158230 0x7f24e00cf6e0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 -- 172.21.15.99:0/582602950 >> 172.21.15.99:0/582602950 conn(0x7f24e00b3330 msgr2=0x7f24e00b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 -- 172.21.15.99:0/582602950 shutdown_connections 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.933+0000 7f24e677e640 1 -- 172.21.15.99:0/582602950 wait complete. 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.934+0000 7f24e677e640 1 Processor -- start 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.934+0000 7f24e677e640 1 -- start start 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e677e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 0x7f24e014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e677e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00d1770 0x7f24e01519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e677e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e0158230 0x7f24e0151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e677e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00cc0f0 con 0x7f24e00cd110 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e677e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00cbf70 con 0x7f24e0158230 2024-03-04T10:06:57.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e677e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00cdc10 con 0x7f24e00d1770 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24dffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 0x7f24e014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24dffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 0x7f24e014fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52048/0 (socket says 172.21.15.99:52048) 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24dffff640 1 -- 172.21.15.99:0/3695688001 learned_addr learned my addr 172.21.15.99:0/3695688001 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24e4d1b640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e0158230 0x7f24e0151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.935+0000 7f24df7fe640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00d1770 0x7f24e01519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dffff640 1 -- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00d1770 msgr2=0x7f24e01519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dffff640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00d1770 0x7f24e01519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dffff640 1 -- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e0158230 msgr2=0x7f24e0151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dffff640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e0158230 0x7f24e0151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dffff640 1 -- 172.21.15.99:0/3695688001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00b6630 con 0x7f24e00cd110 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dffff640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 0x7f24e014fc60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f24d0000c00 tx=0x7f24d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0040070 con 0x7f24e00cd110 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d0039a00 con 0x7f24e00cd110 2024-03-04T10:06:57.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d0035040 con 0x7f24e00cd110 2024-03-04T10:06:57.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e00b6910 con 0x7f24e00cd110 2024-03-04T10:06:57.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.936+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e015f990 con 0x7f24e00cd110 2024-03-04T10:06:57.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.937+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e00b6e30 con 0x7f24e00cd110 2024-03-04T10:06:57.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.937+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f24d004b060 con 0x7f24e00cd110 2024-03-04T10:06:57.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.938+0000 7f24dcff9640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f24cc06b4d0 0x7f24cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:57.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.938+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f24d00abe60 con 0x7f24e00cd110 2024-03-04T10:06:57.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.938+0000 7f24df7fe640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f24cc06b4d0 0x7f24cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:57.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.938+0000 7f24df7fe640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f24cc06b4d0 0x7f24cc06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f24e006c060 tx=0x7f24d4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:57.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:57.942+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f24e00b6e30 con 0x7f24e00cd110 2024-03-04T10:06:58.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:58.182+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"} v 0) v1 -- 0x7f24e0074f60 con 0x7f24e00cd110 2024-03-04T10:06:59.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.127+0000 7f24dcff9640 1 -- 172.21.15.99:0/3695688001 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "true"}]=0 freezing mds.6729 v51) v1 ==== 112+0+0 (secure 0 0 0) 0x7f24d007aa80 con 0x7f24e00cd110 2024-03-04T10:06:59.128 INFO:teuthology.orchestra.run.smithi099.stderr:freezing mds.6729 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f24cc06b4d0 msgr2=0x7f24cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f24cc06b4d0 0x7f24cc06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f24e006c060 tx=0x7f24d4008040 comp rx=0 tx=0).stop 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 msgr2=0x7f24e014fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 0x7f24e014fc60 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f24d0000c00 tx=0x7f24d00046c0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 shutdown_connections 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f24cc06b4d0 0x7f24cc06d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24e0158230 0x7f24e0151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24e00d1770 0x7f24e01519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 --2- 172.21.15.99:0/3695688001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24e00cd110 0x7f24e014fc60 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.129+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 >> 172.21.15.99:0/3695688001 conn(0x7f24e00b3330 msgr2=0x7f24e0154e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.130+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 shutdown_connections 2024-03-04T10:06:59.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.130+0000 7f24e677e640 1 -- 172.21.15.99:0/3695688001 wait complete. 2024-03-04T10:06:59.144 DEBUG:teuthology.orchestra.run.smithi099:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-04T10:06:59.146 DEBUG:teuthology.orchestra.run.smithi099:> timeout 30 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --connect-timeout=60 tell mds.4:0 flush journal 2024-03-04T10:06:59.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-04T10:06:59.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.255+0000 7f621de0c640 1 Processor -- start 2024-03-04T10:06:59.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.255+0000 7f621de0c640 1 -- start start 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.255+0000 7f621de0c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 0x7f6218161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.255+0000 7f621de0c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180d2400 con 0x7f62180b5700 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.255+0000 7f621de0c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180d26e0 con 0x7f62180b9410 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.255+0000 7f621de0c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62180d29c0 con 0x7f62180d34a0 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62177fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f62180b5700 0x7f62180d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48864/0 (socket says 172.21.15.99:48864) 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62177fe640 1 -- 172.21.15.99:0/1872993251 learned_addr learned my addr 172.21.15.99:0/1872993251 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f6216ffd640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 0x7f6218161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076143186 0 0) 0x7f62180d2400 con 0x7f62180b5700 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6200003520 con 0x7f62180b5700 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1647454731 0 0) 0x7f6200003520 con 0x7f62180b5700 2024-03-04T10:06:59.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 msgr2=0x7f6218161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 0x7f6218161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 >> v1:172.21.15.162:6789/0 conn(0x7f62180b9410 legacy=0x7f6218162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180d2ca0 con 0x7f62180b5700 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6208001670 con 0x7f62180b5700 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 >> v1:172.21.15.99:6789/0 conn(0x7f62180b5700 legacy=0x7f62180d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 0x7f62000039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6200003f60 0x7f6200004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6200004d40 0x7f62000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180d29c0 con 0x7f6200004d40 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620000bb50 con 0x7f6200003f60 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620000bd50 con 0x7f62180d34a0 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.256+0000 7f6217fff640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6200004d40 0x7f62000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6216ffd640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6200003f60 0x7f6200004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f62177fe640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 0x7f62000039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6217fff640 1 -- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 msgr2=0x7f62000039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6217fff640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180d34a0 0x7f62000039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6217fff640 1 -- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6200003f60 msgr2=0x7f6200004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6217fff640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6200003f60 0x7f6200004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6217fff640 1 -- 172.21.15.99:0/1872993251 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200003520 con 0x7f6200004d40 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6216ffd640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6200003f60 0x7f6200004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f6217fff640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6200004d40 0x7f62000051c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6210001e50 tx=0x7f6200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3722348d70c85541 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.257+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210001a70 con 0x7f6200004d40 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6210003b70 con 0x7f6200004d40 2024-03-04T10:06:59.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f62167fc640 1 -- 172.21.15.99:0/1872993251 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62100050d0 con 0x7f6200004d40 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f621de0c640 1 -- 172.21.15.99:0/1872993251 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6200004d40 msgr2=0x7f62000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f621de0c640 1 --2- 172.21.15.99:0/1872993251 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6200004d40 0x7f62000051c0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6210001e50 tx=0x7f6200005830 comp rx=0 tx=0).stop 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f6217fff640 1 -- 172.21.15.99:0/1872993251 reap_dead start 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f621de0c640 1 -- 172.21.15.99:0/1872993251 shutdown_connections 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.258+0000 7f621de0c640 1 -- 172.21.15.99:0/1872993251 >> 172.21.15.99:0/1872993251 conn(0x7f62180d0490 msgr2=0x7f62180d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.259+0000 7f621de0c640 1 -- 172.21.15.99:0/1872993251 shutdown_connections 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.259+0000 7f621de0c640 1 -- 172.21.15.99:0/1872993251 wait complete. 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.260+0000 7f621de0c640 1 Processor -- start 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.260+0000 7f621de0c640 1 -- start start 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f621de0c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f62180c74e0 0x7f62180cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f621de0c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180caae0 0x7f62180caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f621de0c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62180c78c0 0x7f62180c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f621de0c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b5c60 con 0x7f62180c74e0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f621de0c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b5ae0 con 0x7f62180c78c0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f621de0c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62180b5de0 con 0x7f62180caae0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f62180c74e0 0x7f62180cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f62180c74e0 0x7f62180cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52076/0 (socket says 172.21.15.99:52076) 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 -- 172.21.15.99:0/2914877930 learned_addr learned my addr 172.21.15.99:0/2914877930 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f6217fff640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62180c78c0 0x7f62180c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f6216ffd640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180caae0 0x7f62180caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 -- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180caae0 msgr2=0x7f62180caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180caae0 0x7f62180caf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 -- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62180c78c0 msgr2=0x7f62180c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62180c78c0 0x7f62180c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 -- 172.21.15.99:0/2914877930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62180cb4a0 con 0x7f62180c74e0 2024-03-04T10:06:59.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f6216ffd640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f62180caae0 0x7f62180caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f6217fff640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f62180c78c0 0x7f62180c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f62177fe640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f62180c74e0 0x7f62180cc800 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f620800eaf0 tx=0x7f6208001700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f61fffff640 1 -- 172.21.15.99:0/2914877930 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620800c8d0 con 0x7f62180c74e0 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.261+0000 7f61fffff640 1 -- 172.21.15.99:0/2914877930 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208018070 con 0x7f62180c74e0 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.262+0000 7f61fffff640 1 -- 172.21.15.99:0/2914877930 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620800ccd0 con 0x7f62180c74e0 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.262+0000 7f621de0c640 1 -- 172.21.15.99:0/2914877930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62180cb760 con 0x7f62180c74e0 2024-03-04T10:06:59.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.262+0000 7f621de0c640 1 -- 172.21.15.99:0/2914877930 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62180cbd20 con 0x7f62180c74e0 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.263+0000 7f61fffff640 1 -- 172.21.15.99:0/2914877930 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6208013420 con 0x7f62180c74e0 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.264+0000 7f61fffff640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f620406b500 0x7f620406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.264+0000 7f61fffff640 1 -- 172.21.15.99:0/2914877930 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f620808a0a0 con 0x7f62180c74e0 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.264+0000 7f6216ffd640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f620406b500 0x7f620406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 Processor -- start 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 -- start start 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f6216ffd640 1 --2- 172.21.15.99:0/2914877930 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f620406b500 0x7f620406d9c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f62181f6db0 tx=0x7f620c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b0f76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b1274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b0f7c10 0x55db9b127a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db9b060f40 con 0x55db9b124b10 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db9b009d10 con 0x55db9b124730 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.265+0000 7f62209df740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db9b05cfe0 con 0x55db9b0f7c10 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b1274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b1274c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52082/0 (socket says 172.21.15.99:52082) 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 -- 172.21.15.99:0/3128695433 learned_addr learned my addr 172.21.15.99:0/3128695433 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f62177fe640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b0f76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6217fff640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b0f7c10 0x55db9b127a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 -- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b0f7c10 msgr2=0x55db9b127a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b0f7c10 0x55db9b127a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 -- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 msgr2=0x55db9b0f76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b0f76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 -- 172.21.15.99:0/3128695433 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db9ae41a30 con 0x55db9b124b10 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6217fff640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b0f7c10 0x55db9b127a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f62177fe640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b0f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b1274c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f620c013df0 tx=0x7f620c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb7fdb608052203 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f61fd7fa640 1 -- 172.21.15.99:0/3128695433 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c023440 con 0x55db9b124b10 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f61fd7fa640 1 -- 172.21.15.99:0/3128695433 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c028070 con 0x55db9b124b10 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.266+0000 7f61fd7fa640 1 -- 172.21.15.99:0/3128695433 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c0237e0 con 0x55db9b124b10 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 -- 172.21.15.99:0/3128695433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 msgr2=0x55db9b1274c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b1274c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f620c013df0 tx=0x7f620c016000 comp rx=0 tx=0).stop 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 -- 172.21.15.99:0/3128695433 shutdown_connections 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b0f7c10 0x55db9b127a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b1274c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 --2- 172.21.15.99:0/3128695433 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b0f76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 -- 172.21.15.99:0/3128695433 >> 172.21.15.99:0/3128695433 conn(0x55db9b0f9940 msgr2=0x55db9b1424c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 -- 172.21.15.99:0/3128695433 shutdown_connections 2024-03-04T10:06:59.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.267+0000 7f62209df740 1 -- 172.21.15.99:0/3128695433 wait complete. 2024-03-04T10:06:59.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.268+0000 7f62209df740 1 -- start start 2024-03-04T10:06:59.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.268+0000 7f62209df740 1 Processor -- start 2024-03-04T10:06:59.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 0x55db9b142c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124b10 0x55db9b117970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b12a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db9b0fbfe0 con 0x55db9b124730 2024-03-04T10:06:59.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db9b01d0c0 con 0x55db9b124b10 2024-03-04T10:06:59.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db9b0fe820 con 0x55db9b128660 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62209df740 10 client.0 _ll_register_callbacks cb 0x55db9b058ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 0x55db9b142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 0x55db9b142c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52092/0 (socket says 172.21.15.99:52092) 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 -- 172.21.15.99:0/3082236095 learned_addr learned my addr 172.21.15.99:0/3082236095 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124b10 0x55db9b117970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f6217fff640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b12a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 -- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 msgr2=0x55db9b12a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b12a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 -- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124b10 msgr2=0x55db9b117970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124b10 0x55db9b117970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 -- 172.21.15.99:0/3082236095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db9ad75620 con 0x55db9b124730 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f6217fff640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b12a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124b10 0x55db9b117970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f62177fe640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 0x55db9b142c00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f62080882c0 tx=0x7f6208058de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25198dbccd28e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f61f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f61f67fc640 1 -- 172.21.15.99:0/3082236095 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6208060820 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.269+0000 7f61f67fc640 1 -- 172.21.15.99:0/3082236095 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f62080609c0 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f620c028070 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f61f67fc640 1 -- 172.21.15.99:0/3082236095 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62080579f0 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 10 client.6819 fetch_fsmap learned FSMap version 51 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55db9ad35320 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f61f67fc640 1 -- 172.21.15.99:0/3082236095 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c028070 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f61f67fc640 1 -- 172.21.15.99:0/3082236095 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6208057b90 con 0x55db9b124730 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 10 client.6819 fetch_fsmap finished waiting for FSMap version 51 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 10 client.6819 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 20 client.6819 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b1182b0 0x55db9b1186f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 4 client.6819 mds_command: new command op to 6729 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 --> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db9b06e480 con 0x55db9b1182b0 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.270+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b1182b0 0x55db9b1186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f61f67fc640 10 client.6819 ms_handle_connect on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b1182b0 0x55db9b1186f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee370a210627a48 server_cookie=c4b18b203586f17f in_seq=0 out_seq=0 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f61f67fc640 1 -- 172.21.15.99:0/3082236095 <== mds.0 v2:172.21.15.99:6834/3022309305 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55db9b06e480 con 0x55db9b1182b0 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f61f67fc640 10 client.6819 handle_command_reply: tid=0 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f62209df740 1 client.6819 shutdown 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 msgr2=0x55db9b142c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.271+0000 7f62209df740 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 0x55db9b142c00 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f62080882c0 tx=0x7f6208058de0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 shutdown_connections 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b1182b0 0x55db9b1186f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b12a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124b10 0x55db9b117970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 --2- 172.21.15.99:0/3082236095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124730 0x55db9b142c00 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 >> 172.21.15.99:0/3082236095 conn(0x55db9b0f7c10 msgr2=0x55db9b0fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f61f67fc640 0 client.6819 ms_handle_reset on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 shutdown_connections 2024-03-04T10:06:59.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 1 -- 172.21.15.99:0/3082236095 wait complete. 2024-03-04T10:06:59.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.272+0000 7f62209df740 20 client.6819 trim_cache size 0 max 16384 2024-03-04T10:06:59.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.278+0000 7f46c8b09640 1 Processor -- start 2024-03-04T10:06:59.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.278+0000 7f46c8b09640 1 -- start start 2024-03-04T10:06:59.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c8b09640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f46c4143720 0x7f46c41441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c8b09640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f46c4071270 0x7f46c40b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c8b09640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c4071a50 con 0x7f46c40b47a0 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c8b09640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c406d2e0 con 0x7f46c4071270 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c8b09640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c40b4bd0 con 0x7f46c4143720 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c1d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f46c4071270 0x7f46c40b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c2d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f46c40b47a0 0x7f46c415e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48878/0 (socket says 172.21.15.99:48878) 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c1d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f46c4071270 0x7f46c40b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52106/0 (socket says 172.21.15.99:52106) 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c2d9d640 1 -- 172.21.15.99:0/3785126087 learned_addr learned my addr 172.21.15.99:0/3785126087 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c259c640 1 --2- 172.21.15.99:0/3785126087 >> v2:172.21.15.162:3300/0 conn(0x7f46c4143720 0x7f46c41441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c1d9b640 1 -- 172.21.15.99:0/3785126087 >> v2:172.21.15.162:3300/0 conn(0x7f46c4143720 msgr2=0x7f46c41441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c1d9b640 1 --2- 172.21.15.99:0/3785126087 >> v2:172.21.15.162:3300/0 conn(0x7f46c4143720 0x7f46c41441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c1d9b640 1 -- 172.21.15.99:0/3785126087 >> v1:172.21.15.99:6789/0 conn(0x7f46c40b47a0 legacy=0x7f46c415e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c159a640 1 -- 172.21.15.99:0/3785126087 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385197538 0 0) 0x7f46c4071a50 con 0x7f46c40b47a0 2024-03-04T10:06:59.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.279+0000 7f46c1d9b640 1 -- 172.21.15.99:0/3785126087 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4071d30 con 0x7f46c4071270 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.280+0000 7f46c1d9b640 1 --2- 172.21.15.99:0/3785126087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c40b4260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f46c4072710 tx=0x7f46b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549c2ad45f11415 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.280+0000 7f46c159a640 1 -- 172.21.15.99:0/3785126087 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b80315a0 con 0x7f46c4071270 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.280+0000 7f46c159a640 1 -- 172.21.15.99:0/3785126087 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b8037070 con 0x7f46c4071270 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.280+0000 7f46c159a640 1 -- 172.21.15.99:0/3785126087 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8031960 con 0x7f46c4071270 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 -- 172.21.15.99:0/3785126087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 msgr2=0x7f46c40b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 --2- 172.21.15.99:0/3785126087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c40b4260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f46c4072710 tx=0x7f46b802b840 comp rx=0 tx=0).stop 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 -- 172.21.15.99:0/3785126087 shutdown_connections 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 --2- 172.21.15.99:0/3785126087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c40b4260 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 --2- 172.21.15.99:0/3785126087 >> v2:172.21.15.162:3300/0 conn(0x7f46c4143720 0x7f46c41441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 -- 172.21.15.99:0/3785126087 >> 172.21.15.99:0/3785126087 conn(0x7f46c40b2800 msgr2=0x7f46c40b2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 -- 172.21.15.99:0/3785126087 shutdown_connections 2024-03-04T10:06:59.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.281+0000 7f46c8b09640 1 -- 172.21.15.99:0/3785126087 wait complete. 2024-03-04T10:06:59.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.283+0000 7f46c8b09640 1 Processor -- start 2024-03-04T10:06:59.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.283+0000 7f46c8b09640 1 -- start start 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.284+0000 7f46c8b09640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c4136a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.284+0000 7f46c8b09640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46c40b47a0 0x7f46c4137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.284+0000 7f46c8b09640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46c4143720 0x7f46c4137560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c8b09640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c406d530 con 0x7f46c4071270 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c8b09640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4072010 con 0x7f46c4143720 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c8b09640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c4143b00 con 0x7f46c40b47a0 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c4136a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c4136a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52118/0 (socket says 172.21.15.99:52118) 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 -- 172.21.15.99:0/391659235 learned_addr learned my addr 172.21.15.99:0/391659235 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c2d9d640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46c4143720 0x7f46c4137560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c1d9b640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46c40b47a0 0x7f46c4137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 -- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46c40b47a0 msgr2=0x7f46c4137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46c40b47a0 0x7f46c4137020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 -- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46c4143720 msgr2=0x7f46c4137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46c4143720 0x7f46c4137560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 -- 172.21.15.99:0/391659235 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c41418e0 con 0x7f46c4071270 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c2d9d640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46c4143720 0x7f46c4137560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c1d9b640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46c40b47a0 0x7f46c4137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c259c640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46c4071270 0x7f46c4136a20 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f46b400d620 tx=0x7f46b400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b400b620 con 0x7f46c4071270 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b4019070 con 0x7f46c4071270 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c8b09640 1 -- 172.21.15.99:0/391659235 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c4139e10 con 0x7f46c4071270 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b400ba00 con 0x7f46c4071270 2024-03-04T10:06:59.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.285+0000 7f46c8b09640 1 -- 172.21.15.99:0/391659235 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c414afc0 con 0x7f46c4071270 2024-03-04T10:06:59.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.286+0000 7f46c8b09640 1 -- 172.21.15.99:0/391659235 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f46c4074f60 con 0x7f46c4071270 2024-03-04T10:06:59.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.287+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f46b400bba0 con 0x7f46c4071270 2024-03-04T10:06:59.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f62209df740 1 Processor -- start 2024-03-04T10:06:59.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f46b2ffd640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f62209df740 1 -- start start 2024-03-04T10:06:59.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f46b408a790 con 0x7f46c4071270 2024-03-04T10:06:59.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+515 (secure 0 0 0) 0x7f46b408ab70 con 0x7f46c4071270 2024-03-04T10:06:59.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f46c8b09640 1 -- 172.21.15.99:0/391659235 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f46c4072500 con 0x7f46c4071270 2024-03-04T10:06:59.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f46c1d9b640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f46a406b4d0 0x7f46a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.288+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 7 ==== log(1 entries from seq 692 at 2024-03-04T10:06:58.183490+0000) v1 ==== 309+0+0 (secure 0 0 0) 0x7f46b4088bf0 con 0x7f46c4071270 2024-03-04T10:06:59.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f46c1d9b640 1 --2- 172.21.15.99:0/391659235 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f46a406b4d0 0x7f46a406d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f46b8036000 tx=0x7f46b8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b142930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62209df740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b123420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b123420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:60032/0 (socket says 172.21.15.99:60032) 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b123420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52126/0 (socket says 172.21.15.99:52126) 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b0f7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62177fe640 1 -- 172.21.15.99:0/3481532175 learned_addr learned my addr 172.21.15.99:0/3481532175 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218201260 con 0x55db9b124b10 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6218200420 con 0x55db9b124730 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62181ff820 con 0x55db9b128660 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6217fff640 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b0f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 -- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 msgr2=0x55db9b0f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b0f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 -- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 msgr2=0x55db9b142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b142930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.289+0000 7f6216ffd640 1 -- 172.21.15.99:0/3481532175 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db9b127fd0 con 0x55db9b124b10 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.290+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b123420 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x55db9b1281e0 tx=0x7f620c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33559817a3bd889 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.290+0000 7f61fcff9640 1 -- 172.21.15.99:0/3481532175 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c03a070 con 0x55db9b124b10 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.290+0000 7f61fcff9640 1 -- 172.21.15.99:0/3481532175 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c028050 con 0x55db9b124b10 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.290+0000 7f61fcff9640 1 -- 172.21.15.99:0/3481532175 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620c035d20 con 0x55db9b124b10 2024-03-04T10:06:59.297 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.290+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 msgr2=0x55db9b123420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.290+0000 7f62209df740 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b123420 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x55db9b1281e0 tx=0x7f620c018000 comp rx=0 tx=0).stop 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 shutdown_connections 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b128660 0x55db9b0f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b124b10 0x55db9b123420 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 --2- 172.21.15.99:0/3481532175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 >> 172.21.15.99:0/3481532175 conn(0x55db9b0f7c10 msgr2=0x55db9b1410e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 shutdown_connections 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 -- 172.21.15.99:0/3481532175 wait complete. 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 -- start start 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.291+0000 7f62209df740 1 Processor -- start 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b197440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b124b10 0x55db9b1a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b128660 0x55db9b145800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db9b117060 con 0x55db9b128660 2024-03-04T10:06:59.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62181ffe20 con 0x55db9b124730 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db9b142ef0 con 0x55db9b124b10 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62209df740 10 client.0 _ll_register_callbacks cb 0x7f6208001600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b128660 0x55db9b145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b128660 0x55db9b145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52142/0 (socket says 172.21.15.99:52142) 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 -- 172.21.15.99:0/3342416233 learned_addr learned my addr 172.21.15.99:0/3342416233 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b124b10 0x55db9b1a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b124b10 msgr2=0x55db9b1a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b124b10 0x55db9b1a27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 msgr2=0x55db9b197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b197440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6217fff640 1 -- 172.21.15.99:0/3342416233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db9b122020 con 0x55db9b128660 2024-03-04T10:06:59.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55db9b124730 0x55db9b197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.292+0000 7f6216ffd640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55db9b124b10 0x55db9b1a27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f6217fff640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55db9b128660 0x55db9b145800 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6210005040 tx=0x7f6210001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afaf4aad7be5200d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f61f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f61f67fc640 1 -- 172.21.15.99:0/3342416233 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6210006b50 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f61f67fc640 1 -- 172.21.15.99:0/3342416233 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6210006cf0 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 1 -- 172.21.15.99:0/3342416233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f620c028050 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f61f67fc640 1 -- 172.21.15.99:0/3342416233 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f621004d050 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 10 client.6834 fetch_fsmap learned FSMap version 51 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 1 -- 172.21.15.99:0/3342416233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) v3 -- 0x55db9b122ba0 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f61f67fc640 1 -- 172.21.15.99:0/3342416233 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f620c028050 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f61f67fc640 1 -- 172.21.15.99:0/3342416233 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 51) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6210050020 con 0x55db9b128660 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 10 client.6834 fetch_fsmap finished waiting for FSMap version 51 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 10 client.6834 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-03-04T10:06:59.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 20 client.6834 populate_metadata read hostname 'smithi099' 2024-03-04T10:06:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 4 client.6834 mds_command: new command op to 6729 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:06:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62209df740 1 -- 172.21.15.99:0/3342416233 --> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55db9b0687d0 con 0x55db9b145d40 2024-03-04T10:06:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.293+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.294+0000 7f61f67fc640 10 client.6834 ms_handle_connect on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.294+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=474a9e0fcaad9c server_cookie=b00e3625a9d3991e in_seq=0 out_seq=0 2024-03-04T10:06:59.302 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:06:59.296+0000 7f934e85b640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:06:59.302 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:06:59.296+0000 7f934e85b640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f934e85b640 time 2024-03-04T10:06:59.297030+0000 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f935ab81517] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.303 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:06:59.297+0000 7f934e85b640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f934e85b640 time 2024-03-04T10:06:59.297030+0000 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.304 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f935ab81517] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.305 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr:*** Caught signal (Aborted) ** 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7f934e85b640 thread_name:MR_Finisher 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f935a254db0] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f935a2a154c] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: raise() 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: abort() 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f935ab815d4] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.306 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 21: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr: 22: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.307 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:06:59.298+0000 7f934e85b640 -1 *** Caught signal (Aborted) ** 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7f934e85b640 thread_name:MR_Finisher 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f935a254db0] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f935a2a154c] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: raise() 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: abort() 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f935ab815d4] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.308 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 21: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 22: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: -180> 2024-03-04T10:06:57.785+0000 7f9352863640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:59.309 INFO:tasks.ceph.mds.c.smithi099.stderr: -11> 2024-03-04T10:06:59.296+0000 7f934e85b640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: -9> 2024-03-04T10:06:59.296+0000 7f934e85b640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: -3> 2024-03-04T10:06:59.297+0000 7f934e85b640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f934e85b640 time 2024-03-04T10:06:59.297030+0000 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f935ab81517] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.310 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 0> 2024-03-04T10:06:59.298+0000 7f934e85b640 -1 *** Caught signal (Aborted) ** 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7f934e85b640 thread_name:MR_Finisher 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.311 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f935a254db0] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f935a2a154c] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: raise() 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: abort() 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f935ab815d4] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.312 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 21: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 22: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.313 INFO:tasks.ceph.mds.c.smithi099.stderr: -180> 2024-03-04T10:06:57.785+0000 7f9352863640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:06:59.314 INFO:teuthology.orchestra.run.smithi099.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-04T10:06:59.315 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-04T10:06:59.315 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: -11> 2024-03-04T10:06:59.296+0000 7f934e85b640 -1 mds.0.cache.den(0x1 a) newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: -9> 2024-03-04T10:06:59.296+0000 7f934e85b640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: -3> 2024-03-04T10:06:59.297+0000 7f934e85b640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f934e85b640 time 2024-03-04T10:06:59.297030+0000 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f935ab81517] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.316 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: 0> 2024-03-04T10:06:59.298+0000 7f934e85b640 -1 *** Caught signal (Aborted) ** 2024-03-04T10:06:59.317 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7f934e85b640 thread_name:MR_Finisher 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f935a254db0] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f935a2a154c] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: raise() 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: abort() 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f935ab815d4] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x55714300e4f1] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x55714321dd7d] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: ceph-mds(+0x37bfec) [0x55714322ffec] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: (CDir::_omap_commit(int)+0x93a) [0x55714323754a] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 10: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x265) [0x55714322e845] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 11: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0xa61) [0x557143053991] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 12: (MDLog::try_expire(LogSegment*, int)+0x62) [0x5571432b80f2] 2024-03-04T10:06:59.318 INFO:tasks.ceph.mds.c.smithi099.stderr: 13: (MDLog::trim_all()+0x708) [0x5571432bb0c8] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 14: ceph-mds(+0x18e0c1) [0x5571430420c1] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 15: ceph-mds(+0x133dcd) [0x557142fe7dcd] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 16: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 17: ceph-mds(+0x134072) [0x557142fe8072] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 18: (MDSContext::complete(int)+0x5c) [0x5571432b0bdc] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 19: (MDSIOContextBase::complete(int)+0x31c) [0x5571432b102c] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 20: (Finisher::finisher_thread_entry()+0x175) [0x7f935ac2c9a5] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 21: /lib64/libc.so.6(+0x9f802) [0x7f935a29f802] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 22: /lib64/libc.so.6(+0x3f450) [0x7f935a23f450] 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:06:59.319 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.322+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.322+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.322+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.323+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.323+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.323+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T10:06:59.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.323+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.418+0000 7f029ac27640 1 Processor -- start 2024-03-04T10:06:59.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.418+0000 7f029ac27640 1 -- start start 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f029ac27640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f029415d660 0x7f029415e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f029ac27640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f02940aeb20 0x7f0294071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f029ac27640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f029415da40 con 0x7f0294071890 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f029ac27640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0294071ce0 con 0x7f029415d660 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f029ac27640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029415cbc0 con 0x7f02940aeb20 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02989c3640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f029415d660 0x7f029415e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02991c4640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0294071890 0x7f0294150960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:48882/0 (socket says 172.21.15.99:48882) 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02991c4640 1 -- 172.21.15.99:0/2836603684 learned_addr learned my addr 172.21.15.99:0/2836603684 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f0293fff640 1 --2- 172.21.15.99:0/2836603684 >> v2:172.21.15.162:3300/0 conn(0x7f02940aeb20 0x7f0294071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02989c3640 1 -- 172.21.15.99:0/2836603684 >> v2:172.21.15.162:3300/0 conn(0x7f02940aeb20 msgr2=0x7f0294071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02989c3640 1 --2- 172.21.15.99:0/2836603684 >> v2:172.21.15.162:3300/0 conn(0x7f02940aeb20 0x7f0294071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02989c3640 1 -- 172.21.15.99:0/2836603684 >> v1:172.21.15.99:6789/0 conn(0x7f0294071890 legacy=0x7f0294150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02937fe640 1 -- 172.21.15.99:0/2836603684 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904472251 0 0) 0x7f029415da40 con 0x7f0294071890 2024-03-04T10:06:59.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.419+0000 7f02989c3640 1 -- 172.21.15.99:0/2836603684 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029415ebb0 con 0x7f029415d660 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.420+0000 7f02989c3640 1 --2- 172.21.15.99:0/2836603684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f029415e670 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f029406d200 tx=0x7f028402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a88a438178acbe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.420+0000 7f02937fe640 1 -- 172.21.15.99:0/2836603684 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284038070 con 0x7f029415d660 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.420+0000 7f02937fe640 1 -- 172.21.15.99:0/2836603684 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284031710 con 0x7f029415d660 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.420+0000 7f02937fe640 1 -- 172.21.15.99:0/2836603684 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284031ae0 con 0x7f029415d660 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 -- 172.21.15.99:0/2836603684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 msgr2=0x7f029415e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 --2- 172.21.15.99:0/2836603684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f029415e670 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f029406d200 tx=0x7f028402b7b0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 -- 172.21.15.99:0/2836603684 shutdown_connections 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 --2- 172.21.15.99:0/2836603684 >> v2:172.21.15.162:3300/0 conn(0x7f02940aeb20 0x7f0294071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 --2- 172.21.15.99:0/2836603684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f029415e670 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 -- 172.21.15.99:0/2836603684 >> 172.21.15.99:0/2836603684 conn(0x7f029406acd0 msgr2=0x7f029406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 -- 172.21.15.99:0/2836603684 shutdown_connections 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.421+0000 7f029ac27640 1 -- 172.21.15.99:0/2836603684 wait complete. 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.422+0000 7f029ac27640 1 Processor -- start 2024-03-04T10:06:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.422+0000 7f029ac27640 1 -- start start 2024-03-04T10:06:59.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0294071890 0x7f02940b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02940aeb20 0x7f0294141f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f02940b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029415acd0 con 0x7f029415d660 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029415a9f0 con 0x7f0294071890 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029415afb0 con 0x7f02940aeb20 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f02940b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f02940b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:52158/0 (socket says 172.21.15.99:52158) 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 -- 172.21.15.99:0/642529191 learned_addr learned my addr 172.21.15.99:0/642529191 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02989c3640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0294071890 0x7f02940b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f0293fff640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02940aeb20 0x7f0294141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 -- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02940aeb20 msgr2=0x7f0294141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02940aeb20 0x7f0294141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 -- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0294071890 msgr2=0x7f02940b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0294071890 0x7f02940b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 -- 172.21.15.99:0/642529191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294142fc0 con 0x7f029415d660 2024-03-04T10:06:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f0293fff640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02940aeb20 0x7f0294141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02991c4640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f02940b31a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f028c00cb60 tx=0x7f028c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c017070 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c00ad60 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02940cd700 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.423+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c012640 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.424+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02940cdcc0 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.424+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0294074f60 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.425+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f028c01a070 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.425+0000 7f02917fa640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f028006b4d0 0x7f028006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.425+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f028c08a1f0 con 0x7f029415d660 2024-03-04T10:06:59.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.425+0000 7f02989c3640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f028006b4d0 0x7f028006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:06:59.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.426+0000 7f02989c3640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f028006b4d0 0x7f028006d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0284035000 tx=0x7f0284008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:06:59.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.429+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f028c058e90 con 0x7f029415d660 2024-03-04T10:06:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.524+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.524+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.524+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T10:06:59.615 INFO:tasks.ceph.mds.c.smithi099.stderr:daemon-helper: command crashed with signal 6 2024-03-04T10:06:59.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.637+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0294072460 con 0x7f029415d660 2024-03-04T10:06:59.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.638+0000 7f02917fa640 1 -- 172.21.15.99:0/642529191 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f028c058790 con 0x7f029415d660 2024-03-04T10:06:59.639 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:06:59.640 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":51,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:58.183569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:06:59.640 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 51 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f028006b4d0 msgr2=0x7f028006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f028006b4d0 0x7f028006d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0284035000 tx=0x7f0284008030 comp rx=0 tx=0).stop 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 msgr2=0x7f02940b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f02940b31a0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f028c00cb60 tx=0x7f028c00a430 comp rx=0 tx=0).stop 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 shutdown_connections 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f028006b4d0 0x7f028006d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f029415d660 0x7f02940b31a0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f02940aeb20 0x7f0294141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 --2- 172.21.15.99:0/642529191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0294071890 0x7f02940b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:06:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 >> 172.21.15.99:0/642529191 conn(0x7f029406acd0 msgr2=0x7f02940b02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:06:59.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 shutdown_connections 2024-03-04T10:06:59.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.640+0000 7f029ac27640 1 -- 172.21.15.99:0/642529191 wait complete. 2024-03-04T10:06:59.652 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-04T10:06:59.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.925+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:06:59.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.925+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T10:06:59.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:06:59.925+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:00.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:00.132+0000 7f46b2ffd640 1 -- 172.21.15.99:0/391659235 <== mon.0 v2:172.21.15.99:3300/0 8 ==== log(2 entries from seq 693 at 2024-03-04T10:06:59.127872+0000) v1 ==== 635+0+0 (secure 0 0 0) 0x7f46b4056e60 con 0x7f46c4071270 2024-03-04T10:07:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: cluster: 2024-03-04T10:07:00.133 INFO:teuthology.orchestra.run.smithi099.stdout: id: 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: health: HEALTH_OK 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: services: 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: mon: 3 daemons, quorum a,b,c (age 3m) 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: mgr: x(active, since 3m), standbys: y 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: mds: 1/1 daemons up, 3 standby 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: data: 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: volumes: 1/1 healthy 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: pools: 3 pools, 97 pgs 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: objects: 29 objects, 602 KiB 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: usage: 219 MiB used, 720 GiB / 720 GiB avail 2024-03-04T10:07:00.134 INFO:teuthology.orchestra.run.smithi099.stdout: pgs: 97 active+clean 2024-03-04T10:07:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: io: 2024-03-04T10:07:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: client: 8.6 KiB/s rd, 3.5 KiB/s wr, 7 op/s rd, 4 op/s wr 2024-03-04T10:07:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:00.135 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:00.135 INFO:teuthology.orchestra.run.smithi099.stdout:2024-03-04T10:06:59.297008+0000 mds.c [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x557145c29180] 2024-03-04T10:07:00.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:00.726+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:00.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:00.726+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T10:07:00.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:00.726+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:02.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:02.327+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:02.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:02.327+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T10:07:02.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:02.327+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:04.654 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:07:04.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.766+0000 7fa54f068640 1 Processor -- start 2024-03-04T10:07:04.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.766+0000 7fa54f068640 1 -- start start 2024-03-04T10:07:04.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54f068640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa548154a80 0x7fa54815e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54f068640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa5480cf650 0x7fa5480cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54f068640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5481616c0 0x7fa548161b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54f068640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5480cdaa0 con 0x7fa548154a80 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54f068640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5480cdd80 con 0x7fa5480cf650 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54f068640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480ce060 con 0x7fa5481616c0 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa548154a80 0x7fa54815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa548154a80 0x7fa54815e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:34282/0 (socket says 172.21.15.99:34282) 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 -- 172.21.15.99:0/4155082954 learned_addr learned my addr 172.21.15.99:0/4155082954 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54d605640 1 --2- 172.21.15.99:0/4155082954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5481616c0 0x7fa548161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa547fff640 1 --2- 172.21.15.99:0/4155082954 >> v2:172.21.15.162:3300/0 conn(0x7fa5480cf650 0x7fa5480cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 -- 172.21.15.99:0/4155082954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5481616c0 msgr2=0x7fa548161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:04.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 --2- 172.21.15.99:0/4155082954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5481616c0 0x7fa548161b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 -- 172.21.15.99:0/4155082954 >> v2:172.21.15.162:3300/0 conn(0x7fa5480cf650 msgr2=0x7fa5480cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:04.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 --2- 172.21.15.99:0/4155082954 >> v2:172.21.15.162:3300/0 conn(0x7fa5480cf650 0x7fa5480cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.767+0000 7fa54ce04640 1 -- 172.21.15.99:0/4155082954 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5480ce340 con 0x7fa548154a80 2024-03-04T10:07:04.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.768+0000 7fa54ce04640 1 --2- 172.21.15.99:0/4155082954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa548154a80 0x7fa54815e910 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa548067960 tx=0x7fa53c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55ce729172827c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:04.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.768+0000 7fa5477fe640 1 -- 172.21.15.99:0/4155082954 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c035020 con 0x7fa548154a80 2024-03-04T10:07:04.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.768+0000 7fa5477fe640 1 -- 172.21.15.99:0/4155082954 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c031870 con 0x7fa548154a80 2024-03-04T10:07:04.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.768+0000 7fa5477fe640 1 -- 172.21.15.99:0/4155082954 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c031c40 con 0x7fa548154a80 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 -- 172.21.15.99:0/4155082954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa548154a80 msgr2=0x7fa54815e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 --2- 172.21.15.99:0/4155082954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa548154a80 0x7fa54815e910 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa548067960 tx=0x7fa53c02b910 comp rx=0 tx=0).stop 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 -- 172.21.15.99:0/4155082954 shutdown_connections 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 --2- 172.21.15.99:0/4155082954 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa5481616c0 0x7fa548161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 --2- 172.21.15.99:0/4155082954 >> v2:172.21.15.162:3300/0 conn(0x7fa5480cf650 0x7fa5480cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 --2- 172.21.15.99:0/4155082954 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa548154a80 0x7fa54815e910 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 -- 172.21.15.99:0/4155082954 >> 172.21.15.99:0/4155082954 conn(0x7fa54806f5f0 msgr2=0x7fa54806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 -- 172.21.15.99:0/4155082954 shutdown_connections 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.769+0000 7fa54f068640 1 -- 172.21.15.99:0/4155082954 wait complete. 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.770+0000 7fa54f068640 1 Processor -- start 2024-03-04T10:07:04.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.770+0000 7fa54f068640 1 -- start start 2024-03-04T10:07:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54f068640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5480cf650 0x7fa5480b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54f068640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa548154a80 0x7fa548153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54f068640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 0x7fa5480b5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54f068640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548154fe0 con 0x7fa5481616c0 2024-03-04T10:07:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54f068640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548154e60 con 0x7fa5480cf650 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54f068640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5480ceb40 con 0x7fa548154a80 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 0x7fa5480b5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 0x7fa5480b5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:34286/0 (socket says 172.21.15.99:34286) 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 -- 172.21.15.99:0/342347224 learned_addr learned my addr 172.21.15.99:0/342347224 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54ce04640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5480cf650 0x7fa5480b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa547fff640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa548154a80 0x7fa548153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 -- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa548154a80 msgr2=0x7fa548153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa548154a80 0x7fa548153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 -- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5480cf650 msgr2=0x7fa5480b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5480cf650 0x7fa5480b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.771+0000 7fa54d605640 1 -- 172.21.15.99:0/342347224 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa548153d70 con 0x7fa5481616c0 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.772+0000 7fa54d605640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 0x7fa5480b5550 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa540009dd0 tx=0x7fa54000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:04.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.772+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540018020 con 0x7fa5481616c0 2024-03-04T10:07:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.772+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa540014480 con 0x7fa5481616c0 2024-03-04T10:07:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.772+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5480b5ec0 con 0x7fa5481616c0 2024-03-04T10:07:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.772+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540014870 con 0x7fa5481616c0 2024-03-04T10:07:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.772+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5480b6480 con 0x7fa5481616c0 2024-03-04T10:07:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.773+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa548074f60 con 0x7fa5481616c0 2024-03-04T10:07:04.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.773+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa540005d10 con 0x7fa5481616c0 2024-03-04T10:07:04.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.774+0000 7fa5457fa640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa52c06b4d0 0x7fa52c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:04.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.774+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa54008ae60 con 0x7fa5481616c0 2024-03-04T10:07:04.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.774+0000 7fa54ce04640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa52c06b4d0 0x7fa52c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:04.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.774+0000 7fa54ce04640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa52c06b4d0 0x7fa52c06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa53c036000 tx=0x7fa53c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:04.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.778+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa540022070 con 0x7fa5481616c0 2024-03-04T10:07:04.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.985+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5480cfb80 con 0x7fa5481616c0 2024-03-04T10:07:04.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.986+0000 7fa5457fa640 1 -- 172.21.15.99:0/342347224 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4943 (secure 0 0 0) 0x7fa540010040 con 0x7fa5481616c0 2024-03-04T10:07:04.987 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:04.987 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":51,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:58.183569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:07:04.988 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 51 2024-03-04T10:07:04.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa52c06b4d0 msgr2=0x7fa52c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:04.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa52c06b4d0 0x7fa52c06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa53c036000 tx=0x7fa53c03c040 comp rx=0 tx=0).stop 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 msgr2=0x7fa5480b5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 0x7fa5480b5550 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa540009dd0 tx=0x7fa54000c490 comp rx=0 tx=0).stop 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 shutdown_connections 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa52c06b4d0 0x7fa52c06d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa5481616c0 0x7fa5480b5550 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa548154a80 0x7fa548153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 --2- 172.21.15.99:0/342347224 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa5480cf650 0x7fa5480b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 >> 172.21.15.99:0/342347224 conn(0x7fa54806f5f0 msgr2=0x7fa5480b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 shutdown_connections 2024-03-04T10:07:04.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:04.988+0000 7fa54f068640 1 -- 172.21.15.99:0/342347224 wait complete. 2024-03-04T10:07:05.000 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-04T10:07:05.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:05.530+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:05.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:05.530+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-04T10:07:05.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:05.530+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:05.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-04T10:07:09.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:09.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:10.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:07:10.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.111+0000 7f4163f24640 1 Processor -- start 2024-03-04T10:07:10.112 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.111+0000 7f4163f24640 1 -- start start 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4163f24640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f415c15de00 0x7f415c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4163f24640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 0x7f415c0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4163f24640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c15e1e0 con 0x7f415c0b70d0 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4163f24640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415c0d2820 con 0x7f415c15de00 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4163f24640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0d1c10 con 0x7f415c0b9380 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f41624c1640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f415c0b70d0 0x7f415c0b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:50682/0 (socket says 172.21.15.99:50682) 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f41624c1640 1 -- 172.21.15.99:0/2095843188 learned_addr learned my addr 172.21.15.99:0/2095843188 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2095843188 >> v2:172.21.15.162:3300/0 conn(0x7f415c15de00 0x7f415c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f41614bf640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 0x7f415c0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860792425 0 0) 0x7f415c15e1e0 con 0x7f415c0b70d0 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4148003520 con 0x7f415c0b70d0 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 753430680 0 0) 0x7f4148003520 con 0x7f415c0b70d0 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 msgr2=0x7f415c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 0x7f415c0b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 >> v2:172.21.15.162:3300/0 conn(0x7f415c15de00 msgr2=0x7f415c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 --2- 172.21.15.99:0/2095843188 >> v2:172.21.15.162:3300/0 conn(0x7f415c15de00 0x7f415c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0d1ef0 con 0x7f415c0b70d0 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f41580019f0 con 0x7f415c0b70d0 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 >> v1:172.21.15.99:6789/0 conn(0x7f415c0b70d0 legacy=0x7f415c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41480039b0 0x7f4148003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 0x7f4148004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.112+0000 7f4160cbe640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4148004d40 0x7f41480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0d2820 con 0x7f41480039b0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0d1c10 con 0x7f4148004d40 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414800b9c0 con 0x7f415c0b9380 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41480039b0 0x7f4148003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f41624c1640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4148004d40 0x7f41480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f41614bf640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 0x7f4148004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4161cc0640 1 -- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 msgr2=0x7f4148004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0b9380 0x7f4148004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4161cc0640 1 -- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4148004d40 msgr2=0x7f41480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4148004d40 0x7f41480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.113+0000 7f4161cc0640 1 -- 172.21.15.99:0/2095843188 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4148003520 con 0x7f41480039b0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41480039b0 0x7f4148003dd0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4154009ac0 tx=0x7f4148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b9938d762717c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154039070 con 0x7f41480039b0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415402c710 con 0x7f41480039b0 2024-03-04T10:07:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4160cbe640 1 -- 172.21.15.99:0/2095843188 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154009690 con 0x7f41480039b0 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4163f24640 1 -- 172.21.15.99:0/2095843188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41480039b0 msgr2=0x7f4148003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4163f24640 1 --2- 172.21.15.99:0/2095843188 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41480039b0 0x7f4148003dd0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f4154009ac0 tx=0x7f4148005830 comp rx=0 tx=0).stop 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f41624c1640 1 -- 172.21.15.99:0/2095843188 reap_dead start 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4163f24640 1 -- 172.21.15.99:0/2095843188 shutdown_connections 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4163f24640 1 -- 172.21.15.99:0/2095843188 >> 172.21.15.99:0/2095843188 conn(0x7f415c0d0470 msgr2=0x7f415c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4163f24640 1 -- 172.21.15.99:0/2095843188 shutdown_connections 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.114+0000 7f4163f24640 1 -- 172.21.15.99:0/2095843188 wait complete. 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 Processor -- start 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 -- start start 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41480039b0 0x7f415c1f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0cd5a0 0x7f415c0cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 0x7f415c0cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0b97e0 con 0x7f415c0b1f50 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0d24b0 con 0x7f41480039b0 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f4163f24640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0b5b80 con 0x7f415c0cd5a0 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f41624c1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 0x7f415c0cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f41624c1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 0x7f415c0cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:34310/0 (socket says 172.21.15.99:34310) 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.116+0000 7f41624c1640 1 -- 172.21.15.99:0/2254360356 learned_addr learned my addr 172.21.15.99:0/2254360356 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41480039b0 0x7f415c1f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41614bf640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0cd5a0 0x7f415c0cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41624c1640 1 -- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0cd5a0 msgr2=0x7f415c0cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41624c1640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0cd5a0 0x7f415c0cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41624c1640 1 -- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41480039b0 msgr2=0x7f415c1f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41624c1640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41480039b0 0x7f415c1f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41624c1640 1 -- 172.21.15.99:0/2254360356 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c1f4460 con 0x7f415c0b1f50 2024-03-04T10:07:10.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41480039b0 0x7f415c1f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41624c1640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 0x7f415c0cb860 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f415c06dc00 tx=0x7f415800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415800ec10 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4158018070 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c15a7a0 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4158013420 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.117+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c0b4da0 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.118+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c074f60 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.119+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f41580135c0 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.119+0000 7f41527fc640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f414006b500 0x7f414006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.119+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f415808a6c0 con 0x7f415c0b1f50 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.119+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f414006b500 0x7f414006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:10.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.120+0000 7f4161cc0640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f414006b500 0x7f414006d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4154036000 tx=0x7f4154030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:10.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.124+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f415c1f4460 con 0x7f415c0b1f50 2024-03-04T10:07:10.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.329+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f415c0cf680 con 0x7f415c0b1f50 2024-03-04T10:07:10.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.330+0000 7f41527fc640 1 -- 172.21.15.99:0/2254360356 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+4943 (secure 0 0 0) 0x7f41580593f0 con 0x7f415c0b1f50 2024-03-04T10:07:10.331 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:10.332 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":51,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:06:58.183569+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:07:10.332 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 51 2024-03-04T10:07:10.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f414006b500 msgr2=0x7f414006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f414006b500 0x7f414006d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4154036000 tx=0x7f4154030000 comp rx=0 tx=0).stop 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 msgr2=0x7f415c0cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 0x7f415c0cb860 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f415c06dc00 tx=0x7f415800e530 comp rx=0 tx=0).stop 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 shutdown_connections 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f414006b500 0x7f414006d9c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f415c0b1f50 0x7f415c0cb860 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f415c0cd5a0 0x7f415c0cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 --2- 172.21.15.99:0/2254360356 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41480039b0 0x7f415c1f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 >> 172.21.15.99:0/2254360356 conn(0x7f415c0d0470 msgr2=0x7f415c0b8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 shutdown_connections 2024-03-04T10:07:10.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:10.332+0000 7f4163f24640 1 -- 172.21.15.99:0/2254360356 wait complete. 2024-03-04T10:07:10.344 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-04T10:07:11.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:11.937+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:11.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:11.937+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-04T10:07:11.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:11.937+0000 7f61f67fc640 1 client.6834 ms_handle_refused on v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:12.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~7s 2024-03-04T10:07:15.345 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:07:15.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.456+0000 7f16cf390640 1 Processor -- start 2024-03-04T10:07:15.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.456+0000 7f16cf390640 1 -- start start 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cf390640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f16c80b39c0 0x7f16c8161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cf390640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f16c8161ca0 0x7f16c806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cf390640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c806b500 con 0x7f16c8158410 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cf390640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c81620f0 con 0x7f16c80b39c0 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cf390640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c806b7e0 con 0x7f16c8161ca0 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f16c80b39c0 0x7f16c8161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f16c80b39c0 0x7f16c8161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49128/0 (socket says 172.21.15.99:49128) 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 -- 172.21.15.99:0/1905745131 learned_addr learned my addr 172.21.15.99:0/1905745131 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cd92d640 1 --2- 172.21.15.99:0/1905745131 >> v2:172.21.15.162:3300/0 conn(0x7f16c8161ca0 0x7f16c806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 -- 172.21.15.99:0/1905745131 >> v2:172.21.15.162:3300/0 conn(0x7f16c8161ca0 msgr2=0x7f16c806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 --2- 172.21.15.99:0/1905745131 >> v2:172.21.15.162:3300/0 conn(0x7f16c8161ca0 0x7f16c806de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 -- 172.21.15.99:0/1905745131 >> v1:172.21.15.162:6789/0 conn(0x7f16c8158410 legacy=0x7f16c806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.457+0000 7f16cc92b640 1 -- 172.21.15.99:0/1905745131 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c806bac0 con 0x7f16c80b39c0 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.458+0000 7f16cc92b640 1 --2- 172.21.15.99:0/1905745131 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c80b39c0 0x7f16c8161760 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f16c80af960 tx=0x7f16b802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab0b57da3f65aa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.458+0000 7f16bffff640 1 -- 172.21.15.99:0/1905745131 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b802f850 con 0x7f16c80b39c0 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.458+0000 7f16bffff640 1 -- 172.21.15.99:0/1905745131 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b802f9f0 con 0x7f16c80b39c0 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.458+0000 7f16bffff640 1 -- 172.21.15.99:0/1905745131 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b802fdc0 con 0x7f16c80b39c0 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 -- 172.21.15.99:0/1905745131 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c80b39c0 msgr2=0x7f16c8161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 --2- 172.21.15.99:0/1905745131 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c80b39c0 0x7f16c8161760 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f16c80af960 tx=0x7f16b802ba30 comp rx=0 tx=0).stop 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 -- 172.21.15.99:0/1905745131 shutdown_connections 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 --2- 172.21.15.99:0/1905745131 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c80b39c0 0x7f16c8161760 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 --2- 172.21.15.99:0/1905745131 >> v2:172.21.15.162:3300/0 conn(0x7f16c8161ca0 0x7f16c806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 -- 172.21.15.99:0/1905745131 >> 172.21.15.99:0/1905745131 conn(0x7f16c80cfef0 msgr2=0x7f16c80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 -- 172.21.15.99:0/1905745131 shutdown_connections 2024-03-04T10:07:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.459+0000 7f16cf390640 1 -- 172.21.15.99:0/1905745131 wait complete. 2024-03-04T10:07:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.460+0000 7f16cf390640 1 Processor -- start 2024-03-04T10:07:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.460+0000 7f16cf390640 1 -- start start 2024-03-04T10:07:15.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cf390640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c80b39c0 0x7f16c8146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cf390640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 0x7f16c81f7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cf390640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16c8161ca0 0x7f16c81f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cf390640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c80b3e50 con 0x7f16c8158410 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cf390640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c806c080 con 0x7f16c80b39c0 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cf390640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c806c5a0 con 0x7f16c8161ca0 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 0x7f16c81f7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 0x7f16c81f7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49130/0 (socket says 172.21.15.99:49130) 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 -- 172.21.15.99:0/996550191 learned_addr learned my addr 172.21.15.99:0/996550191 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cd92d640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16c8161ca0 0x7f16c81f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cd12c640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c80b39c0 0x7f16c8146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 -- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16c8161ca0 msgr2=0x7f16c81f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16c8161ca0 0x7f16c81f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 -- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c80b39c0 msgr2=0x7f16c8146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c80b39c0 0x7f16c8146a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 -- 172.21.15.99:0/996550191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c80ce1f0 con 0x7f16c8158410 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.461+0000 7f16cc92b640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 0x7f16c81f7010 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f16b8000c00 tx=0x7f16b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.462+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8047070 con 0x7f16c8158410 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.462+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8037cb0 con 0x7f16c8158410 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.462+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8037e60 con 0x7f16c8158410 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.462+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c80ce4d0 con 0x7f16c8158410 2024-03-04T10:07:15.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.462+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c80cea90 con 0x7f16c8158410 2024-03-04T10:07:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.463+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c8074f60 con 0x7f16c8158410 2024-03-04T10:07:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.463+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f16b804a070 con 0x7f16c8158410 2024-03-04T10:07:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.464+0000 7f16bdffb640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.464+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f16b80ac700 con 0x7f16c8158410 2024-03-04T10:07:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.464+0000 7f16cd12c640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.465+0000 7f16cd12c640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f16ac06b4d0 0x7f16ac06d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f16c4001780 tx=0x7f16c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:15.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.468+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f16b8033030 con 0x7f16c8158410 2024-03-04T10:07:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.675+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16c8066e10 con 0x7f16c8158410 2024-03-04T10:07:15.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.676+0000 7f16bdffb640 1 -- 172.21.15.99:0/996550191 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4991 (secure 0 0 0) 0x7f16b807ad50 con 0x7f16c8158410 2024-03-04T10:07:15.677 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:15.677 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":52,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6471,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:14.945626+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"c","rank":0,"incarnation":47,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/3022309305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3022309305},{"type":"v1","addr":"172.21.15.99:6835","nonce":3022309305}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:07:14.945626+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:07:15.678 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 52 2024-03-04T10:07:15.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f16ac06b4d0 msgr2=0x7f16ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f16ac06b4d0 0x7f16ac06d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f16c4001780 tx=0x7f16c4009040 comp rx=0 tx=0).stop 2024-03-04T10:07:15.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 msgr2=0x7f16c81f7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 0x7f16c81f7010 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f16b8000c00 tx=0x7f16b8004730 comp rx=0 tx=0).stop 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 shutdown_connections 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f16ac06b4d0 0x7f16ac06d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16c8161ca0 0x7f16c81f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f16c8158410 0x7f16c81f7010 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 --2- 172.21.15.99:0/996550191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c80b39c0 0x7f16c8146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 >> 172.21.15.99:0/996550191 conn(0x7f16c80cfef0 msgr2=0x7f16c80b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 shutdown_connections 2024-03-04T10:07:15.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.678+0000 7f16cf390640 1 -- 172.21.15.99:0/996550191 wait complete. 2024-03-04T10:07:15.690 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-03-04T10:07:15.791 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-03-04T10:07:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.891+0000 7fda27ff1640 1 Processor -- start 2024-03-04T10:07:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.892+0000 7fda27ff1640 1 -- start start 2024-03-04T10:07:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.892+0000 7fda27ff1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b5700 0x7fda200d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.892+0000 7fda27ff1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fda200b9410 0x7fda2006c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.892+0000 7fda27ff1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda200d2400 con 0x7fda200d34a0 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.892+0000 7fda27ff1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200d26e0 con 0x7fda200b9410 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.892+0000 7fda27ff1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200d29c0 con 0x7fda200b5700 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda25d8d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b5700 0x7fda200d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2658e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fda200b9410 0x7fda2006c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2558c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fda200d34a0 0x7fda2006c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:57616/0 (socket says 172.21.15.99:57616) 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda25d8d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b5700 0x7fda200d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:42872/0 (socket says 172.21.15.99:42872) 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2658e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fda200b9410 0x7fda2006c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:56044/0 (socket says 172.21.15.99:56044) 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2558c640 1 -- 172.21.15.99:0/147946247 learned_addr learned my addr 172.21.15.99:0/147946247 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2658e640 1 -- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b5700 msgr2=0x7fda200d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2658e640 1 --2- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b5700 0x7fda200d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2658e640 1 -- 172.21.15.99:0/147946247 >> v1:172.21.15.162:6789/0 conn(0x7fda200d34a0 legacy=0x7fda2006c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.893+0000 7fda2658e640 1 -- 172.21.15.99:0/147946247 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200d2ca0 con 0x7fda200b9410 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.894+0000 7fda2658e640 1 --2- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200b9410 0x7fda2006c7f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fda18009d90 tx=0x7fda1802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5bfa7381c0d7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.894+0000 7fda24d8b640 1 -- 172.21.15.99:0/147946247 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18039070 con 0x7fda200b9410 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.894+0000 7fda24d8b640 1 -- 172.21.15.99:0/147946247 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1802cbc0 con 0x7fda200b9410 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.894+0000 7fda24d8b640 1 -- 172.21.15.99:0/147946247 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18034440 con 0x7fda200b9410 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 -- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200b9410 msgr2=0x7fda2006c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 --2- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200b9410 0x7fda2006c7f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fda18009d90 tx=0x7fda1802be10 comp rx=0 tx=0).stop 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 -- 172.21.15.99:0/147946247 shutdown_connections 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 --2- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200b9410 0x7fda2006c7f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 --2- 172.21.15.99:0/147946247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b5700 0x7fda200d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 -- 172.21.15.99:0/147946247 >> 172.21.15.99:0/147946247 conn(0x7fda200d0490 msgr2=0x7fda200d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 -- 172.21.15.99:0/147946247 shutdown_connections 2024-03-04T10:07:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.895+0000 7fda27ff1640 1 -- 172.21.15.99:0/147946247 wait complete. 2024-03-04T10:07:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.896+0000 7fda27ff1640 1 Processor -- start 2024-03-04T10:07:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.896+0000 7fda27ff1640 1 -- start start 2024-03-04T10:07:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 0x7fda2015e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b9410 0x7fda200ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200d34a0 0x7fda200cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2006ceb0 con 0x7fda200b5700 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2006cd30 con 0x7fda200d34a0 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200b5ae0 con 0x7fda200b9410 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 0x7fda2015e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 0x7fda2015e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49144/0 (socket says 172.21.15.99:49144) 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 -- 172.21.15.99:0/3933187316 learned_addr learned my addr 172.21.15.99:0/3933187316 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda2558c640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b9410 0x7fda200ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda2658e640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200d34a0 0x7fda200cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 -- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b9410 msgr2=0x7fda200ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b9410 0x7fda200ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 -- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200d34a0 msgr2=0x7fda200cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200d34a0 0x7fda200cea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 -- 172.21.15.99:0/3933187316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200aa7d0 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda2658e640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200d34a0 0x7fda200cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda25d8d640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 0x7fda2015e4f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fda1000e9b0 tx=0x7fda1000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10017070 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1000c9a0 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200aaab0 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1000cd90 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.897+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200ab070 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.898+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda200674d0 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.899+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fda1001a070 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.899+0000 7fda167fc640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0406b4d0 0x7fda0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.899+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fda1008a840 con 0x7fda200b5700 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.899+0000 7fda2558c640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0406b4d0 0x7fda0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.900+0000 7fda2558c640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0406b4d0 0x7fda0406d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fda200ad920 tx=0x7fda1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:15.903+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fda200674d0 con 0x7fda200b5700 2024-03-04T10:07:16.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.076+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) v1 -- 0x7fda20161700 con 0x7fda200b5700 2024-03-04T10:07:16.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.079+0000 7fda167fc640 1 -- 172.21.15.99:0/3933187316 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda10059460 con 0x7fda200b5700 2024-03-04T10:07:16.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0406b4d0 msgr2=0x7fda0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:16.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0406b4d0 0x7fda0406d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fda200ad920 tx=0x7fda1c009040 comp rx=0 tx=0).stop 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 msgr2=0x7fda2015e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 0x7fda2015e4f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fda1000e9b0 tx=0x7fda1000ee70 comp rx=0 tx=0).stop 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 shutdown_connections 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0406b4d0 0x7fda0406d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200d34a0 0x7fda200cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200b9410 0x7fda200ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 --2- 172.21.15.99:0/3933187316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda200b5700 0x7fda2015e4f0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 >> 172.21.15.99:0/3933187316 conn(0x7fda200d0490 msgr2=0x7fda20160240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 shutdown_connections 2024-03-04T10:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.081+0000 7fda27ff1640 1 -- 172.21.15.99:0/3933187316 wait complete. 2024-03-04T10:07:16.093 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 4:0 false 2024-03-04T10:07:16.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 Processor -- start 2024-03-04T10:07:16.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 -- start start 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f83580ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f835815e810 0x7f835806c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f835815bae0 con 0x7f835806bce0 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835806c680 con 0x7f835815e810 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.195+0000 7f835e748640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83580adb90 con 0x7f835815b700 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8357fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f83580ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f835cce5640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f835815e810 0x7f835806c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f83577fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f835806bce0 0x7f835815e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:57624/0 (socket says 172.21.15.99:57624) 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f83577fe640 1 -- 172.21.15.99:0/1723257966 learned_addr learned my addr 172.21.15.99:0/1723257966 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8357fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f83580ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:42896/0 (socket says 172.21.15.99:42896) 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8357fff640 1 -- 172.21.15.99:0/1723257966 >> v2:172.21.15.162:3300/0 conn(0x7f835815e810 msgr2=0x7f835806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8357fff640 1 --2- 172.21.15.99:0/1723257966 >> v2:172.21.15.162:3300/0 conn(0x7f835815e810 0x7f835806c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8357fff640 1 -- 172.21.15.99:0/1723257966 >> v1:172.21.15.162:6789/0 conn(0x7f835806bce0 legacy=0x7f835815e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8356ffd640 1 -- 172.21.15.99:0/1723257966 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572676523 0 0) 0x7f835815bae0 con 0x7f835806bce0 2024-03-04T10:07:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f8357fff640 1 -- 172.21.15.99:0/1723257966 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835815ee60 con 0x7f835815b700 2024-03-04T10:07:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.196+0000 7f835cce5640 1 --2- 172.21.15.99:0/1723257966 >> v2:172.21.15.162:3300/0 conn(0x7f835815e810 0x7f835806c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.197+0000 7f8357fff640 1 --2- 172.21.15.99:0/1723257966 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f83580ae020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83480097a0 tx=0x7f834802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0c67b7b0a9254a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.197+0000 7f8356ffd640 1 -- 172.21.15.99:0/1723257966 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348035020 con 0x7f835815b700 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.197+0000 7f8356ffd640 1 -- 172.21.15.99:0/1723257966 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8348031830 con 0x7f835815b700 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.197+0000 7f8356ffd640 1 -- 172.21.15.99:0/1723257966 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348031c00 con 0x7f835815b700 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 -- 172.21.15.99:0/1723257966 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 msgr2=0x7f83580ae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 --2- 172.21.15.99:0/1723257966 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f83580ae020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f83480097a0 tx=0x7f834802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 -- 172.21.15.99:0/1723257966 shutdown_connections 2024-03-04T10:07:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 --2- 172.21.15.99:0/1723257966 >> v2:172.21.15.162:3300/0 conn(0x7f835815e810 0x7f835806c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 --2- 172.21.15.99:0/1723257966 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f83580ae020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 -- 172.21.15.99:0/1723257966 >> 172.21.15.99:0/1723257966 conn(0x7f8358071bc0 msgr2=0x7f8358071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:16.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 -- 172.21.15.99:0/1723257966 shutdown_connections 2024-03-04T10:07:16.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.198+0000 7f835e748640 1 -- 172.21.15.99:0/1723257966 wait complete. 2024-03-04T10:07:16.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.199+0000 7f835e748640 1 Processor -- start 2024-03-04T10:07:16.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.199+0000 7f835e748640 1 -- start start 2024-03-04T10:07:16.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835e748640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f835806bce0 0x7f83580c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835e748640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f835814cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835e748640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 0x7f835806f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835e748640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83580ae560 con 0x7f835815e810 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835e748640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83580ad820 con 0x7f835806bce0 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835e748640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83580ae6e0 con 0x7f835815b700 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 0x7f835806f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 0x7f835806f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49148/0 (socket says 172.21.15.99:49148) 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 -- 172.21.15.99:0/2867643062 learned_addr learned my addr 172.21.15.99:0/2867643062 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f83577fe640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f835814cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f8357fff640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f835806bce0 0x7f83580c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 -- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 msgr2=0x7f835814cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f835814cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 -- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f835806bce0 msgr2=0x7f83580c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f835806bce0 0x7f83580c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 -- 172.21.15.99:0/2867643062 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580c53b0 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.200+0000 7f835cce5640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 0x7f835806f3f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f835000efc0 tx=0x7f835000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.201+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350018030 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.201+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835000cdf0 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.201+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83580c5690 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.201+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350003bb0 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.201+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83580c5c50 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.202+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8358074f60 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.202+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f835001b070 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.202+0000 7f8354ff9640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.202+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f835008ab90 con 0x7f835815e810 2024-03-04T10:07:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.203+0000 7f8357fff640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:16.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.203+0000 7f8357fff640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8348036000 tx=0x7f834803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:16.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.207+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8350010040 con 0x7f835815e810 2024-03-04T10:07:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:16.443+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"} v 0) v1 -- 0x7f83580b1500 con 0x7f835815e810 2024-03-04T10:07:17.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.158+0000 7f8354ff9640 1 -- 172.21.15.99:0/2867643062 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "4:0", "val": "false"}]=0 unfreezing mds.6729 v53) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8350059270 con 0x7f835815e810 2024-03-04T10:07:17.160 INFO:teuthology.orchestra.run.smithi099.stderr:unfreezing mds.6729 2024-03-04T10:07:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.160+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f834406b4d0 msgr2=0x7f834406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.160+0000 7f835e748640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f834406b4d0 0x7f834406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8348036000 tx=0x7f834803d040 comp rx=0 tx=0).stop 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.160+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 msgr2=0x7f835806f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.160+0000 7f835e748640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 0x7f835806f3f0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f835000efc0 tx=0x7f835000c490 comp rx=0 tx=0).stop 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.160+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 shutdown_connections 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.160+0000 7f835e748640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f834406b4d0 0x7f834406d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.161+0000 7f835e748640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f835815e810 0x7f835806f3f0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.161+0000 7f835e748640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f835815b700 0x7f835814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.161+0000 7f835e748640 1 --2- 172.21.15.99:0/2867643062 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f835806bce0 0x7f83580c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.161+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 >> 172.21.15.99:0/2867643062 conn(0x7f8358071bc0 msgr2=0x7f835815aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.161+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 shutdown_connections 2024-03-04T10:07:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.161+0000 7f835e748640 1 -- 172.21.15.99:0/2867643062 wait complete. 2024-03-04T10:07:17.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -n kernel.core_pattern 2024-03-04T10:07:17.205 INFO:teuthology.orchestra.run.smithi099.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T10:07:17.206 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:07:17.207 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-03-04T10:07:17.277 INFO:teuthology.orchestra.run.smithi099.stdout:1709546819.60247.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i c', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/proc/self/exe', platform: 'x86_64' 2024-03-04T10:07:17.279 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1709546819.60247.core'] 2024-03-04T10:07:17.279 INFO:tasks.cephfs.cephfs_test_case:Found core file 1709546819.60247.core, deleting it 2024-03-04T10:07:17.280 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1709546819.60247.core 2024-03-04T10:07:17.348 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:07:17.348 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T10:07:17.348 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:07:17.349 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:07:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:07:17.349 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-03-04T10:07:17.350 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:07:17.350 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:07:17.391 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:07:17.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:07:17.466 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:07:17.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.491+0000 7f2c05a12640 1 Processor -- start 2024-03-04T10:07:17.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.491+0000 7f2c05a12640 1 -- start start 2024-03-04T10:07:17.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2c05a12640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2c000d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2c05a12640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c000cc760 con 0x7f2c000af7b0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2c05a12640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c000ce720 con 0x7f2c0015ac40 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2c05a12640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0015b020 con 0x7f2c000cf080 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2bfe7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2c000af7b0 0x7f2c000ad7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35812/0 (socket says 172.21.15.99:35812) 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/2935317765 learned_addr learned my addr 172.21.15.99:0/2935317765 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.492+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701726457 0 0) 0x7f2c000cc760 con 0x7f2c000af7b0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be8003520 con 0x7f2c000af7b0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bff7fe640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2c000d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1870013375 0 0) 0x7f2be8003520 con 0x7f2c000af7b0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 msgr2=0x7f2c000d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2c000d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 >> v1:172.21.15.162:6789/0 conn(0x7f2c0015ac40 legacy=0x7f2c000ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c000cbe30 con 0x7f2c000af7b0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f2bf0001670 con 0x7f2c000af7b0 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 >> v1:172.21.15.99:6789/0 conn(0x7f2c000af7b0 legacy=0x7f2c000ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be80039e0 0x7f2be8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2be8004800 0x7f2be8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2be80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0015b020 con 0x7f2be8004800 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be800bb30 con 0x7f2be80039e0 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be800df40 con 0x7f2c000cf080 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2be8004800 0x7f2be8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfeffd640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be80039e0 0x7f2be8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bff7fe640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2be80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 msgr2=0x7f2be80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2be80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be80039e0 msgr2=0x7f2be8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be80039e0 0x7f2be8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/2935317765 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be8003520 con 0x7f2be8004800 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.493+0000 7f2bff7fe640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000cf080 0x7f2be80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.494+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2be8004800 0x7f2be8004c80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0001050 tx=0x7f2be8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff53c61a2c9aab33 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.494+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0007d30 con 0x7f2be8004800 2024-03-04T10:07:17.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.494+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf0030070 con 0x7f2be8004800 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.494+0000 7f2bfdffb640 1 -- 172.21.15.99:0/2935317765 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0005d30 con 0x7f2be8004800 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2c05a12640 1 -- 172.21.15.99:0/2935317765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2be8004800 msgr2=0x7f2be8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2c05a12640 1 --2- 172.21.15.99:0/2935317765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2be8004800 0x7f2be8004c80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0001050 tx=0x7f2be8005830 comp rx=0 tx=0).stop 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2bff7fe640 1 -- 172.21.15.99:0/2935317765 reap_dead start 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2c05a12640 1 -- 172.21.15.99:0/2935317765 shutdown_connections 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2c05a12640 1 -- 172.21.15.99:0/2935317765 >> 172.21.15.99:0/2935317765 conn(0x7f2c000d0770 msgr2=0x7f2c000d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2c05a12640 1 -- 172.21.15.99:0/2935317765 shutdown_connections 2024-03-04T10:07:17.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.495+0000 7f2c05a12640 1 -- 172.21.15.99:0/2935317765 wait complete. 2024-03-04T10:07:17.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.496+0000 7f2c05a12640 1 Processor -- start 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.496+0000 7f2c05a12640 1 -- start start 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be8004800 0x7f2c001545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 0x7f2c0015c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000b41a0 0x7f2c000b5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c000ce340 con 0x7f2c000b32b0 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c000cc3f0 con 0x7f2be8004800 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c000ce4c0 con 0x7f2c000b41a0 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 0x7f2c0015c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 0x7f2c0015c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49188/0 (socket says 172.21.15.99:49188) 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/1170826430 learned_addr learned my addr 172.21.15.99:0/1170826430 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfeffd640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be8004800 0x7f2c001545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bff7fe640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000b41a0 0x7f2c000b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000b41a0 msgr2=0x7f2c000b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000b41a0 0x7f2c000b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be8004800 msgr2=0x7f2c001545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be8004800 0x7f2c001545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 -- 172.21.15.99:0/1170826430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c000c5390 con 0x7f2c000b32b0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2bfe7fc640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 0x7f2c0015c610 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000c380 tx=0x7f2bf0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0001c80 con 0x7f2c000b32b0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf0004bf0 con 0x7f2c000b32b0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c000c5670 con 0x7f2c000b32b0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf0007070 con 0x7f2c000b32b0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.497+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c000c5c30 con 0x7f2c000b32b0 2024-03-04T10:07:17.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.498+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0006cd70 con 0x7f2c000b32b0 2024-03-04T10:07:17.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.499+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2bf0048020 con 0x7f2c000b32b0 2024-03-04T10:07:17.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.499+0000 7f2be7fff640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2bec06b4d0 0x7f2bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:17.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.499+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f2bf00ab800 con 0x7f2c000b32b0 2024-03-04T10:07:17.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.499+0000 7f2bfeffd640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2bec06b4d0 0x7f2bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:17.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.500+0000 7f2bfeffd640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2bec06b4d0 0x7f2bec06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4005590 tx=0x7f2bf4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:17.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.504+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2bf007a440 con 0x7f2c000b32b0 2024-03-04T10:07:17.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.709+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c0006cd70 con 0x7f2c000b32b0 2024-03-04T10:07:17.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.709+0000 7f2be7fff640 1 -- 172.21.15.99:0/1170826430 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4152 (secure 0 0 0) 0x7f2bf0079d40 con 0x7f2c000b32b0 2024-03-04T10:07:17.713 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:17.713 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":54,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:17.162044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:07:17.714 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 54 2024-03-04T10:07:17.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2bec06b4d0 msgr2=0x7f2bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2bec06b4d0 0x7f2bec06d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2bf4005590 tx=0x7f2bf4009040 comp rx=0 tx=0).stop 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 msgr2=0x7f2c0015c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 0x7f2c0015c610 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f2bf000c380 tx=0x7f2bf0001a40 comp rx=0 tx=0).stop 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 shutdown_connections 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2bec06b4d0 0x7f2bec06d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c000b41a0 0x7f2c000b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c000b32b0 0x7f2c0015c610 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 --2- 172.21.15.99:0/1170826430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2be8004800 0x7f2c001545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.711+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 >> 172.21.15.99:0/1170826430 conn(0x7f2c000d0770 msgr2=0x7f2c000cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:17.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.712+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 shutdown_connections 2024-03-04T10:07:17.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:17.712+0000 7f2c05a12640 1 -- 172.21.15.99:0/1170826430 wait complete. 2024-03-04T10:07:18.128 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:07:17.162044+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 68, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6471}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6471': {'gid': 6471, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.99:6833/665274836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 665274836}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 665274836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:07:18.128 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6471:up:replay 2024-03-04T10:07:19.129 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:07:19.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.239+0000 7f011668c640 1 Processor -- start 2024-03-04T10:07:19.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.239+0000 7f011668c640 1 -- start start 2024-03-04T10:07:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f011668c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 0x7f01100cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f011668c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011006fd20 con 0x7f011006b3e0 2024-03-04T10:07:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f011668c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110070000 con 0x7f011015dea0 2024-03-04T10:07:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f011668c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011006fad0 con 0x7f01100cc8d0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f010f7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f011006b3e0 0x7f01100cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35828/0 (socket says 172.21.15.99:35828) 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f010f7fe640 1 -- 172.21.15.99:0/34730600 learned_addr learned my addr 172.21.15.99:0/34730600 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718575301 0 0) 0x7f011006fd20 con 0x7f011006b3e0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00f8003520 con 0x7f011006b3e0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f0114c29640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 0x7f01100cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.240+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 476487662 0 0) 0x7f00f8003520 con 0x7f011006b3e0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 msgr2=0x7f01100cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 0x7f01100cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 >> v1:172.21.15.162:6789/0 conn(0x7f011015dea0 legacy=0x7f011006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01100702e0 con 0x7f011006b3e0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f0104001690 con 0x7f011006b3e0 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 >> v1:172.21.15.99:6789/0 conn(0x7f011006b3e0 legacy=0x7f01100cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f00f80039e0 0x7f00f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00f8004800 0x7f00f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 0x7f00f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011006fad0 con 0x7f00f80039e0 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f800ba70 con 0x7f00f8004800 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f800bc70 con 0x7f01100cc8d0 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010ffff640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f00f80039e0 0x7f00f8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010f7fe640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00f8004800 0x7f00f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f0114c29640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 0x7f00f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010ffff640 1 -- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 msgr2=0x7f00f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010ffff640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01100cc8d0 0x7f00f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010ffff640 1 -- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00f8004800 msgr2=0x7f00f8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010ffff640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f00f8004800 0x7f00f8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.241+0000 7f010ffff640 1 -- 172.21.15.99:0/34730600 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8003520 con 0x7f00f80039e0 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f010ffff640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f00f80039e0 0x7f00f8003e00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0100001720 tx=0x7f00f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd721329a2cbde40 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100001370 con 0x7f00f80039e0 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01000052d0 con 0x7f00f80039e0 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f010effd640 1 -- 172.21.15.99:0/34730600 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100003830 con 0x7f00f80039e0 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f011668c640 1 -- 172.21.15.99:0/34730600 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f00f80039e0 msgr2=0x7f00f8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f011668c640 1 --2- 172.21.15.99:0/34730600 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f00f80039e0 0x7f00f8003e00 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f0100001720 tx=0x7f00f8005830 comp rx=0 tx=0).stop 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f0114c29640 1 -- 172.21.15.99:0/34730600 reap_dead start 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f011668c640 1 -- 172.21.15.99:0/34730600 shutdown_connections 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.242+0000 7f011668c640 1 -- 172.21.15.99:0/34730600 >> 172.21.15.99:0/34730600 conn(0x7f01100d0710 msgr2=0x7f01100d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.243+0000 7f011668c640 1 -- 172.21.15.99:0/34730600 shutdown_connections 2024-03-04T10:07:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.243+0000 7f011668c640 1 -- 172.21.15.99:0/34730600 wait complete. 2024-03-04T10:07:19.249 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.244+0000 7f011668c640 1 Processor -- start 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.244+0000 7f011668c640 1 -- start start 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00f80039e0 0x7f01100b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 0x7f01100c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0110154c80 0x7f01100b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01100d1380 con 0x7f01100c6c60 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01100d1200 con 0x7f0110154c80 2024-03-04T10:07:19.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01100d1500 con 0x7f00f80039e0 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 0x7f01100c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 0x7f01100c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49206/0 (socket says 172.21.15.99:49206) 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 -- 172.21.15.99:0/912681016 learned_addr learned my addr 172.21.15.99:0/912681016 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 -- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00f80039e0 msgr2=0x7f01100b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00f80039e0 0x7f01100b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 -- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0110154c80 msgr2=0x7f01100b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0110154c80 0x7f01100b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 -- 172.21.15.99:0/912681016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01100c8830 con 0x7f01100c6c60 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010f7fe640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 0x7f01100c9bb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f010400e780 tx=0x7f010400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010400c5e0 con 0x7f01100c6c60 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0104018070 con 0x7f01100c6c60 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01100c8b10 con 0x7f01100c6c60 2024-03-04T10:07:19.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010400c9c0 con 0x7f01100c6c60 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.245+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01100c90d0 con 0x7f01100c6c60 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.246+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0110074f60 con 0x7f01100c6c60 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.247+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f010400cb60 con 0x7f01100c6c60 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.247+0000 7f010cff9640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.247+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f010408a200 con 0x7f01100c6c60 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.247+0000 7f010ffff640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.248+0000 7f010ffff640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f00fc06b4d0 0x7f00fc06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0100033000 tx=0x7f010002f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.251+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0104058e40 con 0x7f01100c6c60 2024-03-04T10:07:19.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.458+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f011006ae70 con 0x7f01100c6c60 2024-03-04T10:07:19.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.458+0000 7f010cff9640 1 -- 172.21.15.99:0/912681016 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f0104058740 con 0x7f01100c6c60 2024-03-04T10:07:19.461 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:19.461 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":56,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:18.171700+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:07:19.461 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 56 2024-03-04T10:07:19.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f00fc06b4d0 msgr2=0x7f00fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f00fc06b4d0 0x7f00fc06d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0100033000 tx=0x7f010002f000 comp rx=0 tx=0).stop 2024-03-04T10:07:19.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 msgr2=0x7f01100c9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:19.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 0x7f01100c9bb0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f010400e780 tx=0x7f010400ec40 comp rx=0 tx=0).stop 2024-03-04T10:07:19.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 shutdown_connections 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0110154c80 0x7f01100b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01100c6c60 0x7f01100c9bb0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 --2- 172.21.15.99:0/912681016 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f00f80039e0 0x7f01100b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 >> 172.21.15.99:0/912681016 conn(0x7f01100d0710 msgr2=0x7f01100ad4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 shutdown_connections 2024-03-04T10:07:19.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:19.460+0000 7f011668c640 1 -- 172.21.15.99:0/912681016 wait complete. 2024-03-04T10:07:19.876 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:07:18.171700+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 68, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6471}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6471': {'gid': 6471, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.99:6833/665274836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 665274836}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 665274836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:07:19.877 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6471:up:rejoin 2024-03-04T10:07:20.878 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:07:20.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.987+0000 7f9e34197640 1 Processor -- start 2024-03-04T10:07:20.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e34197640 1 -- start start 2024-03-04T10:07:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e34197640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9e2c0d23b0 0x7f9e2c0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e34197640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2c0cfb40 con 0x7f9e2c0cbf80 2024-03-04T10:07:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e34197640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2c0cfe20 con 0x7f9e2c15bbb0 2024-03-04T10:07:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e34197640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2c0d2880 con 0x7f9e2c0d23b0 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e31732640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9e2c0cbf80 0x7f9e2c0d1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35834/0 (socket says 172.21.15.99:35834) 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.988+0000 7f9e31732640 1 -- 172.21.15.99:0/2746596400 learned_addr learned my addr 172.21.15.99:0/2746596400 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e32734640 1 --2- 172.21.15.99:0/2746596400 >> v2:172.21.15.162:3300/0 conn(0x7f9e2c0d23b0 0x7f9e2c0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706939046 0 0) 0x7f9e2c0cfb40 con 0x7f9e2c0cbf80 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e18003520 con 0x7f9e2c0cbf80 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3636186576 0 0) 0x7f9e18003520 con 0x7f9e2c0cbf80 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 >> v2:172.21.15.162:3300/0 conn(0x7f9e2c0d23b0 msgr2=0x7f9e2c0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 --2- 172.21.15.99:0/2746596400 >> v2:172.21.15.162:3300/0 conn(0x7f9e2c0d23b0 0x7f9e2c0cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 >> v1:172.21.15.162:6789/0 conn(0x7f9e2c15bbb0 legacy=0x7f9e2c0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c0d0100 con 0x7f9e2c0cbf80 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f9e1c001690 con 0x7f9e2c0cbf80 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 >> v1:172.21.15.99:6789/0 conn(0x7f9e2c0cbf80 legacy=0x7f9e2c0d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e180039b0 0x7f9e18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e180051c0 0x7f9e1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0d2880 con 0x7f9e18004800 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1800c060 con 0x7f9e180039b0 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1800c260 con 0x7f9e180051c0 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e31732640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e31f33640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e180039b0 0x7f9e18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.989+0000 7f9e32734640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e180051c0 0x7f9e1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e31732640 1 -- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e180051c0 msgr2=0x7f9e1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e31732640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e180051c0 0x7f9e1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e31732640 1 -- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e180039b0 msgr2=0x7f9e18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e31732640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e180039b0 0x7f9e18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e31732640 1 -- 172.21.15.99:0/2746596400 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18003520 con 0x7f9e18004800 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e32734640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e180051c0 0x7f9e1800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:07:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e32734640 1 -- 172.21.15.99:0/2746596400 reap_dead start 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e31732640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e18004c80 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c210 tx=0x7f9e18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dff330822e32eea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c007d30 con 0x7f9e18004800 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c030070 con 0x7f9e18004800 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.990+0000 7f9e30f31640 1 -- 172.21.15.99:0/2746596400 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c005d30 con 0x7f9e18004800 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 -- 172.21.15.99:0/2746596400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 msgr2=0x7f9e18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e18004c80 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c210 tx=0x7f9e18005830 comp rx=0 tx=0).stop 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 -- 172.21.15.99:0/2746596400 shutdown_connections 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 --2- 172.21.15.99:0/2746596400 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e18004c80 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 -- 172.21.15.99:0/2746596400 >> 172.21.15.99:0/2746596400 conn(0x7f9e2c0b3480 msgr2=0x7f9e2c0b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 -- 172.21.15.99:0/2746596400 shutdown_connections 2024-03-04T10:07:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.991+0000 7f9e34197640 1 -- 172.21.15.99:0/2746596400 wait complete. 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.992+0000 7f9e34197640 1 Processor -- start 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.992+0000 7f9e34197640 1 -- start start 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e34197640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e2c0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e34197640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e2c1f1cd0 0x7f9e2c0aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e34197640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e2c0aaf10 0x7f9e2c159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e34197640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c15bf90 con 0x7f9e18004800 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e34197640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0cc3c0 con 0x7f9e2c1f1cd0 2024-03-04T10:07:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e34197640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2c0ab370 con 0x7f9e2c0aaf10 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e31f33640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e2c0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e31f33640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e2c0aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49216/0 (socket says 172.21.15.99:49216) 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e31f33640 1 -- 172.21.15.99:0/3162129098 learned_addr learned my addr 172.21.15.99:0/3162129098 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e31732640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e2c1f1cd0 0x7f9e2c0aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.993+0000 7f9e32734640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e2c0aaf10 0x7f9e2c159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e31f33640 1 -- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e2c0aaf10 msgr2=0x7f9e2c159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e31f33640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e2c0aaf10 0x7f9e2c159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e31f33640 1 -- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e2c1f1cd0 msgr2=0x7f9e2c0aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e31f33640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e2c1f1cd0 0x7f9e2c0aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e31f33640 1 -- 172.21.15.99:0/3162129098 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2c0b04d0 con 0x7f9e18004800 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e32734640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e2c0aaf10 0x7f9e2c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e31f33640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e2c0aff90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800f960 tx=0x7f9e2800fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28003b80 con 0x7f9e18004800 2024-03-04T10:07:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e28018070 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2c0a88f0 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2800d7f0 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.994+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2c0a8eb0 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.995+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c074f60 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.995+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9e2800d990 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.996+0000 7f9e227fc640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9e1006b4d0 0x7f9e1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.996+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f9e2808a5a0 con 0x7f9e18004800 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.996+0000 7f9e31732640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9e1006b4d0 0x7f9e1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:20.996+0000 7f9e31732640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9e1006b4d0 0x7f9e1006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c034000 tx=0x7f9e1c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.000+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9e2801a310 con 0x7f9e18004800 2024-03-04T10:07:21.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.212+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2c0d1580 con 0x7f9e18004800 2024-03-04T10:07:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.213+0000 7f9e227fc640 1 -- 172.21.15.99:0/3162129098 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f9e280591e0 con 0x7f9e18004800 2024-03-04T10:07:21.215 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:07:21.215 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:19.179041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":15,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:07:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 57 2024-03-04T10:07:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.214+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9e1006b4d0 msgr2=0x7f9e1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.214+0000 7f9e34197640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9e1006b4d0 0x7f9e1006d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c034000 tx=0x7f9e1c009030 comp rx=0 tx=0).stop 2024-03-04T10:07:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.214+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 msgr2=0x7f9e2c0aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.214+0000 7f9e34197640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e2c0aff90 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9e2800f960 tx=0x7f9e2800fe20 comp rx=0 tx=0).stop 2024-03-04T10:07:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 shutdown_connections 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9e1006b4d0 0x7f9e1006d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9e2c0aaf10 0x7f9e2c159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9e2c1f1cd0 0x7f9e2c0aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 --2- 172.21.15.99:0/3162129098 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9e18004800 0x7f9e2c0aff90 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 >> 172.21.15.99:0/3162129098 conn(0x7f9e2c0b3480 msgr2=0x7f9e2c0b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 shutdown_connections 2024-03-04T10:07:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.215+0000 7f9e34197640 1 -- 172.21.15.99:0/3162129098 wait complete. 2024-03-04T10:07:21.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:06:35.354323+0000', 'modified': '2024-03-04T10:07:19.179041+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 68, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6471}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6471': {'gid': 6471, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.99:6833/665274836', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 665274836}, {'type': 'v1', 'addr': '172.21.15.99:6833', 'nonce': 665274836}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:07:21.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:07:21.630 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T10:07:21.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.730+0000 7fd22ddf1640 1 Processor -- start 2024-03-04T10:07:21.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd22ddf1640 1 -- start start 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd22ddf1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2280b5b80 0x7fd2280b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd22ddf1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22815e1e0 con 0x7fd2280b2e60 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd22ddf1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280d1c10 con 0x7fd22815de00 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd22ddf1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2280b98c0 con 0x7fd2280b5b80 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd227fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd2280b5b80 0x7fd2280b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd21effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd2280b2e60 0x7fd2280b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:35838/0 (socket says 172.21.15.99:35838) 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.731+0000 7fd21effd640 1 -- 172.21.15.99:0/2517185693 learned_addr learned my addr 172.21.15.99:0/2517185693 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.732+0000 7fd227fff640 1 -- 172.21.15.99:0/2517185693 >> v1:172.21.15.162:6789/0 conn(0x7fd22815de00 legacy=0x7fd2280b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.732+0000 7fd227fff640 1 -- 172.21.15.99:0/2517185693 >> v1:172.21.15.99:6789/0 conn(0x7fd2280b2e60 legacy=0x7fd2280b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.732+0000 7fd227fff640 1 -- 172.21.15.99:0/2517185693 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2280d1ef0 con 0x7fd2280b5b80 2024-03-04T10:07:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.732+0000 7fd226ffd640 1 -- 172.21.15.99:0/2517185693 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247651853 0 0) 0x7fd22815e1e0 con 0x7fd2280b2e60 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.732+0000 7fd227fff640 1 --2- 172.21.15.99:0/2517185693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b5b80 0x7fd2280b9380 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd220009dc0 tx=0x7fd22002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ca7ab33f52c05f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd226ffd640 1 -- 172.21.15.99:0/2517185693 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220039070 con 0x7fd2280b5b80 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd226ffd640 1 -- 172.21.15.99:0/2517185693 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22002cc40 con 0x7fd2280b5b80 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd226ffd640 1 -- 172.21.15.99:0/2517185693 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220034440 con 0x7fd2280b5b80 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2517185693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b5b80 msgr2=0x7fd2280b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2517185693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b5b80 0x7fd2280b9380 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd220009dc0 tx=0x7fd22002be90 comp rx=0 tx=0).stop 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2517185693 shutdown_connections 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2517185693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b5b80 0x7fd2280b9380 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2517185693 >> 172.21.15.99:0/2517185693 conn(0x7fd2280d0470 msgr2=0x7fd2280d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2517185693 shutdown_connections 2024-03-04T10:07:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.733+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2517185693 wait complete. 2024-03-04T10:07:21.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.734+0000 7fd22ddf1640 1 Processor -- start 2024-03-04T10:07:21.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.734+0000 7fd22ddf1640 1 -- start start 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd22ddf1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 0x7fd228156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd22ddf1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2280b5b80 0x7fd22815a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd22ddf1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd22815de00 0x7fd2280b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd22ddf1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2280d2820 con 0x7fd2280b2e60 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd22ddf1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2280d24b0 con 0x7fd2280b5b80 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd22ddf1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2280b72a0 con 0x7fd22815de00 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd2277fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 0x7fd228156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd2277fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 0x7fd228156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49240/0 (socket says 172.21.15.99:49240) 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd2277fe640 1 -- 172.21.15.99:0/2684168087 learned_addr learned my addr 172.21.15.99:0/2684168087 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.735+0000 7fd21effd640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2280b5b80 0x7fd22815a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd227fff640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd22815de00 0x7fd2280b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd2277fe640 1 -- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd22815de00 msgr2=0x7fd2280b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd2277fe640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd22815de00 0x7fd2280b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd2277fe640 1 -- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2280b5b80 msgr2=0x7fd22815a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd2277fe640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2280b5b80 0x7fd22815a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd2277fe640 1 -- 172.21.15.99:0/2684168087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2281f3b10 con 0x7fd2280b2e60 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd2277fe640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 0x7fd228156e50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c430 tx=0x7fd21800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd224ff9640 1 -- 172.21.15.99:0/2684168087 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218011790 con 0x7fd2280b2e60 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd224ff9640 1 -- 172.21.15.99:0/2684168087 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218017070 con 0x7fd2280b2e60 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2281f3df0 con 0x7fd2280b2e60 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd224ff9640 1 -- 172.21.15.99:0/2684168087 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218011bb0 con 0x7fd2280b2e60 2024-03-04T10:07:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.736+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2281f43b0 con 0x7fd2280b2e60 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.737+0000 7fd224ff9640 1 -- 172.21.15.99:0/2684168087 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd21801c460 con 0x7fd2280b2e60 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.738+0000 7fd224ff9640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd20806b4d0 0x7fd20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.738+0000 7fd224ff9640 1 -- 172.21.15.99:0/2684168087 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fd21808a3c0 con 0x7fd2280b2e60 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.738+0000 7fd21effd640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd20806b4d0 0x7fd20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.739+0000 7fd2309d3740 1 Processor -- start 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.739+0000 7fd21effd640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd20806b4d0 0x7fd20806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd2280d0090 tx=0x7fd210009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.739+0000 7fd2309d3740 1 -- start start 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.739+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b701c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 0x56147b702170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd2309d3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147b63c7c0 con 0x56147b6d2380 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147b5e5b50 con 0x56147b6ff280 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56147b638860 con 0x56147b6feea0 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 0x56147b702170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 0x56147b702170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49252/0 (socket says 172.21.15.99:49252) 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 -- 172.21.15.99:0/638348993 learned_addr learned my addr 172.21.15.99:0/638348993 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd21effd640 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b701c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 -- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 msgr2=0x56147b6d1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd2277fe640 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6d1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 -- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 msgr2=0x56147b701c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b701c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 -- 172.21.15.99:0/638348993 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56147b5f99a0 con 0x56147b6d2380 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd227fff640 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 0x56147b702170 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd220009ee0 tx=0x7fd22002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fe06240692d78c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd21d7fa640 1 -- 172.21.15.99:0/638348993 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220039040 con 0x56147b6d2380 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd21d7fa640 1 -- 172.21.15.99:0/638348993 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22003f440 con 0x56147b6d2380 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.740+0000 7fd21d7fa640 1 -- 172.21.15.99:0/638348993 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22003f830 con 0x56147b6d2380 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 -- 172.21.15.99:0/638348993 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 msgr2=0x56147b702170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 0x56147b702170 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd220009ee0 tx=0x7fd22002e000 comp rx=0 tx=0).stop 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 -- 172.21.15.99:0/638348993 shutdown_connections 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6d2380 0x56147b702170 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b701c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 --2- 172.21.15.99:0/638348993 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6d1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 -- 172.21.15.99:0/638348993 >> 172.21.15.99:0/638348993 conn(0x56147b6d40b0 msgr2=0x56147b71cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 -- 172.21.15.99:0/638348993 shutdown_connections 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.741+0000 7fd2309d3740 1 -- 172.21.15.99:0/638348993 wait complete. 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 -- start start 2024-03-04T10:07:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 Processor -- start 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6feea0 0x56147b72d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b6f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6f7c20 0x56147b6f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147b6d6750 con 0x56147b6ff280 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147b5f8f90 con 0x56147b6feea0 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56147b6d8f90 con 0x56147b6f7c20 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd2309d3740 10 client.0 _ll_register_callbacks cb 0x56147b61ad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd21effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b6f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd21effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b6f74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49260/0 (socket says 172.21.15.99:49260) 2024-03-04T10:07:21.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.742+0000 7fd21effd640 1 -- 172.21.15.99:0/4284247225 learned_addr learned my addr 172.21.15.99:0/4284247225 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2277fe640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6feea0 0x56147b72d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd227fff640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6f7c20 0x56147b6f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd21effd640 1 -- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6f7c20 msgr2=0x56147b6f80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd21effd640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6f7c20 0x56147b6f80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd21effd640 1 -- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6feea0 msgr2=0x56147b72d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd21effd640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6feea0 0x56147b72d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd21effd640 1 -- 172.21.15.99:0/4284247225 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56147b3512d0 con 0x56147b6ff280 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd21effd640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b6f74f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd210005b20 tx=0x7fd210000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadbdcf225a11486 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2067fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2067fc640 1 -- 172.21.15.99:0/4284247225 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210021430 con 0x56147b6ff280 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2067fc640 1 -- 172.21.15.99:0/4284247225 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fd210020d50 con 0x56147b6ff280 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd22003f440 con 0x56147b6ff280 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2067fc640 1 -- 172.21.15.99:0/4284247225 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2100215d0 con 0x56147b6ff280 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 10 client.6906 fetch_fsmap learned FSMap version 57 2024-03-04T10:07:21.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x56147b41b6b0 con 0x56147b6ff280 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2067fc640 1 -- 172.21.15.99:0/4284247225 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22003f440 con 0x56147b6ff280 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2067fc640 1 -- 172.21.15.99:0/4284247225 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd210021770 con 0x56147b6ff280 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 10 client.6906 fetch_fsmap finished waiting for FSMap version 57 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 10 client.6906 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 20 client.6906 populate_metadata read hostname 'smithi099' 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b704310 0x56147b7046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 4 client.6906 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.743+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 --> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56147b649d00 con 0x56147b704310 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2277fe640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b704310 0x56147b7046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2067fc640 10 client.6906 ms_handle_connect on v2:172.21.15.99:6832/665274836 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2277fe640 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b704310 0x56147b7046f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d534b425196e6ce1 server_cookie=9207d2c9b7226c14 in_seq=0 out_seq=0 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2067fc640 1 -- 172.21.15.99:0/4284247225 <== mds.0 v2:172.21.15.99:6832/665274836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56147b649d00 con 0x56147b704310 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2067fc640 10 client.6906 handle_command_reply: tid=0 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2309d3740 1 client.6906 shutdown 2024-03-04T10:07:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 msgr2=0x56147b6f74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.744+0000 7fd2309d3740 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b6f74f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd210005b20 tx=0x7fd210000ed0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 shutdown_connections 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6f7c20 0x56147b6f80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b6f74f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b704310 0x56147b7046f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 --2- 172.21.15.99:0/4284247225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6feea0 0x56147b72d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2067fc640 0 client.6906 ms_handle_reset on v2:172.21.15.99:6832/665274836 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 >> 172.21.15.99:0/4284247225 conn(0x56147b6d2380 msgr2=0x56147b6d5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 shutdown_connections 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 1 -- 172.21.15.99:0/4284247225 wait complete. 2024-03-04T10:07:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.745+0000 7fd2309d3740 20 client.6906 trim_cache size 0 max 16384 2024-03-04T10:07:21.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.760+0000 7fd2309d3740 1 Processor -- start 2024-03-04T10:07:21.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.760+0000 7fd2309d3740 1 -- start start 2024-03-04T10:07:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 0x56147b6d7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b6f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2309d3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2282007f0 con 0x56147b6f7c20 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2281ff910 con 0x56147b6ff280 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2281fede0 con 0x56147b6feea0 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2277fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 0x56147b6d7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2277fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 0x56147b6d7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49274/0 (socket says 172.21.15.99:49274) 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.761+0000 7fd2277fe640 1 -- 172.21.15.99:0/1689813488 learned_addr learned my addr 172.21.15.99:0/1689813488 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2277fe640 1 -- 172.21.15.99:0/1689813488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 msgr2=0x56147b6f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2277fe640 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2277fe640 1 -- 172.21.15.99:0/1689813488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 msgr2=0x56147b6f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2277fe640 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b6f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2277fe640 1 -- 172.21.15.99:0/1689813488 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56147b6d10f0 con 0x56147b6f7c20 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2277fe640 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 0x56147b6d7d40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd218058800 tx=0x7fd218014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdb0acc43db4708 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd21cff9640 1 -- 172.21.15.99:0/1689813488 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218088310 con 0x56147b6f7c20 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd21cff9640 1 -- 172.21.15.99:0/1689813488 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218086070 con 0x56147b6f7c20 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd21cff9640 1 -- 172.21.15.99:0/1689813488 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd218056b70 con 0x56147b6f7c20 2024-03-04T10:07:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2309d3740 1 -- 172.21.15.99:0/1689813488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 msgr2=0x56147b6d7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.762+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 0x56147b6d7d40 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd218058800 tx=0x7fd218014040 comp rx=0 tx=0).stop 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 -- 172.21.15.99:0/1689813488 shutdown_connections 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6ff280 0x56147b6f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b6f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1689813488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6f7c20 0x56147b6d7d40 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 -- 172.21.15.99:0/1689813488 >> 172.21.15.99:0/1689813488 conn(0x56147b6d2380 msgr2=0x56147b6f7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 -- 172.21.15.99:0/1689813488 shutdown_connections 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 -- 172.21.15.99:0/1689813488 wait complete. 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 -- start start 2024-03-04T10:07:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.763+0000 7fd2309d3740 1 Processor -- start 2024-03-04T10:07:21.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6f7c20 0x56147b76e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b71e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b71ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x56147b703990 con 0x56147b6ff280 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2281ff310 con 0x56147b6f7c20 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x56147b6fa0a0 con 0x56147b6feea0 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2309d3740 10 client.0 _ll_register_callbacks cb 0x7fd220037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b71ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b71ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:49284/0 (socket says 172.21.15.99:49284) 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 -- 172.21.15.99:0/1156510719 learned_addr learned my addr 172.21.15.99:0/1156510719 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 -- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 msgr2=0x56147b71e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b71e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 -- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6f7c20 msgr2=0x56147b76e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2277fe640 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6f7c20 0x56147b76e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6f7c20 0x56147b76e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 -- 172.21.15.99:0/1156510719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56147b732140 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd227fff640 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b71ea10 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd220009eb0 tx=0x7fd220004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc7a48638e9865a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.764+0000 7fd2067fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2067fc640 1 -- 172.21.15.99:0/1156510719 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd220004040 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2067fc640 1 -- 172.21.15.99:0/1156510719 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fd2200041e0 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd218086070 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2067fc640 1 -- 172.21.15.99:0/1156510719 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd22005a050 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 10 client.6912 fetch_fsmap learned FSMap version 57 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x56147b6cd6d0 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2067fc640 1 -- 172.21.15.99:0/1156510719 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd218086070 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2067fc640 1 -- 172.21.15.99:0/1156510719 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd22005d020 con 0x56147b6ff280 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 10 client.6912 fetch_fsmap finished waiting for FSMap version 57 2024-03-04T10:07:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 10 client.6912 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 20 client.6912 populate_metadata read hostname 'smithi099' 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b6f87c0 0x56147b6f3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 4 client.6912 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 --> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56147b6f9bc0 con 0x56147b6f87c0 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2277fe640 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b6f87c0 0x56147b6f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2067fc640 10 client.6912 ms_handle_connect on v2:172.21.15.99:6832/665274836 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.765+0000 7fd2277fe640 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b6f87c0 0x56147b6f3750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5ea275527d3497b server_cookie=8925d63c81933b36 in_seq=0 out_seq=0 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2067fc640 1 -- 172.21.15.99:0/1156510719 <== mds.0 v2:172.21.15.99:6832/665274836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x56147b6f9bc0 con 0x56147b6f87c0 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2067fc640 10 client.6912 handle_command_reply: tid=0 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 client.6912 shutdown 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 msgr2=0x56147b71ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b71ea10 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd220009eb0 tx=0x7fd220004a40 comp rx=0 tx=0).stop 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 shutdown_connections 2024-03-04T10:07:21.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x56147b6ff280 0x56147b71ea10 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56147b6feea0 0x56147b71e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x56147b6f87c0 0x56147b6f3750 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 --2- 172.21.15.99:0/1156510719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56147b6f7c20 0x56147b76e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2067fc640 0 client.6912 ms_handle_reset on v2:172.21.15.99:6832/665274836 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 >> 172.21.15.99:0/1156510719 conn(0x56147b6d2380 msgr2=0x56147b7031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 shutdown_connections 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 1 -- 172.21.15.99:0/1156510719 wait complete. 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd2309d3740 20 client.6912 trim_cache size 0 max 16384 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:07:21.774 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 6471, 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.602705383, 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 57, 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 69, 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 68, 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 50.768071395 2024-03-04T10:07:21.775 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:07:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd20806b4d0 msgr2=0x7fd20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd20806b4d0 0x7fd20806d990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd2280d0090 tx=0x7fd210009040 comp rx=0 tx=0).stop 2024-03-04T10:07:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 msgr2=0x7fd228156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:07:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 0x7fd228156e50 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c430 tx=0x7fd21800c8f0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 shutdown_connections 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd20806b4d0 0x7fd20806d990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd22815de00 0x7fd2280b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd2280b5b80 0x7fd22815a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 --2- 172.21.15.99:0/2684168087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd2280b2e60 0x7fd228156e50 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 >> 172.21.15.99:0/2684168087 conn(0x7fd2280d0470 msgr2=0x7fd2280d2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 shutdown_connections 2024-03-04T10:07:21.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:21.766+0000 7fd22ddf1640 1 -- 172.21.15.99:0/2684168087 wait complete. 2024-03-04T10:07:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.738+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_RECONNECTING pgs=14 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f61f67fc640 0 client.6834 ms_handle_remote_reset on v2:172.21.15.99:6834/3022309305 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:07:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:24.739+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:07:39.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:39.748+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:39.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:39.748+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:07:39.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:39.748+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:07:39.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:39.748+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:07:39.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:39.748+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:07:39.841 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:39.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:07:54.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:54.751+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:07:54.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:54.751+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:07:54.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:54.751+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:07:54.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:54.751+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:07:54.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:07:54.751+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:08:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:09.757+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:09.757+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:08:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:09.757+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:08:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:09.757+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:08:09.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:09.757+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:08:09.913 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:09.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:24.760+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:24.760+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:08:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:24.760+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:08:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:24.760+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:08:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:24.760+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:08:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:39.768+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:39.769+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:08:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:39.769+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:08:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:39.769+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:08:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:39.769+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:08:40.039 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:40.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:08:54.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:54.771+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:54.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:54.771+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-03-04T10:08:54.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:54.771+0000 7f62177fe640 1 -- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 msgr2=0x55db9b15b7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-04T10:08:54.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:54.771+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T10:08:54.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:54.771+0000 7f62177fe640 1 --2- 172.21.15.99:0/3342416233 >> [v2:172.21.15.99:6834/3022309305,v1:172.21.15.99:6835/3022309305] conn(0x55db9b145d40 0x55db9b15b7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-04T10:08:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-04T10:08:59.181 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:08:59.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.291+0000 7f3fee24b640 1 Processor -- start 2024-03-04T10:08:59.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.291+0000 7f3fee24b640 1 -- start start 2024-03-04T10:08:59.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fee24b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3fe815d660 0x7f3fe815e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:08:59.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fee24b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe815da40 con 0x7f3fe80aeb20 2024-03-04T10:08:59.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fee24b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe815ebb0 con 0x7f3fe8071a60 2024-03-04T10:08:59.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fee24b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe815cbc0 con 0x7f3fe815d660 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fe77fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3fe815d660 0x7f3fe815e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fe6ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3fe80aeb20 0x7f3fe8071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40838/0 (socket says 172.21.15.99:40838) 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.292+0000 7f3fe6ffd640 1 -- 172.21.15.99:0/3933915881 learned_addr learned my addr 172.21.15.99:0/3933915881 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.293+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3933915881 >> v1:172.21.15.162:6789/0 conn(0x7f3fe8071a60 legacy=0x7f3fe8150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.293+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3933915881 >> v1:172.21.15.99:6789/0 conn(0x7f3fe80aeb20 legacy=0x7f3fe8071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.293+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3933915881 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516767475 0 0) 0x7f3fe815da40 con 0x7f3fe80aeb20 2024-03-04T10:08:59.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.293+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3933915881 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80af090 con 0x7f3fe815d660 2024-03-04T10:08:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.293+0000 7f3fe77fe640 1 --2- 172.21.15.99:0/3933915881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe815d660 0x7f3fe815e670 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800a940 tx=0x7f3fd802cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5beedb15cd5a37d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:08:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3933915881 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd803a070 con 0x7f3fe815d660 2024-03-04T10:08:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3933915881 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd802dab0 con 0x7f3fe815d660 2024-03-04T10:08:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fe67fc640 1 -- 172.21.15.99:0/3933915881 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd800b040 con 0x7f3fe815d660 2024-03-04T10:08:59.301 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 -- 172.21.15.99:0/3933915881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe815d660 msgr2=0x7f3fe815e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3933915881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe815d660 0x7f3fe815e670 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f3fd800a940 tx=0x7f3fd802cd00 comp rx=0 tx=0).stop 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 -- 172.21.15.99:0/3933915881 shutdown_connections 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3933915881 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe815d660 0x7f3fe815e670 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 -- 172.21.15.99:0/3933915881 >> 172.21.15.99:0/3933915881 conn(0x7f3fe806acd0 msgr2=0x7f3fe806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 -- 172.21.15.99:0/3933915881 shutdown_connections 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.294+0000 7f3fee24b640 1 -- 172.21.15.99:0/3933915881 wait complete. 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.295+0000 7f3fee24b640 1 Processor -- start 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 -- start start 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 0x7f3fe80adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe80aeb20 0x7f3fe8152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe815d660 0x7f3fe8152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe815afb0 con 0x7f3fe8071a60 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe815acd0 con 0x7f3fe815d660 2024-03-04T10:08:59.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fee24b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe815a660 con 0x7f3fe80aeb20 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fe77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 0x7f3fe80adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fe77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 0x7f3fe80adf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56172/0 (socket says 172.21.15.99:56172) 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.296+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3567996491 learned_addr learned my addr 172.21.15.99:0/3567996491 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe6ffd640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe80aeb20 0x7f3fe8152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe80aeb20 msgr2=0x7f3fe8152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe77fe640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe80aeb20 0x7f3fe8152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe815d660 msgr2=0x7f3fe8152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe77fe640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe815d660 0x7f3fe8152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe77fe640 1 -- 172.21.15.99:0/3567996491 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe81553c0 con 0x7f3fe8071a60 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fe77fe640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 0x7f3fe80adf40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8038040 tx=0x7f3fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd803a040 con 0x7f3fe8071a60 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd8045070 con 0x7f3fe8071a60 2024-03-04T10:08:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8040660 con 0x7f3fe8071a60 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe81556a0 con 0x7f3fe8071a60 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.297+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe8155c60 con 0x7f3fe8071a60 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.298+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe8074f60 con 0x7f3fe8071a60 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.299+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3fd8040800 con 0x7f3fe8071a60 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.299+0000 7f3fcffff640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.299+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f3fd80acaa0 con 0x7f3fe8071a60 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.299+0000 7f3fe6ffd640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:08:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.300+0000 7f3fe6ffd640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0011e0 tx=0x7f3fdc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:08:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.303+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3fd807b6e0 con 0x7f3fe8071a60 2024-03-04T10:08:59.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.509+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3fe8072460 con 0x7f3fe8071a60 2024-03-04T10:08:59.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.510+0000 7f3fcffff640 1 -- 172.21.15.99:0/3567996491 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3fd807afe0 con 0x7f3fe8071a60 2024-03-04T10:08:59.511 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:08:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 msgr2=0x7f3fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:08:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0011e0 tx=0x7f3fdc009040 comp rx=0 tx=0).stop 2024-03-04T10:08:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 msgr2=0x7f3fe80adf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:08:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 0x7f3fe80adf40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8038040 tx=0x7f3fd80046c0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 shutdown_connections 2024-03-04T10:08:59.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fd406b4d0 0x7f3fd406d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe815d660 0x7f3fe8152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe80aeb20 0x7f3fe8152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.511+0000 7f3fee24b640 1 --2- 172.21.15.99:0/3567996491 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe8071a60 0x7f3fe80adf40 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:08:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.512+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 >> 172.21.15.99:0/3567996491 conn(0x7f3fe806acd0 msgr2=0x7f3fe80b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:08:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.512+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 shutdown_connections 2024-03-04T10:08:59.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:08:59.512+0000 7f3fee24b640 1 -- 172.21.15.99:0/3567996491 wait complete. 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:08:59.524 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:08:59.525 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:08:59.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.525 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:08:59.540 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:08:59.540 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:08:59.541 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42446sec preferred_lft 42446sec 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:08:59.542 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:08:59.543 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:08:59.543 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:08:59.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.543 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:08:59.607 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:08:59.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.609 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:08:59.664 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:08:59.665 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:08:59.665 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:08:59.665 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:08:59.665 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:08:59.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.665 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:08:59.665 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:08:59.665 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:08:59.666 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:08:59.666 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:08:59.666 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:08:59.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.760 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:08:59.775 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:08:59.775 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:08:59.775 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:08:59.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:08:59.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:08:59.842 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:08:59.896 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:08:59.897 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:08:59.897 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:08:59.951 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:08:59.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:08:59.952 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:09:00.037 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:09:00.037 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:09:00.045 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:09:00.045 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:09:00.045 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:09:00.045 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:09:00.045 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:09:00.046 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:09:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:09:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:09:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:09:00.046 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:09:00.047 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:09:00.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:00.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:00.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:00.074 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:09:00.130 INFO:teuthology.orchestra.run.smithi099.stdout:client5693 2024-03-04T10:09:00.131 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:00.131 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:00.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:00.187 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat a/ && find a/') 2024-03-04T10:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout: File: a/ 2024-03-04T10:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 2 Blocks: 0 IO Block: 65536 directory 2024-03-04T10:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 29h/41d Inode: 1099511627776 Links: 4 2024-03-04T10:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-04T10:09:00.243 INFO:teuthology.orchestra.run.smithi099.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-04T10:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-04 10:06:41.795836535 +0000 2024-03-04T10:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-04 10:06:41.799836439 +0000 2024-03-04T10:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-04 10:06:41.799836439 +0000 2024-03-04T10:09:00.244 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-04 10:06:41.795836535 +0000 2024-03-04T10:09:00.248 INFO:teuthology.orchestra.run.smithi099.stdout:a/ 2024-03-04T10:09:00.249 INFO:teuthology.orchestra.run.smithi099.stdout:a/c 2024-03-04T10:09:00.249 INFO:teuthology.orchestra.run.smithi099.stdout:a/c/d 2024-03-04T10:09:00.249 INFO:teuthology.orchestra.run.smithi099.stdout:a/b 2024-03-04T10:09:00.249 INFO:teuthology.orchestra.run.smithi099.stdout:a/b/d 2024-03-04T10:09:00.249 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T10:09:00.249 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-03-04T10:09:00.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.390+0000 7ff5e1a27640 1 Processor -- start 2024-03-04T10:09:00.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.390+0000 7ff5e1a27640 1 -- start start 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5e1a27640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0b5700 0x7ff5dc0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5e1a27640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff5dc0d34a0 0x7ff5dc06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5e1a27640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5dc0d2400 con 0x7ff5dc0b9410 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5e1a27640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5dc0d26e0 con 0x7ff5dc0d34a0 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5e1a27640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0d29c0 con 0x7ff5dc0b5700 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff5dc0d34a0 0x7ff5dc06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff5dc0d34a0 0x7ff5dc06c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56176/0 (socket says 172.21.15.99:56176) 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 -- 172.21.15.99:0/3741203222 learned_addr learned my addr 172.21.15.99:0/3741203222 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5daffd640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0b5700 0x7ff5dc0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 -- 172.21.15.99:0/3741203222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0b5700 msgr2=0x7ff5dc0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0b5700 0x7ff5dc0d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 -- 172.21.15.99:0/3741203222 >> v1:172.21.15.162:6789/0 conn(0x7ff5dc0b9410 legacy=0x7ff5dc06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5da7fc640 1 -- 172.21.15.99:0/3741203222 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0d2ca0 con 0x7ff5dc0d34a0 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.391+0000 7ff5daffd640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0b5700 0x7ff5dc0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.392+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0d34a0 0x7ff5dc06c0e0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009870 tx=0x7ff5d002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d844ea7dc7edb0c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.392+0000 7ff5d9ffb640 1 -- 172.21.15.99:0/3741203222 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00317c0 con 0x7ff5dc0d34a0 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.392+0000 7ff5d9ffb640 1 -- 172.21.15.99:0/3741203222 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0037070 con 0x7ff5dc0d34a0 2024-03-04T10:09:00.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.392+0000 7ff5d9ffb640 1 -- 172.21.15.99:0/3741203222 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0031b80 con 0x7ff5dc0d34a0 2024-03-04T10:09:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.392+0000 7ff5e1a27640 1 -- 172.21.15.99:0/3741203222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0d34a0 msgr2=0x7ff5dc06c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.392+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0d34a0 0x7ff5dc06c0e0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009870 tx=0x7ff5d002b860 comp rx=0 tx=0).stop 2024-03-04T10:09:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.393+0000 7ff5e1a27640 1 -- 172.21.15.99:0/3741203222 shutdown_connections 2024-03-04T10:09:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.393+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0d34a0 0x7ff5dc06c0e0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.393+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/3741203222 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0b5700 0x7ff5dc0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.393+0000 7ff5e1a27640 1 -- 172.21.15.99:0/3741203222 >> 172.21.15.99:0/3741203222 conn(0x7ff5dc0d0490 msgr2=0x7ff5dc0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.393+0000 7ff5e1a27640 1 -- 172.21.15.99:0/3741203222 shutdown_connections 2024-03-04T10:09:00.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.393+0000 7ff5e1a27640 1 -- 172.21.15.99:0/3741203222 wait complete. 2024-03-04T10:09:00.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.394+0000 7ff5e1a27640 1 Processor -- start 2024-03-04T10:09:00.398 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.394+0000 7ff5e1a27640 1 -- start start 2024-03-04T10:09:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5dc0b5700 0x7ff5dc15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 0x7ff5dc14c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0d34a0 0x7ff5dc0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc06ceb0 con 0x7ff5dc0b9410 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc06cd30 con 0x7ff5dc0b5700 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc0b5ae0 con 0x7ff5dc0d34a0 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 0x7ff5dc14c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 0x7ff5dc14c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56188/0 (socket says 172.21.15.99:56188) 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 -- 172.21.15.99:0/410035953 learned_addr learned my addr 172.21.15.99:0/410035953 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0d34a0 0x7ff5dc0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5daffd640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5dc0b5700 0x7ff5dc15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 -- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0d34a0 msgr2=0x7ff5dc0b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0d34a0 0x7ff5dc0b21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 -- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5dc0b5700 msgr2=0x7ff5dc15d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5dc0b5700 0x7ff5dc15d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 -- 172.21.15.99:0/410035953 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0b2730 con 0x7ff5dc0b9410 2024-03-04T10:09:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 0x7ff5dc14c880 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009870 tx=0x7ff5d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5c3fff640 1 -- 172.21.15.99:0/410035953 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0032860 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5c3fff640 1 -- 172.21.15.99:0/410035953 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0032a00 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5dc0c7f00 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5c3fff640 1 -- 172.21.15.99:0/410035953 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0032df0 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.395+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5dc0c84c0 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.397+0000 7ff5c3fff640 1 -- 172.21.15.99:0/410035953 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff5d004d3f0 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.397+0000 7ff5c3fff640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff5c806b4d0 0x7ff5c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.397+0000 7ff5c3fff640 1 -- 172.21.15.99:0/410035953 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7ff5d00ac140 con 0x7ff5dc0b9410 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.397+0000 7ff5daffd640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff5c806b4d0 0x7ff5c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.398+0000 7ff5daffd640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff5c806b4d0 0x7ff5c806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0075b0 tx=0x7ff5cc007540 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.398+0000 7ff5e460a740 1 Processor -- start 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.398+0000 7ff5e460a740 1 -- start start 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3b33800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b33d40 0x5624e3b63b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5e460a740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a9ce00 con 0x5624e3b60c40 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a45b10 con 0x5624e3b60860 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a98ea0 con 0x5624e3b33d40 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b635f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56196/0 (socket says 172.21.15.99:56196) 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1077373777 learned_addr learned my addr 172.21.15.99:0/1077373777 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5daffd640 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3b33800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.399+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b33d40 0x5624e3b63b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b33d40 msgr2=0x5624e3b63b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b33d40 0x5624e3b63b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 msgr2=0x5624e3b33800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3b33800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1077373777 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624e387b7d0 con 0x5624e3b60c40 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b33d40 0x5624e3b63b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b635f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00aa3d0 tx=0x7ff5d0079fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215b953881d5b159 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5c17fa640 1 -- 172.21.15.99:0/1077373777 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0092020 con 0x5624e3b60c40 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5c17fa640 1 -- 172.21.15.99:0/1077373777 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0037050 con 0x5624e3b60c40 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.400+0000 7ff5c17fa640 1 -- 172.21.15.99:0/1077373777 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d007e7d0 con 0x5624e3b60c40 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 -- 172.21.15.99:0/1077373777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 msgr2=0x5624e3b635f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b635f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7ff5d00aa3d0 tx=0x7ff5d0079fe0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 -- 172.21.15.99:0/1077373777 shutdown_connections 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b33d40 0x5624e3b63b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b635f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1077373777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3b33800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 -- 172.21.15.99:0/1077373777 >> 172.21.15.99:0/1077373777 conn(0x5624e3b35a70 msgr2=0x5624e3b7e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 -- 172.21.15.99:0/1077373777 shutdown_connections 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 -- 172.21.15.99:0/1077373777 wait complete. 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 -- start start 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.401+0000 7ff5e460a740 1 Processor -- start 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b78c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b8e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b79570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a98ea0 con 0x5624e3b60c40 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a590a0 con 0x5624e3b8e670 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a9ce00 con 0x5624e3b60860 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5e460a740 10 client.0 _ll_register_callbacks cb 0x5624e3a58a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b8e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b8e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56200/0 (socket says 172.21.15.99:56200) 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 -- 172.21.15.99:0/678798897 learned_addr learned my addr 172.21.15.99:0/678798897 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5daffd640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b78c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b79570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 -- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 msgr2=0x5624e3b78c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b78c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 -- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 msgr2=0x5624e3b79570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b79570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5da7fc640 1 -- 172.21.15.99:0/678798897 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624e37b13f0 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.402+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b79570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b8e130 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff5d007a380 tx=0x7ff5d004a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d465ddcf8609f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5ba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/678798897 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0096020 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/678798897 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7ff5d0082900 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5d0037050 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/678798897 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5d00a0050 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 10 client.6933 fetch_fsmap learned FSMap version 57 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5624e37710f0 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/678798897 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0037050 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/678798897 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff5d00a3020 con 0x5624e3b60c40 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 10 client.6933 fetch_fsmap finished waiting for FSMap version 57 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 10 client.6933 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 20 client.6933 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:00.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7d620 0x5624e3b7da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 4 client.6933 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 --> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624e3aaa340 con 0x5624e3b7d620 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.403+0000 7ff5daffd640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7d620 0x5624e3b7da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5ba7fc640 10 client.6933 ms_handle_connect on v2:172.21.15.99:6832/665274836 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5daffd640 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7d620 0x5624e3b7da00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bb1061b574c96fa server_cookie=2cc0efa62a67e930 in_seq=0 out_seq=0 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/678798897 <== mds.0 v2:172.21.15.99:6832/665274836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5624e3aaa340 con 0x5624e3b7d620 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5ba7fc640 10 client.6933 handle_command_reply: tid=0 2024-03-04T10:09:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 client.6933 shutdown 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 msgr2=0x5624e3b8e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b8e130 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7ff5d007a380 tx=0x7ff5d004a000 comp rx=0 tx=0).stop 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 shutdown_connections 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7d620 0x5624e3b7da00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b79570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b8e130 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5ba7fc640 0 client.6933 ms_handle_reset on v2:172.21.15.99:6832/665274836 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 --2- 172.21.15.99:0/678798897 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b78c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 >> 172.21.15.99:0/678798897 conn(0x5624e3b33d40 msgr2=0x5624e3b38500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 shutdown_connections 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 1 -- 172.21.15.99:0/678798897 wait complete. 2024-03-04T10:09:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.404+0000 7ff5e460a740 20 client.6933 trim_cache size 0 max 16384 2024-03-04T10:09:00.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.417+0000 7ff5e460a740 1 Processor -- start 2024-03-04T10:09:00.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.417+0000 7ff5e460a740 1 -- start start 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b7eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b328c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b5f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5e460a740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc200820 con 0x5624e3b60c40 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc1ff940 con 0x5624e3b8e670 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc1e74e0 con 0x5624e3b60860 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5da7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56214/0 (socket says 172.21.15.99:56214) 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.418+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1111779892 learned_addr learned my addr 172.21.15.99:0/1111779892 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1111779892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 msgr2=0x5624e3b7eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b7eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1111779892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 msgr2=0x5624e3b5f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b5f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5da7fc640 1 -- 172.21.15.99:0/1111779892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624e3b798e0 con 0x5624e3b60c40 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b328c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0078930 tx=0x7ff5d00ab000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66eb287d15b35cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5c0ff9640 1 -- 172.21.15.99:0/1111779892 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0096070 con 0x5624e3b60c40 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5c0ff9640 1 -- 172.21.15.99:0/1111779892 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0037070 con 0x5624e3b60c40 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5c0ff9640 1 -- 172.21.15.99:0/1111779892 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0092040 con 0x5624e3b60c40 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 -- 172.21.15.99:0/1111779892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 msgr2=0x5624e3b328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b328c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0078930 tx=0x7ff5d00ab000 comp rx=0 tx=0).stop 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 -- 172.21.15.99:0/1111779892 shutdown_connections 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b8e670 0x5624e3b5f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b60c40 0x5624e3b328c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 --2- 172.21.15.99:0/1111779892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60860 0x5624e3b7eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 -- 172.21.15.99:0/1111779892 >> 172.21.15.99:0/1111779892 conn(0x5624e3b33d40 msgr2=0x5624e3b2edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.419+0000 7ff5e460a740 1 -- 172.21.15.99:0/1111779892 shutdown_connections 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.420+0000 7ff5e460a740 1 -- 172.21.15.99:0/1111779892 wait complete. 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.420+0000 7ff5e460a740 1 -- start start 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.420+0000 7ff5e460a740 1 Processor -- start 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3be7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60c40 0x5624e3b63900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 0x5624e3b63e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624e3a45b10 con 0x5624e3b8e670 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5dc1ff340 con 0x5624e3b60860 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624e3b7c950 con 0x5624e3b60c40 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5e460a740 10 client.0 _ll_register_callbacks cb 0x7ff5d0006cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 0x5624e3b63e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 0x5624e3b63e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56222/0 (socket says 172.21.15.99:56222) 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 -- 172.21.15.99:0/737805558 learned_addr learned my addr 172.21.15.99:0/737805558 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5da7fc640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60c40 0x5624e3b63900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5daffd640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3be7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 -- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60c40 msgr2=0x5624e3b63900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60c40 0x5624e3b63900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 -- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 msgr2=0x5624e3be7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3be7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 -- 172.21.15.99:0/737805558 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624e3ba1510 con 0x5624e3b8e670 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5daffd640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3be7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5db7fe640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 0x5624e3b63e40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x5624e3b36280 tx=0x7ff5d4004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b792a9c1202bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5ba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.421+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/737805558 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4012e80 con 0x5624e3b8e670 2024-03-04T10:09:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/737805558 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7ff5d4007030 con 0x5624e3b8e670 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5d0037070 con 0x5624e3b8e670 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/737805558 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5d4023050 con 0x5624e3b8e670 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 10 client.6939 fetch_fsmap learned FSMap version 57 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5624e3be8310 con 0x5624e3b8e670 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/737805558 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d0037070 con 0x5624e3b8e670 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/737805558 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff5d4026020 con 0x5624e3b8e670 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 10 client.6939 fetch_fsmap finished waiting for FSMap version 57 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 10 client.6939 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.a 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 20 client.6939 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7da00 0x5624e3b9e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 4 client.6939 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 --> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5624e3aa4690 con 0x5624e3b7da00 2024-03-04T10:09:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5daffd640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7da00 0x5624e3b9e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5ba7fc640 10 client.6939 ms_handle_connect on v2:172.21.15.99:6832/665274836 2024-03-04T10:09:00.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.422+0000 7ff5daffd640 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7da00 0x5624e3b9e780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db1e5ad5df45351c server_cookie=a2c1448d71f4abe4 in_seq=0 out_seq=0 2024-03-04T10:09:00.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.436+0000 7ff5ba7fc640 1 -- 172.21.15.99:0/737805558 <== mds.0 v2:172.21.15.99:6832/665274836 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5624e3aa4690 con 0x5624e3b7da00 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.436+0000 7ff5ba7fc640 10 client.6939 handle_command_reply: tid=0 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 client.6939 shutdown 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 msgr2=0x5624e3b63e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 0x5624e3b63e40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x5624e3b36280 tx=0x7ff5d4004c40 comp rx=0 tx=0).stop 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 shutdown_connections 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5624e3b8e670 0x5624e3b63e40 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5624e3b60c40 0x5624e3b63900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.99:6832/665274836,v1:172.21.15.99:6833/665274836] conn(0x5624e3b7da00 0x5624e3b9e780 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 --2- 172.21.15.99:0/737805558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5624e3b60860 0x5624e3be7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5ba7fc640 0 client.6939 ms_handle_reset on v2:172.21.15.99:6832/665274836 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 >> 172.21.15.99:0/737805558 conn(0x5624e3b33d40 msgr2=0x5624e3b376c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 shutdown_connections 2024-03-04T10:09:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 1 -- 172.21.15.99:0/737805558 wait complete. 2024-03-04T10:09:00.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e460a740 20 client.6939 trim_cache size 0 max 16384 2024-03-04T10:09:00.440 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:09:00.440 INFO:teuthology.orchestra.run.smithi099.stdout: "message": "", 2024-03-04T10:09:00.440 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0 2024-03-04T10:09:00.440 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff5c806b4d0 msgr2=0x7ff5c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff5c806b4d0 0x7ff5c806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0075b0 tx=0x7ff5cc007540 comp rx=0 tx=0).stop 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 msgr2=0x7ff5dc14c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 0x7ff5dc14c880 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009870 tx=0x7ff5d00046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 shutdown_connections 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff5c806b4d0 0x7ff5c806d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff5dc0d34a0 0x7ff5dc0b21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff5dc0b9410 0x7ff5dc14c880 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 --2- 172.21.15.99:0/410035953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5dc0b5700 0x7ff5dc15d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.437+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 >> 172.21.15.99:0/410035953 conn(0x7ff5dc0d0490 msgr2=0x7ff5dc15fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.438+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 shutdown_connections 2024-03-04T10:09:00.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.438+0000 7ff5e1a27640 1 -- 172.21.15.99:0/410035953 wait complete. 2024-03-04T10:09:00.450 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:09:00.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:09:00.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:09:00.541 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:09:00.541 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:00.541 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:00.556 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:00.556 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:00.611 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:00.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.749+0000 7fa2ec5df640 1 Processor -- start 2024-03-04T10:09:00.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.749+0000 7fa2ec5df640 1 -- start start 2024-03-04T10:09:00.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.750+0000 7fa2ec5df640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa2e40cc8d0 0x7fa2e40cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.750+0000 7fa2ec5df640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e406fd20 con 0x7fa2e415dea0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.750+0000 7fa2ec5df640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e4070000 con 0x7fa2e406b3e0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.750+0000 7fa2ec5df640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e406fad0 con 0x7fa2e40cc8d0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.750+0000 7fa2ea37b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa2e415dea0 0x7fa2e406acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40846/0 (socket says 172.21.15.99:40846) 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.750+0000 7fa2ea37b640 1 -- 172.21.15.99:0/1005238691 learned_addr learned my addr 172.21.15.99:0/1005238691 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2eab7c640 1 --2- 172.21.15.99:0/1005238691 >> v2:172.21.15.162:3300/0 conn(0x7fa2e40cc8d0 0x7fa2e40cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927963315 0 0) 0x7fa2e406fd20 con 0x7fa2e415dea0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d0003520 con 0x7fa2e415dea0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1587712236 0 0) 0x7fa2d0003520 con 0x7fa2e415dea0 2024-03-04T10:09:00.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 >> v2:172.21.15.162:3300/0 conn(0x7fa2e40cc8d0 msgr2=0x7fa2e40cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 --2- 172.21.15.99:0/1005238691 >> v2:172.21.15.162:3300/0 conn(0x7fa2e40cc8d0 0x7fa2e40cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 >> v1:172.21.15.162:6789/0 conn(0x7fa2e406b3e0 legacy=0x7fa2e40cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40702e0 con 0x7fa2e415dea0 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fa2e00019f0 con 0x7fa2e415dea0 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 >> v1:172.21.15.99:6789/0 conn(0x7fa2e415dea0 legacy=0x7fa2e406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d00039b0 0x7fa2d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e406fad0 con 0x7fa2d0004800 2024-03-04T10:09:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d000c060 con 0x7fa2d00039b0 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d000c260 con 0x7fa2d00051c0 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9b7a640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2ea37b640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d00039b0 0x7fa2d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2eab7c640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9b7a640 1 -- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2d00051c0 msgr2=0x7fa2d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9b7a640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9b7a640 1 -- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d00039b0 msgr2=0x7fa2d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9b7a640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d00039b0 0x7fa2d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2e9b7a640 1 -- 172.21.15.99:0/1005238691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0003520 con 0x7fa2d0004800 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2eab7c640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2d00051c0 0x7fa2d000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.751+0000 7fa2eab7c640 1 -- 172.21.15.99:0/1005238691 reap_dead start 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.752+0000 7fa2e9b7a640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4001750 tx=0x7fa2d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c885f7638454cb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.752+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4001370 con 0x7fa2d0004800 2024-03-04T10:09:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.752+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d40052d0 con 0x7fa2d0004800 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.752+0000 7fa2e9379640 1 -- 172.21.15.99:0/1005238691 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4003830 con 0x7fa2d0004800 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1005238691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2d0004800 msgr2=0x7fa2d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4001750 tx=0x7fa2d0005830 comp rx=0 tx=0).stop 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1005238691 shutdown_connections 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1005238691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2d0004800 0x7fa2d0004c80 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1005238691 >> 172.21.15.99:0/1005238691 conn(0x7fa2e40d0710 msgr2=0x7fa2e40d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1005238691 shutdown_connections 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.753+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1005238691 wait complete. 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.754+0000 7fa2ec5df640 1 Processor -- start 2024-03-04T10:09:00.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.754+0000 7fa2ec5df640 1 -- start start 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d0004800 0x7fa2e40ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2e40c5250 0x7fa2e40c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 0x7fa2e40c7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40d1380 con 0x7fa2e40c77e0 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40d1200 con 0x7fa2d0004800 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40d1500 con 0x7fa2e40c5250 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 0x7fa2e40c7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 0x7fa2e40c7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56230/0 (socket says 172.21.15.99:56230) 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 -- 172.21.15.99:0/1517693680 learned_addr learned my addr 172.21.15.99:0/1517693680 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2e9b7a640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2e40c5250 0x7fa2e40c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ea37b640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d0004800 0x7fa2e40ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 -- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2e40c5250 msgr2=0x7fa2e40c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2e40c5250 0x7fa2e40c56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 -- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d0004800 msgr2=0x7fa2e40ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d0004800 0x7fa2e40ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 -- 172.21.15.99:0/1517693680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40c5d10 con 0x7fa2e40c77e0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2eab7c640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 0x7fa2e40c7c90 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40cc690 tx=0x7fa2dc011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc004440 con 0x7fa2e40c77e0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc01b070 con 0x7fa2e40c77e0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e40c5ff0 con 0x7fa2e40c77e0 2024-03-04T10:09:00.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc003850 con 0x7fa2e40c77e0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.755+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e40c65b0 con 0x7fa2e40c77e0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.756+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4074f60 con 0x7fa2e40c77e0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.757+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa2dc0039f0 con 0x7fa2e40c77e0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.757+0000 7fa2daffd640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2c806b500 0x7fa2c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.757+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fa2dc08a1a0 con 0x7fa2e40c77e0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.758+0000 7fa2ea37b640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2c806b500 0x7fa2c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.758+0000 7fa2ea37b640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2c806b500 0x7fa2c806d9c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0001330 tx=0x7fa2e0006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.762+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa2dc024020 con 0x7fa2e40c77e0 2024-03-04T10:09:00.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.934+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2e40cecf0 con 0x7fa2e40c77e0 2024-03-04T10:09:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.936+0000 7fa2daffd640 1 -- 172.21.15.99:0/1517693680 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17348 (secure 0 0 0) 0x7fa2dc0122d0 con 0x7fa2e40c77e0 2024-03-04T10:09:00.937 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:00.937 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":70,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:07:22.185227+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1f","mappings":[{"from":1,"to":2}]},{"pgid":"10.21","mappings":[{"from":4,"to":3}]},{"pgid":"10.34","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.3b","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6834/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6835/2687500379":"2024-03-05T10:06:46.776427+0000","172.21.15.99:6834/2687500379":"2024-03-05T10:06:46.776427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2c806b500 msgr2=0x7fa2c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2c806b500 0x7fa2c806d9c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0001330 tx=0x7fa2e0006040 comp rx=0 tx=0).stop 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 msgr2=0x7fa2e40c7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 0x7fa2e40c7c90 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40cc690 tx=0x7fa2dc011920 comp rx=0 tx=0).stop 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 shutdown_connections 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2c806b500 0x7fa2c806d9c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2e40c77e0 0x7fa2e40c7c90 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2e40c5250 0x7fa2e40c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 --2- 172.21.15.99:0/1517693680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2d0004800 0x7fa2e40ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 >> 172.21.15.99:0/1517693680 conn(0x7fa2e40d0710 msgr2=0x7fa2e40cbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 shutdown_connections 2024-03-04T10:09:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:00.938+0000 7fa2ec5df640 1 -- 172.21.15.99:0/1517693680 wait complete. 2024-03-04T10:09:00.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:06:33.589849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:06:34.331851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:09:00.950 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:09:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.051+0000 7f5cb4bcd640 1 Processor -- start 2024-03-04T10:09:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.051+0000 7f5cb4bcd640 1 -- start start 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cb4bcd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5cb00cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cb4bcd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb0162450 con 0x7f5cb01616e0 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cb4bcd640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb00cc2d0 con 0x7f5cb015da20 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cb4bcd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb00d0000 con 0x7f5cb00cf660 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5caed9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5cb01616e0 0x7f5cb0161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40860/0 (socket says 172.21.15.99:40860) 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5caed9d640 1 -- 172.21.15.99:0/348907664 learned_addr learned my addr 172.21.15.99:0/348907664 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750068663 0 0) 0x7f5cb0162450 con 0x7f5cb01616e0 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5cb00cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c98003520 con 0x7f5cb01616e0 2024-03-04T10:09:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2610395790 0 0) 0x7f5c98003520 con 0x7f5cb01616e0 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 msgr2=0x7f5cb00cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5cb00cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 >> v1:172.21.15.162:6789/0 conn(0x7f5cb015da20 legacy=0x7f5cb00ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb00cc5b0 con 0x7f5cb01616e0 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5ca8001670 con 0x7f5cb01616e0 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 >> v1:172.21.15.99:6789/0 conn(0x7f5cb01616e0 legacy=0x7f5cb0161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c980039e0 0x7f5c98003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5c98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5c98005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9800bac0 con 0x7f5c98004800 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9800bc70 con 0x7f5c980039e0 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9800be70 con 0x7f5cb00cf660 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.052+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5c98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cae59c640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c980039e0 0x7f5c98003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5caed9d640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5c98005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cadd9b640 1 -- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 msgr2=0x7f5c98005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5c98005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cadd9b640 1 -- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c980039e0 msgr2=0x7f5c98003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c980039e0 0x7f5c98003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cadd9b640 1 -- 172.21.15.99:0/348907664 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98003520 con 0x7f5c98004800 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5caed9d640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb00cf660 0x7f5c98005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.053+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5c98004c80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009ac0 tx=0x7f5c98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22d83f2cbb147cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4039070 con 0x7f5c98004800 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca402c6b0 con 0x7f5c98004800 2024-03-04T10:09:01.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cad59a640 1 -- 172.21.15.99:0/348907664 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4009690 con 0x7f5c98004800 2024-03-04T10:09:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/348907664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 msgr2=0x7f5c98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/348907664 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5c98004c80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009ac0 tx=0x7f5c98005830 comp rx=0 tx=0).stop 2024-03-04T10:09:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5caed9d640 1 -- 172.21.15.99:0/348907664 reap_dead start 2024-03-04T10:09:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/348907664 shutdown_connections 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/348907664 >> 172.21.15.99:0/348907664 conn(0x7f5cb00b33a0 msgr2=0x7f5cb00b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/348907664 shutdown_connections 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.054+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/348907664 wait complete. 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 Processor -- start 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 -- start start 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5cb01523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb015f0b0 0x7f5cb00c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cb0152bb0 0x7f5cb0153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb015de00 con 0x7f5c98004800 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb00cc920 con 0x7f5cb0152bb0 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cb4bcd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb00cfac0 con 0x7f5cb015f0b0 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cae59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5cb01523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cae59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5cb01523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56256/0 (socket says 172.21.15.99:56256) 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.056+0000 7f5cae59c640 1 -- 172.21.15.99:0/2828199836 learned_addr learned my addr 172.21.15.99:0/2828199836 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cae59c640 1 -- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb015f0b0 msgr2=0x7f5cb00c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb015f0b0 0x7f5cb00c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5caed9d640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cb0152bb0 0x7f5cb0153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cae59c640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb015f0b0 0x7f5cb00c98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cae59c640 1 -- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cb0152bb0 msgr2=0x7f5cb0153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cae59c640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cb0152bb0 0x7f5cb0153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cae59c640 1 -- 172.21.15.99:0/2828199836 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb01553d0 con 0x7f5c98004800 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cae59c640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5cb01523b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000fbc0 tx=0x7f5ca000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca000dc30 con 0x7f5c98004800 2024-03-04T10:09:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca0018070 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb015f540 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca0003970 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.057+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb015fb00 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.058+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb0160210 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.058+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5ca0003b10 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.059+0000 7f5c9effd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c9006b500 0x7f5c9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.059+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f5ca008a720 con 0x7f5c98004800 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.059+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c9006b500 0x7f5c9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.060+0000 7f5cadd9b640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c9006b500 0x7f5c9006d9c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4036000 tx=0x7f5ca4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.063+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5ca00592d0 con 0x7f5c98004800 2024-03-04T10:09:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.233+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5cb00cbbb0 con 0x7f5c98004800 2024-03-04T10:09:01.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.233+0000 7f5c9effd640 1 -- 172.21.15.99:0/2828199836 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ca0058bd0 con 0x7f5c98004800 2024-03-04T10:09:01.235 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c9006b500 msgr2=0x7f5c9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c9006b500 0x7f5c9006d9c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4036000 tx=0x7f5ca4030000 comp rx=0 tx=0).stop 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 msgr2=0x7f5cb01523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5cb01523b0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f5ca000fbc0 tx=0x7f5ca000d520 comp rx=0 tx=0).stop 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 shutdown_connections 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c9006b500 0x7f5c9006d9c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5cb0152bb0 0x7f5cb0153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5cb015f0b0 0x7f5cb00c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 --2- 172.21.15.99:0/2828199836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c98004800 0x7f5cb01523b0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 >> 172.21.15.99:0/2828199836 conn(0x7f5cb00b33a0 msgr2=0x7f5cb00b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 shutdown_connections 2024-03-04T10:09:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.235+0000 7f5cb4bcd640 1 -- 172.21.15.99:0/2828199836 wait complete. 2024-03-04T10:09:01.247 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:09:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.349+0000 7f8ea9438640 1 Processor -- start 2024-03-04T10:09:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.349+0000 7f8ea9438640 1 -- start start 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea9438640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8ea40b5700 0x7f8ea40d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea9438640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b9410 0x7f8ea406c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea9438640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40d2400 con 0x7f8ea40d34a0 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea9438640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea40d26e0 con 0x7f8ea40b5700 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea9438640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40d29c0 con 0x7f8ea40b9410 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea2ffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8ea40b5700 0x7f8ea40d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b9410 0x7f8ea406c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea27fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f8ea40d34a0 0x7f8ea406c0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:39370/0 (socket says 172.21.15.99:39370) 2024-03-04T10:09:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea27fc640 1 -- 172.21.15.99:0/2200295334 learned_addr learned my addr 172.21.15.99:0/2200295334 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea2ffd640 1 -- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b9410 msgr2=0x7f8ea406c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea2ffd640 1 --2- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b9410 0x7f8ea406c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea2ffd640 1 -- 172.21.15.99:0/2200295334 >> v1:172.21.15.162:6789/0 conn(0x7f8ea40d34a0 legacy=0x7f8ea406c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.350+0000 7f8ea2ffd640 1 -- 172.21.15.99:0/2200295334 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40d2ca0 con 0x7f8ea40b5700 2024-03-04T10:09:01.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.351+0000 7f8ea2ffd640 1 --2- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b5700 0x7f8ea40d1b00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8e940097a0 tx=0x7f8e9402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed60915f08516816 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.351+0000 7f8ea1ffb640 1 -- 172.21.15.99:0/2200295334 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e940315a0 con 0x7f8ea40b5700 2024-03-04T10:09:01.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.351+0000 7f8ea1ffb640 1 -- 172.21.15.99:0/2200295334 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e94037070 con 0x7f8ea40b5700 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.351+0000 7f8ea1ffb640 1 -- 172.21.15.99:0/2200295334 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e94031960 con 0x7f8ea40b5700 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 -- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b5700 msgr2=0x7f8ea40d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 --2- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b5700 0x7f8ea40d1b00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8e940097a0 tx=0x7f8e9402b840 comp rx=0 tx=0).stop 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 -- 172.21.15.99:0/2200295334 shutdown_connections 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 --2- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b9410 0x7f8ea406c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 --2- 172.21.15.99:0/2200295334 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b5700 0x7f8ea40d1b00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 -- 172.21.15.99:0/2200295334 >> 172.21.15.99:0/2200295334 conn(0x7f8ea40d0490 msgr2=0x7f8ea40d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 -- 172.21.15.99:0/2200295334 shutdown_connections 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.352+0000 7f8ea9438640 1 -- 172.21.15.99:0/2200295334 wait complete. 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.353+0000 7f8ea9438640 1 Processor -- start 2024-03-04T10:09:01.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.353+0000 7f8ea9438640 1 -- start start 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea9438640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b5700 0x7f8ea40cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea9438640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b9410 0x7f8ea415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea9438640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 0x7f8ea4153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea9438640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea406ceb0 con 0x7f8ea40d34a0 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea9438640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea406cd30 con 0x7f8ea40b9410 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea9438640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40b5ae0 con 0x7f8ea40b5700 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 0x7f8ea4153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 0x7f8ea4153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56268/0 (socket says 172.21.15.99:56268) 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 -- 172.21.15.99:0/558533882 learned_addr learned my addr 172.21.15.99:0/558533882 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea2ffd640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b5700 0x7f8ea40cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea27fc640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b9410 0x7f8ea415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 -- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b5700 msgr2=0x7f8ea40cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b5700 0x7f8ea40cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 -- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b9410 msgr2=0x7f8ea415d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b9410 0x7f8ea415d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 -- 172.21.15.99:0/558533882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea40cb900 con 0x7f8ea40d34a0 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.354+0000 7f8ea37fe640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 0x7f8ea4153c80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00efc0 tx=0x7f8e9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.355+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c018020 con 0x7f8ea40d34a0 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.355+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c014480 con 0x7f8ea40d34a0 2024-03-04T10:09:01.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.355+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c014870 con 0x7f8ea40d34a0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.355+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea40cbbe0 con 0x7f8ea40d34a0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.355+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea40cc480 con 0x7f8ea40d34a0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.356+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea4074f60 con 0x7f8ea40d34a0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.356+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8e9c005d10 con 0x7f8ea40d34a0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.357+0000 7f8e8b7fe640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8e9006b4d0 0x7f8e9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.357+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f8e9c08af40 con 0x7f8ea40d34a0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.357+0000 7f8ea2ffd640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8e9006b4d0 0x7f8e9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.357+0000 7f8ea2ffd640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8e9006b4d0 0x7f8e9006d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8e94036000 tx=0x7f8e9403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.361+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8e9c022070 con 0x7f8ea40d34a0 2024-03-04T10:09:01.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.532+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8ea40cc650 con 0x7f8ea40d34a0 2024-03-04T10:09:01.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.532+0000 7f8e8b7fe640 1 -- 172.21.15.99:0/558533882 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e9c010040 con 0x7f8ea40d34a0 2024-03-04T10:09:01.534 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8e9006b4d0 msgr2=0x7f8e9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8e9006b4d0 0x7f8e9006d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f8e94036000 tx=0x7f8e9403d040 comp rx=0 tx=0).stop 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 msgr2=0x7f8ea4153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 0x7f8ea4153c80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00efc0 tx=0x7f8e9c00c490 comp rx=0 tx=0).stop 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 shutdown_connections 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8e9006b4d0 0x7f8e9006d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ea40d34a0 0x7f8ea4153c80 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ea40b9410 0x7f8ea415d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 --2- 172.21.15.99:0/558533882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ea40b5700 0x7f8ea40cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 >> 172.21.15.99:0/558533882 conn(0x7f8ea40d0490 msgr2=0x7f8ea40c6b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 shutdown_connections 2024-03-04T10:09:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.534+0000 7f8ea9438640 1 -- 172.21.15.99:0/558533882 wait complete. 2024-03-04T10:09:01.546 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:09:01.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.647+0000 7f4fb809a640 1 Processor -- start 2024-03-04T10:09:01.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.647+0000 7f4fb809a640 1 -- start start 2024-03-04T10:09:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb809a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f4fb01450e0 0x7f4fb00cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb809a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4fb00cc510 0x7f4fb00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb809a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb00cebd0 con 0x7f4fb015c9b0 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb809a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb00cf7e0 con 0x7f4fb00cc510 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb809a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb00ceeb0 con 0x7f4fb01450e0 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb6637640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4fb00cc510 0x7f4fb00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb5635640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f4fb015c9b0 0x7f4fb00cbe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40862/0 (socket says 172.21.15.99:40862) 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb6637640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f4fb00cc510 0x7f4fb00b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56274/0 (socket says 172.21.15.99:56274) 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb5635640 1 -- 172.21.15.99:0/167533964 learned_addr learned my addr 172.21.15.99:0/167533964 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb5e36640 1 --2- 172.21.15.99:0/167533964 >> v2:172.21.15.162:3300/0 conn(0x7f4fb01450e0 0x7f4fb00cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb6637640 1 -- 172.21.15.99:0/167533964 >> v2:172.21.15.162:3300/0 conn(0x7f4fb01450e0 msgr2=0x7f4fb00cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb6637640 1 --2- 172.21.15.99:0/167533964 >> v2:172.21.15.162:3300/0 conn(0x7f4fb01450e0 0x7f4fb00cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb6637640 1 -- 172.21.15.99:0/167533964 >> v1:172.21.15.99:6789/0 conn(0x7f4fb015c9b0 legacy=0x7f4fb00cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb6637640 1 -- 172.21.15.99:0/167533964 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb00cf190 con 0x7f4fb00cc510 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.648+0000 7f4fb4e34640 1 -- 172.21.15.99:0/167533964 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213677292 0 0) 0x7f4fb00cebd0 con 0x7f4fb015c9b0 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.649+0000 7f4fb6637640 1 --2- 172.21.15.99:0/167533964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb00cc510 0x7f4fb00b3250 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00b23d0 tx=0x7f4fa802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede424d682d02fa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.649+0000 7f4fb4e34640 1 -- 172.21.15.99:0/167533964 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8039070 con 0x7f4fb00cc510 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.649+0000 7f4fb4e34640 1 -- 172.21.15.99:0/167533964 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa802cc80 con 0x7f4fb00cc510 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.649+0000 7f4fb4e34640 1 -- 172.21.15.99:0/167533964 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8034660 con 0x7f4fb00cc510 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 -- 172.21.15.99:0/167533964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb00cc510 msgr2=0x7f4fb00b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 --2- 172.21.15.99:0/167533964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb00cc510 0x7f4fb00b3250 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00b23d0 tx=0x7f4fa802bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 -- 172.21.15.99:0/167533964 shutdown_connections 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 --2- 172.21.15.99:0/167533964 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb00cc510 0x7f4fb00b3250 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 --2- 172.21.15.99:0/167533964 >> v2:172.21.15.162:3300/0 conn(0x7f4fb01450e0 0x7f4fb00cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 -- 172.21.15.99:0/167533964 >> 172.21.15.99:0/167533964 conn(0x7f4fb006acd0 msgr2=0x7f4fb006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 -- 172.21.15.99:0/167533964 shutdown_connections 2024-03-04T10:09:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.650+0000 7f4fb809a640 1 -- 172.21.15.99:0/167533964 wait complete. 2024-03-04T10:09:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 Processor -- start 2024-03-04T10:09:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 -- start start 2024-03-04T10:09:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fb00cc510 0x7f4fb013d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 0x7f4fb01f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fb015c9b0 0x7f4fb015b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb01454c0 con 0x7f4fb01450e0 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb015ce60 con 0x7f4fb015c9b0 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.651+0000 7f4fb809a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb00ccd10 con 0x7f4fb00cc510 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 0x7f4fb01f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 0x7f4fb01f53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56282/0 (socket says 172.21.15.99:56282) 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 -- 172.21.15.99:0/2247090820 learned_addr learned my addr 172.21.15.99:0/2247090820 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 -- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fb00cc510 msgr2=0x7f4fb013d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fb00cc510 0x7f4fb013d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 -- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fb015c9b0 msgr2=0x7f4fb015b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb6637640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fb015c9b0 0x7f4fb015b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fb015c9b0 0x7f4fb015b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 -- 172.21.15.99:0/2247090820 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb00a8790 con 0x7f4fb01450e0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb6637640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fb015c9b0 0x7f4fb015b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb5635640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 0x7f4fb01f53e0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c430 tx=0x7f4fa000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa00074e0 con 0x7f4fb01450e0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa00047e0 con 0x7f4fb01450e0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb00a96a0 con 0x7f4fb01450e0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0004bd0 con 0x7f4fb01450e0 2024-03-04T10:09:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.652+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb00a93c0 con 0x7f4fb01450e0 2024-03-04T10:09:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.653+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb0074f60 con 0x7f4fb01450e0 2024-03-04T10:09:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.653+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4fa0011070 con 0x7f4fb01450e0 2024-03-04T10:09:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.654+0000 7f4fa67fc640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f9406b500 0x7f4f9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.654+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f4fa008a030 con 0x7f4fb01450e0 2024-03-04T10:09:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.654+0000 7f4fb5e36640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f9406b500 0x7f4f9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.654+0000 7f4fb5e36640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f9406b500 0x7f4f9406d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0098f0 tx=0x7f4fac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.658+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4fb00a8790 con 0x7f4fb01450e0 2024-03-04T10:09:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.827+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4fb00cfda0 con 0x7f4fb01450e0 2024-03-04T10:09:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.827+0000 7f4fa67fc640 1 -- 172.21.15.99:0/2247090820 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4fa000e280 con 0x7f4fb01450e0 2024-03-04T10:09:01.829 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:09:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f9406b500 msgr2=0x7f4f9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f9406b500 0x7f4f9406d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0098f0 tx=0x7f4fac008040 comp rx=0 tx=0).stop 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 msgr2=0x7f4fb01f53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 0x7f4fb01f53e0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000c430 tx=0x7f4fa000c8f0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 shutdown_connections 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4f9406b500 0x7f4f9406d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fb015c9b0 0x7f4fb015b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4fb01450e0 0x7f4fb01f53e0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 --2- 172.21.15.99:0/2247090820 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4fb00cc510 0x7f4fb013d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 >> 172.21.15.99:0/2247090820 conn(0x7f4fb006acd0 msgr2=0x7f4fb0144410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 shutdown_connections 2024-03-04T10:09:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.829+0000 7f4fb809a640 1 -- 172.21.15.99:0/2247090820 wait complete. 2024-03-04T10:09:01.841 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:09:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.940+0000 7feb19589640 1 Processor -- start 2024-03-04T10:09:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.940+0000 7feb19589640 1 -- start start 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb19589640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7feb1415de00 0x7feb140b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb19589640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb140b9380 0x7feb140b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb19589640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1415e1e0 con 0x7feb140b70d0 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb19589640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb140d2820 con 0x7feb140b9380 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb19589640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb140d1c10 con 0x7feb1415de00 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb127fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb140b9380 0x7feb140b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb137fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7feb140b70d0 0x7feb140b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40864/0 (socket says 172.21.15.99:40864) 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.941+0000 7feb137fe640 1 -- 172.21.15.99:0/2465051858 learned_addr learned my addr 172.21.15.99:0/2465051858 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb127fc640 1 -- 172.21.15.99:0/2465051858 >> v2:172.21.15.162:3300/0 conn(0x7feb1415de00 msgr2=0x7feb140b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb12ffd640 1 --2- 172.21.15.99:0/2465051858 >> v2:172.21.15.162:3300/0 conn(0x7feb1415de00 0x7feb140b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb127fc640 1 --2- 172.21.15.99:0/2465051858 >> v2:172.21.15.162:3300/0 conn(0x7feb1415de00 0x7feb140b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb127fc640 1 -- 172.21.15.99:0/2465051858 >> v1:172.21.15.99:6789/0 conn(0x7feb140b70d0 legacy=0x7feb140b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb127fc640 1 -- 172.21.15.99:0/2465051858 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140d1ef0 con 0x7feb140b9380 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb11ffb640 1 -- 172.21.15.99:0/2465051858 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561491542 0 0) 0x7feb1415e1e0 con 0x7feb140b70d0 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb12ffd640 1 --2- 172.21.15.99:0/2465051858 >> v2:172.21.15.162:3300/0 conn(0x7feb1415de00 0x7feb140b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.942+0000 7feb127fc640 1 --2- 172.21.15.99:0/2465051858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b9380 0x7feb140b6b90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7feb140ce240 tx=0x7feb0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7530a66dc53c2640 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb11ffb640 1 -- 172.21.15.99:0/2465051858 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08035020 con 0x7feb140b9380 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb11ffb640 1 -- 172.21.15.99:0/2465051858 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08031830 con 0x7feb140b9380 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb11ffb640 1 -- 172.21.15.99:0/2465051858 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08031c00 con 0x7feb140b9380 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 -- 172.21.15.99:0/2465051858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b9380 msgr2=0x7feb140b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 --2- 172.21.15.99:0/2465051858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b9380 0x7feb140b6b90 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7feb140ce240 tx=0x7feb0802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 -- 172.21.15.99:0/2465051858 shutdown_connections 2024-03-04T10:09:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 --2- 172.21.15.99:0/2465051858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b9380 0x7feb140b6b90 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 --2- 172.21.15.99:0/2465051858 >> v2:172.21.15.162:3300/0 conn(0x7feb1415de00 0x7feb140b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 -- 172.21.15.99:0/2465051858 >> 172.21.15.99:0/2465051858 conn(0x7feb140d0470 msgr2=0x7feb140d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 -- 172.21.15.99:0/2465051858 shutdown_connections 2024-03-04T10:09:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.943+0000 7feb19589640 1 -- 172.21.15.99:0/2465051858 wait complete. 2024-03-04T10:09:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.944+0000 7feb19589640 1 Processor -- start 2024-03-04T10:09:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.944+0000 7feb19589640 1 -- start start 2024-03-04T10:09:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb19589640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 0x7feb1414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb19589640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb140b9380 0x7feb141f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb19589640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb1415de00 0x7feb140b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb19589640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb140b97e0 con 0x7feb140b70d0 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb19589640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb140d24b0 con 0x7feb140b9380 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb19589640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb140b5b80 con 0x7feb1415de00 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 0x7feb1414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb12ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 0x7feb1414d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56294/0 (socket says 172.21.15.99:56294) 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb12ffd640 1 -- 172.21.15.99:0/219726318 learned_addr learned my addr 172.21.15.99:0/219726318 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb127fc640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb140b9380 0x7feb141f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb137fe640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb1415de00 0x7feb140b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb12ffd640 1 -- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb1415de00 msgr2=0x7feb140b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.945+0000 7feb12ffd640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb1415de00 0x7feb140b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb12ffd640 1 -- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb140b9380 msgr2=0x7feb141f2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb12ffd640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb140b9380 0x7feb141f2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb12ffd640 1 -- 172.21.15.99:0/219726318 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1414dcf0 con 0x7feb140b70d0 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb127fc640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb140b9380 0x7feb141f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb12ffd640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 0x7feb1414d7b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7feb0400ce00 tx=0x7feb0400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0400f890 con 0x7feb140b70d0 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0400fa30 con 0x7feb140b70d0 2024-03-04T10:09:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb140cadf0 con 0x7feb140b70d0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0400fe40 con 0x7feb140b70d0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.946+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb140cb690 con 0x7feb140b70d0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.947+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb140cafc0 con 0x7feb140b70d0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.947+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7feb04018400 con 0x7feb140b70d0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.947+0000 7feafb7fe640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb0006b4d0 0x7feb0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.948+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7feb0408b230 con 0x7feb140b70d0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.948+0000 7feb127fc640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb0006b4d0 0x7feb0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.948+0000 7feb127fc640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb0006b4d0 0x7feb0006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feb08036000 tx=0x7feb08008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:01.952+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feb04014030 con 0x7feb140b70d0 2024-03-04T10:09:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.134+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feb14074f60 con 0x7feb140b70d0 2024-03-04T10:09:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.135+0000 7feafb7fe640 1 -- 172.21.15.99:0/219726318 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v70) v1 ==== 80+0+244 (secure 0 0 0) 0x7feb0400ae20 con 0x7feb140b70d0 2024-03-04T10:09:02.137 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6835/3022309305 2024-03-05T10:07:17.162027+0000 2024-03-04T10:09:02.137 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6834/3022309305 2024-03-05T10:07:17.162027+0000 2024-03-04T10:09:02.137 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6835/2687500379 2024-03-05T10:06:46.776427+0000 2024-03-04T10:09:02.137 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6834/2687500379 2024-03-05T10:06:46.776427+0000 2024-03-04T10:09:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:listed 4 entries 2024-03-04T10:09:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb0006b4d0 msgr2=0x7feb0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb0006b4d0 0x7feb0006d990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7feb08036000 tx=0x7feb08008030 comp rx=0 tx=0).stop 2024-03-04T10:09:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 msgr2=0x7feb1414d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 0x7feb1414d7b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7feb0400ce00 tx=0x7feb0400a650 comp rx=0 tx=0).stop 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 shutdown_connections 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb0006b4d0 0x7feb0006d990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb1415de00 0x7feb140b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb140b9380 0x7feb141f2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 --2- 172.21.15.99:0/219726318 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb140b70d0 0x7feb1414d7b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 >> 172.21.15.99:0/219726318 conn(0x7feb140d0470 msgr2=0x7feb140b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 shutdown_connections 2024-03-04T10:09:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.137+0000 7feb19589640 1 -- 172.21.15.99:0/219726318 wait complete. 2024-03-04T10:09:02.149 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:09:02.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:02.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:02.179 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:09:02.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:02.179 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:09:02.234 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:09:02.234 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:02.254 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:09:02.254 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:02.254 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:02.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:09:02.289 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:09:02.289 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:02.313 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:09:02.313 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:02.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.428+0000 7f531ce75640 1 Processor -- start 2024-03-04T10:09:02.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.428+0000 7f531ce75640 1 -- start start 2024-03-04T10:09:02.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f531ce75640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f53180b5b80 0x7f53180b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f531ce75640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531815e1e0 con 0x7f53180b2e60 2024-03-04T10:09:02.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f531ce75640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180d1c10 con 0x7f531815de00 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f531ce75640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53180b98c0 con 0x7f53180b5b80 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f5316d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f53180b5b80 0x7f53180b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f530ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f53180b2e60 0x7f53180b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40880/0 (socket says 172.21.15.99:40880) 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f530ffff640 1 -- 172.21.15.99:0/2656983505 learned_addr learned my addr 172.21.15.99:0/2656983505 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f5316d9d640 1 -- 172.21.15.99:0/2656983505 >> v1:172.21.15.162:6789/0 conn(0x7f531815de00 legacy=0x7f53180b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f5316d9d640 1 -- 172.21.15.99:0/2656983505 >> v1:172.21.15.99:6789/0 conn(0x7f53180b2e60 legacy=0x7f53180b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.429+0000 7f5316d9d640 1 -- 172.21.15.99:0/2656983505 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53180d1ef0 con 0x7f53180b5b80 2024-03-04T10:09:02.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f5316d9d640 1 --2- 172.21.15.99:0/2656983505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f53180b5b80 0x7f53180b9380 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5310009dc0 tx=0x7f531002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1664436a477b54e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:02.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f5315d9b640 1 -- 172.21.15.99:0/2656983505 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310039070 con 0x7f53180b5b80 2024-03-04T10:09:02.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f5315d9b640 1 -- 172.21.15.99:0/2656983505 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531002cc40 con 0x7f53180b5b80 2024-03-04T10:09:02.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f5315d9b640 1 -- 172.21.15.99:0/2656983505 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310034440 con 0x7f53180b5b80 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 -- 172.21.15.99:0/2656983505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f53180b5b80 msgr2=0x7f53180b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 --2- 172.21.15.99:0/2656983505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f53180b5b80 0x7f53180b9380 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f5310009dc0 tx=0x7f531002be90 comp rx=0 tx=0).stop 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 -- 172.21.15.99:0/2656983505 shutdown_connections 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 --2- 172.21.15.99:0/2656983505 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f53180b5b80 0x7f53180b9380 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 -- 172.21.15.99:0/2656983505 >> 172.21.15.99:0/2656983505 conn(0x7f53180d0470 msgr2=0x7f53180d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 -- 172.21.15.99:0/2656983505 shutdown_connections 2024-03-04T10:09:02.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.430+0000 7f531ce75640 1 -- 172.21.15.99:0/2656983505 wait complete. 2024-03-04T10:09:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 Processor -- start 2024-03-04T10:09:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 -- start start 2024-03-04T10:09:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53180b2e60 0x7f5318153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53180b5b80 0x7f53181580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 0x7f5318158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180d2820 con 0x7f531815de00 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180d24b0 con 0x7f53180b2e60 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.432+0000 7f531ce75640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53180b72a0 con 0x7f53180b5b80 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 0x7f5318158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 0x7f5318158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56318/0 (socket says 172.21.15.99:56318) 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 -- 172.21.15.99:0/1731645902 learned_addr learned my addr 172.21.15.99:0/1731645902 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 -- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53180b5b80 msgr2=0x7f53181580f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53180b5b80 0x7f53181580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 -- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53180b2e60 msgr2=0x7f5318153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53180b2e60 0x7f5318153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 -- 172.21.15.99:0/1731645902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181543d0 con 0x7f531815de00 2024-03-04T10:09:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f5316d9d640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 0x7f5318158630 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5310037040 tx=0x7f5310004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310044020 con 0x7f531815de00 2024-03-04T10:09:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5310034e30 con 0x7f531815de00 2024-03-04T10:09:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53180cb6d0 con 0x7f531815de00 2024-03-04T10:09:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310039040 con 0x7f531815de00 2024-03-04T10:09:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.433+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53180c9050 con 0x7f531815de00 2024-03-04T10:09:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.434+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53180cbbf0 con 0x7f531815de00 2024-03-04T10:09:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.435+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5310005d10 con 0x7f531815de00 2024-03-04T10:09:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.435+0000 7f530effd640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52f806b4d0 0x7f52f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.435+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f53100acdd0 con 0x7f531815de00 2024-03-04T10:09:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.435+0000 7f531659c640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52f806b4d0 0x7f52f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.436+0000 7f531659c640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52f806b4d0 0x7f52f806d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f53180cf2a0 tx=0x7f5308007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:02.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.439+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f531004e070 con 0x7f531815de00 2024-03-04T10:09:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.647+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5318074f60 con 0x7f531815de00 2024-03-04T10:09:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.648+0000 7f530effd640 1 -- 172.21.15.99:0/1731645902 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f53100ae020 con 0x7f531815de00 2024-03-04T10:09:02.649 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:02.649 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:19.179041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":15,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:09:02.650 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 57 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.649+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52f806b4d0 msgr2=0x7f52f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.649+0000 7f531ce75640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52f806b4d0 0x7f52f806d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f53180cf2a0 tx=0x7f5308007990 comp rx=0 tx=0).stop 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.649+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 msgr2=0x7f5318158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.649+0000 7f531ce75640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 0x7f5318158630 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f5310037040 tx=0x7f5310004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 shutdown_connections 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f52f806b4d0 0x7f52f806d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f531815de00 0x7f5318158630 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f53180b5b80 0x7f53181580f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 --2- 172.21.15.99:0/1731645902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53180b2e60 0x7f5318153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 >> 172.21.15.99:0/1731645902 conn(0x7f53180d0470 msgr2=0x7f53180b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 shutdown_connections 2024-03-04T10:09:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.650+0000 7f531ce75640 1 -- 172.21.15.99:0/1731645902 wait complete. 2024-03-04T10:09:02.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:02.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.760+0000 7fb9c0485640 1 Processor -- start 2024-03-04T10:09:02.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.760+0000 7fb9c0485640 1 -- start start 2024-03-04T10:09:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.760+0000 7fb9c0485640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b80cf650 0x7fb9b80cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9c0485640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb9b8070de0 0x7fb9b8071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9c0485640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b815f020 con 0x7fb9b8154a80 2024-03-04T10:09:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9c0485640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b80cfff0 con 0x7fb9b8070de0 2024-03-04T10:09:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9c0485640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80cd010 con 0x7fb9b80cf650 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bea22640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb9b8070de0 0x7fb9b8071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bda20640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b80cf650 0x7fb9b80cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9be221640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb9b8154a80 0x7fb9b815e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:39376/0 (socket says 172.21.15.99:39376) 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9be221640 1 -- 172.21.15.99:0/3832972295 learned_addr learned my addr 172.21.15.99:0/3832972295 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bda20640 1 -- 172.21.15.99:0/3832972295 >> v2:172.21.15.162:3300/0 conn(0x7fb9b8070de0 msgr2=0x7fb9b8071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bda20640 1 --2- 172.21.15.99:0/3832972295 >> v2:172.21.15.162:3300/0 conn(0x7fb9b8070de0 0x7fb9b8071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bda20640 1 -- 172.21.15.99:0/3832972295 >> v1:172.21.15.162:6789/0 conn(0x7fb9b8154a80 legacy=0x7fb9b815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bda20640 1 -- 172.21.15.99:0/3832972295 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b80cdaa0 con 0x7fb9b80cf650 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bd21f640 1 -- 172.21.15.99:0/3832972295 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214543879 0 0) 0x7fb9b4001110 con 0x7fb9b8154a80 2024-03-04T10:09:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.761+0000 7fb9bea22640 1 --2- 172.21.15.99:0/3832972295 >> v2:172.21.15.162:3300/0 conn(0x7fb9b8070de0 0x7fb9b8071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.762+0000 7fb9bda20640 1 --2- 172.21.15.99:0/3832972295 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b80cf650 0x7fb9b80cfab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80097a0 tx=0x7fb9a802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae27884c922f4b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.762+0000 7fb9bd21f640 1 -- 172.21.15.99:0/3832972295 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8035020 con 0x7fb9b80cf650 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.762+0000 7fb9bd21f640 1 -- 172.21.15.99:0/3832972295 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8031830 con 0x7fb9b80cf650 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.762+0000 7fb9bd21f640 1 -- 172.21.15.99:0/3832972295 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8031c00 con 0x7fb9b80cf650 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 -- 172.21.15.99:0/3832972295 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b80cf650 msgr2=0x7fb9b80cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 --2- 172.21.15.99:0/3832972295 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b80cf650 0x7fb9b80cfab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80097a0 tx=0x7fb9a802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 -- 172.21.15.99:0/3832972295 shutdown_connections 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 --2- 172.21.15.99:0/3832972295 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b80cf650 0x7fb9b80cfab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 --2- 172.21.15.99:0/3832972295 >> v2:172.21.15.162:3300/0 conn(0x7fb9b8070de0 0x7fb9b8071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 -- 172.21.15.99:0/3832972295 >> 172.21.15.99:0/3832972295 conn(0x7fb9b806f5f0 msgr2=0x7fb9b806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 -- 172.21.15.99:0/3832972295 shutdown_connections 2024-03-04T10:09:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.763+0000 7fb9c0485640 1 -- 172.21.15.99:0/3832972295 wait complete. 2024-03-04T10:09:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.764+0000 7fb9c0485640 1 Processor -- start 2024-03-04T10:09:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.764+0000 7fb9c0485640 1 -- start start 2024-03-04T10:09:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9b8070de0 0x7fb9b8155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 0x7fb9b81588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b8154a80 0x7fb9b80c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80ce340 con 0x7fb9b80cf650 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80ce060 con 0x7fb9b8070de0 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b80ce620 con 0x7fb9b8154a80 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 0x7fb9b81588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 0x7fb9b81588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56326/0 (socket says 172.21.15.99:56326) 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 -- 172.21.15.99:0/1800505691 learned_addr learned my addr 172.21.15.99:0/1800505691 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9be221640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9b8070de0 0x7fb9b8155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bea22640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b8154a80 0x7fb9b80c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 -- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b8154a80 msgr2=0x7fb9b80c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b8154a80 0x7fb9b80c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 -- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9b8070de0 msgr2=0x7fb9b8155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9b8070de0 0x7fb9b8155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 -- 172.21.15.99:0/1800505691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b8159320 con 0x7fb9b80cf650 2024-03-04T10:09:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9be221640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9b8070de0 0x7fb9b8155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9bda20640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 0x7fb9b81588d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8000c00 tx=0x7fb9a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8035070 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8004230 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b8154ef0 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.765+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a803a630 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.766+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b81554b0 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.766+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b8074f60 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.767+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb9a8038020 con 0x7fb9b80cf650 2024-03-04T10:09:02.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.767+0000 7fb9aeffd640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb99c06b4d0 0x7fb99c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:02.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.767+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fb9a80ac170 con 0x7fb9b80cf650 2024-03-04T10:09:02.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.768+0000 7fb9be221640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb99c06b4d0 0x7fb99c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:02.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.768+0000 7fb9be221640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb99c06b4d0 0x7fb99c06d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40013b0 tx=0x7fb9b4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:02.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.772+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb9a8040070 con 0x7fb9b80cf650 2024-03-04T10:09:02.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.943+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9b806c6b0 con 0x7fb9b80cf650 2024-03-04T10:09:02.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.944+0000 7fb9aeffd640 1 -- 172.21.15.99:0/1800505691 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17348 (secure 0 0 0) 0x7fb9a807ad10 con 0x7fb9b80cf650 2024-03-04T10:09:02.946 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:02.946 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":70,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:07:22.185227+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1f","mappings":[{"from":1,"to":2}]},{"pgid":"10.21","mappings":[{"from":4,"to":3}]},{"pgid":"10.34","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.3b","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6834/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6835/2687500379":"2024-03-05T10:06:46.776427+0000","172.21.15.99:6834/2687500379":"2024-03-05T10:06:46.776427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:02.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb99c06b4d0 msgr2=0x7fb99c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb99c06b4d0 0x7fb99c06d990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40013b0 tx=0x7fb9b4009040 comp rx=0 tx=0).stop 2024-03-04T10:09:02.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 msgr2=0x7fb9b81588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 0x7fb9b81588d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8000c00 tx=0x7fb9a80046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 shutdown_connections 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb99c06b4d0 0x7fb99c06d990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9b8154a80 0x7fb9b80c6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9b80cf650 0x7fb9b81588d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 --2- 172.21.15.99:0/1800505691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb9b8070de0 0x7fb9b8155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 >> 172.21.15.99:0/1800505691 conn(0x7fb9b806f5f0 msgr2=0x7fb9b80d1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 shutdown_connections 2024-03-04T10:09:02.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:02.946+0000 7fb9c0485640 1 -- 172.21.15.99:0/1800505691 wait complete. 2024-03-04T10:09:02.958 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:06:33.589849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:06:34.331851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:09:02.958 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:09:03.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.059+0000 7fa1d11c6640 1 Processor -- start 2024-03-04T10:09:03.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1d11c6640 1 -- start start 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1d11c6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa1cc15a810 0x7fa1cc0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1d11c6640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa1cc06d3c0 0x7fa1cc0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1d11c6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1cc06d8a0 con 0x7fa1cc0cbfb0 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1d11c6640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1cc06db80 con 0x7fa1cc15a810 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1d11c6640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1cc06de60 con 0x7fa1cc06d3c0 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.060+0000 7fa1cad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa1cc15a810 0x7fa1cc0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa1cc15a810 0x7fa1cc0b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56342/0 (socket says 172.21.15.99:56342) 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 -- 172.21.15.99:0/17219009 learned_addr learned my addr 172.21.15.99:0/17219009 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:03.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1ca59c640 1 --2- 172.21.15.99:0/17219009 >> v2:172.21.15.162:3300/0 conn(0x7fa1cc06d3c0 0x7fa1cc0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 -- 172.21.15.99:0/17219009 >> v2:172.21.15.162:3300/0 conn(0x7fa1cc06d3c0 msgr2=0x7fa1cc0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 --2- 172.21.15.99:0/17219009 >> v2:172.21.15.162:3300/0 conn(0x7fa1cc06d3c0 0x7fa1cc0b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 -- 172.21.15.99:0/17219009 >> v1:172.21.15.162:6789/0 conn(0x7fa1cc0cbfb0 legacy=0x7fa1cc15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 -- 172.21.15.99:0/17219009 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc06e140 con 0x7fa1cc15a810 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1ca59c640 1 --2- 172.21.15.99:0/17219009 >> v2:172.21.15.162:3300/0 conn(0x7fa1cc06d3c0 0x7fa1cc0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.061+0000 7fa1cad9d640 1 --2- 172.21.15.99:0/17219009 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0b4190 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0b4d00 tx=0x7fa1bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a612cfe7a32f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1c9d9b640 1 -- 172.21.15.99:0/17219009 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc035020 con 0x7fa1cc15a810 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1c9d9b640 1 -- 172.21.15.99:0/17219009 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc031870 con 0x7fa1cc15a810 2024-03-04T10:09:03.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1c9d9b640 1 -- 172.21.15.99:0/17219009 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc031c40 con 0x7fa1cc15a810 2024-03-04T10:09:03.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 -- 172.21.15.99:0/17219009 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 msgr2=0x7fa1cc0b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/17219009 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0b4190 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0b4d00 tx=0x7fa1bc02b910 comp rx=0 tx=0).stop 2024-03-04T10:09:03.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 -- 172.21.15.99:0/17219009 shutdown_connections 2024-03-04T10:09:03.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/17219009 >> v2:172.21.15.162:3300/0 conn(0x7fa1cc06d3c0 0x7fa1cc0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/17219009 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0b4190 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 -- 172.21.15.99:0/17219009 >> 172.21.15.99:0/17219009 conn(0x7fa1cc06acd0 msgr2=0x7fa1cc06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 -- 172.21.15.99:0/17219009 shutdown_connections 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.062+0000 7fa1d11c6640 1 -- 172.21.15.99:0/17219009 wait complete. 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 Processor -- start 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 -- start start 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa1cc06d3c0 0x7fa1cc14d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1cc0cbfb0 0x7fa1cc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc15e3c0 con 0x7fa1cc15a810 2024-03-04T10:09:03.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc0b3c50 con 0x7fa1cc0cbfb0 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1d11c6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1cc15e540 con 0x7fa1cc06d3c0 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.064+0000 7fa1cb59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56348/0 (socket says 172.21.15.99:56348) 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 -- 172.21.15.99:0/2239559754 learned_addr learned my addr 172.21.15.99:0/2239559754 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cad9d640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa1cc06d3c0 0x7fa1cc14d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1ca59c640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1cc0cbfb0 0x7fa1cc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 -- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa1cc06d3c0 msgr2=0x7fa1cc14d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa1cc06d3c0 0x7fa1cc14d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 -- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1cc0cbfb0 msgr2=0x7fa1cc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1cc0cbfb0 0x7fa1cc0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 -- 172.21.15.99:0/2239559754 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc1f0890 con 0x7fa1cc15a810 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1ca59c640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1cc0cbfb0 0x7fa1cc0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1cb59e640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0ce8b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400ea20 tx=0x7fa1c400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4004750 con 0x7fa1cc15a810 2024-03-04T10:09:03.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c4011470 con 0x7fa1cc15a810 2024-03-04T10:09:03.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c40118a0 con 0x7fa1cc15a810 2024-03-04T10:09:03.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1cc1f0b50 con 0x7fa1cc15a810 2024-03-04T10:09:03.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.065+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1cc1f1110 con 0x7fa1cc15a810 2024-03-04T10:09:03.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.066+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1cc074f60 con 0x7fa1cc15a810 2024-03-04T10:09:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.067+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa1c4011a40 con 0x7fa1cc15a810 2024-03-04T10:09:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.067+0000 7fa1b37fe640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa1b806b4d0 0x7fa1b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.067+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fa1c408cdf0 con 0x7fa1cc15a810 2024-03-04T10:09:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.067+0000 7fa1cad9d640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa1b806b4d0 0x7fa1b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.068+0000 7fa1cad9d640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa1b806b4d0 0x7fa1b806d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc036000 tx=0x7fa1bc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.071+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa1c4016030 con 0x7fa1cc15a810 2024-03-04T10:09:03.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.241+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1cc0afae0 con 0x7fa1cc15a810 2024-03-04T10:09:03.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.241+0000 7fa1b37fe640 1 -- 172.21.15.99:0/2239559754 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1c405ba30 con 0x7fa1cc15a810 2024-03-04T10:09:03.243 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:09:03.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa1b806b4d0 msgr2=0x7fa1b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa1b806b4d0 0x7fa1b806d990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc036000 tx=0x7fa1bc03c040 comp rx=0 tx=0).stop 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 msgr2=0x7fa1cc0ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0ce8b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400ea20 tx=0x7fa1c400eee0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 shutdown_connections 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa1b806b4d0 0x7fa1b806d990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa1cc15a810 0x7fa1cc0ce8b0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1cc0cbfb0 0x7fa1cc0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 --2- 172.21.15.99:0/2239559754 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa1cc06d3c0 0x7fa1cc14d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 >> 172.21.15.99:0/2239559754 conn(0x7fa1cc06acd0 msgr2=0x7fa1cc14f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 shutdown_connections 2024-03-04T10:09:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.243+0000 7fa1d11c6640 1 -- 172.21.15.99:0/2239559754 wait complete. 2024-03-04T10:09:03.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:09:03.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.356+0000 7ff93643d640 1 Processor -- start 2024-03-04T10:09:03.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.356+0000 7ff93643d640 1 -- start start 2024-03-04T10:09:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.357+0000 7ff93643d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 0x7ff9300b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.357+0000 7ff93643d640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9300ca970 con 0x7ff9300cec40 2024-03-04T10:09:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.357+0000 7ff93643d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff930161d90 con 0x7ff9300cf020 2024-03-04T10:09:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.357+0000 7ff93643d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff930162070 con 0x7ff93015de40 2024-03-04T10:09:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.357+0000 7ff92ffff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff9300cec40 0x7ff93015d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40892/0 (socket says 172.21.15.99:40892) 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.357+0000 7ff92ffff640 1 -- 172.21.15.99:0/1670941344 learned_addr learned my addr 172.21.15.99:0/1670941344 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 0x7ff9300b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241927523 0 0) 0x7ff9300ca970 con 0x7ff9300cec40 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff918003520 con 0x7ff9300cec40 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298337751 0 0) 0x7ff918003520 con 0x7ff9300cec40 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 msgr2=0x7ff9300b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 0x7ff9300b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 >> v1:172.21.15.162:6789/0 conn(0x7ff9300cf020 legacy=0x7ff9300d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff930162350 con 0x7ff9300cec40 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7ff920001690 con 0x7ff9300cec40 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 >> v1:172.21.15.99:6789/0 conn(0x7ff9300cec40 legacy=0x7ff93015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9180039e0 0x7ff918003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff918004800 0x7ff918004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 0x7ff9180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff930162070 con 0x7ff918004800 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800baa0 con 0x7ff9180039e0 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800bca0 con 0x7ff93015de40 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff918004800 0x7ff918004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92ffff640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9180039e0 0x7ff918003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff9349da640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 0x7ff9180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 -- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 msgr2=0x7ff9180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff93015de40 0x7ff9180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 -- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9180039e0 msgr2=0x7ff918003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9180039e0 0x7ff918003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.358+0000 7ff92f7fe640 1 -- 172.21.15.99:0/1670941344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918003520 con 0x7ff918004800 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.359+0000 7ff92f7fe640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff918004800 0x7ff918004c80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff93015e2a0 tx=0x7ff918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f4de63de988ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.359+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924039070 con 0x7ff918004800 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.359+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92402c630 con 0x7ff918004800 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.359+0000 7ff92effd640 1 -- 172.21.15.99:0/1670941344 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924009690 con 0x7ff918004800 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff93643d640 1 -- 172.21.15.99:0/1670941344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff918004800 msgr2=0x7ff918004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff93643d640 1 --2- 172.21.15.99:0/1670941344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff918004800 0x7ff918004c80 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff93015e2a0 tx=0x7ff918005830 comp rx=0 tx=0).stop 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff9349da640 1 -- 172.21.15.99:0/1670941344 reap_dead start 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff93643d640 1 -- 172.21.15.99:0/1670941344 shutdown_connections 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff93643d640 1 -- 172.21.15.99:0/1670941344 >> 172.21.15.99:0/1670941344 conn(0x7ff9300d0580 msgr2=0x7ff9300d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff93643d640 1 -- 172.21.15.99:0/1670941344 shutdown_connections 2024-03-04T10:09:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.360+0000 7ff93643d640 1 -- 172.21.15.99:0/1670941344 wait complete. 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.361+0000 7ff93643d640 1 Processor -- start 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.361+0000 7ff93643d640 1 -- start start 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff93643d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff918004800 0x7ff930156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff93643d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9300aa3b0 0x7ff93015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff93643d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 0x7ff93015ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff93643d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9300d3300 con 0x7ff9300aa830 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff93643d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9300d1bf0 con 0x7ff918004800 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff93643d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9300d3480 con 0x7ff9300aa3b0 2024-03-04T10:09:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 0x7ff93015ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 0x7ff93015ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56360/0 (socket says 172.21.15.99:56360) 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 -- 172.21.15.99:0/2723099513 learned_addr learned my addr 172.21.15.99:0/2723099513 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff92ffff640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff918004800 0x7ff930156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff92f7fe640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9300aa3b0 0x7ff93015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 -- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9300aa3b0 msgr2=0x7ff93015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9300aa3b0 0x7ff93015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 -- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff918004800 msgr2=0x7ff930156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff918004800 0x7ff930156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 -- 172.21.15.99:0/2723099513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93015a6c0 con 0x7ff9300aa830 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff92ffff640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff918004800 0x7ff930156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff9349da640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 0x7ff93015ba30 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff9300d1900 tx=0x7ff92800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.362+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92801a070 con 0x7ff9300aa830 2024-03-04T10:09:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.363+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff928006d60 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.363+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff928003b90 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.363+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93015a9a0 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.363+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93015af60 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.363+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff930074f60 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.364+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff92801d070 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.364+0000 7ff92cff9640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.364+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7ff928089ac0 con 0x7ff9300aa830 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.364+0000 7ff92ffff640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.365+0000 7ff92ffff640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff91c06b4d0 0x7ff91c06d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff920000c00 tx=0x7ff920009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.368+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff928058670 con 0x7ff9300aa830 2024-03-04T10:09:03.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.538+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9300d1930 con 0x7ff9300aa830 2024-03-04T10:09:03.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.538+0000 7ff92cff9640 1 -- 172.21.15.99:0/2723099513 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff928057f70 con 0x7ff9300aa830 2024-03-04T10:09:03.540 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff91c06b4d0 msgr2=0x7ff91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff91c06b4d0 0x7ff91c06d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff920000c00 tx=0x7ff920009040 comp rx=0 tx=0).stop 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 msgr2=0x7ff93015ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 0x7ff93015ba30 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff9300d1900 tx=0x7ff92800fd40 comp rx=0 tx=0).stop 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 shutdown_connections 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff9300aa830 0x7ff93015ba30 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff9300aa3b0 0x7ff93015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 --2- 172.21.15.99:0/2723099513 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff918004800 0x7ff930156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 >> 172.21.15.99:0/2723099513 conn(0x7ff9300d0580 msgr2=0x7ff9301605d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 shutdown_connections 2024-03-04T10:09:03.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.540+0000 7ff93643d640 1 -- 172.21.15.99:0/2723099513 wait complete. 2024-03-04T10:09:03.552 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:09:03.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.653+0000 7f0a16c97640 1 Processor -- start 2024-03-04T10:09:03.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.654+0000 7f0a16c97640 1 -- start start 2024-03-04T10:09:03.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.654+0000 7f0a16c97640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0a100b48a0 0x7f0a100b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.654+0000 7f0a16c97640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1006d8a0 con 0x7f0a1015a810 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.654+0000 7f0a16c97640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1006db80 con 0x7f0a1006d3c0 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.654+0000 7f0a16c97640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1006de60 con 0x7f0a100b48a0 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a14a33640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0a1015a810 0x7f0a100b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33286/0 (socket says 172.21.15.99:33286) 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a14a33640 1 -- 172.21.15.99:0/2373492418 learned_addr learned my addr 172.21.15.99:0/2373492418 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0ffff640 1 --2- 172.21.15.99:0/2373492418 >> v2:172.21.15.162:3300/0 conn(0x7f0a100b48a0 0x7f0a100b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021806204 0 0) 0x7f0a1006d8a0 con 0x7f0a1015a810 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a00003520 con 0x7f0a1015a810 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3727384382 0 0) 0x7f0a00003520 con 0x7f0a1015a810 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 >> v2:172.21.15.162:3300/0 conn(0x7f0a100b48a0 msgr2=0x7f0a100b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 --2- 172.21.15.99:0/2373492418 >> v2:172.21.15.162:3300/0 conn(0x7f0a100b48a0 0x7f0a100b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 >> v1:172.21.15.162:6789/0 conn(0x7f0a1006d3c0 legacy=0x7f0a100b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1006e140 con 0x7f0a1015a810 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f0a04001670 con 0x7f0a1015a810 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 >> v1:172.21.15.99:6789/0 conn(0x7f0a1015a810 legacy=0x7f0a100b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a000039b0 0x7f0a00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a000043b0 0x7f0a00004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a00005080 0x7f0a0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1006de60 con 0x7f0a00005080 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0000bfe0 con 0x7f0a000043b0 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0000c1e0 con 0x7f0a000039b0 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a15234640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a00005080 0x7f0a0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a0ffff640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a000043b0 0x7f0a00004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.655+0000 7f0a14a33640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a000039b0 0x7f0a00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 -- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a000039b0 msgr2=0x7f0a00003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a000039b0 0x7f0a00003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 -- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a000043b0 msgr2=0x7f0a00004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a000043b0 0x7f0a00004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 -- 172.21.15.99:0/2373492418 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a00003520 con 0x7f0a00005080 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 -- 172.21.15.99:0/2373492418 reap_dead start 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a15234640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a00005080 0x7f0a0000b9c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0a08001f30 tx=0x7f0a00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddb4541d5d431f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08001b30 con 0x7f0a00005080 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08003b90 con 0x7f0a00005080 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.656+0000 7f0a0f7fe640 1 -- 172.21.15.99:0/2373492418 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08005070 con 0x7f0a00005080 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 -- 172.21.15.99:0/2373492418 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a00005080 msgr2=0x7f0a0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a00005080 0x7f0a0000b9c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f0a08001f30 tx=0x7f0a00005830 comp rx=0 tx=0).stop 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 -- 172.21.15.99:0/2373492418 shutdown_connections 2024-03-04T10:09:03.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 --2- 172.21.15.99:0/2373492418 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a00005080 0x7f0a0000b9c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 -- 172.21.15.99:0/2373492418 >> 172.21.15.99:0/2373492418 conn(0x7f0a1006acd0 msgr2=0x7f0a1006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 -- 172.21.15.99:0/2373492418 shutdown_connections 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.657+0000 7f0a16c97640 1 -- 172.21.15.99:0/2373492418 wait complete. 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.658+0000 7f0a16c97640 1 Processor -- start 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.658+0000 7f0a16c97640 1 -- start start 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a16c97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a00005080 0x7f0a1015c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a16c97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a10154020 0x7f0a10159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a16c97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 0x7f0a10153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a16c97640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1015e3c0 con 0x7f0a1015da20 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a16c97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a100cc1c0 con 0x7f0a00005080 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a16c97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1015e540 con 0x7f0a10154020 2024-03-04T10:09:03.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 0x7f0a10153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 0x7f0a10153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57022/0 (socket says 172.21.15.99:57022) 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 -- 172.21.15.99:0/1399910179 learned_addr learned my addr 172.21.15.99:0/1399910179 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a14a33640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a00005080 0x7f0a1015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a0ffff640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a10154020 0x7f0a10159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 -- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a10154020 msgr2=0x7f0a10159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a10154020 0x7f0a10159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 -- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a00005080 msgr2=0x7f0a1015c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.659+0000 7f0a15234640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a00005080 0x7f0a1015c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a15234640 1 -- 172.21.15.99:0/1399910179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1015c980 con 0x7f0a1015da20 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a15234640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 0x7f0a10153430 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0a08002050 tx=0x7f0a080015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08006b50 con 0x7f0a1015da20 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08006cf0 con 0x7f0a1015da20 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0802d6c0 con 0x7f0a1015da20 2024-03-04T10:09:03.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1015cc60 con 0x7f0a1015da20 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.660+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1015d220 con 0x7f0a1015da20 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.661+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a10074f60 con 0x7f0a1015da20 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.661+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0a08040070 con 0x7f0a1015da20 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.662+0000 7f0a0d7fa640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.662+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f0a080ab8b0 con 0x7f0a1015da20 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.662+0000 7f0a14a33640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.662+0000 7f0a14a33640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f09f406b4d0 0x7f09f406d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0a04000c00 tx=0x7f0a040078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.666+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0a0807a520 con 0x7f0a1015da20 2024-03-04T10:09:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.835+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0a100cbc60 con 0x7f0a1015da20 2024-03-04T10:09:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.835+0000 7f0a0d7fa640 1 -- 172.21.15.99:0/1399910179 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0a08079e20 con 0x7f0a1015da20 2024-03-04T10:09:03.837 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f09f406b4d0 msgr2=0x7f09f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f09f406b4d0 0x7f09f406d990 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0a04000c00 tx=0x7f0a040078f0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 msgr2=0x7f0a10153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 0x7f0a10153430 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f0a08002050 tx=0x7f0a080015b0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 shutdown_connections 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a1015da20 0x7f0a10153430 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a10154020 0x7f0a10159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 --2- 172.21.15.99:0/1399910179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a00005080 0x7f0a1015c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 >> 172.21.15.99:0/1399910179 conn(0x7f0a1006acd0 msgr2=0x7f0a100b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 shutdown_connections 2024-03-04T10:09:03.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.837+0000 7f0a16c97640 1 -- 172.21.15.99:0/1399910179 wait complete. 2024-03-04T10:09:03.849 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:03.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.948+0000 7feb57959640 1 Processor -- start 2024-03-04T10:09:03.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.948+0000 7feb57959640 1 -- start start 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb57959640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb5015dea0 0x7feb500cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb57959640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7feb500cc8d0 0x7feb5006acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb57959640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5006fd20 con 0x7feb500d1200 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb57959640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5006fad0 con 0x7feb5015dea0 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb57959640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb50070000 con 0x7feb500cc8d0 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb556f5640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb5015dea0 0x7feb500cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb556f5640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7feb5015dea0 0x7feb500cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57034/0 (socket says 172.21.15.99:57034) 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb556f5640 1 -- 172.21.15.99:0/4088323811 learned_addr learned my addr 172.21.15.99:0/4088323811 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb54ef4640 1 --2- 172.21.15.99:0/4088323811 >> v2:172.21.15.162:3300/0 conn(0x7feb500cc8d0 0x7feb5006acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb556f5640 1 -- 172.21.15.99:0/4088323811 >> v2:172.21.15.162:3300/0 conn(0x7feb500cc8d0 msgr2=0x7feb5006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb556f5640 1 --2- 172.21.15.99:0/4088323811 >> v2:172.21.15.162:3300/0 conn(0x7feb500cc8d0 0x7feb5006acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.949+0000 7feb556f5640 1 -- 172.21.15.99:0/4088323811 >> v1:172.21.15.162:6789/0 conn(0x7feb500d1200 legacy=0x7feb500cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:03.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.950+0000 7feb556f5640 1 -- 172.21.15.99:0/4088323811 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500702e0 con 0x7feb5015dea0 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.950+0000 7feb556f5640 1 --2- 172.21.15.99:0/4088323811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb5015dea0 0x7feb500cad60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7feb5006e310 tx=0x7feb4c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8897ef26bb4e2c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.950+0000 7feb47fff640 1 -- 172.21.15.99:0/4088323811 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c02cb20 con 0x7feb5015dea0 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.950+0000 7feb47fff640 1 -- 172.21.15.99:0/4088323811 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c02ccc0 con 0x7feb5015dea0 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.950+0000 7feb47fff640 1 -- 172.21.15.99:0/4088323811 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c034610 con 0x7feb5015dea0 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 -- 172.21.15.99:0/4088323811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb5015dea0 msgr2=0x7feb500cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 --2- 172.21.15.99:0/4088323811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb5015dea0 0x7feb500cad60 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7feb5006e310 tx=0x7feb4c02bf80 comp rx=0 tx=0).stop 2024-03-04T10:09:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 -- 172.21.15.99:0/4088323811 shutdown_connections 2024-03-04T10:09:03.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 --2- 172.21.15.99:0/4088323811 >> v2:172.21.15.162:3300/0 conn(0x7feb500cc8d0 0x7feb5006acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 --2- 172.21.15.99:0/4088323811 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb5015dea0 0x7feb500cad60 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 -- 172.21.15.99:0/4088323811 >> 172.21.15.99:0/4088323811 conn(0x7feb500d0710 msgr2=0x7feb500d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:03.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 -- 172.21.15.99:0/4088323811 shutdown_connections 2024-03-04T10:09:03.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.951+0000 7feb57959640 1 -- 172.21.15.99:0/4088323811 wait complete. 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.952+0000 7feb57959640 1 Processor -- start 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.952+0000 7feb57959640 1 -- start start 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb57959640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 0x7feb501f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb57959640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb500d1200 0x7feb500c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb57959640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb5015dea0 0x7feb50151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb57959640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5006b390 con 0x7feb500cc8d0 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb57959640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5006b210 con 0x7feb5015dea0 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb57959640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5006b510 con 0x7feb500d1200 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 0x7feb501f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 0x7feb501f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57044/0 (socket says 172.21.15.99:57044) 2024-03-04T10:09:03.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 -- 172.21.15.99:0/3013861344 learned_addr learned my addr 172.21.15.99:0/3013861344 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 -- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb500d1200 msgr2=0x7feb500c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb55ef6640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb5015dea0 0x7feb50151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb500d1200 0x7feb500c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 -- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb5015dea0 msgr2=0x7feb50151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb5015dea0 0x7feb50151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 -- 172.21.15.99:0/3013861344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500c4ef0 con 0x7feb500cc8d0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.953+0000 7feb556f5640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 0x7feb501f4210 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7feb4c000f80 tx=0x7feb4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.954+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c043020 con 0x7feb500cc8d0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.954+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c03e450 con 0x7feb500cc8d0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.954+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb500c51d0 con 0x7feb500cc8d0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.954+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c009510 con 0x7feb500cc8d0 2024-03-04T10:09:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.954+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb500c5790 con 0x7feb500cc8d0 2024-03-04T10:09:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.955+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb50074f60 con 0x7feb500cc8d0 2024-03-04T10:09:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.955+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7feb4c046020 con 0x7feb500cc8d0 2024-03-04T10:09:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.956+0000 7feb45ffb640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.956+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7feb4c0acc70 con 0x7feb500cc8d0 2024-03-04T10:09:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.956+0000 7feb54ef4640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.956+0000 7feb54ef4640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7feb40006fd0 tx=0x7feb40008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:03.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:03.960+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feb4c04d030 con 0x7feb500cc8d0 2024-03-04T10:09:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.164+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb500caa40 con 0x7feb500cc8d0 2024-03-04T10:09:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.165+0000 7feb45ffb640 1 -- 172.21.15.99:0/3013861344 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4944 (secure 0 0 0) 0x7feb4c07b8b0 con 0x7feb500cc8d0 2024-03-04T10:09:04.166 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:04.166 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:19.179041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":15,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:09:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 57 2024-03-04T10:09:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.166+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb3406b4d0 msgr2=0x7feb3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7feb40006fd0 tx=0x7feb40008040 comp rx=0 tx=0).stop 2024-03-04T10:09:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 msgr2=0x7feb501f4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 0x7feb501f4210 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7feb4c000f80 tx=0x7feb4c004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 shutdown_connections 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb5015dea0 0x7feb50151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb500d1200 0x7feb500c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 --2- 172.21.15.99:0/3013861344 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb500cc8d0 0x7feb501f4210 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 >> 172.21.15.99:0/3013861344 conn(0x7feb500d0710 msgr2=0x7feb500b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 shutdown_connections 2024-03-04T10:09:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.167+0000 7feb57959640 1 -- 172.21.15.99:0/3013861344 wait complete. 2024-03-04T10:09:04.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:04.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.277+0000 7fdf5b309640 1 Processor -- start 2024-03-04T10:09:04.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.278+0000 7fdf5b309640 1 -- start start 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.278+0000 7fdf5b309640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fdf540cc8d0 0x7fdf540cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.278+0000 7fdf5b309640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5406fd20 con 0x7fdf5406b3e0 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.278+0000 7fdf5b309640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf54070000 con 0x7fdf5415dea0 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.278+0000 7fdf5b309640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5406fad0 con 0x7fdf540cc8d0 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf588a4640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdf5406b3e0 0x7fdf540cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33292/0 (socket says 172.21.15.99:33292) 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf588a4640 1 -- 172.21.15.99:0/1937487008 learned_addr learned my addr 172.21.15.99:0/1937487008 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf598a6640 1 --2- 172.21.15.99:0/1937487008 >> v2:172.21.15.162:3300/0 conn(0x7fdf540cc8d0 0x7fdf540cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354884886 0 0) 0x7fdf5406fd20 con 0x7fdf5406b3e0 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf40003520 con 0x7fdf5406b3e0 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605190062 0 0) 0x7fdf40003520 con 0x7fdf5406b3e0 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 >> v2:172.21.15.162:3300/0 conn(0x7fdf540cc8d0 msgr2=0x7fdf540cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 --2- 172.21.15.99:0/1937487008 >> v2:172.21.15.162:3300/0 conn(0x7fdf540cc8d0 0x7fdf540cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 >> v1:172.21.15.162:6789/0 conn(0x7fdf5415dea0 legacy=0x7fdf5406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540702e0 con 0x7fdf5406b3e0 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fdf50001670 con 0x7fdf5406b3e0 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 >> v1:172.21.15.99:6789/0 conn(0x7fdf5406b3e0 legacy=0x7fdf540cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf400039b0 0x7fdf40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf40004800 0x7fdf40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5406fad0 con 0x7fdf400039b0 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4000c060 con 0x7fdf40004800 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4000c260 con 0x7fdf400051c0 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf590a5640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf400039b0 0x7fdf40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf588a4640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf40004800 0x7fdf40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.279+0000 7fdf598a6640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf590a5640 1 -- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400051c0 msgr2=0x7fdf4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf590a5640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf590a5640 1 -- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf40004800 msgr2=0x7fdf40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf590a5640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf40004800 0x7fdf40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf590a5640 1 -- 172.21.15.99:0/1937487008 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf40003520 con 0x7fdf400039b0 2024-03-04T10:09:04.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf598a6640 1 -- 172.21.15.99:0/1937487008 reap_dead start 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf590a5640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf400039b0 0x7fdf40003dd0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fdf48001750 tx=0x7fdf40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f8bcce4c865389 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48001370 con 0x7fdf400039b0 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf480052d0 con 0x7fdf400039b0 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.280+0000 7fdf47fff640 1 -- 172.21.15.99:0/1937487008 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48003830 con 0x7fdf400039b0 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 -- 172.21.15.99:0/1937487008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf400039b0 msgr2=0x7fdf40003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf400039b0 0x7fdf40003dd0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fdf48001750 tx=0x7fdf40005830 comp rx=0 tx=0).stop 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 -- 172.21.15.99:0/1937487008 shutdown_connections 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 --2- 172.21.15.99:0/1937487008 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf400039b0 0x7fdf40003dd0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 -- 172.21.15.99:0/1937487008 >> 172.21.15.99:0/1937487008 conn(0x7fdf540d0710 msgr2=0x7fdf540d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 -- 172.21.15.99:0/1937487008 shutdown_connections 2024-03-04T10:09:04.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.281+0000 7fdf5b309640 1 -- 172.21.15.99:0/1937487008 wait complete. 2024-03-04T10:09:04.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.282+0000 7fdf5b309640 1 Processor -- start 2024-03-04T10:09:04.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.282+0000 7fdf5b309640 1 -- start start 2024-03-04T10:09:04.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf5b309640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400039b0 0x7fdf540c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf5b309640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 0x7fdf541f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf5b309640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf541f4260 0x7fdf541f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf5b309640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d1380 con 0x7fdf540c6810 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf5b309640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d1200 con 0x7fdf541f4260 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf5b309640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540d1500 con 0x7fdf400039b0 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 0x7fdf541f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 0x7fdf541f3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57072/0 (socket says 172.21.15.99:57072) 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 -- 172.21.15.99:0/2995522774 learned_addr learned my addr 172.21.15.99:0/2995522774 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf590a5640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400039b0 0x7fdf540c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf598a6640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf541f4260 0x7fdf541f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 -- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400039b0 msgr2=0x7fdf540c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400039b0 0x7fdf540c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 -- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf541f4260 msgr2=0x7fdf541f6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf541f4260 0x7fdf541f6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 -- 172.21.15.99:0/2995522774 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540c56f0 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.283+0000 7fdf588a4640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 0x7fdf541f3d20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fdf500072f0 tx=0x7fdf500077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.284+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5001a070 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.284+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5000f960 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.284+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf540c4650 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.284+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5000fd50 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.284+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf540c4c10 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.285+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf54074f60 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.285+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdf5001d070 con 0x7fdf540c6810 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.286+0000 7fdf45ffb640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b4d0 0x7fdf3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.286+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fdf50089d80 con 0x7fdf540c6810 2024-03-04T10:09:04.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.286+0000 7fdf590a5640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b4d0 0x7fdf3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.286+0000 7fdf590a5640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b4d0 0x7fdf3806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdf48033000 tx=0x7fdf48009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.290+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdf50058930 con 0x7fdf540c6810 2024-03-04T10:09:04.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.463+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf540ce210 con 0x7fdf540c6810 2024-03-04T10:09:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.464+0000 7fdf45ffb640 1 -- 172.21.15.99:0/2995522774 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17348 (secure 0 0 0) 0x7fdf50058230 con 0x7fdf540c6810 2024-03-04T10:09:04.466 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:04.466 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":70,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:07:22.185227+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1f","mappings":[{"from":1,"to":2}]},{"pgid":"10.21","mappings":[{"from":4,"to":3}]},{"pgid":"10.34","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.3b","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6834/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6835/2687500379":"2024-03-05T10:06:46.776427+0000","172.21.15.99:6834/2687500379":"2024-03-05T10:06:46.776427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b4d0 msgr2=0x7fdf3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b4d0 0x7fdf3806d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdf48033000 tx=0x7fdf48009030 comp rx=0 tx=0).stop 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 msgr2=0x7fdf541f3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 0x7fdf541f3d20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fdf500072f0 tx=0x7fdf500077b0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 shutdown_connections 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b4d0 0x7fdf3806d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf541f4260 0x7fdf541f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf540c6810 0x7fdf541f3d20 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 --2- 172.21.15.99:0/2995522774 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf400039b0 0x7fdf540c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.466+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 >> 172.21.15.99:0/2995522774 conn(0x7fdf540d0710 msgr2=0x7fdf540d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.467+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 shutdown_connections 2024-03-04T10:09:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.467+0000 7fdf5b309640 1 -- 172.21.15.99:0/2995522774 wait complete. 2024-03-04T10:09:04.479 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:09:04.479 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:04.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 Processor -- start 2024-03-04T10:09:04.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 -- start start 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f362015d660 0x7f362015e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f36200aeb20 0x7f3620071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3620071890 0x7f3620150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362015da40 con 0x7f3620071890 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362015ebb0 con 0x7f362015d660 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.578+0000 7f3625dd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36200af090 con 0x7f36200aeb20 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3620071890 0x7f3620150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3620071890 0x7f3620150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57084/0 (socket says 172.21.15.99:57084) 2024-03-04T10:09:04.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 -- 172.21.15.99:0/2231563177 learned_addr learned my addr 172.21.15.99:0/2231563177 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 -- 172.21.15.99:0/2231563177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f36200aeb20 msgr2=0x7f3620071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 --2- 172.21.15.99:0/2231563177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f36200aeb20 0x7f3620071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 -- 172.21.15.99:0/2231563177 >> v2:172.21.15.162:3300/0 conn(0x7f362015d660 msgr2=0x7f362015e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361f7fe640 1 --2- 172.21.15.99:0/2231563177 >> v2:172.21.15.162:3300/0 conn(0x7f362015d660 0x7f362015e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 --2- 172.21.15.99:0/2231563177 >> v2:172.21.15.162:3300/0 conn(0x7f362015d660 0x7f362015e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.579+0000 7f361ffff640 1 -- 172.21.15.99:0/2231563177 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362015a9f0 con 0x7f3620071890 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f361ffff640 1 --2- 172.21.15.99:0/2231563177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3620071890 0x7f3620150960 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f36200b4e10 tx=0x7f361802dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9886b8e107c2042 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f361effd640 1 -- 172.21.15.99:0/2231563177 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361802e960 con 0x7f3620071890 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f361effd640 1 -- 172.21.15.99:0/2231563177 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361802eb00 con 0x7f3620071890 2024-03-04T10:09:04.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f361effd640 1 -- 172.21.15.99:0/2231563177 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618008500 con 0x7f3620071890 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 -- 172.21.15.99:0/2231563177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3620071890 msgr2=0x7f3620150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 --2- 172.21.15.99:0/2231563177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3620071890 0x7f3620150960 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f36200b4e10 tx=0x7f361802dfc0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 -- 172.21.15.99:0/2231563177 shutdown_connections 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 --2- 172.21.15.99:0/2231563177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3620071890 0x7f3620150960 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 --2- 172.21.15.99:0/2231563177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f36200aeb20 0x7f3620071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 --2- 172.21.15.99:0/2231563177 >> v2:172.21.15.162:3300/0 conn(0x7f362015d660 0x7f362015e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 -- 172.21.15.99:0/2231563177 >> 172.21.15.99:0/2231563177 conn(0x7f362006acd0 msgr2=0x7f362006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 -- 172.21.15.99:0/2231563177 shutdown_connections 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.580+0000 7f3625dd4640 1 -- 172.21.15.99:0/2231563177 wait complete. 2024-03-04T10:09:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 Processor -- start 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 -- start start 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3620071890 0x7f36201f1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36200aeb20 0x7f36200cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 0x7f3620151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362015a660 con 0x7f362015d660 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362015afb0 con 0x7f36200aeb20 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f3625dd4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362015b4d0 con 0x7f3620071890 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f361ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 0x7f3620151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f361ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 0x7f3620151930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57100/0 (socket says 172.21.15.99:57100) 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.582+0000 7f361ffff640 1 -- 172.21.15.99:0/3912768958 learned_addr learned my addr 172.21.15.99:0/3912768958 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361f7fe640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3620071890 0x7f36201f1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f3617fff640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36200aeb20 0x7f36200cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361ffff640 1 -- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3620071890 msgr2=0x7f36201f1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361ffff640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3620071890 0x7f36201f1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361ffff640 1 -- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36200aeb20 msgr2=0x7f36200cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361ffff640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36200aeb20 0x7f36200cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361ffff640 1 -- 172.21.15.99:0/3912768958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620154e20 con 0x7f362015d660 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361ffff640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 0x7f3620151930 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3618038040 tx=0x7f3618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618045020 con 0x7f362015d660 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618040450 con 0x7f362015d660 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36201f2300 con 0x7f362015d660 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361800a620 con 0x7f362015d660 2024-03-04T10:09:04.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.583+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36201550c0 con 0x7f362015d660 2024-03-04T10:09:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.584+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3620074f60 con 0x7f362015d660 2024-03-04T10:09:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.584+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3618048020 con 0x7f362015d660 2024-03-04T10:09:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.585+0000 7f361cff9640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f360006b4d0 0x7f360006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.585+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f36180aef80 con 0x7f362015d660 2024-03-04T10:09:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.585+0000 7f361f7fe640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f360006b4d0 0x7f360006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.585+0000 7f361f7fe640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f360006b4d0 0x7f360006d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f361000a970 tx=0x7f3610008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.589+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f361804f030 con 0x7f362015d660 2024-03-04T10:09:04.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.798+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3620072460 con 0x7f362015d660 2024-03-04T10:09:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.798+0000 7f361cff9640 1 -- 172.21.15.99:0/3912768958 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f361807dbc0 con 0x7f362015d660 2024-03-04T10:09:04.800 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:04.800 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:19.179041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":15,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:09:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 57 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.800+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f360006b4d0 msgr2=0x7f360006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.800+0000 7f3625dd4640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f360006b4d0 0x7f360006d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f361000a970 tx=0x7f3610008040 comp rx=0 tx=0).stop 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.800+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 msgr2=0x7f3620151930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.800+0000 7f3625dd4640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 0x7f3620151930 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3618038040 tx=0x7f3618004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 shutdown_connections 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f360006b4d0 0x7f360006d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f362015d660 0x7f3620151930 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36200aeb20 0x7f36200cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 --2- 172.21.15.99:0/3912768958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3620071890 0x7f36201f1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 >> 172.21.15.99:0/3912768958 conn(0x7f362006acd0 msgr2=0x7f3620154090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 shutdown_connections 2024-03-04T10:09:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.801+0000 7f3625dd4640 1 -- 172.21.15.99:0/3912768958 wait complete. 2024-03-04T10:09:04.813 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.912+0000 7f87f979f640 1 Processor -- start 2024-03-04T10:09:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.912+0000 7f87f979f640 1 -- start start 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.912+0000 7f87f979f640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f87f41613a0 0x7f87f4099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.912+0000 7f87f979f640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f40d1230 con 0x7f87f4099f40 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.912+0000 7f87f979f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f409a320 con 0x7f87f4160fc0 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.912+0000 7f87f979f640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f40cfa80 con 0x7f87f41613a0 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f87f41613a0 0x7f87f4099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f8f9e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f87f4099f40 0x7f87f40cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33302/0 (socket says 172.21.15.99:33302) 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f87f41613a0 0x7f87f4099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57116/0 (socket says 172.21.15.99:57116) 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f8f9e640 1 -- 172.21.15.99:0/48224282 learned_addr learned my addr 172.21.15.99:0/48224282 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:04.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f37fe640 1 -- 172.21.15.99:0/48224282 >> v1:172.21.15.162:6789/0 conn(0x7f87f4160fc0 legacy=0x7f87f40d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f37fe640 1 -- 172.21.15.99:0/48224282 >> v1:172.21.15.99:6789/0 conn(0x7f87f4099f40 legacy=0x7f87f40cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f37fe640 1 -- 172.21.15.99:0/48224282 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40cfd60 con 0x7f87f41613a0 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.913+0000 7f87f2ffd640 1 -- 172.21.15.99:0/48224282 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836797156 0 0) 0x7f87f40d1230 con 0x7f87f4099f40 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f37fe640 1 --2- 172.21.15.99:0/48224282 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f41613a0 0x7f87f4099a00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f87e80097a0 tx=0x7f87e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3f66f21003f568 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f2ffd640 1 -- 172.21.15.99:0/48224282 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8034070 con 0x7f87f41613a0 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f2ffd640 1 -- 172.21.15.99:0/48224282 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e8030610 con 0x7f87f41613a0 2024-03-04T10:09:04.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f2ffd640 1 -- 172.21.15.99:0/48224282 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e80309e0 con 0x7f87f41613a0 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 -- 172.21.15.99:0/48224282 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f41613a0 msgr2=0x7f87f4099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 --2- 172.21.15.99:0/48224282 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f41613a0 0x7f87f4099a00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f87e80097a0 tx=0x7f87e802b960 comp rx=0 tx=0).stop 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 -- 172.21.15.99:0/48224282 shutdown_connections 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 --2- 172.21.15.99:0/48224282 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f41613a0 0x7f87f4099a00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 -- 172.21.15.99:0/48224282 >> 172.21.15.99:0/48224282 conn(0x7f87f40b3200 msgr2=0x7f87f40b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 -- 172.21.15.99:0/48224282 shutdown_connections 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.914+0000 7f87f979f640 1 -- 172.21.15.99:0/48224282 wait complete. 2024-03-04T10:09:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.915+0000 7f87f979f640 1 Processor -- start 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 -- start start 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f4099f40 0x7f87f4152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 0x7f87f40b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f41613a0 0x7f87f40b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40d0600 con 0x7f87f4160fc0 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40d0320 con 0x7f87f4099f40 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f979f640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40cc470 con 0x7f87f41613a0 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 0x7f87f40b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 0x7f87f40b5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57132/0 (socket says 172.21.15.99:57132) 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.916+0000 7f87f37fe640 1 -- 172.21.15.99:0/2872874087 learned_addr learned my addr 172.21.15.99:0/2872874087 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f3fff640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f4099f40 0x7f87f4152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f8f9e640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f41613a0 0x7f87f40b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f37fe640 1 -- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f41613a0 msgr2=0x7f87f40b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f37fe640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f41613a0 0x7f87f40b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f37fe640 1 -- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f4099f40 msgr2=0x7f87f4152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f37fe640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f4099f40 0x7f87f4152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f37fe640 1 -- 172.21.15.99:0/2872874087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40b4b40 con 0x7f87f4160fc0 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f3fff640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f4099f40 0x7f87f4152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f37fe640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 0x7f87f40b5be0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f87e802bf70 tx=0x7f87e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8034040 con 0x7f87f4160fc0 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e803acc0 con 0x7f87f4160fc0 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f40b4e20 con 0x7f87f4160fc0 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8044630 con 0x7f87f4160fc0 2024-03-04T10:09:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.917+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f40b53e0 con 0x7f87f4160fc0 2024-03-04T10:09:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.918+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f404f100 con 0x7f87f4160fc0 2024-03-04T10:09:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.918+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f87e8037070 con 0x7f87f4160fc0 2024-03-04T10:09:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.919+0000 7f87f0ff9640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87e006b4d0 0x7f87e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.919+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f87e80ac530 con 0x7f87f4160fc0 2024-03-04T10:09:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.919+0000 7f87f3fff640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87e006b4d0 0x7f87e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.920+0000 7f87f3fff640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87e006b4d0 0x7f87e006d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f87e4007fd0 tx=0x7f87e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:04.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:04.923+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f87e803f070 con 0x7f87f4160fc0 2024-03-04T10:09:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.130+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f41532a0 con 0x7f87f4160fc0 2024-03-04T10:09:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.130+0000 7f87f0ff9640 1 -- 172.21.15.99:0/2872874087 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4944 (secure 0 0 0) 0x7f87e807b050 con 0x7f87f4160fc0 2024-03-04T10:09:05.132 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:05.132 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:06:35.354323+0000","modified":"2024-03-04T10:07:19.179041+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":15,"addr":"172.21.15.99:6833/665274836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":665274836},{"type":"v1","addr":"172.21.15.99:6833","nonce":665274836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-04T10:09:05.132 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 57 2024-03-04T10:09:05.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87e006b4d0 msgr2=0x7f87e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87e006b4d0 0x7f87e006d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f87e4007fd0 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 msgr2=0x7f87f40b5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 0x7f87f40b5be0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f87e802bf70 tx=0x7f87e80046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 shutdown_connections 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f87e006b4d0 0x7f87e006d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f87f41613a0 0x7f87f40b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f87f4160fc0 0x7f87f40b5be0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 --2- 172.21.15.99:0/2872874087 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f87f4099f40 0x7f87f4152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 >> 172.21.15.99:0/2872874087 conn(0x7f87f40b3200 msgr2=0x7f87f406f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 shutdown_connections 2024-03-04T10:09:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.132+0000 7f87f979f640 1 -- 172.21.15.99:0/2872874087 wait complete. 2024-03-04T10:09:05.144 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:05.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.243+0000 7fafae759640 1 Processor -- start 2024-03-04T10:09:05.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.243+0000 7fafae759640 1 -- start start 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafae759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa80aeb20 0x7fafa815d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafae759640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafa8071350 0x7fafa80717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafae759640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80af090 con 0x7fafa815e670 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafae759640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80cfe20 con 0x7fafa8071350 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafae759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa815a9f0 con 0x7fafa80aeb20 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafaccf6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafa8071350 0x7fafa80717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafa77fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fafa815e670 0x7fafa80cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33310/0 (socket says 172.21.15.99:33310) 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.244+0000 7fafa77fe640 1 -- 172.21.15.99:0/312478819 learned_addr learned my addr 172.21.15.99:0/312478819 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:05.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafa7fff640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa80aeb20 0x7fafa815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafaccf6640 1 -- 172.21.15.99:0/312478819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa80aeb20 msgr2=0x7fafa815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafaccf6640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa80aeb20 0x7fafa815d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafaccf6640 1 -- 172.21.15.99:0/312478819 >> v1:172.21.15.99:6789/0 conn(0x7fafa815e670 legacy=0x7fafa80cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafaccf6640 1 -- 172.21.15.99:0/312478819 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa815acd0 con 0x7fafa8071350 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafa7fff640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa80aeb20 0x7fafa815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafaccf6640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa8071350 0x7fafa80717e0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009d90 tx=0x7fafa002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e924494bf602d540 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.245+0000 7fafa6ffd640 1 -- 172.21.15.99:0/312478819 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0039070 con 0x7fafa8071350 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafa6ffd640 1 -- 172.21.15.99:0/312478819 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa002cbc0 con 0x7fafa8071350 2024-03-04T10:09:05.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafa6ffd640 1 -- 172.21.15.99:0/312478819 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0034440 con 0x7fafa8071350 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 -- 172.21.15.99:0/312478819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa8071350 msgr2=0x7fafa80717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa8071350 0x7fafa80717e0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fafa0009d90 tx=0x7fafa002be10 comp rx=0 tx=0).stop 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 -- 172.21.15.99:0/312478819 shutdown_connections 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa8071350 0x7fafa80717e0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 --2- 172.21.15.99:0/312478819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa80aeb20 0x7fafa815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 -- 172.21.15.99:0/312478819 >> 172.21.15.99:0/312478819 conn(0x7fafa806acd0 msgr2=0x7fafa806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:05.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 -- 172.21.15.99:0/312478819 shutdown_connections 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.246+0000 7fafae759640 1 -- 172.21.15.99:0/312478819 wait complete. 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.247+0000 7fafae759640 1 Processor -- start 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.247+0000 7fafae759640 1 -- start start 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafae759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa8071350 0x7fafa814a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafae759640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 0x7fafa815bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafae759640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa815e670 0x7fafa815c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafae759640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa815b4d0 con 0x7fafa80aeb20 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafae759640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa815a660 con 0x7fafa815e670 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafae759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa814dfb0 con 0x7fafa8071350 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafa77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 0x7fafa815bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafa77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 0x7fafa815bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57136/0 (socket says 172.21.15.99:57136) 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafa77fe640 1 -- 172.21.15.99:0/4191552709 learned_addr learned my addr 172.21.15.99:0/4191552709 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafa7fff640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa8071350 0x7fafa814a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.248+0000 7fafaccf6640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa815e670 0x7fafa815c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa77fe640 1 -- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa8071350 msgr2=0x7fafa814a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa77fe640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa8071350 0x7fafa814a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa77fe640 1 -- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa815e670 msgr2=0x7fafa815c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa77fe640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa815e670 0x7fafa815c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa77fe640 1 -- 172.21.15.99:0/4191552709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa814a790 con 0x7fafa80aeb20 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafaccf6640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa815e670 0x7fafa815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa77fe640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 0x7fafa815bbb0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00c3c0 tx=0x7faf9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c017070 con 0x7fafa80aeb20 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9c005ce0 con 0x7fafa80aeb20 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa814b670 con 0x7fafa80aeb20 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c0072f0 con 0x7fafa80aeb20 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.249+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa814bbf0 con 0x7fafa80aeb20 2024-03-04T10:09:05.253 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.250+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa8074f60 con 0x7fafa80aeb20 2024-03-04T10:09:05.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.250+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7faf9c01a070 con 0x7fafa80aeb20 2024-03-04T10:09:05.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.251+0000 7fafa4ff9640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf9406b500 0x7faf9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.251+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7faf9c089780 con 0x7fafa80aeb20 2024-03-04T10:09:05.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.251+0000 7fafa7fff640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf9406b500 0x7faf9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.251+0000 7fafa7fff640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf9406b500 0x7faf9406d9c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7faf9800ab60 tx=0x7faf98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.255+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fafa814a790 con 0x7fafa80aeb20 2024-03-04T10:09:05.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.425+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafa815df50 con 0x7fafa80aeb20 2024-03-04T10:09:05.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.426+0000 7fafa4ff9640 1 -- 172.21.15.99:0/4191552709 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17348 (secure 0 0 0) 0x7faf9c057ce0 con 0x7fafa80aeb20 2024-03-04T10:09:05.428 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:05.428 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":70,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:07:22.185227+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:06:33.589849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-04T10:06:34.331851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.1f","mappings":[{"from":1,"to":2}]},{"pgid":"10.21","mappings":[{"from":4,"to":3}]},{"pgid":"10.34","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":4,"to":5}]},{"pgid":"10.3b","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6834/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6835/2687500379":"2024-03-05T10:06:46.776427+0000","172.21.15.99:6834/2687500379":"2024-03-05T10:06:46.776427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:05.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf9406b500 msgr2=0x7faf9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf9406b500 0x7faf9406d9c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7faf9800ab60 tx=0x7faf98008040 comp rx=0 tx=0).stop 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 msgr2=0x7fafa815bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 0x7fafa815bbb0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7faf9c00c3c0 tx=0x7faf9c00c880 comp rx=0 tx=0).stop 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 shutdown_connections 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf9406b500 0x7faf9406d9c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa815e670 0x7fafa815c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa80aeb20 0x7fafa815bbb0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 --2- 172.21.15.99:0/4191552709 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa8071350 0x7fafa814a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 >> 172.21.15.99:0/4191552709 conn(0x7fafa806acd0 msgr2=0x7fafa80b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 shutdown_connections 2024-03-04T10:09:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.428+0000 7fafae759640 1 -- 172.21.15.99:0/4191552709 wait complete. 2024-03-04T10:09:05.440 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:09:05.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.541+0000 7f64d72a4640 1 Processor -- start 2024-03-04T10:09:05.542 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.541+0000 7f64d72a4640 1 -- start start 2024-03-04T10:09:05.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d72a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f64d00cf650 0x7f64d00cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d72a4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64d0070de0 0x7f64d0071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d72a4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d015f020 con 0x7f64d0154a80 2024-03-04T10:09:05.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d72a4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d00cd010 con 0x7f64d0070de0 2024-03-04T10:09:05.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d72a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d00cdaa0 con 0x7f64d00cf650 2024-03-04T10:09:05.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64d0070de0 0x7f64d0071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64d0070de0 0x7f64d0071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57140/0 (socket says 172.21.15.99:57140) 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 -- 172.21.15.99:0/1002253905 learned_addr learned my addr 172.21.15.99:0/1002253905 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 -- 172.21.15.99:0/1002253905 >> v2:172.21.15.162:3300/0 conn(0x7f64d00cf650 msgr2=0x7f64d00cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d483f640 1 --2- 172.21.15.99:0/1002253905 >> v2:172.21.15.162:3300/0 conn(0x7f64d00cf650 0x7f64d00cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 --2- 172.21.15.99:0/1002253905 >> v2:172.21.15.162:3300/0 conn(0x7f64d00cf650 0x7f64d00cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 -- 172.21.15.99:0/1002253905 >> v1:172.21.15.162:6789/0 conn(0x7f64d0154a80 legacy=0x7f64d015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.542+0000 7f64d5841640 1 -- 172.21.15.99:0/1002253905 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d00cdd80 con 0x7f64d0070de0 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64d5841640 1 --2- 172.21.15.99:0/1002253905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d0070de0 0x7f64d0071270 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f64d0067960 tx=0x7f64c802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83d5e9689281b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64c7fff640 1 -- 172.21.15.99:0/1002253905 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c802ca80 con 0x7f64d0070de0 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64c7fff640 1 -- 172.21.15.99:0/1002253905 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c802cc20 con 0x7f64d0070de0 2024-03-04T10:09:05.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64c7fff640 1 -- 172.21.15.99:0/1002253905 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c80343f0 con 0x7f64d0070de0 2024-03-04T10:09:05.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64d72a4640 1 -- 172.21.15.99:0/1002253905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d0070de0 msgr2=0x7f64d0071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1002253905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d0070de0 0x7f64d0071270 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f64d0067960 tx=0x7f64c802bee0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64d72a4640 1 -- 172.21.15.99:0/1002253905 shutdown_connections 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.543+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1002253905 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d0070de0 0x7f64d0071270 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.544+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1002253905 >> v2:172.21.15.162:3300/0 conn(0x7f64d00cf650 0x7f64d00cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.544+0000 7f64d72a4640 1 -- 172.21.15.99:0/1002253905 >> 172.21.15.99:0/1002253905 conn(0x7f64d006f5f0 msgr2=0x7f64d006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.544+0000 7f64d72a4640 1 -- 172.21.15.99:0/1002253905 shutdown_connections 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.544+0000 7f64d72a4640 1 -- 172.21.15.99:0/1002253905 wait complete. 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 Processor -- start 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 -- start start 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64d0070de0 0x7f64d00b6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 0x7f64d0153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64d0154a80 0x7f64d01613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d00ce620 con 0x7f64d00cf650 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d00ce340 con 0x7f64d0154a80 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d72a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d0154e60 con 0x7f64d0070de0 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.545+0000 7f64d483f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 0x7f64d0153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 0x7f64d0153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57144/0 (socket says 172.21.15.99:57144) 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 -- 172.21.15.99:0/1771107771 learned_addr learned my addr 172.21.15.99:0/1771107771 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 -- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64d0070de0 msgr2=0x7f64d00b6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64d0070de0 0x7f64d00b6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 -- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64d0154a80 msgr2=0x7f64d01613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64d0154a80 0x7f64d01613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 -- 172.21.15.99:0/1771107771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d0155630 con 0x7f64d00cf650 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d483f640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 0x7f64d0153dc0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00c7f0 tx=0x7f64cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc015070 con 0x7f64d00cf650 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc004240 con 0x7f64d00cf650 2024-03-04T10:09:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc007620 con 0x7f64d00cf650 2024-03-04T10:09:05.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d0155910 con 0x7f64d00cf650 2024-03-04T10:09:05.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.546+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d0155ed0 con 0x7f64d00cf650 2024-03-04T10:09:05.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.547+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d006cd80 con 0x7f64d00cf650 2024-03-04T10:09:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.548+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f64cc0079d0 con 0x7f64d00cf650 2024-03-04T10:09:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.548+0000 7f64c5ffb640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64b406b500 0x7f64b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.548+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6418+0+0 (secure 0 0 0) 0x7f64cc08a540 con 0x7f64d00cf650 2024-03-04T10:09:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.548+0000 7f64d5040640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64b406b500 0x7f64b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.549+0000 7f64d5040640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64b406b500 0x7f64b406d9c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.552+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f64d006cd80 con 0x7f64d00cf650 2024-03-04T10:09:05.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.757+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f64d0074f60 con 0x7f64d00cf650 2024-03-04T10:09:05.759 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:09:05.758+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:09:05.762 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.761+0000 7f64c5ffb640 1 -- 172.21.15.99:0/1771107771 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f64cc059180 con 0x7f64d00cf650 2024-03-04T10:09:05.762 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64b406b500 msgr2=0x7f64b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64b406b500 0x7f64b406d9c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f64c0006fd0 tx=0x7f64c0006bc0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 msgr2=0x7f64d0153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 0x7f64d0153dc0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00c7f0 tx=0x7f64cc00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 shutdown_connections 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64b406b500 0x7f64b406d9c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64d0154a80 0x7f64d01613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64d00cf650 0x7f64d0153dc0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 --2- 172.21.15.99:0/1771107771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64d0070de0 0x7f64d00b6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 >> 172.21.15.99:0/1771107771 conn(0x7f64d006f5f0 msgr2=0x7f64d006d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 shutdown_connections 2024-03-04T10:09:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.764+0000 7f64d72a4640 1 -- 172.21.15.99:0/1771107771 wait complete. 2024-03-04T10:09:05.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T10:09:05.802 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:09:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.882+0000 7f282290a640 1 Processor -- start 2024-03-04T10:09:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.882+0000 7f282290a640 1 -- start start 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f282290a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c15a810 0x7f281c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f282290a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f281c06d3c0 0x7f281c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f282290a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c06d8a0 con 0x7f281c0cbfb0 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f282290a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c06db80 con 0x7f281c06d3c0 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f282290a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c06de60 con 0x7f281c15a810 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f281b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f281c06d3c0 0x7f281c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f281b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f281c06d3c0 0x7f281c0b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57156/0 (socket says 172.21.15.99:57156) 2024-03-04T10:09:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f281b7fe640 1 -- 172.21.15.99:0/641916410 learned_addr learned my addr 172.21.15.99:0/641916410 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.883+0000 7f281bfff640 1 --2- 172.21.15.99:0/641916410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c15a810 0x7f281c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281b7fe640 1 -- 172.21.15.99:0/641916410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c15a810 msgr2=0x7f281c0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281b7fe640 1 --2- 172.21.15.99:0/641916410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c15a810 0x7f281c0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281b7fe640 1 -- 172.21.15.99:0/641916410 >> v1:172.21.15.162:6789/0 conn(0x7f281c0cbfb0 legacy=0x7f281c15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281affd640 1 -- 172.21.15.99:0/641916410 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920243662 0 0) 0x7f281c06d8a0 con 0x7f281c0cbfb0 2024-03-04T10:09:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281b7fe640 1 -- 172.21.15.99:0/641916410 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c06e140 con 0x7f281c06d3c0 2024-03-04T10:09:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281b7fe640 1 --2- 172.21.15.99:0/641916410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c0b46d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f281000bb10 tx=0x7f281002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd7242c37b39aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.884+0000 7f281affd640 1 -- 172.21.15.99:0/641916410 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281003b070 con 0x7f281c06d3c0 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f281affd640 1 -- 172.21.15.99:0/641916410 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2810036440 con 0x7f281c06d3c0 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f281affd640 1 -- 172.21.15.99:0/641916410 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810036810 con 0x7f281c06d3c0 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 -- 172.21.15.99:0/641916410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 msgr2=0x7f281c0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 --2- 172.21.15.99:0/641916410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c0b46d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f281000bb10 tx=0x7f281002e150 comp rx=0 tx=0).stop 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 -- 172.21.15.99:0/641916410 shutdown_connections 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 --2- 172.21.15.99:0/641916410 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c0b46d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 --2- 172.21.15.99:0/641916410 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c15a810 0x7f281c0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 -- 172.21.15.99:0/641916410 >> 172.21.15.99:0/641916410 conn(0x7f281c06acd0 msgr2=0x7f281c06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 -- 172.21.15.99:0/641916410 shutdown_connections 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.885+0000 7f282290a640 1 -- 172.21.15.99:0/641916410 wait complete. 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.886+0000 7f282290a640 1 Processor -- start 2024-03-04T10:09:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.886+0000 7f282290a640 1 -- start start 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f282290a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f282290a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c0cbfb0 0x7f281c0a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f282290a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f281c15a810 0x7f281c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f282290a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c15e3c0 con 0x7f281c06d3c0 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f282290a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0b3c10 con 0x7f281c15a810 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f282290a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c15e540 con 0x7f281c0cbfb0 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c156160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57168/0 (socket says 172.21.15.99:57168) 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 -- 172.21.15.99:0/4294712071 learned_addr learned my addr 172.21.15.99:0/4294712071 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281b7fe640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c0cbfb0 0x7f281c0a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f2820ea7640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f281c15a810 0x7f281c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 -- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c0cbfb0 msgr2=0x7f281c0a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c0cbfb0 0x7f281c0a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 -- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f281c15a810 msgr2=0x7f281c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f281c15a810 0x7f281c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.887+0000 7f281bfff640 1 -- 172.21.15.99:0/4294712071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c15ae40 con 0x7f281c06d3c0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.888+0000 7f281bfff640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c156160 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f280c00e8a0 tx=0x7f280c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.888+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c017070 con 0x7f281c06d3c0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.888+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c00c940 con 0x7f281c06d3c0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.888+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c15b120 con 0x7f281c06d3c0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.888+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c00cd30 con 0x7f281c06d3c0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.888+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c1566a0 con 0x7f281c06d3c0 2024-03-04T10:09:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.889+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c15b830 con 0x7f281c06d3c0 2024-03-04T10:09:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.889+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f280c01a070 con 0x7f281c06d3c0 2024-03-04T10:09:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.890+0000 7f2818ff9640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f280806b4d0 0x7f280806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.890+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6504+0+0 (secure 0 0 0) 0x7f280c08a6c0 con 0x7f281c06d3c0 2024-03-04T10:09:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.890+0000 7f281b7fe640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f280806b4d0 0x7f280806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.890+0000 7f281b7fe640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f280806b4d0 0x7f280806d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2810038000 tx=0x7f2810032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:05.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:05.894+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f280c0592b0 con 0x7f281c06d3c0 2024-03-04T10:09:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.102+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f281c074f60 con 0x7f281c06d3c0 2024-03-04T10:09:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.765+0000 7f2818ff9640 1 -- 172.21.15.99:0/4294712071 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7f280c058bb0 con 0x7f281c06d3c0 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f280806b4d0 msgr2=0x7f280806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f280806b4d0 0x7f280806d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2810038000 tx=0x7f2810032000 comp rx=0 tx=0).stop 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 msgr2=0x7f281c156160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c156160 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f280c00e8a0 tx=0x7f280c00ed60 comp rx=0 tx=0).stop 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 shutdown_connections 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f280806b4d0 0x7f280806d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f281c15a810 0x7f281c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f281c0cbfb0 0x7f281c0a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 --2- 172.21.15.99:0/4294712071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f281c06d3c0 0x7f281c156160 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 >> 172.21.15.99:0/4294712071 conn(0x7f281c06acd0 msgr2=0x7f281c156b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 shutdown_connections 2024-03-04T10:09:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.767+0000 7f282290a640 1 -- 172.21.15.99:0/4294712071 wait complete. 2024-03-04T10:09:06.781 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T10:09:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 Processor -- start 2024-03-04T10:09:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 -- start start 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15bf10 0x7fc85c0cead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc85c0cf010 0x7fc85c15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc85c0b73e0 0x7fc85c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85c15d3e0 con 0x7fc85c0b73e0 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85c15ce20 con 0x7fc85c0cf010 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.928+0000 7fc8608cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c15d100 con 0x7fc85c15bf10 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc85c0b73e0 0x7fc85c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc85c0b73e0 0x7fc85c15c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57172/0 (socket says 172.21.15.99:57172) 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 -- 172.21.15.99:0/2025688326 learned_addr learned my addr 172.21.15.99:0/2025688326 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85a59c640 1 --2- 172.21.15.99:0/2025688326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15bf10 0x7fc85c0cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc853fff640 1 --2- 172.21.15.99:0/2025688326 >> v2:172.21.15.162:3300/0 conn(0x7fc85c0cf010 0x7fc85c15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 -- 172.21.15.99:0/2025688326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15bf10 msgr2=0x7fc85c0cead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 --2- 172.21.15.99:0/2025688326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15bf10 0x7fc85c0cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 -- 172.21.15.99:0/2025688326 >> v2:172.21.15.162:3300/0 conn(0x7fc85c0cf010 msgr2=0x7fc85c15c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 --2- 172.21.15.99:0/2025688326 >> v2:172.21.15.162:3300/0 conn(0x7fc85c0cf010 0x7fc85c15c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.929+0000 7fc85ad9d640 1 -- 172.21.15.99:0/2025688326 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c0ce2d0 con 0x7fc85c0b73e0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc85ad9d640 1 --2- 172.21.15.99:0/2025688326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0b73e0 0x7fc85c15c830 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc85c0cf3f0 tx=0x7fc85402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71dc074201529c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc859d9b640 1 -- 172.21.15.99:0/2025688326 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85402ca70 con 0x7fc85c0b73e0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc859d9b640 1 -- 172.21.15.99:0/2025688326 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85402cc10 con 0x7fc85c0b73e0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc859d9b640 1 -- 172.21.15.99:0/2025688326 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540343f0 con 0x7fc85c0b73e0 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 -- 172.21.15.99:0/2025688326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0b73e0 msgr2=0x7fc85c15c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 --2- 172.21.15.99:0/2025688326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0b73e0 0x7fc85c15c830 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc85c0cf3f0 tx=0x7fc85402bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 -- 172.21.15.99:0/2025688326 shutdown_connections 2024-03-04T10:09:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 --2- 172.21.15.99:0/2025688326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0b73e0 0x7fc85c15c830 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 --2- 172.21.15.99:0/2025688326 >> v2:172.21.15.162:3300/0 conn(0x7fc85c0cf010 0x7fc85c15c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 --2- 172.21.15.99:0/2025688326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15bf10 0x7fc85c0cead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 -- 172.21.15.99:0/2025688326 >> 172.21.15.99:0/2025688326 conn(0x7fc85c0b34f0 msgr2=0x7fc85c0b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 -- 172.21.15.99:0/2025688326 shutdown_connections 2024-03-04T10:09:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.930+0000 7fc8608cd640 1 -- 172.21.15.99:0/2025688326 wait complete. 2024-03-04T10:09:06.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 Processor -- start 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 -- start start 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 0x7fc85c14fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc85c15bf10 0x7fc85c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15a130 0x7fc85c1f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0cc8b0 con 0x7fc85c0cf010 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0cc730 con 0x7fc85c15bf10 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc8608cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c0cca30 con 0x7fc85c15a130 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc85a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 0x7fc85c14fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc85a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 0x7fc85c14fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57178/0 (socket says 172.21.15.99:57178) 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.932+0000 7fc85a59c640 1 -- 172.21.15.99:0/1455766886 learned_addr learned my addr 172.21.15.99:0/1455766886 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85a59c640 1 -- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15a130 msgr2=0x7fc85c1f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85ad9d640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15a130 0x7fc85c1f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85a59c640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15a130 0x7fc85c1f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85a59c640 1 -- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc85c15bf10 msgr2=0x7fc85c159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc853fff640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc85c15bf10 0x7fc85c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85a59c640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc85c15bf10 0x7fc85c159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85a59c640 1 -- 172.21.15.99:0/1455766886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85c1f5e10 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc85a59c640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 0x7fc85c14fd50 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c7f0 tx=0x7fc84c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c015070 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc84c004220 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c0b5270 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c007620 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.933+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c0b5830 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.934+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c15a540 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.934+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc84c0079d0 con 0x7fc85c0cf010 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.935+0000 7fc852ffd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc83c06b4d0 0x7fc83c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.935+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6504+0+0 (secure 0 0 0) 0x7fc84c08a500 con 0x7fc85c0cf010 2024-03-04T10:09:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.935+0000 7fc853fff640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc83c06b4d0 0x7fc83c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.935+0000 7fc853fff640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc83c06b4d0 0x7fc83c06d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc85c159740 tx=0x7fc844008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:06.939+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc85c15a540 con 0x7fc85c0cf010 2024-03-04T10:09:07.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.110+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc85c1606f0 con 0x7fc85c0cf010 2024-03-04T10:09:07.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.773+0000 7fc852ffd640 1 -- 172.21.15.99:0/1455766886 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc84c0590f0 con 0x7fc85c0cf010 2024-03-04T10:09:07.775 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' removed 2024-03-04T10:09:07.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc83c06b4d0 msgr2=0x7fc83c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc83c06b4d0 0x7fc83c06d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc85c159740 tx=0x7fc844008040 comp rx=0 tx=0).stop 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 msgr2=0x7fc85c14fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 0x7fc85c14fd50 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00c7f0 tx=0x7fc84c00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 shutdown_connections 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc83c06b4d0 0x7fc83c06d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc85c15a130 0x7fc85c1f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc85c15bf10 0x7fc85c159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 --2- 172.21.15.99:0/1455766886 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc85c0cf010 0x7fc85c14fd50 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 >> 172.21.15.99:0/1455766886 conn(0x7fc85c0b34f0 msgr2=0x7fc85c0cf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 shutdown_connections 2024-03-04T10:09:07.778 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.776+0000 7fc8608cd640 1 -- 172.21.15.99:0/1455766886 wait complete. 2024-03-04T10:09:07.792 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T10:09:07.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.898+0000 7f8ef29cd640 1 Processor -- start 2024-03-04T10:09:07.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.898+0000 7f8ef29cd640 1 -- start start 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8ef29cd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8eec0cdc40 0x7f8eec06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8ef29cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec06b210 0x7f8eec0b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8ef29cd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec0d02b0 con 0x7f8eec15bd90 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8ef29cd640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eec0ce0a0 con 0x7f8eec0cdc40 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8ef29cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec0d0590 con 0x7f8eec06b210 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8eeb7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8eec0cdc40 0x7f8eec06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8eebfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8eec15bd90 0x7f8eec0cfa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33334/0 (socket says 172.21.15.99:33334) 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.899+0000 7f8eebfff640 1 -- 172.21.15.99:0/1172946961 learned_addr learned my addr 172.21.15.99:0/1172946961 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8ef0f6a640 1 --2- 172.21.15.99:0/1172946961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec06b210 0x7f8eec0b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/1172946961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec06b210 msgr2=0x7f8eec0b41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:07.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8eeb7fe640 1 --2- 172.21.15.99:0/1172946961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec06b210 0x7f8eec0b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/1172946961 >> v1:172.21.15.99:6789/0 conn(0x7f8eec15bd90 legacy=0x7f8eec0cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/1172946961 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec0b36c0 con 0x7f8eec0cdc40 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8eeaffd640 1 -- 172.21.15.99:0/1172946961 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084341324 0 0) 0x7f8eec0d02b0 con 0x7f8eec15bd90 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.900+0000 7f8eeb7fe640 1 --2- 172.21.15.99:0/1172946961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec06acd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00097a0 tx=0x7f8ee002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e7d3a1965081f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8eeaffd640 1 -- 172.21.15.99:0/1172946961 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0035020 con 0x7f8eec0cdc40 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8eeaffd640 1 -- 172.21.15.99:0/1172946961 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee0031830 con 0x7f8eec0cdc40 2024-03-04T10:09:07.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8eeaffd640 1 -- 172.21.15.99:0/1172946961 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0031c00 con 0x7f8eec0cdc40 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 -- 172.21.15.99:0/1172946961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 msgr2=0x7f8eec06acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/1172946961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec06acd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00097a0 tx=0x7f8ee002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 -- 172.21.15.99:0/1172946961 shutdown_connections 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/1172946961 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec06b210 0x7f8eec0b41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/1172946961 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec06acd0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 -- 172.21.15.99:0/1172946961 >> 172.21.15.99:0/1172946961 conn(0x7f8eec0b22e0 msgr2=0x7f8eec0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 -- 172.21.15.99:0/1172946961 shutdown_connections 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.901+0000 7f8ef29cd640 1 -- 172.21.15.99:0/1172946961 wait complete. 2024-03-04T10:09:07.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.902+0000 7f8ef29cd640 1 Processor -- start 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.902+0000 7f8ef29cd640 1 -- start start 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef29cd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8eec06b210 0x7f8eec1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef29cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef29cd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec15bd90 0x7f8eec0c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef29cd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec15c170 con 0x7f8eec0cdc40 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef29cd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec0b3c80 con 0x7f8eec06b210 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef29cd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec0b46e0 con 0x7f8eec15bd90 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8eeb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8eeb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec0c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57200/0 (socket says 172.21.15.99:57200) 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/3497144379 learned_addr learned my addr 172.21.15.99:0/3497144379 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8ef0f6a640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec15bd90 0x7f8eec0c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.903+0000 7f8eebfff640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8eec06b210 0x7f8eec1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec15bd90 msgr2=0x7f8eec0c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8eeb7fe640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec15bd90 0x7f8eec0c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8eec06b210 msgr2=0x7f8eec1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8eeb7fe640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8eec06b210 0x7f8eec1547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8eeb7fe640 1 -- 172.21.15.99:0/3497144379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec15cf00 con 0x7f8eec0cdc40 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8eeb7fe640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec0c64d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0000c00 tx=0x7f8ee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0035050 con 0x7f8eec0cdc40 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee0004230 con 0x7f8eec0cdc40 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee003b800 con 0x7f8eec0cdc40 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec15d1e0 con 0x7f8eec0cdc40 2024-03-04T10:09:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.904+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec15d7a0 con 0x7f8eec0cdc40 2024-03-04T10:09:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.905+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec074f60 con 0x7f8eec0cdc40 2024-03-04T10:09:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.905+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8ee0038020 con 0x7f8eec0cdc40 2024-03-04T10:09:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.906+0000 7f8ee8ff9640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ed806b4d0 0x7f8ed806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.906+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6037+0+0 (secure 0 0 0) 0x7f8ee00ac150 con 0x7f8eec0cdc40 2024-03-04T10:09:07.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.906+0000 7f8eebfff640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ed806b4d0 0x7f8ed806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:07.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.906+0000 7f8eebfff640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ed806b4d0 0x7f8ed806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8edc000f60 tx=0x7f8edc007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:07.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:07.910+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8ee0040070 con 0x7f8eec0cdc40 2024-03-04T10:09:08.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.084+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8eec0b31b0 con 0x7f8eec0cdc40 2024-03-04T10:09:08.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.779+0000 7f8ee8ff9640 1 -- 172.21.15.99:0/3497144379 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8ee007ae80 con 0x7f8eec0cdc40 2024-03-04T10:09:08.780 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' removed 2024-03-04T10:09:08.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ed806b4d0 msgr2=0x7f8ed806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:08.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ed806b4d0 0x7f8ed806d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8edc000f60 tx=0x7f8edc007c00 comp rx=0 tx=0).stop 2024-03-04T10:09:08.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 msgr2=0x7f8eec0c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:08.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec0c64d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0000c00 tx=0x7f8ee00046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 shutdown_connections 2024-03-04T10:09:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ed806b4d0 0x7f8ed806d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8eec15bd90 0x7f8eec0c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8eec0cdc40 0x7f8eec0c64d0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 --2- 172.21.15.99:0/3497144379 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8eec06b210 0x7f8eec1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.782+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 >> 172.21.15.99:0/3497144379 conn(0x7f8eec0b22e0 msgr2=0x7f8eec0b4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.784+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 shutdown_connections 2024-03-04T10:09:08.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.784+0000 7f8ef29cd640 1 -- 172.21.15.99:0/3497144379 wait complete. 2024-03-04T10:09:08.799 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T10:09:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.909+0000 7f2fc9c0c640 1 Processor -- start 2024-03-04T10:09:08.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.909+0000 7f2fc9c0c640 1 -- start start 2024-03-04T10:09:08.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc9c0c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2fc40cdd70 0x7f2fc40cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:08.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc9c0c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc406ae00 con 0x7f2fc40ccfe0 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc9c0c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc406b0e0 con 0x7f2fc40cc900 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc9c0c640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40ccde0 con 0x7f2fc40cdd70 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2fc40cdd70 0x7f2fc40cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc3fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2fc40ccfe0 0x7f2fc415db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33344/0 (socket says 172.21.15.99:33344) 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc3fff640 1 -- 172.21.15.99:0/1732232196 learned_addr learned my addr 172.21.15.99:0/1732232196 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc37fe640 1 -- 172.21.15.99:0/1732232196 >> v1:172.21.15.162:6789/0 conn(0x7f2fc40cc900 legacy=0x7f2fc415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc37fe640 1 -- 172.21.15.99:0/1732232196 >> v1:172.21.15.99:6789/0 conn(0x7f2fc40ccfe0 legacy=0x7f2fc415db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:08.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.910+0000 7f2fc37fe640 1 -- 172.21.15.99:0/1732232196 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc406b3c0 con 0x7f2fc40cdd70 2024-03-04T10:09:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc37fe640 1 --2- 172.21.15.99:0/1732232196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40cfa00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40097a0 tx=0x7f2fb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22fb744ec960ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1732232196 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb40315a0 con 0x7f2fc40cdd70 2024-03-04T10:09:08.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1732232196 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4037070 con 0x7f2fc40cdd70 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc27fc640 1 -- 172.21.15.99:0/1732232196 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4031960 con 0x7f2fc40cdd70 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/1732232196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 msgr2=0x7f2fc40cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/1732232196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40cfa00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40097a0 tx=0x7f2fb402b840 comp rx=0 tx=0).stop 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/1732232196 shutdown_connections 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/1732232196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40cfa00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/1732232196 >> 172.21.15.99:0/1732232196 conn(0x7f2fc40b22e0 msgr2=0x7f2fc40b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/1732232196 shutdown_connections 2024-03-04T10:09:08.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.911+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/1732232196 wait complete. 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 Processor -- start 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 -- start start 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc40cc900 0x7f2fc41e8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc40ccfe0 0x7f2fc41505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b2e90 con 0x7f2fc40cdd70 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc415e230 con 0x7f2fc40cc900 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.913+0000 7f2fc9c0c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b3010 con 0x7f2fc40ccfe0 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57216/0 (socket says 172.21.15.99:57216) 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 -- 172.21.15.99:0/614078733 learned_addr learned my addr 172.21.15.99:0/614078733 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc37fe640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc40cc900 0x7f2fc41e8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:08.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc2ffd640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc40ccfe0 0x7f2fc41505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 -- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc40ccfe0 msgr2=0x7f2fc41505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc40ccfe0 0x7f2fc41505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 -- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc40cc900 msgr2=0x7f2fc41e8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc40cc900 0x7f2fc41e8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 -- 172.21.15.99:0/614078733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4150af0 con 0x7f2fc40cdd70 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc2ffd640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc40ccfe0 0x7f2fc41505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc3fff640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40b3d70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00cbf0 tx=0x7f2fbc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc017070 con 0x7f2fc40cdd70 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc00ad40 con 0x7f2fc40cdd70 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc012600 con 0x7f2fc40cdd70 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc41e7c80 con 0x7f2fc40cdd70 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.914+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc41e8240 con 0x7f2fc40cdd70 2024-03-04T10:09:08.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.915+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc4074f60 con 0x7f2fc40cdd70 2024-03-04T10:09:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.916+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2fbc01a070 con 0x7f2fc40cdd70 2024-03-04T10:09:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.916+0000 7f2fabfff640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.916+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f2fbc089d40 con 0x7f2fc40cdd70 2024-03-04T10:09:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.916+0000 7f2fc37fe640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:08.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.917+0000 7f2fc37fe640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4036000 tx=0x7f2fb403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:08.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:08.921+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2fbc058630 con 0x7f2fc40cdd70 2024-03-04T10:09:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.090+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2fc40ce480 con 0x7f2fc40cdd70 2024-03-04T10:09:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.093+0000 7f2fabfff640 1 -- 172.21.15.99:0/614078733 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fbc05c6d0 con 0x7f2fc40cdd70 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 msgr2=0x7f2fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4036000 tx=0x7f2fb403d040 comp rx=0 tx=0).stop 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 msgr2=0x7f2fc40b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40b3d70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00cbf0 tx=0x7f2fbc00a430 comp rx=0 tx=0).stop 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 shutdown_connections 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2fb006b4d0 0x7f2fb006d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2fc40cdd70 0x7f2fc40b3d70 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2fc40ccfe0 0x7f2fc41505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 --2- 172.21.15.99:0/614078733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2fc40cc900 0x7f2fc41e8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.095+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 >> 172.21.15.99:0/614078733 conn(0x7f2fc40b22e0 msgr2=0x7f2fc406e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:09.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.096+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 shutdown_connections 2024-03-04T10:09:09.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.096+0000 7f2fc9c0c640 1 -- 172.21.15.99:0/614078733 wait complete. 2024-03-04T10:09:09.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_journal_corrupt_dentry_first 2024-03-04T10:09:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.208+0000 7fede4e9a640 1 Processor -- start 2024-03-04T10:09:09.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.208+0000 7fede4e9a640 1 -- start start 2024-03-04T10:09:09.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fede4e9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 0x7fede00728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fede4e9a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fede00700a0 0x7fede015d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fede4e9a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede015cec0 con 0x7fede015bba0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fede4e9a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fede015c100 con 0x7fede00700a0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fede4e9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede015c3e0 con 0x7fede006bce0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedddd9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fede015bba0 0x7fede006c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33346/0 (socket says 172.21.15.99:33346) 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedddd9b640 1 -- 172.21.15.99:0/1523144728 learned_addr learned my addr 172.21.15.99:0/1523144728 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedde59c640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 0x7fede00728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedded9d640 1 --2- 172.21.15.99:0/1523144728 >> v2:172.21.15.162:3300/0 conn(0x7fede00700a0 0x7fede015d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731053905 0 0) 0x7fede015cec0 con 0x7fede015bba0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc8003520 con 0x7fede015bba0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677789055 0 0) 0x7fedc8003520 con 0x7fede015bba0 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 msgr2=0x7fede00728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 0x7fede00728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 >> v2:172.21.15.162:3300/0 conn(0x7fede00700a0 msgr2=0x7fede015d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 --2- 172.21.15.99:0/1523144728 >> v2:172.21.15.162:3300/0 conn(0x7fede00700a0 0x7fede015d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede015c6c0 con 0x7fede015bba0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fedd4001670 con 0x7fede015bba0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 >> v1:172.21.15.99:6789/0 conn(0x7fede015bba0 legacy=0x7fede006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fedc80039b0 0x7fedc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 0x7fedc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fedc8004d40 0x7fedc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede015c100 con 0x7fedc8004d40 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede015c3e0 con 0x7fedc80039b0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc800b9c0 con 0x7fede006bce0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedded9d640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fedc8004d40 0x7fedc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedde59c640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fedc80039b0 0x7fedc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.209+0000 7fedddd9b640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 0x7fedc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.210+0000 7fedded9d640 1 -- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 msgr2=0x7fedc8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.210+0000 7fedded9d640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fede006bce0 0x7fedc8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.210+0000 7fedded9d640 1 -- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fedc80039b0 msgr2=0x7fedc8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.210+0000 7fedded9d640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fedc80039b0 0x7fedc8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.210+0000 7fedded9d640 1 -- 172.21.15.99:0/1523144728 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8003520 con 0x7fedc8004d40 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.210+0000 7fedded9d640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fedc8004d40 0x7fedc80051c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009fc0 tx=0x7fedc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e05a17d8f454de server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8039070 con 0x7fedc8004d40 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd802cc60 con 0x7fedc8004d40 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7feddd59a640 1 -- 172.21.15.99:0/1523144728 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd8009c10 con 0x7fedc8004d40 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fede4e9a640 1 -- 172.21.15.99:0/1523144728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fedc8004d40 msgr2=0x7fedc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fede4e9a640 1 --2- 172.21.15.99:0/1523144728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fedc8004d40 0x7fedc80051c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fedd8009fc0 tx=0x7fedc8005830 comp rx=0 tx=0).stop 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fedded9d640 1 -- 172.21.15.99:0/1523144728 reap_dead start 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fede4e9a640 1 -- 172.21.15.99:0/1523144728 shutdown_connections 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fede4e9a640 1 -- 172.21.15.99:0/1523144728 >> 172.21.15.99:0/1523144728 conn(0x7fede0071bc0 msgr2=0x7fede0071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fede4e9a640 1 -- 172.21.15.99:0/1523144728 shutdown_connections 2024-03-04T10:09:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.211+0000 7fede4e9a640 1 -- 172.21.15.99:0/1523144728 wait complete. 2024-03-04T10:09:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.212+0000 7fede4e9a640 1 Processor -- start 2024-03-04T10:09:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.212+0000 7fede4e9a640 1 -- start start 2024-03-04T10:09:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fede4e9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fedc8004d40 0x7fede01554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fede4e9a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 0x7fede015a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fede4e9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fede00cf4d0 0x7fede015aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fede4e9a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede00728f0 con 0x7fede00cf070 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fede4e9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede00cc3e0 con 0x7fede00cf4d0 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fede4e9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fede0072a70 con 0x7fedc8004d40 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fedddd9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 0x7fede015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fedddd9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 0x7fede015a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57224/0 (socket says 172.21.15.99:57224) 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.213+0000 7fedddd9b640 1 -- 172.21.15.99:0/942521133 learned_addr learned my addr 172.21.15.99:0/942521133 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedde59c640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fedc8004d40 0x7fede01554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedddd9b640 1 -- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fedc8004d40 msgr2=0x7fede01554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedded9d640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fede00cf4d0 0x7fede015aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedddd9b640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fedc8004d40 0x7fede01554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedddd9b640 1 -- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fede00cf4d0 msgr2=0x7fede015aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedddd9b640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fede00cf4d0 0x7fede015aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedddd9b640 1 -- 172.21.15.99:0/942521133 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede0155fd0 con 0x7fede00cf070 2024-03-04T10:09:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedddd9b640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 0x7fede015a760 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fedd4007370 tx=0x7fedd4007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd401a070 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd400fa80 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede00cdfd0 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd400fe70 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.214+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede00ce590 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.215+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede0074f60 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.215+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fedd401d070 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.216+0000 7fedceffd640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.216+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fedd4089cd0 con 0x7fede00cf070 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.216+0000 7fedde59c640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.217+0000 7fedde59c640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fedc006b4d0 0x7fedc006d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009980 tx=0x7fedd0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.220+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fedd4058c10 con 0x7fede00cf070 2024-03-04T10:09:09.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.391+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"} v 0) v1 -- 0x7fede00af560 con 0x7fede00cf070 2024-03-04T10:09:09.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.391+0000 7fedceffd640 1 -- 172.21.15.99:0/942521133 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_journal_corrupt_dentry_first"}]=0 v21) v1 ==== 122+0+0 (secure 0 0 0) 0x7fedd4058510 con 0x7fede00cf070 2024-03-04T10:09:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fedc006b4d0 msgr2=0x7fedc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fedc006b4d0 0x7fedc006d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009980 tx=0x7fedd0008040 comp rx=0 tx=0).stop 2024-03-04T10:09:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 msgr2=0x7fede015a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 0x7fede015a760 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fedd4007370 tx=0x7fedd4007830 comp rx=0 tx=0).stop 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 shutdown_connections 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fede00cf4d0 0x7fede015aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fede00cf070 0x7fede015a760 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 --2- 172.21.15.99:0/942521133 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fedc8004d40 0x7fede01554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 >> 172.21.15.99:0/942521133 conn(0x7fede0071bc0 msgr2=0x7fede006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 shutdown_connections 2024-03-04T10:09:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.393+0000 7fede4e9a640 1 -- 172.21.15.99:0/942521133 wait complete. 2024-03-04T10:09:09.405 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit' 2024-03-04T10:09:09.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.504+0000 7f8e0ae12640 1 Processor -- start 2024-03-04T10:09:09.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.504+0000 7f8e0ae12640 1 -- start start 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8e0ae12640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8e0415bb20 0x7f8e0415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8e0ae12640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0406d8b0 con 0x7f8e0406d3b0 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8e0ae12640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0406db90 con 0x7f8e0415b030 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8e0ae12640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e040b2720 con 0x7f8e0415bb20 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfbfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8e0406d3b0 0x7f8e0415b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33356/0 (socket says 172.21.15.99:33356) 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfbfff640 1 -- 172.21.15.99:0/1538787903 learned_addr learned my addr 172.21.15.99:0/1538787903 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328663413 0 0) 0x7f8e0406d8b0 con 0x7f8e0406d3b0 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dec003520 con 0x7f8e0406d3b0 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8e093af640 1 --2- 172.21.15.99:0/1538787903 >> v2:172.21.15.162:3300/0 conn(0x7f8e0415bb20 0x7f8e0415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2327304432 0 0) 0x7f8dec003520 con 0x7f8e0406d3b0 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 >> v2:172.21.15.162:3300/0 conn(0x7f8e0415bb20 msgr2=0x7f8e0415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 --2- 172.21.15.99:0/1538787903 >> v2:172.21.15.162:3300/0 conn(0x7f8e0415bb20 0x7f8e0415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 >> v1:172.21.15.162:6789/0 conn(0x7f8e0415b030 legacy=0x7f8e0415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0406de70 con 0x7f8e0406d3b0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8df4001670 con 0x7f8e0406d3b0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 >> v1:172.21.15.99:6789/0 conn(0x7f8e0406d3b0 legacy=0x7f8e0415b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dec0039b0 0x7f8dec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dec004800 0x7f8dec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.505+0000 7f8dfb7fe640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8dec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e040b2720 con 0x7f8dec0052e0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec00c1b0 con 0x7f8dec004800 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dec00c3b0 con 0x7f8dec0039b0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e093af640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8dec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e08bae640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dec0039b0 0x7f8dec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8dfbfff640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dec004800 0x7f8dec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e093af640 1 -- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dec0039b0 msgr2=0x7f8dec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e093af640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8dec0039b0 0x7f8dec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e093af640 1 -- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dec004800 msgr2=0x7f8dec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e093af640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8dec004800 0x7f8dec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.506+0000 7f8e093af640 1 -- 172.21.15.99:0/1538787903 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec003520 con 0x7f8dec0052e0 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e093af640 1 -- 172.21.15.99:0/1538787903 reap_dead start 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e093af640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8dec00b9c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009f90 tx=0x7f8dec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2230aa00c9100c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc009c10 con 0x7f8dec0052e0 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc02cc20 con 0x7f8dec0052e0 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8dfb7fe640 1 -- 172.21.15.99:0/1538787903 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc003990 con 0x7f8dec0052e0 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 -- 172.21.15.99:0/1538787903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 msgr2=0x7f8dec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8dec00b9c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009f90 tx=0x7f8dec005830 comp rx=0 tx=0).stop 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 -- 172.21.15.99:0/1538787903 shutdown_connections 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/1538787903 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8dec00b9c0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 -- 172.21.15.99:0/1538787903 >> 172.21.15.99:0/1538787903 conn(0x7f8e0406acd0 msgr2=0x7f8e0406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 -- 172.21.15.99:0/1538787903 shutdown_connections 2024-03-04T10:09:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.507+0000 7f8e0ae12640 1 -- 172.21.15.99:0/1538787903 wait complete. 2024-03-04T10:09:09.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.508+0000 7f8e0ae12640 1 Processor -- start 2024-03-04T10:09:09.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 -- start start 2024-03-04T10:09:09.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8e04145730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8e04145c70 0x7f8e04140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e04140da0 0x7f8e040af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0406e810 con 0x7f8dec0052e0 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0406e690 con 0x7f8e04140da0 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e0ae12640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0415c120 con 0x7f8e04145c70 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e08bae640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8e04145730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e08bae640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8e04145730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57242/0 (socket says 172.21.15.99:57242) 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.509+0000 7f8e08bae640 1 -- 172.21.15.99:0/3704245866 learned_addr learned my addr 172.21.15.99:0/3704245866 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8dfbfff640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8e04145c70 0x7f8e04140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e093af640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e04140da0 0x7f8e040af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e08bae640 1 -- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8e04145c70 msgr2=0x7f8e04140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e08bae640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8e04145c70 0x7f8e04140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e08bae640 1 -- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e04140da0 msgr2=0x7f8e040af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e08bae640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e04140da0 0x7f8e040af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e08bae640 1 -- 172.21.15.99:0/3704245866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04149310 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e08bae640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8e04145730 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000f5f0 tx=0x7f8e0000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000052e0 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00018070 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e041495f0 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00003810 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.510+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e04149bb0 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.511+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e04074f60 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.511+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8e000039b0 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.512+0000 7f8df97fa640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8df006b500 0x7f8df006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.512+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f8e00089f10 con 0x7f8dec0052e0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.512+0000 7f8dfbfff640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8df006b500 0x7f8df006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.513+0000 7f8dfbfff640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8df006b500 0x7f8df006d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8df40010e0 tx=0x7f8df4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.516+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8e00058e50 con 0x7f8dec0052e0 2024-03-04T10:09:09.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.758+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]} v 0) v1 -- 0x7f8e0406b250 con 0x7f8dec0052e0 2024-03-04T10:09:09.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.780+0000 7f8df97fa640 1 -- 172.21.15.99:0/3704245866 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_precommit"]}]=0 v274) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8e00058750 con 0x7f8dec0052e0 2024-03-04T10:09:09.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8df006b500 msgr2=0x7f8df006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8df006b500 0x7f8df006d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8df40010e0 tx=0x7f8df4009040 comp rx=0 tx=0).stop 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 msgr2=0x7f8e04145730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8e04145730 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000f5f0 tx=0x7f8e0000fab0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 shutdown_connections 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8df006b500 0x7f8df006d9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8e04140da0 0x7f8e040af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8e04145c70 0x7f8e04140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 --2- 172.21.15.99:0/3704245866 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8dec0052e0 0x7f8e04145730 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 >> 172.21.15.99:0/3704245866 conn(0x7f8e0406acd0 msgr2=0x7f8e040b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 shutdown_connections 2024-03-04T10:09:09.784 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.782+0000 7f8e0ae12640 1 -- 172.21.15.99:0/3704245866 wait complete. 2024-03-04T10:09:09.794 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-03-04T10:09:09.795 INFO:tasks.cephfs_test_runner:Starting test: test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) That the MDS won't write a dentry with new damage to CDentry::first 2024-03-04T10:09:09.795 INFO:tasks.cephfs_test_runner:test_dentry_first_preflush (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:09:09.795 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.894+0000 7fdc76153640 1 Processor -- start 2024-03-04T10:09:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.894+0000 7fdc76153640 1 -- start start 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc76153640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc7015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc76153640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdc7015b410 0x7fdc7015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc76153640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7015be00 con 0x7fdc7015b030 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc76153640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc700b2720 con 0x7fdc7015b410 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc76153640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7006d8b0 con 0x7fdc7006d3b0 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc6ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdc7015b410 0x7fdc7015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc6ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdc7015b410 0x7fdc7015b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57250/0 (socket says 172.21.15.99:57250) 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc6ffff640 1 -- 172.21.15.99:0/94492862 learned_addr learned my addr 172.21.15.99:0/94492862 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.895+0000 7fdc6effd640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc7015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6ffff640 1 -- 172.21.15.99:0/94492862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 msgr2=0x7fdc7015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6ffff640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc7015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6ffff640 1 -- 172.21.15.99:0/94492862 >> v1:172.21.15.162:6789/0 conn(0x7fdc7015b030 legacy=0x7fdc7015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6ffff640 1 -- 172.21.15.99:0/94492862 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7006db90 con 0x7fdc7015b410 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6effd640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc7015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6ffff640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc7015b8c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009e50 tx=0x7fdc6802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7cc67533a25013 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6e7fc640 1 -- 172.21.15.99:0/94492862 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6802c9e0 con 0x7fdc7015b410 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6e7fc640 1 -- 172.21.15.99:0/94492862 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6802cb80 con 0x7fdc7015b410 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.896+0000 7fdc6e7fc640 1 -- 172.21.15.99:0/94492862 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680343f0 con 0x7fdc7015b410 2024-03-04T10:09:09.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 -- 172.21.15.99:0/94492862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 msgr2=0x7fdc7015b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc7015b8c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009e50 tx=0x7fdc6802be40 comp rx=0 tx=0).stop 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 -- 172.21.15.99:0/94492862 shutdown_connections 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc7015b8c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 --2- 172.21.15.99:0/94492862 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc7015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 -- 172.21.15.99:0/94492862 >> 172.21.15.99:0/94492862 conn(0x7fdc7006acd0 msgr2=0x7fdc7006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 -- 172.21.15.99:0/94492862 shutdown_connections 2024-03-04T10:09:09.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.897+0000 7fdc76153640 1 -- 172.21.15.99:0/94492862 wait complete. 2024-03-04T10:09:09.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.898+0000 7fdc76153640 1 Processor -- start 2024-03-04T10:09:09.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.898+0000 7fdc76153640 1 -- start start 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc76153640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc70071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc76153640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc7015b030 0x7fdc70071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc76153640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc700aa600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc76153640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7006e690 con 0x7fdc7015b410 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc76153640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7006e150 con 0x7fdc7015b030 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc76153640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7006e810 con 0x7fdc7006d3b0 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc700aa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc700aa600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57260/0 (socket says 172.21.15.99:57260) 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 -- 172.21.15.99:0/1987346154 learned_addr learned my addr 172.21.15.99:0/1987346154 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6f7fe640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc70071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6effd640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc7015b030 0x7fdc70071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 -- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 msgr2=0x7fdc70071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc70071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 -- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc7015b030 msgr2=0x7fdc70071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc7015b030 0x7fdc70071890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.899+0000 7fdc6ffff640 1 -- 172.21.15.99:0/1987346154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7014f840 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.900+0000 7fdc6ffff640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc700aa600 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fdc68036040 tx=0x7fdc68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.900+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68043020 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.900+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc6803e450 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.900+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7014fae0 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.900+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68009510 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.900+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc701500a0 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.901+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc70074f60 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.901+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdc68046020 con 0x7fdc7015b410 2024-03-04T10:09:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.902+0000 7fdc57fff640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:09.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.902+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fdc680ac8a0 con 0x7fdc7015b410 2024-03-04T10:09:09.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.902+0000 7fdc6f7fe640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:09.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.903+0000 7fdc6f7fe640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdc700cec70 tx=0x7fdc60007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:09.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:09.906+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdc6804d030 con 0x7fdc7015b410 2024-03-04T10:09:10.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.076+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc7015e260 con 0x7fdc7015b410 2024-03-04T10:09:10.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.077+0000 7fdc57fff640 1 -- 172.21.15.99:0/1987346154 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13516 (secure 0 0 0) 0x7fdc6807b220 con 0x7fdc7015b410 2024-03-04T10:09:10.079 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:10.079 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":73,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:08.774700+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6833/665274836":"2024-03-05T10:09:05.758670+0000","172.21.15.99:6832/665274836":"2024-03-05T10:09:05.758670+0000","172.21.15.99:6835/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6834/3022309305":"2024-03-05T10:07:17.162027+0000","172.21.15.99:6835/2687500379":"2024-03-05T10:06:46.776427+0000","172.21.15.99:6834/2687500379":"2024-03-05T10:06:46.776427+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc5c06b4d0 msgr2=0x7fdc5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdc700cec70 tx=0x7fdc60007e30 comp rx=0 tx=0).stop 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 msgr2=0x7fdc700aa600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc700aa600 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fdc68036040 tx=0x7fdc68004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 shutdown_connections 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdc7015b410 0x7fdc700aa600 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdc7015b030 0x7fdc70071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 --2- 172.21.15.99:0/1987346154 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdc7006d3b0 0x7fdc70071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 >> 172.21.15.99:0/1987346154 conn(0x7fdc7006acd0 msgr2=0x7fdc7015c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:10.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 shutdown_connections 2024-03-04T10:09:10.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.079+0000 7fdc76153640 1 -- 172.21.15.99:0/1987346154 wait complete. 2024-03-04T10:09:10.091 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:09:10.092 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:09:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 Processor -- start 2024-03-04T10:09:10.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 -- start start 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f415df00 0x7fd5f406b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd5f406fa80 0x7fd5f40cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd5f40cfa40 0x7fd5f40cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f40cb290 con 0x7fd5f406fa80 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f40701d0 con 0x7fd5f40cfa40 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5f9e61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f40704b0 con 0x7fd5f415df00 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5eaffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd5f406fa80 0x7fd5f40cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.196+0000 7fd5eaffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd5f406fa80 0x7fd5f40cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57262/0 (socket says 172.21.15.99:57262) 2024-03-04T10:09:10.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 -- 172.21.15.99:0/1755679388 learned_addr learned my addr 172.21.15.99:0/1755679388 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 -- 172.21.15.99:0/1755679388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f415df00 msgr2=0x7fd5f406b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5f37fe640 1 --2- 172.21.15.99:0/1755679388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f415df00 0x7fd5f406b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5f3fff640 1 --2- 172.21.15.99:0/1755679388 >> v2:172.21.15.162:3300/0 conn(0x7fd5f40cfa40 0x7fd5f40cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 --2- 172.21.15.99:0/1755679388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f415df00 0x7fd5f406b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 -- 172.21.15.99:0/1755679388 >> v2:172.21.15.162:3300/0 conn(0x7fd5f40cfa40 msgr2=0x7fd5f40cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 --2- 172.21.15.99:0/1755679388 >> v2:172.21.15.162:3300/0 conn(0x7fd5f40cfa40 0x7fd5f40cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 -- 172.21.15.99:0/1755679388 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f40cc8e0 con 0x7fd5f406fa80 2024-03-04T10:09:10.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.197+0000 7fd5eaffd640 1 --2- 172.21.15.99:0/1755679388 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f406fa80 0x7fd5f40cad50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40cc6a0 tx=0x7fd5dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d08191dc933652b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f2ffd640 1 -- 172.21.15.99:0/1755679388 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc035020 con 0x7fd5f406fa80 2024-03-04T10:09:10.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f2ffd640 1 -- 172.21.15.99:0/1755679388 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5dc031870 con 0x7fd5f406fa80 2024-03-04T10:09:10.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f2ffd640 1 -- 172.21.15.99:0/1755679388 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc031c40 con 0x7fd5f406fa80 2024-03-04T10:09:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 -- 172.21.15.99:0/1755679388 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f406fa80 msgr2=0x7fd5f40cad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/1755679388 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f406fa80 0x7fd5f40cad50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fd5f40cc6a0 tx=0x7fd5dc02b910 comp rx=0 tx=0).stop 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 -- 172.21.15.99:0/1755679388 shutdown_connections 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/1755679388 >> v2:172.21.15.162:3300/0 conn(0x7fd5f40cfa40 0x7fd5f40cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/1755679388 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f406fa80 0x7fd5f40cad50 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/1755679388 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f415df00 0x7fd5f406b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 -- 172.21.15.99:0/1755679388 >> 172.21.15.99:0/1755679388 conn(0x7fd5f40b2250 msgr2=0x7fd5f40b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 -- 172.21.15.99:0/1755679388 shutdown_connections 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.198+0000 7fd5f9e61640 1 -- 172.21.15.99:0/1755679388 wait complete. 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.199+0000 7fd5f9e61640 1 Processor -- start 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 -- start start 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f406fa80 0x7fd5f40ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f40cfa40 0x7fd5f40c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 0x7fd5f40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f406fee0 con 0x7fd5f415df00 2024-03-04T10:09:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f40ccea0 con 0x7fd5f40cfa40 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f9e61640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f40d0410 con 0x7fd5f406fa80 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 0x7fd5f40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 0x7fd5f40d0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57264/0 (socket says 172.21.15.99:57264) 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.200+0000 7fd5f3fff640 1 -- 172.21.15.99:0/2281581642 learned_addr learned my addr 172.21.15.99:0/2281581642 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f3fff640 1 -- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f406fa80 msgr2=0x7fd5f40ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f3fff640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f406fa80 0x7fd5f40ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f3fff640 1 -- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f40cfa40 msgr2=0x7fd5f40c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f3fff640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f40cfa40 0x7fd5f40c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f3fff640 1 -- 172.21.15.99:0/2281581642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f40caac0 con 0x7fd5f415df00 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f3fff640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 0x7fd5f40d0bd0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00cd70 tx=0x7fd5ec00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec003b90 con 0x7fd5f415df00 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec003d30 con 0x7fd5f415df00 2024-03-04T10:09:10.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f40c82b0 con 0x7fd5f415df00 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec004240 con 0x7fd5f415df00 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.201+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f40c9aa0 con 0x7fd5f415df00 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.202+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f40b65c0 con 0x7fd5f415df00 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.202+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd5ec0043e0 con 0x7fd5f415df00 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.203+0000 7fd5f0ff9640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd5d406b4d0 0x7fd5d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.203+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fd5ec08ac80 con 0x7fd5f415df00 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.203+0000 7fd5f37fe640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd5d406b4d0 0x7fd5d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.204+0000 7fd5f37fe640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd5d406b4d0 0x7fd5d406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd5f415b980 tx=0x7fd5e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.207+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd5f40b65c0 con 0x7fd5f415df00 2024-03-04T10:09:10.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:10.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.381+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5f4074f60 con 0x7fd5f415df00 2024-03-04T10:09:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.382+0000 7fd5f0ff9640 1 -- 172.21.15.99:0/2281581642 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5ec059b40 con 0x7fd5f415df00 2024-03-04T10:09:10.383 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:09:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd5d406b4d0 msgr2=0x7fd5d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd5d406b4d0 0x7fd5d406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd5f415b980 tx=0x7fd5e4008040 comp rx=0 tx=0).stop 2024-03-04T10:09:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 msgr2=0x7fd5f40d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 0x7fd5f40d0bd0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00cd70 tx=0x7fd5ec00a660 comp rx=0 tx=0).stop 2024-03-04T10:09:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 shutdown_connections 2024-03-04T10:09:10.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd5d406b4d0 0x7fd5d406d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5f415df00 0x7fd5f40d0bd0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5f40cfa40 0x7fd5f40c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 --2- 172.21.15.99:0/2281581642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5f406fa80 0x7fd5f40ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.383+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 >> 172.21.15.99:0/2281581642 conn(0x7fd5f40b2250 msgr2=0x7fd5f40af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:10.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.384+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 shutdown_connections 2024-03-04T10:09:10.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.384+0000 7fd5f9e61640 1 -- 172.21.15.99:0/2281581642 wait complete. 2024-03-04T10:09:10.396 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-03-04T10:09:10.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 Processor -- start 2024-03-04T10:09:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 -- start start 2024-03-04T10:09:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c0b2e60 0x7f716c0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f716c0b5470 0x7f716c0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716c15e1e0 con 0x7f716c15de00 2024-03-04T10:09:10.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716c0b98c0 con 0x7f716c0b5470 2024-03-04T10:09:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f7170a73640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0b32c0 con 0x7f716c0b2e60 2024-03-04T10:09:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f716ad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f716c0b5470 0x7f716c0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f716ad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f716c0b5470 0x7f716c0b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57272/0 (socket says 172.21.15.99:57272) 2024-03-04T10:09:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.496+0000 7f716ad9d640 1 -- 172.21.15.99:0/4264397214 learned_addr learned my addr 172.21.15.99:0/4264397214 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:10.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.497+0000 7f716ad9d640 1 -- 172.21.15.99:0/4264397214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c0b2e60 msgr2=0x7f716c0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.497+0000 7f716ad9d640 1 --2- 172.21.15.99:0/4264397214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c0b2e60 0x7f716c0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.497+0000 7f716ad9d640 1 -- 172.21.15.99:0/4264397214 >> v1:172.21.15.162:6789/0 conn(0x7f716c15de00 legacy=0x7f716c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.497+0000 7f716ad9d640 1 -- 172.21.15.99:0/4264397214 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c0d1c10 con 0x7f716c0b5470 2024-03-04T10:09:10.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.497+0000 7f716ad9d640 1 --2- 172.21.15.99:0/4264397214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0b5900 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7164009e50 tx=0x7f716402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f566b75c7f8d3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7169d9b640 1 -- 172.21.15.99:0/4264397214 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716402ca70 con 0x7f716c0b5470 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7169d9b640 1 -- 172.21.15.99:0/4264397214 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716402cc10 con 0x7f716c0b5470 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7169d9b640 1 -- 172.21.15.99:0/4264397214 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71640343f0 con 0x7f716c0b5470 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 -- 172.21.15.99:0/4264397214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 msgr2=0x7f716c0b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 --2- 172.21.15.99:0/4264397214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0b5900 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7164009e50 tx=0x7f716402bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 -- 172.21.15.99:0/4264397214 shutdown_connections 2024-03-04T10:09:10.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 --2- 172.21.15.99:0/4264397214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0b5900 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 --2- 172.21.15.99:0/4264397214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c0b2e60 0x7f716c0b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 -- 172.21.15.99:0/4264397214 >> 172.21.15.99:0/4264397214 conn(0x7f716c0d0470 msgr2=0x7f716c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 -- 172.21.15.99:0/4264397214 shutdown_connections 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.498+0000 7f7170a73640 1 -- 172.21.15.99:0/4264397214 wait complete. 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.499+0000 7f7170a73640 1 Processor -- start 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 -- start start 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f716c0b2e60 0x7f716c1f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c15de00 0x7f716c14ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d21d0 con 0x7f716c0b5470 2024-03-04T10:09:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d2820 con 0x7f716c0b2e60 2024-03-04T10:09:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7170a73640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716c0d24b0 con 0x7f716c15de00 2024-03-04T10:09:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7163fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7163fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57286/0 (socket says 172.21.15.99:57286) 2024-03-04T10:09:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f7163fff640 1 -- 172.21.15.99:0/1112379849 learned_addr learned my addr 172.21.15.99:0/1112379849 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:10.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f716a59c640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f716c0b2e60 0x7f716c1f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.500+0000 7f716ad9d640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c15de00 0x7f716c14ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7163fff640 1 -- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c15de00 msgr2=0x7f716c14ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7163fff640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c15de00 0x7f716c14ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7163fff640 1 -- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f716c0b2e60 msgr2=0x7f716c1f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7163fff640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f716c0b2e60 0x7f716c1f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7163fff640 1 -- 172.21.15.99:0/1112379849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c15aa40 con 0x7f716c0b5470 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f716ad9d640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c15de00 0x7f716c14ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7163fff640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0cbd50 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f715400b4a0 tx=0x7f715400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154014070 con 0x7f716c0b5470 2024-03-04T10:09:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7154010610 con 0x7f716c0b5470 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154010a00 con 0x7f716c0b5470 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716c14b3b0 con 0x7f716c0b5470 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.501+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716c15afa0 con 0x7f716c0b5470 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.502+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716c074f60 con 0x7f716c0b5470 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.502+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7154017070 con 0x7f716c0b5470 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.503+0000 7f7162ffd640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f714c06b4d0 0x7f714c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.503+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f7154089a90 con 0x7f716c0b5470 2024-03-04T10:09:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.503+0000 7f716a59c640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f714c06b4d0 0x7f714c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.503+0000 7f716a59c640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f714c06b4d0 0x7f714c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f715c006fd0 tx=0x7f715c009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.507+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f716c15aa40 con 0x7f716c0b5470 2024-03-04T10:09:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.741+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) v1 -- 0x7f716c15b1a0 con 0x7f716c0b5470 2024-03-04T10:09:10.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.781+0000 7f7162ffd640 1 -- 172.21.15.99:0/1112379849 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v275) v1 ==== 144+0+0 (secure 0 0 0) 0x7f7154058380 con 0x7f716c0b5470 2024-03-04T10:09:10.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f714c06b4d0 msgr2=0x7f714c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f714c06b4d0 0x7f714c06d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f715c006fd0 tx=0x7f715c009060 comp rx=0 tx=0).stop 2024-03-04T10:09:10.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 msgr2=0x7f716c0cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0cbd50 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f715400b4a0 tx=0x7f715400b960 comp rx=0 tx=0).stop 2024-03-04T10:09:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 shutdown_connections 2024-03-04T10:09:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f714c06b4d0 0x7f714c06d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f716c15de00 0x7f716c14ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f716c0b5470 0x7f716c0cbd50 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 --2- 172.21.15.99:0/1112379849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f716c0b2e60 0x7f716c1f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 >> 172.21.15.99:0/1112379849 conn(0x7f716c0d0470 msgr2=0x7f716c0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:10.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 shutdown_connections 2024-03-04T10:09:10.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.784+0000 7f7170a73640 1 -- 172.21.15.99:0/1112379849 wait complete. 2024-03-04T10:09:10.796 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T10:09:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.897+0000 7fd163bb9640 1 Processor -- start 2024-03-04T10:09:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.898+0000 7fd163bb9640 1 -- start start 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.898+0000 7fd163bb9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd15c0b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.898+0000 7fd163bb9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c06d8a0 con 0x7fd15c15a810 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.898+0000 7fd163bb9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c06db80 con 0x7fd15c06d3c0 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.898+0000 7fd163bb9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c06de60 con 0x7fd15c0b48a0 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.898+0000 7fd161955640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd15c15a810 0x7fd15c0b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33358/0 (socket says 172.21.15.99:33358) 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161955640 1 -- 172.21.15.99:0/4196374478 learned_addr learned my addr 172.21.15.99:0/4196374478 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd15c0b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391445981 0 0) 0x7fd15c06d8a0 con 0x7fd15c15a810 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd148003520 con 0x7fd15c15a810 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3742143723 0 0) 0x7fd148003520 con 0x7fd15c15a810 2024-03-04T10:09:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 msgr2=0x7fd15c0b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd15c0b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 >> v1:172.21.15.162:6789/0 conn(0x7fd15c06d3c0 legacy=0x7fd15c0b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c06e140 con 0x7fd15c15a810 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fd158001670 con 0x7fd15c15a810 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 >> v1:172.21.15.99:6789/0 conn(0x7fd15c15a810 legacy=0x7fd15c0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd1480039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd148003f20 0x7fd148004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148004d40 0x7fd1480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c06de60 con 0x7fd148003f20 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800ba20 con 0x7fd148004d40 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800bc20 con 0x7fd15c0b48a0 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd148003f20 0x7fd148004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161955640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd1480039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd162156640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148004d40 0x7fd1480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 -- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 msgr2=0x7fd1480039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd1480039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 -- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148004d40 msgr2=0x7fd1480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148004d40 0x7fd1480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161154640 1 -- 172.21.15.99:0/4196374478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148003520 con 0x7fd148003f20 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd161955640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c0b48a0 0x7fd1480039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.899+0000 7fd162156640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd148004d40 0x7fd1480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.900+0000 7fd161154640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd148003f20 0x7fd148004800 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ac0 tx=0x7fd148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0665707fc39b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.900+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c039070 con 0x7fd148003f20 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.900+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c02c600 con 0x7fd148003f20 2024-03-04T10:09:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.900+0000 7fd160953640 1 -- 172.21.15.99:0/4196374478 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c009690 con 0x7fd148003f20 2024-03-04T10:09:10.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd163bb9640 1 -- 172.21.15.99:0/4196374478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd148003f20 msgr2=0x7fd148004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd163bb9640 1 --2- 172.21.15.99:0/4196374478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd148003f20 0x7fd148004800 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009ac0 tx=0x7fd148005830 comp rx=0 tx=0).stop 2024-03-04T10:09:10.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd162156640 1 -- 172.21.15.99:0/4196374478 reap_dead start 2024-03-04T10:09:10.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd163bb9640 1 -- 172.21.15.99:0/4196374478 shutdown_connections 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd163bb9640 1 -- 172.21.15.99:0/4196374478 >> 172.21.15.99:0/4196374478 conn(0x7fd15c06acd0 msgr2=0x7fd15c06b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd163bb9640 1 -- 172.21.15.99:0/4196374478 shutdown_connections 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.901+0000 7fd163bb9640 1 -- 172.21.15.99:0/4196374478 wait complete. 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.902+0000 7fd163bb9640 1 Processor -- start 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.902+0000 7fd163bb9640 1 -- start start 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c14e310 0x7fd15c071320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 0x7fd15c14f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd15c14f9b0 0x7fd15c154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0cc130 con 0x7fd15c071860 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0cbfb0 con 0x7fd15c14f9b0 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1501b0 con 0x7fd15c14e310 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 0x7fd15c14f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 0x7fd15c14f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57294/0 (socket says 172.21.15.99:57294) 2024-03-04T10:09:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 -- 172.21.15.99:0/404899089 learned_addr learned my addr 172.21.15.99:0/404899089 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161955640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c14e310 0x7fd15c071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd162156640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd15c14f9b0 0x7fd15c154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 -- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c14e310 msgr2=0x7fd15c071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c14e310 0x7fd15c071320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 -- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd15c14f9b0 msgr2=0x7fd15c154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd15c14f9b0 0x7fd15c154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 -- 172.21.15.99:0/404899089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c1f1ba0 con 0x7fd15c071860 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161955640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c14e310 0x7fd15c071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd161154640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 0x7fd15c14f470 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fd14c000c00 tx=0x7fd14c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c042020 con 0x7fd15c071860 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c03a040 con 0x7fd15c071860 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c039070 con 0x7fd15c071860 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c1f1e80 con 0x7fd15c071860 2024-03-04T10:09:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.903+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c1f2440 con 0x7fd15c071860 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.904+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c074f60 con 0x7fd15c071860 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.905+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd14c045020 con 0x7fd15c071860 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.905+0000 7fd1527fc640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd14006b500 0x7fd14006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.905+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fd14c0ab550 con 0x7fd15c071860 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.905+0000 7fd161955640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd14006b500 0x7fd14006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.906+0000 7fd161955640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd14006b500 0x7fd14006d9c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd158004840 tx=0x7fd158000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:10.909+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd14c04c070 con 0x7fd15c071860 2024-03-04T10:09:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.082+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd15c06ec30 con 0x7fd15c071860 2024-03-04T10:09:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.085+0000 7fd1527fc640 1 -- 172.21.15.99:0/404899089 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd14c07a490 con 0x7fd15c071860 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd14006b500 msgr2=0x7fd14006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd14006b500 0x7fd14006d9c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd158004840 tx=0x7fd158000ed0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 msgr2=0x7fd15c14f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 0x7fd15c14f470 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fd14c000c00 tx=0x7fd14c004730 comp rx=0 tx=0).stop 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 shutdown_connections 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd14006b500 0x7fd14006d9c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd15c14f9b0 0x7fd15c154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd15c071860 0x7fd15c14f470 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 --2- 172.21.15.99:0/404899089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd15c14e310 0x7fd15c071320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 >> 172.21.15.99:0/404899089 conn(0x7fd15c06acd0 msgr2=0x7fd15c0b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 shutdown_connections 2024-03-04T10:09:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.087+0000 7fd163bb9640 1 -- 172.21.15.99:0/404899089 wait complete. 2024-03-04T10:09:11.099 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:09:11.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:09:11.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:09:11.201 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:09:11.201 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:11.201 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:11.217 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:09:11.217 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:09:11.217 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:11.234 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:09:11.235 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:11.235 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:11.272 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:09:11.272 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:09:11.272 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:11.291 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:09:11.291 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T10:09:11.292 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 Processor -- start 2024-03-04T10:09:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 -- start start 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 0x7fcc8c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcc8c0b5470 0x7fcc8c0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8c15e1e0 con 0x7fcc8c0b9380 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d2820 con 0x7fcc8c0b5470 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc91ca0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d1c10 con 0x7fcc8c15de00 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc82ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fcc8c0b9380 0x7fcc8c0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33360/0 (socket says 172.21.15.99:33360) 2024-03-04T10:09:11.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.411+0000 7fcc82ffd640 1 -- 172.21.15.99:0/2855813091 learned_addr learned my addr 172.21.15.99:0/2855813091 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500026628 0 0) 0x7fcc8c15e1e0 con 0x7fcc8c0b9380 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc78003520 con 0x7fcc8c0b9380 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/2855813091 >> v2:172.21.15.162:3300/0 conn(0x7fcc8c0b5470 0x7fcc8c0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8b7fe640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 0x7fcc8c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971361588 0 0) 0x7fcc78003520 con 0x7fcc8c0b9380 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 msgr2=0x7fcc8c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 0x7fcc8c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 >> v2:172.21.15.162:3300/0 conn(0x7fcc8c0b5470 msgr2=0x7fcc8c0b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 --2- 172.21.15.99:0/2855813091 >> v2:172.21.15.162:3300/0 conn(0x7fcc8c0b5470 0x7fcc8c0b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c0d1ef0 con 0x7fcc8c0b9380 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fcc74001670 con 0x7fcc8c0b9380 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 >> v1:172.21.15.99:6789/0 conn(0x7fcc8c0b9380 legacy=0x7fcc8c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 0x7fcc780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc78003fe0 0x7fcc78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc78004d40 0x7fcc780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d2820 con 0x7fcc78004d40 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d1c10 con 0x7fcc78003fe0 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7800bac0 con 0x7fcc8c15de00 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc78004d40 0x7fcc780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc82ffd640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc78003fe0 0x7fcc78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8b7fe640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 0x7fcc780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 -- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 msgr2=0x7fcc780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c15de00 0x7fcc780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 -- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc78003fe0 msgr2=0x7fcc78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc78003fe0 0x7fcc78004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.412+0000 7fcc8bfff640 1 -- 172.21.15.99:0/2855813091 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc78003520 con 0x7fcc78004d40 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.413+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc78004d40 0x7fcc780051c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fcc84007360 tx=0x7fcc78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6e44218ec0dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.413+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84039070 con 0x7fcc78004d40 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.413+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8402ccf0 con 0x7fcc78004d40 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.413+0000 7fcc8affd640 1 -- 172.21.15.99:0/2855813091 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84009be0 con 0x7fcc78004d40 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc91ca0640 1 -- 172.21.15.99:0/2855813091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc78004d40 msgr2=0x7fcc780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/2855813091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc78004d40 0x7fcc780051c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fcc84007360 tx=0x7fcc78005830 comp rx=0 tx=0).stop 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc8bfff640 1 -- 172.21.15.99:0/2855813091 reap_dead start 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc91ca0640 1 -- 172.21.15.99:0/2855813091 shutdown_connections 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc91ca0640 1 -- 172.21.15.99:0/2855813091 >> 172.21.15.99:0/2855813091 conn(0x7fcc8c0d0470 msgr2=0x7fcc8c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc91ca0640 1 -- 172.21.15.99:0/2855813091 shutdown_connections 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.414+0000 7fcc91ca0640 1 -- 172.21.15.99:0/2855813091 wait complete. 2024-03-04T10:09:11.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.415+0000 7fcc91ca0640 1 Processor -- start 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.415+0000 7fcc91ca0640 1 -- start start 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc91ca0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc8c152ef0 0x7fcc8c15f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc91ca0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c160e20 0x7fcc8c1540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc91ca0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 0x7fcc8c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc91ca0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0b97e0 con 0x7fcc8c1532d0 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc91ca0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0d24b0 con 0x7fcc8c152ef0 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc91ca0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8c0b72a0 con 0x7fcc8c160e20 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 0x7fcc8c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 0x7fcc8c154620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57312/0 (socket says 172.21.15.99:57312) 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 -- 172.21.15.99:0/521061360 learned_addr learned my addr 172.21.15.99:0/521061360 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8b7fe640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc8c152ef0 0x7fcc8c15f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc82ffd640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c160e20 0x7fcc8c1540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 -- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c160e20 msgr2=0x7fcc8c1540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c160e20 0x7fcc8c1540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 -- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc8c152ef0 msgr2=0x7fcc8c15f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc8c152ef0 0x7fcc8c15f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 -- 172.21.15.99:0/521061360 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c15fd80 con 0x7fcc8c1532d0 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.416+0000 7fcc8bfff640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 0x7fcc8c154620 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fcc84007480 tx=0x7fcc84004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.417+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84039070 con 0x7fcc8c1532d0 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.417+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc84044070 con 0x7fcc8c1532d0 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.417+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8c160060 con 0x7fcc8c1532d0 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.417+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84009be0 con 0x7fcc8c1532d0 2024-03-04T10:09:11.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.417+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8c160620 con 0x7fcc8c1532d0 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.418+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8c074f60 con 0x7fcc8c1532d0 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.418+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcc8403e850 con 0x7fcc8c1532d0 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.419+0000 7fcc88ff9640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc6c06b4d0 0x7fcc6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.419+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fcc840abd10 con 0x7fcc8c1532d0 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.419+0000 7fcc8b7fe640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc6c06b4d0 0x7fcc6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.419+0000 7fcc8b7fe640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc6c06b4d0 0x7fcc6c06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c0caae0 tx=0x7fcc7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.423+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcc8404d020 con 0x7fcc8c1532d0 2024-03-04T10:09:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.634+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc8c0ce020 con 0x7fcc8c1532d0 2024-03-04T10:09:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.634+0000 7fcc88ff9640 1 -- 172.21.15.99:0/521061360 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fcc8407a570 con 0x7fcc8c1532d0 2024-03-04T10:09:11.636 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:11.636 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2167244038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2167244038},{"type":"v1","addr":"172.21.15.162:6837","nonce":2167244038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":5483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/1243485909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1243485909},{"type":"v1","addr":"172.21.15.162:6836","nonce":1243485909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6873,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/4052516864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":4052516864},{"type":"v1","addr":"172.21.15.99:6835","nonce":4052516864}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":7050,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/533701532","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":533701532},{"type":"v1","addr":"172.21.15.99:6833","nonce":533701532}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[]} 2024-03-04T10:09:11.636 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 59 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc6c06b4d0 msgr2=0x7fcc6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc6c06b4d0 0x7fcc6c06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c0caae0 tx=0x7fcc7c008040 comp rx=0 tx=0).stop 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 msgr2=0x7fcc8c154620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 0x7fcc8c154620 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fcc84007480 tx=0x7fcc84004ab0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 shutdown_connections 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc6c06b4d0 0x7fcc6c06d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc8c1532d0 0x7fcc8c154620 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc8c160e20 0x7fcc8c1540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 --2- 172.21.15.99:0/521061360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc8c152ef0 0x7fcc8c15f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 >> 172.21.15.99:0/521061360 conn(0x7fcc8c0d0470 msgr2=0x7fcc8c0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:11.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 shutdown_connections 2024-03-04T10:09:11.639 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.636+0000 7fcc91ca0640 1 -- 172.21.15.99:0/521061360 wait complete. 2024-03-04T10:09:11.649 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:09:11.649 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T10:09:11.649 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:09:11.649 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:09:11.650 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:09:11.650 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T10:09:11.650 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:09:11.650 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:09:11.650 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T10:09:11.650 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T10:09:11.651 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T10:09:11.651 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:09:11.651 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T10:09:11.651 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T10:09:11.651 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T10:09:11.651 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:09:11.658 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:09:11.658 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:09:11.660 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:09:11.660 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:09:11.663 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T10:09:11.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T10:09:11.664 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T10:09:11.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T10:09:11.700 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:09:11.701 DEBUG:teuthology.parallel:result is None 2024-03-04T10:09:11.701 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:09:11.701 DEBUG:teuthology.parallel:result is None 2024-03-04T10:09:11.706 INFO:tasks.ceph.mds.d:Started 2024-03-04T10:09:11.706 DEBUG:teuthology.parallel:result is None 2024-03-04T10:09:11.707 INFO:tasks.ceph.mds.b:Started 2024-03-04T10:09:11.707 DEBUG:teuthology.parallel:result is None 2024-03-04T10:09:11.707 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T10:09:11.776 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:09:11.776 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:09:11.790 INFO:tasks.ceph.mds.d.smithi162.stdout:starting mds.d at 2024-03-04T10:09:11.790 INFO:tasks.ceph.mds.b.smithi162.stdout:starting mds.b at 2024-03-04T10:09:11.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.811+0000 7fdcbc044640 1 Processor -- start 2024-03-04T10:09:11.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.812+0000 7fdcbc044640 1 -- start start 2024-03-04T10:09:11.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.812+0000 7fdcbc044640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.812+0000 7fdcbc044640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdcb415bc40 0x7fdcb40cea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.812+0000 7fdcbc044640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb40d0350 con 0x7fdcb40cf570 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.812+0000 7fdcbc044640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb415caa0 con 0x7fdcb415bc40 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.812+0000 7fdcbc044640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb415cf30 con 0x7fdcb40cfa30 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdcb415bc40 0x7fdcb40cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdcb415bc40 0x7fdcb40cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57374/0 (socket says 172.21.15.99:57374) 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 -- 172.21.15.99:0/4154453312 learned_addr learned my addr 172.21.15.99:0/4154453312 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcb9de0640 1 --2- 172.21.15.99:0/4154453312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 -- 172.21.15.99:0/4154453312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 msgr2=0x7fdcb40cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 --2- 172.21.15.99:0/4154453312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 -- 172.21.15.99:0/4154453312 >> v1:172.21.15.162:6789/0 conn(0x7fdcb40cf570 legacy=0x7fdcb40ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.813+0000 7fdcba5e1640 1 -- 172.21.15.99:0/4154453312 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb415c170 con 0x7fdcb415bc40 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcba5e1640 1 --2- 172.21.15.99:0/4154453312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb415bc40 0x7fdcb40cea60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009e50 tx=0x7fdcac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960a3c749be77c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcb8dde640 1 -- 172.21.15.99:0/4154453312 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac02cb00 con 0x7fdcb415bc40 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcb8dde640 1 -- 172.21.15.99:0/4154453312 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcac02cca0 con 0x7fdcb415bc40 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcb8dde640 1 -- 172.21.15.99:0/4154453312 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac034610 con 0x7fdcb415bc40 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcbc044640 1 -- 172.21.15.99:0/4154453312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb415bc40 msgr2=0x7fdcb40cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcbc044640 1 --2- 172.21.15.99:0/4154453312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb415bc40 0x7fdcb40cea60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009e50 tx=0x7fdcac02bf60 comp rx=0 tx=0).stop 2024-03-04T10:09:11.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcbc044640 1 -- 172.21.15.99:0/4154453312 shutdown_connections 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcbc044640 1 --2- 172.21.15.99:0/4154453312 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb415bc40 0x7fdcb40cea60 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcbc044640 1 --2- 172.21.15.99:0/4154453312 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.814+0000 7fdcbc044640 1 -- 172.21.15.99:0/4154453312 >> 172.21.15.99:0/4154453312 conn(0x7fdcb40d0780 msgr2=0x7fdcb40d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.815+0000 7fdcbc044640 1 -- 172.21.15.99:0/4154453312 shutdown_connections 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.815+0000 7fdcbc044640 1 -- 172.21.15.99:0/4154453312 wait complete. 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 Processor -- start 2024-03-04T10:09:11.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 -- start start 2024-03-04T10:09:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 0x7fdcb40b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb415bc40 0x7fdcb40b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb40c75a0 con 0x7fdcb40cf570 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb415c730 con 0x7fdcb415bc40 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcbc044640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb406acd0 con 0x7fdcb40cfa30 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcb9de0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 0x7fdcb40b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcb9de0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 0x7fdcb40b25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57388/0 (socket says 172.21.15.99:57388) 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.816+0000 7fdcb9de0640 1 -- 172.21.15.99:0/195779509 learned_addr learned my addr 172.21.15.99:0/195779509 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcba5e1640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb415bc40 0x7fdcb40b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb95df640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb9de0640 1 -- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 msgr2=0x7fdcb40b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb9de0640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb9de0640 1 -- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb415bc40 msgr2=0x7fdcb40b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb9de0640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb415bc40 0x7fdcb40b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb9de0640 1 -- 172.21.15.99:0/195779509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb40c3e90 con 0x7fdcb40cf570 2024-03-04T10:09:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcba5e1640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb415bc40 0x7fdcb40b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcb9de0640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 0x7fdcb40b25f0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000d5d0 tx=0x7fdcb000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00125f0 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0019070 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00129f0 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb40c4150 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.817+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb40b2b30 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.818+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb406cd70 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.818+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdcb0012b90 con 0x7fdcb40cf570 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.819+0000 7fdcaa7fc640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc9806b500 0x7fdc9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:11.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.819+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fdcb008a580 con 0x7fdcb40cf570 2024-03-04T10:09:11.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.819+0000 7fdcb95df640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc9806b500 0x7fdc9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:11.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.820+0000 7fdcb95df640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc9806b500 0x7fdc9806d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdca4007fd0 tx=0x7fdca4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:11.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:11.823+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdcb00594c0 con 0x7fdcb40cf570 2024-03-04T10:09:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.008+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdcb4073e10 con 0x7fdcb40cf570 2024-03-04T10:09:12.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.098+0000 7fdcaa7fc640 1 -- 172.21.15.99:0/195779509 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdcb0058dc0 con 0x7fdcb40cf570 2024-03-04T10:09:12.100 INFO:teuthology.orchestra.run.smithi099.stderr: removed all blocklist entries 2024-03-04T10:09:12.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc9806b500 msgr2=0x7fdc9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc9806b500 0x7fdc9806d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdca4007fd0 tx=0x7fdca4009040 comp rx=0 tx=0).stop 2024-03-04T10:09:12.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 msgr2=0x7fdcb40b25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 0x7fdcb40b25f0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000d5d0 tx=0x7fdcb000da90 comp rx=0 tx=0).stop 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 shutdown_connections 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdc9806b500 0x7fdc9806d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdcb415bc40 0x7fdcb40b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdcb40cfa30 0x7fdcb40b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 --2- 172.21.15.99:0/195779509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdcb40cf570 0x7fdcb40b25f0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 >> 172.21.15.99:0/195779509 conn(0x7fdcb40d0780 msgr2=0x7fdcb40c9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 shutdown_connections 2024-03-04T10:09:12.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.101+0000 7fdcbc044640 1 -- 172.21.15.99:0/195779509 wait complete. 2024-03-04T10:09:12.115 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T10:09:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.214+0000 7f7e7cebf640 1 Processor -- start 2024-03-04T10:09:12.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.214+0000 7f7e7cebf640 1 -- start start 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7cebf640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7e780cfd60 0x7f7e780cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7cebf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e7806d3b0 0x7f7e780cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7cebf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e780d0140 con 0x7f7e780ce9c0 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7cebf640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7815cbb0 con 0x7f7e780cfd60 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7cebf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815c280 con 0x7f7e7806d3b0 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7e780cfd60 0x7f7e780cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7e780cfd60 0x7f7e780cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57392/0 (socket says 172.21.15.99:57392) 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 -- 172.21.15.99:0/3904829925 learned_addr learned my addr 172.21.15.99:0/3904829925 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e6ffff640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e7806d3b0 0x7f7e780cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 -- 172.21.15.99:0/3904829925 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e7806d3b0 msgr2=0x7f7e780cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e7806d3b0 0x7f7e780cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 -- 172.21.15.99:0/3904829925 >> v1:172.21.15.162:6789/0 conn(0x7f7e780ce9c0 legacy=0x7f7e780cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e7659c640 1 -- 172.21.15.99:0/3904829925 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7815c560 con 0x7f7e780cfd60 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.215+0000 7f7e6ffff640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e7806d3b0 0x7f7e780cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e7659c640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780cfd60 0x7f7e780cbdf0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7e7815cd30 tx=0x7f7e6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8417a9f5cad21480 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e75d9b640 1 -- 172.21.15.99:0/3904829925 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68034070 con 0x7f7e780cfd60 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e75d9b640 1 -- 172.21.15.99:0/3904829925 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68030840 con 0x7f7e780cfd60 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e75d9b640 1 -- 172.21.15.99:0/3904829925 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68030c10 con 0x7f7e780cfd60 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e7cebf640 1 -- 172.21.15.99:0/3904829925 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780cfd60 msgr2=0x7f7e780cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780cfd60 0x7f7e780cbdf0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7e7815cd30 tx=0x7f7e6802b980 comp rx=0 tx=0).stop 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.216+0000 7f7e7cebf640 1 -- 172.21.15.99:0/3904829925 shutdown_connections 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.217+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e7806d3b0 0x7f7e780cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.217+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/3904829925 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780cfd60 0x7f7e780cbdf0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.217+0000 7f7e7cebf640 1 -- 172.21.15.99:0/3904829925 >> 172.21.15.99:0/3904829925 conn(0x7f7e7806acd0 msgr2=0x7f7e7806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:12.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.217+0000 7f7e7cebf640 1 -- 172.21.15.99:0/3904829925 shutdown_connections 2024-03-04T10:09:12.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.217+0000 7f7e7cebf640 1 -- 172.21.15.99:0/3904829925 wait complete. 2024-03-04T10:09:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 Processor -- start 2024-03-04T10:09:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 -- start start 2024-03-04T10:09:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e7806d3b0 0x7f7e78156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 0x7f7e7815e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e780cfd60 0x7f7e781f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815f520 con 0x7f7e780ce9c0 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815bf30 con 0x7f7e7806d3b0 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.218+0000 7f7e7cebf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815f6a0 con 0x7f7e780cfd60 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 0x7f7e7815e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 0x7f7e7815e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57402/0 (socket says 172.21.15.99:57402) 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 -- 172.21.15.99:0/673045177 learned_addr learned my addr 172.21.15.99:0/673045177 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e76d9d640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e780cfd60 0x7f7e781f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e7659c640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e7806d3b0 0x7f7e78156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 -- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e780cfd60 msgr2=0x7f7e781f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e780cfd60 0x7f7e781f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 -- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e7806d3b0 msgr2=0x7f7e78156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e7806d3b0 0x7f7e78156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 -- 172.21.15.99:0/673045177 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7815d2b0 con 0x7f7e780ce9c0 2024-03-04T10:09:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6ffff640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 0x7f7e7815e350 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e810 tx=0x7f7e6000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000f040 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60004030 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7815d590 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60004400 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.219+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7815db50 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.220+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e78074f60 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.221+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7e6000c6d0 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.221+0000 7f7e6effd640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e5806b4d0 0x7f7e5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.221+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7e60089d80 con 0x7f7e780ce9c0 2024-03-04T10:09:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.221+0000 7f7e7659c640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e5806b4d0 0x7f7e5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.222+0000 7f7e7659c640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e5806b4d0 0x7f7e5806d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7e68036000 tx=0x7f7e6803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.225+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e60017020 con 0x7f7e780ce9c0 2024-03-04T10:09:12.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.465+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7e780cc6d0 con 0x7f7e780ce9c0 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.465+0000 7f7e6effd640 1 -- 172.21.15.99:0/673045177 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7e60058dc0 con 0x7f7e780ce9c0 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_dump": [ 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.a", 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg==", 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.467 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.b", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw==", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.468 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.c", 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg==", 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.d", 2024-03-04T10:09:12.469 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw==", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.0", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg==", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.470 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.1", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ==", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.2", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw==", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.471 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.3", 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA==", 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.472 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.4", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw==", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.5", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg==", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.473 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.6", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg==", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.7", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA==", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:09:12.474 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.0", 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ==", 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow r", 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.1", 2024-03-04T10:09:12.475 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA==", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.2", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg==", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:09:12.476 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.3", 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ==", 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.477 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.admin", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBAnOVlV1MhNhAAJTJsIc9qGmeQBnHKjrkPvw==", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.x", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw==", 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.478 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.y", 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ==", 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:12.479 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-03-04T10:09:12.480 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:09:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.467+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e5806b4d0 msgr2=0x7f7e5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.467+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e5806b4d0 0x7f7e5806d990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7e68036000 tx=0x7f7e6803b040 comp rx=0 tx=0).stop 2024-03-04T10:09:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.467+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 msgr2=0x7f7e7815e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.467+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 0x7f7e7815e350 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e810 tx=0x7f7e6000ecd0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 shutdown_connections 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e5806b4d0 0x7f7e5806d990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7e780cfd60 0x7f7e781f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e780ce9c0 0x7f7e7815e350 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 --2- 172.21.15.99:0/673045177 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e7806d3b0 0x7f7e78156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 >> 172.21.15.99:0/673045177 conn(0x7f7e7806acd0 msgr2=0x7f7e780ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 shutdown_connections 2024-03-04T10:09:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.468+0000 7f7e7cebf640 1 -- 172.21.15.99:0/673045177 wait complete. 2024-03-04T10:09:12.482 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.582+0000 7fb55da73640 1 Processor -- start 2024-03-04T10:09:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.582+0000 7fb55da73640 1 -- start start 2024-03-04T10:09:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.582+0000 7fb55da73640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb55815bb20 0x7fb55815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.582+0000 7fb55da73640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806d8b0 con 0x7fb55806d3b0 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.582+0000 7fb55da73640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806db90 con 0x7fb55815b030 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.582+0000 7fb55da73640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580b2720 con 0x7fb55815bb20 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb556ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb55806d3b0 0x7fb55815b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33374/0 (socket says 172.21.15.99:33374) 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb556ffd640 1 -- 172.21.15.99:0/2570319099 learned_addr learned my addr 172.21.15.99:0/2570319099 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888376281 0 0) 0x7fb55806d8b0 con 0x7fb55806d3b0 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb557fff640 1 --2- 172.21.15.99:0/2570319099 >> v2:172.21.15.162:3300/0 conn(0x7fb55815bb20 0x7fb55815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb540003520 con 0x7fb55806d3b0 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067264093 0 0) 0x7fb540003520 con 0x7fb55806d3b0 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 >> v2:172.21.15.162:3300/0 conn(0x7fb55815bb20 msgr2=0x7fb55815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 --2- 172.21.15.99:0/2570319099 >> v2:172.21.15.162:3300/0 conn(0x7fb55815bb20 0x7fb55815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 >> v1:172.21.15.162:6789/0 conn(0x7fb55815b030 legacy=0x7fb55815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55806de70 con 0x7fb55806d3b0 2024-03-04T10:09:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fb54c001670 con 0x7fb55806d3b0 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 >> v1:172.21.15.99:6789/0 conn(0x7fb55806d3b0 legacy=0x7fb55815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb540003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb540004800 0x7fb540004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5400052e0 0x7fb54000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580b2720 con 0x7fb5400039b0 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54000c1b0 con 0x7fb5400052e0 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54000c3b0 con 0x7fb540004800 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb5577fe640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb540003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb556ffd640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb540004800 0x7fb540004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.583+0000 7fb557fff640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5400052e0 0x7fb54000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5577fe640 1 -- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb540004800 msgr2=0x7fb540004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5577fe640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb540004800 0x7fb540004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5577fe640 1 -- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5400052e0 msgr2=0x7fb54000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5577fe640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5400052e0 0x7fb54000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5577fe640 1 -- 172.21.15.99:0/2570319099 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb540003520 con 0x7fb5400039b0 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb557fff640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb5400052e0 0x7fb54000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb556ffd640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb540004800 0x7fb540004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb557fff640 1 -- 172.21.15.99:0/2570319099 reap_dead start 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5577fe640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb540003dd0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fb548001720 tx=0x7fb540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cc3389b9678b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548001370 con 0x7fb5400039b0 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480052d0 con 0x7fb5400039b0 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.584+0000 7fb5567fc640 1 -- 172.21.15.99:0/2570319099 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548003830 con 0x7fb5400039b0 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 -- 172.21.15.99:0/2570319099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 msgr2=0x7fb540003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb540003dd0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fb548001720 tx=0x7fb540005830 comp rx=0 tx=0).stop 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 -- 172.21.15.99:0/2570319099 shutdown_connections 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 --2- 172.21.15.99:0/2570319099 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb540003dd0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 -- 172.21.15.99:0/2570319099 >> 172.21.15.99:0/2570319099 conn(0x7fb55806acd0 msgr2=0x7fb55806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 -- 172.21.15.99:0/2570319099 shutdown_connections 2024-03-04T10:09:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.585+0000 7fb55da73640 1 -- 172.21.15.99:0/2570319099 wait complete. 2024-03-04T10:09:12.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.586+0000 7fb55da73640 1 Processor -- start 2024-03-04T10:09:12.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.586+0000 7fb55da73640 1 -- start start 2024-03-04T10:09:12.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb55da73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb5581502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb55da73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558152f40 0x7fb558154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb55da73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb558155d60 0x7fb5581561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb55da73640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806e810 con 0x7fb5400039b0 2024-03-04T10:09:12.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb55da73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806e690 con 0x7fb558152f40 2024-03-04T10:09:12.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb55da73640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815c120 con 0x7fb558155d60 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb5581502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb5581502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57422/0 (socket says 172.21.15.99:57422) 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 -- 172.21.15.99:0/1396846475 learned_addr learned my addr 172.21.15.99:0/1396846475 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb556ffd640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558152f40 0x7fb558154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb557fff640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb558155d60 0x7fb5581561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 -- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb558155d60 msgr2=0x7fb5581561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb558155d60 0x7fb5581561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 -- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558152f40 msgr2=0x7fb558154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558152f40 0x7fb558154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb5577fe640 1 -- 172.21.15.99:0/1396846475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558150810 con 0x7fb5400039b0 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.587+0000 7fb556ffd640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558152f40 0x7fb558154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.588+0000 7fb5577fe640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb5581502d0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb548000c00 tx=0x7fb548001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.588+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548041020 con 0x7fb5400039b0 2024-03-04T10:09:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.588+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548005480 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.588+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5581554a0 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.588+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548007a30 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.588+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb558150d40 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.589+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558074860 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.589+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb548044020 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.590+0000 7fb53ffff640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb54406b500 0x7fb54406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.590+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb5480abad0 con 0x7fb5400039b0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.590+0000 7fb556ffd640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb54406b500 0x7fb54406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.590+0000 7fb556ffd640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb54406b500 0x7fb54406d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb54c001eb0 tx=0x7fb54c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.594+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb558074860 con 0x7fb5400039b0 2024-03-04T10:09:12.765 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.764+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb55815de40 con 0x7fb5400039b0 2024-03-04T10:09:12.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.765+0000 7fb53ffff640 1 -- 172.21.15.99:0/1396846475 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13129 (secure 0 0 0) 0x7fb54804b030 con 0x7fb5400039b0 2024-03-04T10:09:12.766 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:12.767 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":77,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:12.104670+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb54406b500 msgr2=0x7fb54406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb54406b500 0x7fb54406d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb54c001eb0 tx=0x7fb54c009040 comp rx=0 tx=0).stop 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 msgr2=0x7fb5581502d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb5581502d0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fb548000c00 tx=0x7fb548001120 comp rx=0 tx=0).stop 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 shutdown_connections 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb54406b500 0x7fb54406d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb558155d60 0x7fb5581561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb558152f40 0x7fb558154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 --2- 172.21.15.99:0/1396846475 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb5400039b0 0x7fb5581502d0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 >> 172.21.15.99:0/1396846475 conn(0x7fb55806acd0 msgr2=0x7fb55815cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 shutdown_connections 2024-03-04T10:09:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.767+0000 7fb55da73640 1 -- 172.21.15.99:0/1396846475 wait complete. 2024-03-04T10:09:12.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:09:12.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:09:12.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.881+0000 7f3a39820640 1 Processor -- start 2024-03-04T10:09:12.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.881+0000 7f3a39820640 1 -- start start 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a39820640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3a3406ff60 0x7f3a340b2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a39820640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a340cd820 0x7f3a3415a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a39820640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a34070340 con 0x7f3a340cf190 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a39820640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a340cc050 con 0x7f3a340cd820 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a39820640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3406f530 con 0x7f3a3406ff60 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a337fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a340cd820 0x7f3a3415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a327fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3a340cf190 0x7f3a340cd110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:33380/0 (socket says 172.21.15.99:33380) 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a327fc640 1 -- 172.21.15.99:0/2954255490 learned_addr learned my addr 172.21.15.99:0/2954255490 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a337fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a340cd820 0x7f3a3415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57426/0 (socket says 172.21.15.99:57426) 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.882+0000 7f3a32ffd640 1 --2- 172.21.15.99:0/2954255490 >> v2:172.21.15.162:3300/0 conn(0x7f3a3406ff60 0x7f3a340b2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.883+0000 7f3a337fe640 1 -- 172.21.15.99:0/2954255490 >> v2:172.21.15.162:3300/0 conn(0x7f3a3406ff60 msgr2=0x7f3a340b2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.883+0000 7f3a337fe640 1 --2- 172.21.15.99:0/2954255490 >> v2:172.21.15.162:3300/0 conn(0x7f3a3406ff60 0x7f3a340b2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.883+0000 7f3a337fe640 1 -- 172.21.15.99:0/2954255490 >> v1:172.21.15.99:6789/0 conn(0x7f3a340cf190 legacy=0x7f3a340cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.883+0000 7f3a31ffb640 1 -- 172.21.15.99:0/2954255490 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864292867 0 0) 0x7f3a34070340 con 0x7f3a340cf190 2024-03-04T10:09:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.883+0000 7f3a337fe640 1 -- 172.21.15.99:0/2954255490 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a340cb440 con 0x7f3a340cd820 2024-03-04T10:09:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.883+0000 7f3a337fe640 1 --2- 172.21.15.99:0/2954255490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a3415a350 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3a340ce280 tx=0x7f3a2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d891c51d320481aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a31ffb640 1 -- 172.21.15.99:0/2954255490 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c039070 con 0x7f3a340cd820 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a31ffb640 1 -- 172.21.15.99:0/2954255490 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c02cc80 con 0x7f3a340cd820 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a31ffb640 1 -- 172.21.15.99:0/2954255490 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c034660 con 0x7f3a340cd820 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 -- 172.21.15.99:0/2954255490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 msgr2=0x7f3a3415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 --2- 172.21.15.99:0/2954255490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a3415a350 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3a340ce280 tx=0x7f3a2c02bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 -- 172.21.15.99:0/2954255490 shutdown_connections 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 --2- 172.21.15.99:0/2954255490 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a3415a350 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 --2- 172.21.15.99:0/2954255490 >> v2:172.21.15.162:3300/0 conn(0x7f3a3406ff60 0x7f3a340b2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 -- 172.21.15.99:0/2954255490 >> 172.21.15.99:0/2954255490 conn(0x7f3a340b2250 msgr2=0x7f3a340b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:12.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 -- 172.21.15.99:0/2954255490 shutdown_connections 2024-03-04T10:09:12.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.884+0000 7f3a39820640 1 -- 172.21.15.99:0/2954255490 wait complete. 2024-03-04T10:09:12.891 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.885+0000 7f3a39820640 1 Processor -- start 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.885+0000 7f3a39820640 1 -- start start 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a39820640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a3406ff60 0x7f3a340c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a39820640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a340b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a39820640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a340cf190 0x7f3a34152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a39820640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a340cbce0 con 0x7f3a340cd820 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a39820640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a340cba00 con 0x7f3a340cf190 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a39820640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a340cc4e0 con 0x7f3a3406ff60 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a327fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a340b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a327fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a340b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57442/0 (socket says 172.21.15.99:57442) 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a327fc640 1 -- 172.21.15.99:0/899221413 learned_addr learned my addr 172.21.15.99:0/899221413 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a32ffd640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a3406ff60 0x7f3a340c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.886+0000 7f3a337fe640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a340cf190 0x7f3a34152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a327fc640 1 -- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a3406ff60 msgr2=0x7f3a340c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a327fc640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a3406ff60 0x7f3a340c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a327fc640 1 -- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a340cf190 msgr2=0x7f3a34152a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a327fc640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a340cf190 0x7f3a34152a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a327fc640 1 -- 172.21.15.99:0/899221413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a340b52c0 con 0x7f3a340cd820 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a327fc640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a340b4d80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00c500 tx=0x7f3a1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c018070 con 0x7f3a340cd820 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1c007580 con 0x7f3a340cd820 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a340b5560 con 0x7f3a340cd820 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c004a20 con 0x7f3a340cd820 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.887+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a340b5b20 con 0x7f3a340cd820 2024-03-04T10:09:12.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.888+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a34074f60 con 0x7f3a340cd820 2024-03-04T10:09:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.888+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3a1c01b070 con 0x7f3a340cd820 2024-03-04T10:09:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.889+0000 7f3a2bfff640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a1406b4d0 0x7f3a1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.889+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3a1c089230 con 0x7f3a340cd820 2024-03-04T10:09:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.889+0000 7f3a32ffd640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a1406b4d0 0x7f3a1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.890+0000 7f3a32ffd640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a1406b4d0 0x7f3a1406d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3a340b0cf0 tx=0x7f3a24009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:12.893+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a1c057cb0 con 0x7f3a340cd820 2024-03-04T10:09:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.060+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a340ccef0 con 0x7f3a340cd820 2024-03-04T10:09:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.061+0000 7f3a2bfff640 1 -- 172.21.15.99:0/899221413 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a1c05bd50 con 0x7f3a340cd820 2024-03-04T10:09:13.062 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:09:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.062+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a1406b4d0 msgr2=0x7f3a1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a1406b4d0 0x7f3a1406d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3a340b0cf0 tx=0x7f3a24009190 comp rx=0 tx=0).stop 2024-03-04T10:09:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 msgr2=0x7f3a340b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a340b4d80 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00c500 tx=0x7f3a1c00c9c0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 shutdown_connections 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3a1406b4d0 0x7f3a1406d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a340cf190 0x7f3a34152a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a340cd820 0x7f3a340b4d80 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 --2- 172.21.15.99:0/899221413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a3406ff60 0x7f3a340c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 >> 172.21.15.99:0/899221413 conn(0x7f3a340b2250 msgr2=0x7f3a3415e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 shutdown_connections 2024-03-04T10:09:13.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.063+0000 7f3a39820640 1 -- 172.21.15.99:0/899221413 wait complete. 2024-03-04T10:09:13.076 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T10:09:13.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.174+0000 7f236de85640 1 Processor -- start 2024-03-04T10:09:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.174+0000 7f236de85640 1 -- start start 2024-03-04T10:09:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f236de85640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2368158410 0x7f23680b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f236de85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 0x7f2368161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f236de85640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236806b500 con 0x7f2368161ca0 2024-03-04T10:09:13.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f236de85640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23680b3f00 con 0x7f2368158410 2024-03-04T10:09:13.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f236de85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236806b7e0 con 0x7f236806acd0 2024-03-04T10:09:13.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 0x7f2368161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.175+0000 7f23677fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2368158410 0x7f23680b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f2366ffd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 0x7f2368161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:49208/0 (socket says 172.21.15.99:49208) 2024-03-04T10:09:13.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f2366ffd640 1 -- 172.21.15.99:0/2714431448 learned_addr learned my addr 172.21.15.99:0/2714431448 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:13.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f2366ffd640 1 -- 172.21.15.99:0/2714431448 >> v2:172.21.15.162:3300/0 conn(0x7f2368158410 msgr2=0x7f23680b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f2366ffd640 1 --2- 172.21.15.99:0/2714431448 >> v2:172.21.15.162:3300/0 conn(0x7f2368158410 0x7f23680b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f2366ffd640 1 -- 172.21.15.99:0/2714431448 >> v1:172.21.15.162:6789/0 conn(0x7f2368161ca0 legacy=0x7f236815e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f2366ffd640 1 -- 172.21.15.99:0/2714431448 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236806bac0 con 0x7f236806acd0 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.176+0000 7f23677fe640 1 --2- 172.21.15.99:0/2714431448 >> v2:172.21.15.162:3300/0 conn(0x7f2368158410 0x7f23680b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f2366ffd640 1 --2- 172.21.15.99:0/2714431448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 0x7f2368161760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f235c009870 tx=0x7f235c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa8e2091a7c3f359 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f23667fc640 1 -- 172.21.15.99:0/2714431448 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c02f830 con 0x7f236806acd0 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f23667fc640 1 -- 172.21.15.99:0/2714431448 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c02f9d0 con 0x7f236806acd0 2024-03-04T10:09:13.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f23667fc640 1 -- 172.21.15.99:0/2714431448 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c02fda0 con 0x7f236806acd0 2024-03-04T10:09:13.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f236de85640 1 -- 172.21.15.99:0/2714431448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 msgr2=0x7f2368161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f236de85640 1 --2- 172.21.15.99:0/2714431448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 0x7f2368161760 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f235c009870 tx=0x7f235c02ba10 comp rx=0 tx=0).stop 2024-03-04T10:09:13.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f236de85640 1 -- 172.21.15.99:0/2714431448 shutdown_connections 2024-03-04T10:09:13.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f236de85640 1 --2- 172.21.15.99:0/2714431448 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f236806acd0 0x7f2368161760 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.177+0000 7f236de85640 1 --2- 172.21.15.99:0/2714431448 >> v2:172.21.15.162:3300/0 conn(0x7f2368158410 0x7f23680b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.178+0000 7f236de85640 1 -- 172.21.15.99:0/2714431448 >> 172.21.15.99:0/2714431448 conn(0x7f23680cfef0 msgr2=0x7f23680d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:13.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.178+0000 7f236de85640 1 -- 172.21.15.99:0/2714431448 shutdown_connections 2024-03-04T10:09:13.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.178+0000 7f236de85640 1 -- 172.21.15.99:0/2714431448 wait complete. 2024-03-04T10:09:13.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 Processor -- start 2024-03-04T10:09:13.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 -- start start 2024-03-04T10:09:13.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 0x7f2368155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2368158410 0x7f2368159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2368161ca0 0x7f23680b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236806c5a0 con 0x7f236806acd0 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236806c080 con 0x7f2368158410 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.179+0000 7f236de85640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236806ee60 con 0x7f2368161ca0 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 0x7f2368155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 0x7f2368155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57452/0 (socket says 172.21.15.99:57452) 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 -- 172.21.15.99:0/3335328426 learned_addr learned my addr 172.21.15.99:0/3335328426 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f2366ffd640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2368158410 0x7f2368159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f2367fff640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2368161ca0 0x7f23680b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 -- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2368161ca0 msgr2=0x7f23680b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2368161ca0 0x7f23680b0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 -- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2368158410 msgr2=0x7f2368159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2368158410 0x7f2368159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 -- 172.21.15.99:0/3335328426 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680ce310 con 0x7f236806acd0 2024-03-04T10:09:13.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f2366ffd640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2368158410 0x7f2368159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f23677fe640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 0x7f2368155fc0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f235800ebe0 tx=0x7f235800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358017070 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235800cc30 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23680ce030 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358013470 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.180+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2368158b70 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.181+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2368074f60 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.182+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f235801a070 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.182+0000 7f234ffff640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f235406b4d0 0x7f235406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.182+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2358089f90 con 0x7f236806acd0 2024-03-04T10:09:13.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.182+0000 7f2366ffd640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f235406b4d0 0x7f235406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.183+0000 7f2366ffd640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f235406b4d0 0x7f235406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f235c036000 tx=0x7f235c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:13.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.186+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2358058fe0 con 0x7f236806acd0 2024-03-04T10:09:13.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.359+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2368066e10 con 0x7f236806acd0 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.359+0000 7f234ffff640 1 -- 172.21.15.99:0/3335328426 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f23580588e0 con 0x7f236806acd0 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:09:13.361 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f235406b4d0 msgr2=0x7f235406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f235406b4d0 0x7f235406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f235c036000 tx=0x7f235c008030 comp rx=0 tx=0).stop 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 msgr2=0x7f2368155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 0x7f2368155fc0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f235800ebe0 tx=0x7f235800c520 comp rx=0 tx=0).stop 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 shutdown_connections 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f235406b4d0 0x7f235406d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2368161ca0 0x7f23680b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2368158410 0x7f2368159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 --2- 172.21.15.99:0/3335328426 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f236806acd0 0x7f2368155fc0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 >> 172.21.15.99:0/3335328426 conn(0x7f23680cfef0 msgr2=0x7f23680aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:13.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 shutdown_connections 2024-03-04T10:09:13.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.361+0000 7f236de85640 1 -- 172.21.15.99:0/3335328426 wait complete. 2024-03-04T10:09:13.373 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T10:09:13.373 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T10:09:13.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.475+0000 7fda25b7e640 1 Processor -- start 2024-03-04T10:09:13.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.475+0000 7fda25b7e640 1 -- start start 2024-03-04T10:09:13.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda25b7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fda201438e0 0x7fda200cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda25b7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda20143cc0 0x7fda200ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda25b7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fda200cedc0 0x7fda20071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda25b7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda200b2aa0 con 0x7fda201438e0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda25b7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda2015e5d0 con 0x7fda200cedc0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda25b7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2015e8b0 con 0x7fda20143cc0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fda201438e0 0x7fda200cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fda201438e0 0x7fda200cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33374/0 (socket says 172.21.15.99:33374) 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 -- 172.21.15.99:0/2774814589 learned_addr learned my addr 172.21.15.99:0/2774814589 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1ffff640 1 --2- 172.21.15.99:0/2774814589 >> v2:172.21.15.162:3300/0 conn(0x7fda200cedc0 0x7fda20071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1effd640 1 --2- 172.21.15.99:0/2774814589 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda20143cc0 0x7fda200ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 -- 172.21.15.99:0/2774814589 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda20143cc0 msgr2=0x7fda200ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 --2- 172.21.15.99:0/2774814589 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda20143cc0 0x7fda200ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 -- 172.21.15.99:0/2774814589 >> v2:172.21.15.162:3300/0 conn(0x7fda200cedc0 msgr2=0x7fda20071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 --2- 172.21.15.99:0/2774814589 >> v2:172.21.15.162:3300/0 conn(0x7fda200cedc0 0x7fda20071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.476+0000 7fda1f7fe640 1 -- 172.21.15.99:0/2774814589 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2015eb90 con 0x7fda201438e0 2024-03-04T10:09:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.477+0000 7fda1f7fe640 1 --2- 172.21.15.99:0/2774814589 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda200cb910 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fda200b2a30 tx=0x7fda1002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf58253b6a9069d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.477+0000 7fda1e7fc640 1 -- 172.21.15.99:0/2774814589 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1003b070 con 0x7fda201438e0 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.477+0000 7fda1e7fc640 1 -- 172.21.15.99:0/2774814589 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda10036440 con 0x7fda201438e0 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.477+0000 7fda1e7fc640 1 -- 172.21.15.99:0/2774814589 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10036810 con 0x7fda201438e0 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 -- 172.21.15.99:0/2774814589 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 msgr2=0x7fda200cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 --2- 172.21.15.99:0/2774814589 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda200cb910 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fda200b2a30 tx=0x7fda1002e0e0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 -- 172.21.15.99:0/2774814589 shutdown_connections 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 --2- 172.21.15.99:0/2774814589 >> v2:172.21.15.162:3300/0 conn(0x7fda200cedc0 0x7fda20071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 --2- 172.21.15.99:0/2774814589 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda20143cc0 0x7fda200ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 --2- 172.21.15.99:0/2774814589 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda200cb910 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 -- 172.21.15.99:0/2774814589 >> 172.21.15.99:0/2774814589 conn(0x7fda200b1600 msgr2=0x7fda200b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 -- 172.21.15.99:0/2774814589 shutdown_connections 2024-03-04T10:09:13.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.478+0000 7fda25b7e640 1 -- 172.21.15.99:0/2774814589 wait complete. 2024-03-04T10:09:13.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.479+0000 7fda25b7e640 1 Processor -- start 2024-03-04T10:09:13.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.479+0000 7fda25b7e640 1 -- start start 2024-03-04T10:09:13.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda25b7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200cedc0 0x7fda200c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda25b7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda201f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda25b7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200c50a0 0x7fda200c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda25b7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda200cf250 con 0x7fda201438e0 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda25b7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2015f150 con 0x7fda200c50a0 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda25b7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda200cbe50 con 0x7fda200cedc0 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda201f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda201f7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33386/0 (socket says 172.21.15.99:33386) 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 -- 172.21.15.99:0/1977843617 learned_addr learned my addr 172.21.15.99:0/1977843617 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1f7fe640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200cedc0 0x7fda200c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1ffff640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200c50a0 0x7fda200c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 -- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200cedc0 msgr2=0x7fda200c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200cedc0 0x7fda200c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 -- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200c50a0 msgr2=0x7fda200c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:13.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200c50a0 0x7fda200c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1effd640 1 -- 172.21.15.99:0/1977843617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda200c62e0 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.480+0000 7fda1ffff640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200c50a0 0x7fda200c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.481+0000 7fda1effd640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda201f7260 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fda1400d800 tx=0x7fda1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.481+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1400b6c0 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.481+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda14019070 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.481+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200c65c0 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.481+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1400ba80 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.481+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda200c6b80 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.482+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda20074f60 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.482+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fda1400bc20 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.482+0000 7fda07fff640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0c06b500 0x7fda0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.483+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fda1408a310 con 0x7fda201438e0 2024-03-04T10:09:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.483+0000 7fda1f7fe640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0c06b500 0x7fda0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:13.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.483+0000 7fda1f7fe640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0c06b500 0x7fda0c06d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fda10038000 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:13.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.487+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fda140593e0 con 0x7fda201438e0 2024-03-04T10:09:13.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:13.657+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fda2006f200 con 0x7fda201438e0 2024-03-04T10:09:14.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.105+0000 7fda07fff640 1 -- 172.21.15.99:0/1977843617 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7fda14088e20 con 0x7fda201438e0 2024-03-04T10:09:14.106 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0c06b500 msgr2=0x7fda0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0c06b500 0x7fda0c06d9c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fda10038000 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 msgr2=0x7fda201f7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda201f7260 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fda1400d800 tx=0x7fda1400dcc0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 shutdown_connections 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda0c06b500 0x7fda0c06d9c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda200c50a0 0x7fda200c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda201438e0 0x7fda201f7260 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 --2- 172.21.15.99:0/1977843617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda200cedc0 0x7fda200c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 >> 172.21.15.99:0/1977843617 conn(0x7fda200b1600 msgr2=0x7fda200b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:14.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 shutdown_connections 2024-03-04T10:09:14.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.107+0000 7fda25b7e640 1 -- 172.21.15.99:0/1977843617 wait complete. 2024-03-04T10:09:14.123 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T10:09:14.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.228+0000 7f86996bf640 1 Processor -- start 2024-03-04T10:09:14.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.228+0000 7f86996bf640 1 -- start start 2024-03-04T10:09:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f86996bf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f86940cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f86996bf640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869406ae00 con 0x7f86940cc900 2024-03-04T10:09:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f86996bf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869406b0e0 con 0x7f86940ccfe0 2024-03-04T10:09:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f86996bf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86940ccde0 con 0x7f86940cdd70 2024-03-04T10:09:14.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f86927fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f86940cc900 0x7f869415d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45912/0 (socket says 172.21.15.99:45912) 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f86927fc640 1 -- 172.21.15.99:0/1702803617 learned_addr learned my addr 172.21.15.99:0/1702803617 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917523578 0 0) 0x7f869406ae00 con 0x7f86940cc900 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f867c003520 con 0x7f86940cc900 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.229+0000 7f8692ffd640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f86940cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1872020054 0 0) 0x7f867c003520 con 0x7f86940cc900 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 msgr2=0x7f86940cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f86940cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 >> v1:172.21.15.162:6789/0 conn(0x7f86940ccfe0 legacy=0x7f869415db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869406b3c0 con 0x7f86940cc900 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8688001670 con 0x7f86940cc900 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f86940cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 >> v1:172.21.15.99:6789/0 conn(0x7f86940cc900 legacy=0x7f869415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f867c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f867c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f867c004d40 0x7f867c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86940ccde0 con 0x7f867c0039e0 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00bb50 con 0x7f867c004d40 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c00bd50 con 0x7f86940cdd70 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f867c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f86927fc640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f867c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f86937fe640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f867c004d40 0x7f867c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 -- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 msgr2=0x7f867c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86940cdd70 0x7f867c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 -- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f867c004d40 msgr2=0x7f867c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:14.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f867c004d40 0x7f867c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f8692ffd640 1 -- 172.21.15.99:0/1702803617 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c003520 con 0x7f867c0039e0 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.230+0000 7f86937fe640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f867c004d40 0x7f867c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f8692ffd640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f867c003e00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f86840098a0 tx=0x7f867c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc3a875a5c68606 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868400f040 con 0x7f867c0039e0 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8684004230 con 0x7f867c0039e0 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f8691ffb640 1 -- 172.21.15.99:0/1702803617 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684003610 con 0x7f867c0039e0 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f86996bf640 1 -- 172.21.15.99:0/1702803617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 msgr2=0x7f867c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f86996bf640 1 --2- 172.21.15.99:0/1702803617 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f867c003e00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f86840098a0 tx=0x7f867c005830 comp rx=0 tx=0).stop 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.231+0000 7f86937fe640 1 -- 172.21.15.99:0/1702803617 reap_dead start 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.232+0000 7f86996bf640 1 -- 172.21.15.99:0/1702803617 shutdown_connections 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.232+0000 7f86996bf640 1 -- 172.21.15.99:0/1702803617 >> 172.21.15.99:0/1702803617 conn(0x7f86940b22e0 msgr2=0x7f86940b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:14.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.232+0000 7f86996bf640 1 -- 172.21.15.99:0/1702803617 shutdown_connections 2024-03-04T10:09:14.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.232+0000 7f86996bf640 1 -- 172.21.15.99:0/1702803617 wait complete. 2024-03-04T10:09:14.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.233+0000 7f86996bf640 1 Processor -- start 2024-03-04T10:09:14.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.233+0000 7f86996bf640 1 -- start start 2024-03-04T10:09:14.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.233+0000 7f86996bf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f86941e7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.233+0000 7f86996bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869415e3b0 0x7f86940c4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f86996bf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8694160f80 0x7f86940c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f86996bf640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86940b2e90 con 0x7f867c0039e0 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f86996bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869415e230 con 0x7f869415e3b0 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f86996bf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86940b3010 con 0x7f8694160f80 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f86941e7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f86941e7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33410/0 (socket says 172.21.15.99:33410) 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 -- 172.21.15.99:0/2318591826 learned_addr learned my addr 172.21.15.99:0/2318591826 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 -- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8694160f80 msgr2=0x7f86940c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8694160f80 0x7f86940c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 -- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869415e3b0 msgr2=0x7f86940c4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:14.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869415e3b0 0x7f86940c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 -- 172.21.15.99:0/2318591826 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86941e8030 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f8692ffd640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f86941e7a50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f86840099c0 tx=0x7f8684004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684048070 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8684030430 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684030800 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86940c6d70 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.234+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869415e810 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.235+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8694074f60 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.236+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f868404b070 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.236+0000 7f867b7fe640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f868006b4d0 0x7f868006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.236+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f86840abb50 con 0x7f867c0039e0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.236+0000 7f86927fc640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f868006b4d0 0x7f868006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:14.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.237+0000 7f86927fc640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f868006b4d0 0x7f868006d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8688001cd0 tx=0x7f8688001b80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:14.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.241+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8684034020 con 0x7f867c0039e0 2024-03-04T10:09:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:14.413+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f86940ce480 con 0x7f867c0039e0 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.107+0000 7f867b7fe640 1 -- 172.21.15.99:0/2318591826 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f868407aa20 con 0x7f867c0039e0 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f868006b4d0 msgr2=0x7f868006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f868006b4d0 0x7f868006d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8688001cd0 tx=0x7f8688001b80 comp rx=0 tx=0).stop 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 msgr2=0x7f86941e7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f86941e7a50 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f86840099c0 tx=0x7f8684004730 comp rx=0 tx=0).stop 2024-03-04T10:09:15.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 shutdown_connections 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f868006b4d0 0x7f868006d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8694160f80 0x7f86940c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f869415e3b0 0x7f86940c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 --2- 172.21.15.99:0/2318591826 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f867c0039e0 0x7f86941e7a50 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 >> 172.21.15.99:0/2318591826 conn(0x7f86940b22e0 msgr2=0x7f869406fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 shutdown_connections 2024-03-04T10:09:15.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.110+0000 7f86996bf640 1 -- 172.21.15.99:0/2318591826 wait complete. 2024-03-04T10:09:15.138 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T10:09:15.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.243+0000 7feb0a69a640 1 Processor -- start 2024-03-04T10:09:15.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.243+0000 7feb0a69a640 1 -- start start 2024-03-04T10:09:15.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.244+0000 7feb0a69a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feb040b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.244+0000 7feb0a69a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0406d8a0 con 0x7feb0415a810 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.244+0000 7feb0a69a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0406db80 con 0x7feb040b48a0 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.244+0000 7feb0a69a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0406de60 con 0x7feb0406d3c0 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.244+0000 7feb03fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7feb0415a810 0x7feb040b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45918/0 (socket says 172.21.15.99:45918) 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.244+0000 7feb03fff640 1 -- 172.21.15.99:0/1131198408 learned_addr learned my addr 172.21.15.99:0/1131198408 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130515407 0 0) 0x7feb0406d8a0 con 0x7feb0415a810 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec003520 con 0x7feb0415a810 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feb040b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1638473919 0 0) 0x7feaec003520 con 0x7feb0415a810 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 msgr2=0x7feb040b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:15.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feb040b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 >> v1:172.21.15.162:6789/0 conn(0x7feb040b48a0 legacy=0x7feb040b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0406e140 con 0x7feb0415a810 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7feaf8001670 con 0x7feb0415a810 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feb040b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 >> v1:172.21.15.99:6789/0 conn(0x7feb0415a810 legacy=0x7feb040b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feaec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feaec003fd0 0x7feaec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feaec004d40 0x7feaec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0406de60 con 0x7feaec004d40 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaec00bb20 con 0x7feaec003fd0 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaec00bd20 con 0x7feb0406d3c0 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feaec004d40 0x7feaec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb03fff640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feaec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb037fe640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feaec003fd0 0x7feaec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 -- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 msgr2=0x7feaec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb0406d3c0 0x7feaec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 -- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feaec003fd0 msgr2=0x7feaec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feaec003fd0 0x7feaec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.245+0000 7feb08c37640 1 -- 172.21.15.99:0/1131198408 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec003520 con 0x7feaec004d40 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.246+0000 7feb08c37640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feaec004d40 0x7feaec0051c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7feb0406d7d0 tx=0x7feaec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de319df81e3147bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.246+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc009c80 con 0x7feaec004d40 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.246+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc02ccb0 con 0x7feaec004d40 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.246+0000 7feb02ffd640 1 -- 172.21.15.99:0/1131198408 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc003b90 con 0x7feaec004d40 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb0a69a640 1 -- 172.21.15.99:0/1131198408 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feaec004d40 msgr2=0x7feaec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb0a69a640 1 --2- 172.21.15.99:0/1131198408 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feaec004d40 0x7feaec0051c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7feb0406d7d0 tx=0x7feaec005830 comp rx=0 tx=0).stop 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb08c37640 1 -- 172.21.15.99:0/1131198408 reap_dead start 2024-03-04T10:09:15.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb0a69a640 1 -- 172.21.15.99:0/1131198408 shutdown_connections 2024-03-04T10:09:15.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb0a69a640 1 -- 172.21.15.99:0/1131198408 >> 172.21.15.99:0/1131198408 conn(0x7feb0406acd0 msgr2=0x7feb0406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:15.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb0a69a640 1 -- 172.21.15.99:0/1131198408 shutdown_connections 2024-03-04T10:09:15.252 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.247+0000 7feb0a69a640 1 -- 172.21.15.99:0/1131198408 wait complete. 2024-03-04T10:09:15.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.248+0000 7feb0a69a640 1 Processor -- start 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.248+0000 7feb0a69a640 1 -- start start 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb0a69a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb04071590 0x7feb040ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb0a69a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb040adcc0 0x7feb041592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb0a69a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 0x7feb041597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb0a69a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb040cc130 con 0x7feb04071970 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb0a69a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb040cbfb0 con 0x7feb040adcc0 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb0a69a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb04159d30 con 0x7feb04071590 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 0x7feb041597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 0x7feb041597f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33420/0 (socket says 172.21.15.99:33420) 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 -- 172.21.15.99:0/2072051451 learned_addr learned my addr 172.21.15.99:0/2072051451 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb03fff640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb04071590 0x7feb040ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 -- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb04071590 msgr2=0x7feb040ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb04071590 0x7feb040ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 -- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb040adcc0 msgr2=0x7feb041592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb040adcc0 0x7feb041592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 -- 172.21.15.99:0/2072051451 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb040acf50 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.249+0000 7feb08c37640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 0x7feb041597f0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7feafc007470 tx=0x7feafc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.250+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc02ce60 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.250+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feafc045070 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.250+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb040ad230 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.250+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc009c80 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.250+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb040aefe0 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.251+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb04155500 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.251+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7feafc0046d0 con 0x7feb04071970 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.251+0000 7feb00ff9640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feaf006b500 0x7feaf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:15.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.252+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6001+0+0 (secure 0 0 0) 0x7feafc0ac460 con 0x7feb04071970 2024-03-04T10:09:15.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.252+0000 7feb03fff640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feaf006b500 0x7feaf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:15.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.252+0000 7feb03fff640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feaf006b500 0x7feaf006d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feaf8007c70 tx=0x7feaf8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:15.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.256+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7feb04155500 con 0x7feb04071970 2024-03-04T10:09:15.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:15.460+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feb040af1e0 con 0x7feb04071970 2024-03-04T10:09:16.140 INFO:teuthology.orchestra.run.smithi099.stderr:new fs with metadata pool 11 and data pool 12 2024-03-04T10:09:16.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.135+0000 7feb00ff9640 1 -- 172.21.15.99:0/2072051451 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v62) v1 ==== 174+0+0 (secure 0 0 0) 0x7feb040af1e0 con 0x7feb04071970 2024-03-04T10:09:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.139+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feaf006b500 msgr2=0x7feaf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.139+0000 7feb0a69a640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feaf006b500 0x7feaf006d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feaf8007c70 tx=0x7feaf8009040 comp rx=0 tx=0).stop 2024-03-04T10:09:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.139+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 msgr2=0x7feb041597f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.139+0000 7feb0a69a640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 0x7feb041597f0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7feafc007470 tx=0x7feafc004ab0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.142+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 shutdown_connections 2024-03-04T10:09:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.142+0000 7feb0a69a640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7feaf006b500 0x7feaf006d9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.142+0000 7feb0a69a640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7feb04071970 0x7feb041597f0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.142+0000 7feb0a69a640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb040adcc0 0x7feb041592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.142+0000 7feb0a69a640 1 --2- 172.21.15.99:0/2072051451 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7feb04071590 0x7feb040ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.142+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 >> 172.21.15.99:0/2072051451 conn(0x7feb0406acd0 msgr2=0x7feb0414f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.144+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 shutdown_connections 2024-03-04T10:09:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.144+0000 7feb0a69a640 1 -- 172.21.15.99:0/2072051451 wait complete. 2024-03-04T10:09:16.161 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:16.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.260+0000 7f890d759640 1 Processor -- start 2024-03-04T10:09:16.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.260+0000 7f890d759640 1 -- start start 2024-03-04T10:09:16.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f890d759640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f890815de00 0x7f89080b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f890d759640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f89080b9380 0x7f89080b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f890d759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f890d759640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89080d2820 con 0x7f890815de00 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f890d759640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890815e1e0 con 0x7f89080b9380 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f890d759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89080d1c10 con 0x7f89080b70d0 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f890815de00 0x7f89080b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f890815de00 0x7f89080b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33432/0 (socket says 172.21.15.99:33432) 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 -- 172.21.15.99:0/2127614325 learned_addr learned my addr 172.21.15.99:0/2127614325 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f89077fe640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f89067fc640 1 --2- 172.21.15.99:0/2127614325 >> v2:172.21.15.162:3300/0 conn(0x7f89080b9380 0x7f89080b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 -- 172.21.15.99:0/2127614325 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 msgr2=0x7f89080b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 -- 172.21.15.99:0/2127614325 >> v2:172.21.15.162:3300/0 conn(0x7f89080b9380 msgr2=0x7f89080b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 --2- 172.21.15.99:0/2127614325 >> v2:172.21.15.162:3300/0 conn(0x7f89080b9380 0x7f89080b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f8906ffd640 1 -- 172.21.15.99:0/2127614325 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89080d1ef0 con 0x7f890815de00 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f89077fe640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.261+0000 7f89067fc640 1 --2- 172.21.15.99:0/2127614325 >> v2:172.21.15.162:3300/0 conn(0x7f89080b9380 0x7f89080b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.262+0000 7f8906ffd640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080b2e60 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8908161830 tx=0x7f88f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3b1244b16caaac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.262+0000 7f8905ffb640 1 -- 172.21.15.99:0/2127614325 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8035020 con 0x7f890815de00 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.262+0000 7f8905ffb640 1 -- 172.21.15.99:0/2127614325 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f8031870 con 0x7f890815de00 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.262+0000 7f8905ffb640 1 -- 172.21.15.99:0/2127614325 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8031c40 con 0x7f890815de00 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 -- 172.21.15.99:0/2127614325 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 msgr2=0x7f89080b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080b2e60 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f8908161830 tx=0x7f88f802b910 comp rx=0 tx=0).stop 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 -- 172.21.15.99:0/2127614325 shutdown_connections 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 --2- 172.21.15.99:0/2127614325 >> v2:172.21.15.162:3300/0 conn(0x7f89080b9380 0x7f89080b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 --2- 172.21.15.99:0/2127614325 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080b2e60 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 -- 172.21.15.99:0/2127614325 >> 172.21.15.99:0/2127614325 conn(0x7f89080d0470 msgr2=0x7f89080d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 -- 172.21.15.99:0/2127614325 shutdown_connections 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.263+0000 7f890d759640 1 -- 172.21.15.99:0/2127614325 wait complete. 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.264+0000 7f890d759640 1 Processor -- start 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.264+0000 7f890d759640 1 -- start start 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080abba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89080b9380 0x7f89080aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080aa910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89080b97e0 con 0x7f890815de00 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89080d24b0 con 0x7f89080b9380 2024-03-04T10:09:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89080b59b0 con 0x7f89080b70d0 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080aa910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33448/0 (socket says 172.21.15.99:33448) 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 -- 172.21.15.99:0/508383680 learned_addr learned my addr 172.21.15.99:0/508383680 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f8906ffd640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89067fc640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89080b9380 0x7f89080aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 -- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 msgr2=0x7f89080abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 -- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89080b9380 msgr2=0x7f89080aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89080b9380 0x7f89080aa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 -- 172.21.15.99:0/508383680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89080b2680 con 0x7f890815de00 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f89077fe640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080aa910 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f890000cef0 tx=0x7f890000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890001a070 con 0x7f890815de00 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890000d840 con 0x7f890815de00 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89080b2960 con 0x7f890815de00 2024-03-04T10:09:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890000dc30 con 0x7f890815de00 2024-03-04T10:09:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.265+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89080b2f20 con 0x7f890815de00 2024-03-04T10:09:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.266+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908074f60 con 0x7f890815de00 2024-03-04T10:09:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.267+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f890001d070 con 0x7f890815de00 2024-03-04T10:09:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.267+0000 7f88ef7fe640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88f406b4d0 0x7f88f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.267+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f890008ab90 con 0x7f890815de00 2024-03-04T10:09:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.268+0000 7f8906ffd640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88f406b4d0 0x7f88f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.268+0000 7f8906ffd640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88f406b4d0 0x7f88f406d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f88f8036000 tx=0x7f88f803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.272+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8900059910 con 0x7f890815de00 2024-03-04T10:09:16.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.444+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89080008d0 con 0x7f890815de00 2024-03-04T10:09:16.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.446+0000 7f88ef7fe640 1 -- 172.21.15.99:0/508383680 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16922 (secure 0 0 0) 0x7f8900059210 con 0x7f890815de00 2024-03-04T10:09:16.447 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:16.447 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":80,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:15.461533+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:16.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88f406b4d0 msgr2=0x7f88f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88f406b4d0 0x7f88f406d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f88f8036000 tx=0x7f88f803c040 comp rx=0 tx=0).stop 2024-03-04T10:09:16.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 msgr2=0x7f89080aa910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080aa910 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f890000cef0 tx=0x7f890000a710 comp rx=0 tx=0).stop 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 shutdown_connections 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f88f406b4d0 0x7f88f406d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f890815de00 0x7f89080aa910 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89080b9380 0x7f89080aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 --2- 172.21.15.99:0/508383680 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f89080b70d0 0x7f89080abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 >> 172.21.15.99:0/508383680 conn(0x7f89080d0470 msgr2=0x7f89080b8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 shutdown_connections 2024-03-04T10:09:16.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.448+0000 7f890d759640 1 -- 172.21.15.99:0/508383680 wait complete. 2024-03-04T10:09:16.460 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.559+0000 7fadf7137640 1 Processor -- start 2024-03-04T10:09:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.559+0000 7fadf7137640 1 -- start start 2024-03-04T10:09:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf7137640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fadf015dcd0 0x7fadf00ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf7137640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadf01618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf7137640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf00cf200 con 0x7fadf0161de0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf7137640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf00cc0c0 con 0x7fadf015dcd0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf7137640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf0162230 con 0x7fadf00cfeb0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf56d4640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fadf0161de0 0x7fadf00d18c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45924/0 (socket says 172.21.15.99:45924) 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.560+0000 7fadf56d4640 1 -- 172.21.15.99:0/81730090 learned_addr learned my addr 172.21.15.99:0/81730090 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade7fff640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadf01618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/81730090 >> v2:172.21.15.162:3300/0 conn(0x7fadf015dcd0 0x7fadf00ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183707437 0 0) 0x7fadf00cf200 con 0x7fadf0161de0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd8003520 con 0x7fadf0161de0 2024-03-04T10:09:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173166917 0 0) 0x7fadd8003520 con 0x7fadf0161de0 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 msgr2=0x7fadf01618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadf01618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 >> v2:172.21.15.162:3300/0 conn(0x7fadf015dcd0 msgr2=0x7fadf00ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 --2- 172.21.15.99:0/81730090 >> v2:172.21.15.162:3300/0 conn(0x7fadf015dcd0 0x7fadf00ce0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf00ce680 con 0x7fadf0161de0 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fade80019f0 con 0x7fadf0161de0 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 >> v1:172.21.15.99:6789/0 conn(0x7fadf0161de0 legacy=0x7fadf00d18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadd80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd8003f30 0x7fadd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadd8004d40 0x7fadd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf0162230 con 0x7fadd8004d40 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd800b9c0 con 0x7fadd8003f30 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd800bb40 con 0x7fadf00cfeb0 2024-03-04T10:09:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf56d4640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadd8004d40 0x7fadd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadd80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fade7fff640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd8003f30 0x7fadd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf56d4640 1 -- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 msgr2=0x7fadd80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf56d4640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadd80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf56d4640 1 -- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd8003f30 msgr2=0x7fadd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf56d4640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadd8003f30 0x7fadd8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf56d4640 1 -- 172.21.15.99:0/81730090 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd8003520 con 0x7fadd8004d40 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.561+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf00cfeb0 0x7fadd80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.562+0000 7fadf56d4640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadd8004d40 0x7fadd80051c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fade800a870 tx=0x7fadd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78037e447aa150a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.562+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade800a4c0 con 0x7fadd8004d40 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.562+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade8034630 con 0x7fadd8004d40 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.562+0000 7fade77fe640 1 -- 172.21.15.99:0/81730090 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade8034a20 con 0x7fadd8004d40 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf7137640 1 -- 172.21.15.99:0/81730090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadd8004d40 msgr2=0x7fadd80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf7137640 1 --2- 172.21.15.99:0/81730090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadd8004d40 0x7fadd80051c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fade800a870 tx=0x7fadd8005830 comp rx=0 tx=0).stop 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf56d4640 1 -- 172.21.15.99:0/81730090 reap_dead start 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf7137640 1 -- 172.21.15.99:0/81730090 shutdown_connections 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf7137640 1 -- 172.21.15.99:0/81730090 >> 172.21.15.99:0/81730090 conn(0x7fadf00b3480 msgr2=0x7fadf00b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf7137640 1 -- 172.21.15.99:0/81730090 shutdown_connections 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.563+0000 7fadf7137640 1 -- 172.21.15.99:0/81730090 wait complete. 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.564+0000 7fadf7137640 1 Processor -- start 2024-03-04T10:09:16.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.564+0000 7fadf7137640 1 -- start start 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf7137640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf006d5e0 0x7fadf00b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf7137640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 0x7fadf01f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf7137640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf01f7170 0x7fadf00c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf7137640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf00cef20 con 0x7fadf006d9c0 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf7137640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf00cec40 con 0x7fadf006d5e0 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf7137640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf006bff0 con 0x7fadf01f7170 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 0x7fadf01f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 0x7fadf01f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33466/0 (socket says 172.21.15.99:33466) 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 -- 172.21.15.99:0/3152168462 learned_addr learned my addr 172.21.15.99:0/3152168462 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf006d5e0 0x7fadf00b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf56d4640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf01f7170 0x7fadf00c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 -- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf01f7170 msgr2=0x7fadf00c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf01f7170 0x7fadf00c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 -- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf006d5e0 msgr2=0x7fadf00b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf006d5e0 0x7fadf00b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 -- 172.21.15.99:0/3152168462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf01f7810 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf006d5e0 0x7fadf00b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.565+0000 7fade7fff640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 0x7fadf01f6c30 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fade00118e0 tx=0x7fade0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.566+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade000faf0 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.566+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade000fc90 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.566+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf01f7af0 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.566+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade0014440 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.566+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf00c5ae0 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.567+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf0074f60 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.567+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fade0012040 con 0x7fadf006d9c0 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.567+0000 7fade57fa640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faddc06b500 0x7faddc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.568+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fade008aa40 con 0x7fadf006d9c0 2024-03-04T10:09:16.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.568+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faddc06b500 0x7faddc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.568+0000 7fadf4ed3640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faddc06b500 0x7faddc06d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fadf0160ff0 tx=0x7fadec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.572+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fade0019030 con 0x7fadf006d9c0 2024-03-04T10:09:16.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.745+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadf006d090 con 0x7fadf006d9c0 2024-03-04T10:09:16.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.746+0000 7fade57fa640 1 -- 172.21.15.99:0/3152168462 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16922 (secure 0 0 0) 0x7fade008ae20 con 0x7fadf006d9c0 2024-03-04T10:09:16.747 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:16.748 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":80,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:15.461533+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:16.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faddc06b500 msgr2=0x7faddc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faddc06b500 0x7faddc06d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fadf0160ff0 tx=0x7fadec008040 comp rx=0 tx=0).stop 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 msgr2=0x7fadf01f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 0x7fadf01f6c30 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fade00118e0 tx=0x7fade0011da0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 shutdown_connections 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faddc06b500 0x7faddc06d9c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fadf01f7170 0x7fadf00c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fadf006d9c0 0x7fadf01f6c30 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 --2- 172.21.15.99:0/3152168462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fadf006d5e0 0x7fadf00b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 >> 172.21.15.99:0/3152168462 conn(0x7fadf00b3480 msgr2=0x7fadf006ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 shutdown_connections 2024-03-04T10:09:16.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.748+0000 7fadf7137640 1 -- 172.21.15.99:0/3152168462 wait complete. 2024-03-04T10:09:16.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T10:09:16.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 Processor -- start 2024-03-04T10:09:16.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 -- start start 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b5700 0x7f50ec0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f50ec0d34a0 0x7f50ec06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f50ec0b9410 0x7f50ec06c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec0d2400 con 0x7f50ec0b9410 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec0d26e0 con 0x7f50ec0d34a0 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.863+0000 7f50f26cb640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0d29c0 con 0x7f50ec0b5700 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f50ec0b9410 0x7f50ec06c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f50ec0b9410 0x7f50ec06c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33468/0 (socket says 172.21.15.99:33468) 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 -- 172.21.15.99:0/3754739323 learned_addr learned my addr 172.21.15.99:0/3754739323 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50ebfff640 1 --2- 172.21.15.99:0/3754739323 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b5700 0x7f50ec0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50eb7fe640 1 --2- 172.21.15.99:0/3754739323 >> v2:172.21.15.162:3300/0 conn(0x7f50ec0d34a0 0x7f50ec06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 -- 172.21.15.99:0/3754739323 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b5700 msgr2=0x7f50ec0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 --2- 172.21.15.99:0/3754739323 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b5700 0x7f50ec0d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 -- 172.21.15.99:0/3754739323 >> v2:172.21.15.162:3300/0 conn(0x7f50ec0d34a0 msgr2=0x7f50ec06c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 --2- 172.21.15.99:0/3754739323 >> v2:172.21.15.162:3300/0 conn(0x7f50ec0d34a0 0x7f50ec06c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.864+0000 7f50f0c68640 1 -- 172.21.15.99:0/3754739323 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0d2ca0 con 0x7f50ec0b9410 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f0c68640 1 --2- 172.21.15.99:0/3754739323 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b9410 0x7f50ec06c620 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0d4a70 tx=0x7f50e402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27badfa41caf2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50eaffd640 1 -- 172.21.15.99:0/3754739323 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e402c9e0 con 0x7f50ec0b9410 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50eaffd640 1 -- 172.21.15.99:0/3754739323 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e402cb80 con 0x7f50ec0b9410 2024-03-04T10:09:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50eaffd640 1 -- 172.21.15.99:0/3754739323 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e40343f0 con 0x7f50ec0b9410 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 -- 172.21.15.99:0/3754739323 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b9410 msgr2=0x7f50ec06c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 --2- 172.21.15.99:0/3754739323 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b9410 0x7f50ec06c620 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0d4a70 tx=0x7f50e402be40 comp rx=0 tx=0).stop 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 -- 172.21.15.99:0/3754739323 shutdown_connections 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 --2- 172.21.15.99:0/3754739323 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b9410 0x7f50ec06c620 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 --2- 172.21.15.99:0/3754739323 >> v2:172.21.15.162:3300/0 conn(0x7f50ec0d34a0 0x7f50ec06c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 --2- 172.21.15.99:0/3754739323 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b5700 0x7f50ec0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 -- 172.21.15.99:0/3754739323 >> 172.21.15.99:0/3754739323 conn(0x7f50ec0d0490 msgr2=0x7f50ec0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 -- 172.21.15.99:0/3754739323 shutdown_connections 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.865+0000 7f50f26cb640 1 -- 172.21.15.99:0/3754739323 wait complete. 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 Processor -- start 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 -- start start 2024-03-04T10:09:16.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 0x7f50ec14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b9410 0x7f50ec0c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50ec0d34a0 0x7f50ec0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0b5c60 con 0x7f50ec0b5700 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0b5ae0 con 0x7f50ec0d34a0 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.867+0000 7f50f26cb640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0b5de0 con 0x7f50ec0b9410 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 0x7f50ec14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 0x7f50ec14be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33476/0 (socket says 172.21.15.99:33476) 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 -- 172.21.15.99:0/1341186603 learned_addr learned my addr 172.21.15.99:0/1341186603 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50f0c68640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50ec0d34a0 0x7f50ec0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50eb7fe640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b9410 0x7f50ec0c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 -- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b9410 msgr2=0x7f50ec0c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b9410 0x7f50ec0c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 -- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50ec0d34a0 msgr2=0x7f50ec0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50ec0d34a0 0x7f50ec0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 -- 172.21.15.99:0/1341186603 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0cb8b0 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50ebfff640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 0x7f50ec14be30 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00c5a0 tx=0x7f50dc00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc015070 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc004220 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec0cbb90 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc003610 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.868+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec0cc150 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.869+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ec1586d0 con 0x7f50ec0b5700 2024-03-04T10:09:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.870+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f50dc0037b0 con 0x7f50ec0b5700 2024-03-04T10:09:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.870+0000 7f50e8ff9640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d806b4d0 0x7f50d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.870+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50dc08a0e0 con 0x7f50ec0b5700 2024-03-04T10:09:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.870+0000 7f50eb7fe640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d806b4d0 0x7f50d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.871+0000 7f50eb7fe640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d806b4d0 0x7f50d806d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0cd080 tx=0x7f50e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:16.875+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f50dc058e60 con 0x7f50ec0b5700 2024-03-04T10:09:17.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.083+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f50ec074f60 con 0x7f50ec0b5700 2024-03-04T10:09:17.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.135+0000 7f50e8ff9640 1 -- 172.21.15.99:0/1341186603 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) v1 ==== 118+0+0 (secure 0 0 0) 0x7f50dc058760 con 0x7f50ec0b5700 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d806b4d0 msgr2=0x7f50d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d806b4d0 0x7f50d806d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0cd080 tx=0x7f50e0008040 comp rx=0 tx=0).stop 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 msgr2=0x7f50ec14be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 0x7f50ec14be30 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00c5a0 tx=0x7f50dc00ca60 comp rx=0 tx=0).stop 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 shutdown_connections 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d806b4d0 0x7f50d806d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50ec0d34a0 0x7f50ec0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50ec0b9410 0x7f50ec0c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 --2- 172.21.15.99:0/1341186603 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50ec0b5700 0x7f50ec14be30 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 >> 172.21.15.99:0/1341186603 conn(0x7f50ec0d0490 msgr2=0x7f50ec0b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 shutdown_connections 2024-03-04T10:09:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.137+0000 7f50f26cb640 1 -- 172.21.15.99:0/1341186603 wait complete. 2024-03-04T10:09:17.151 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T10:09:17.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T10:09:17.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.252+0000 7f2cf07a9640 1 Processor -- start 2024-03-04T10:09:17.254 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.252+0000 7f2cf07a9640 1 -- start start 2024-03-04T10:09:17.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.253+0000 7f2cf07a9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2ce81622f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.253+0000 7f2cf07a9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80d2400 con 0x7f2ce80d34a0 2024-03-04T10:09:17.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.253+0000 7f2cf07a9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80d26e0 con 0x7f2ce80b5700 2024-03-04T10:09:17.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.253+0000 7f2cf07a9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80d29c0 con 0x7f2ce80b9410 2024-03-04T10:09:17.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2cedd44640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2ce80d34a0 0x7f2ce8161be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45938/0 (socket says 172.21.15.99:45938) 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2cedd44640 1 -- 172.21.15.99:0/2027289310 learned_addr learned my addr 172.21.15.99:0/2027289310 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ceed46640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2ce81622f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453979913 0 0) 0x7f2ce80d2400 con 0x7f2ce80d34a0 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4003520 con 0x7f2ce80d34a0 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589529700 0 0) 0x7f2cd4003520 con 0x7f2ce80d34a0 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 msgr2=0x7f2ce81622f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2ce81622f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 >> v1:172.21.15.162:6789/0 conn(0x7f2ce80b5700 legacy=0x7f2ce80d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80d2ca0 con 0x7f2ce80d34a0 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f2cd8001670 con 0x7f2ce80d34a0 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 >> v1:172.21.15.99:6789/0 conn(0x7f2ce80d34a0 legacy=0x7f2ce8161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd4004390 0x7f2cd4004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2cd4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80d29c0 con 0x7f2cd4004390 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd400ba70 con 0x7f2cd40039e0 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd400bc70 con 0x7f2ce80b9410 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2cedd44640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd4004390 0x7f2cd4004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2cee545640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2ceed46640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2cd4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2cedd44640 1 -- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 msgr2=0x7f2cd4005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.254+0000 7f2cedd44640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2cd4005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2cedd44640 1 -- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd40039e0 msgr2=0x7f2cd4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2cedd44640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2cedd44640 1 -- 172.21.15.99:0/2027289310 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd4003520 con 0x7f2cd4004390 2024-03-04T10:09:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2ceed46640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce80b9410 0x7f2cd4005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2cee545640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2cedd44640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd4004390 0x7f2cd4004860 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80d18c0 tx=0x7f2cd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f9806fdcdbebbc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8007ca0 con 0x7f2cd4004390 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd8030070 con 0x7f2cd4004390 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.255+0000 7f2ced543640 1 -- 172.21.15.99:0/2027289310 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8005d30 con 0x7f2cd4004390 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2cf07a9640 1 -- 172.21.15.99:0/2027289310 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd4004390 msgr2=0x7f2cd4004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/2027289310 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2cd4004390 0x7f2cd4004860 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80d18c0 tx=0x7f2cd4005830 comp rx=0 tx=0).stop 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2ceed46640 1 -- 172.21.15.99:0/2027289310 reap_dead start 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2cf07a9640 1 -- 172.21.15.99:0/2027289310 shutdown_connections 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2cf07a9640 1 -- 172.21.15.99:0/2027289310 >> 172.21.15.99:0/2027289310 conn(0x7f2ce80d0490 msgr2=0x7f2ce80d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2cf07a9640 1 -- 172.21.15.99:0/2027289310 shutdown_connections 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.256+0000 7f2cf07a9640 1 -- 172.21.15.99:0/2027289310 wait complete. 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.257+0000 7f2cf07a9640 1 Processor -- start 2024-03-04T10:09:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.257+0000 7f2cf07a9640 1 -- start start 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd4004390 0x7f2ce8157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce81526f0 0x7f2ce80cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 0x7f2ce81f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce81622f0 con 0x7f2ce80cd360 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80b5cf0 con 0x7f2cd4004390 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce8162470 con 0x7f2ce81526f0 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 0x7f2ce81f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 0x7f2ce81f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33492/0 (socket says 172.21.15.99:33492) 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 -- 172.21.15.99:0/1474509703 learned_addr learned my addr 172.21.15.99:0/1474509703 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cedd44640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce81526f0 0x7f2ce80cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cee545640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd4004390 0x7f2ce8157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 -- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce81526f0 msgr2=0x7f2ce80cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce81526f0 0x7f2ce80cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 -- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd4004390 msgr2=0x7f2ce8157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd4004390 0x7f2ce8157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 -- 172.21.15.99:0/1474509703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8157840 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2ceed46640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 0x7f2ce81f6fe0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4011490 tx=0x7f2ce4011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4003610 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce401b070 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce81f7da0 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.258+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4003a10 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.259+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce80cae60 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.259+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce8074f60 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.260+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2ce4003bb0 con 0x7f2ce80cd360 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.260+0000 7f2cdeffd640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.260+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2ce408a0b0 con 0x7f2ce80cd360 2024-03-04T10:09:17.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.260+0000 7f2cee545640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:17.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.261+0000 7f2cee545640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0001eb0 tx=0x7f2ce0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:17.266 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.265+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2ce4024020 con 0x7f2ce80cd360 2024-03-04T10:09:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:17.471+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2ce80b60b0 con 0x7f2ce80cd360 2024-03-04T10:09:18.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.149+0000 7f2cdeffd640 1 -- 172.21.15.99:0/1474509703 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ce4058da0 con 0x7f2ce80cd360 2024-03-04T10:09:18.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ccc06b4d0 msgr2=0x7f2ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0001eb0 tx=0x7f2ce0009040 comp rx=0 tx=0).stop 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 msgr2=0x7f2ce81f6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 0x7f2ce81f6fe0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4011490 tx=0x7f2ce4011950 comp rx=0 tx=0).stop 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 shutdown_connections 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ce80cd360 0x7f2ce81f6fe0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2ce81526f0 0x7f2ce80cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 --2- 172.21.15.99:0/1474509703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2cd4004390 0x7f2ce8157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 >> 172.21.15.99:0/1474509703 conn(0x7f2ce80d0490 msgr2=0x7f2ce80b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 shutdown_connections 2024-03-04T10:09:18.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.152+0000 7f2cf07a9640 1 -- 172.21.15.99:0/1474509703 wait complete. 2024-03-04T10:09:18.165 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T10:09:18.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:18.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.267+0000 7fd50ff3b640 1 Processor -- start 2024-03-04T10:09:18.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.267+0000 7fd50ff3b640 1 -- start start 2024-03-04T10:09:18.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.267+0000 7fd50ff3b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd5080cfd60 0x7fd5080cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.267+0000 7fd50ff3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd50806d3b0 0x7fd5080cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50ff3b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd5080ce9c0 0x7fd5080cee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.269 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50ff3b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50815cbb0 con 0x7fd5080cfd60 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50ff3b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080d0140 con 0x7fd5080ce9c0 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50ff3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50815c280 con 0x7fd50806d3b0 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd5080cfd60 0x7fd5080cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd5080cfd60 0x7fd5080cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33504/0 (socket says 172.21.15.99:33504) 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 -- 172.21.15.99:0/1352605873 learned_addr learned my addr 172.21.15.99:0/1352605873 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 -- 172.21.15.99:0/1352605873 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd50806d3b0 msgr2=0x7fd5080cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 --2- 172.21.15.99:0/1352605873 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd50806d3b0 0x7fd5080cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 -- 172.21.15.99:0/1352605873 >> v2:172.21.15.162:3300/0 conn(0x7fd5080ce9c0 msgr2=0x7fd5080cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 --2- 172.21.15.99:0/1352605873 >> v2:172.21.15.162:3300/0 conn(0x7fd5080ce9c0 0x7fd5080cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.268+0000 7fd50dcd7640 1 -- 172.21.15.99:0/1352605873 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50815c560 con 0x7fd5080cfd60 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50dcd7640 1 --2- 172.21.15.99:0/1352605873 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5080cfd60 0x7fd5080cbdf0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fd5080d04c0 tx=0x7fd4f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398f80e71272361d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ccd5640 1 -- 172.21.15.99:0/1352605873 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8035020 con 0x7fd5080cfd60 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ccd5640 1 -- 172.21.15.99:0/1352605873 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8031870 con 0x7fd5080cfd60 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ccd5640 1 -- 172.21.15.99:0/1352605873 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8031c40 con 0x7fd5080cfd60 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 -- 172.21.15.99:0/1352605873 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5080cfd60 msgr2=0x7fd5080cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/1352605873 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5080cfd60 0x7fd5080cbdf0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fd5080d04c0 tx=0x7fd4f802b910 comp rx=0 tx=0).stop 2024-03-04T10:09:18.272 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 -- 172.21.15.99:0/1352605873 shutdown_connections 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/1352605873 >> v2:172.21.15.162:3300/0 conn(0x7fd5080ce9c0 0x7fd5080cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/1352605873 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd50806d3b0 0x7fd5080cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/1352605873 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd5080cfd60 0x7fd5080cbdf0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 -- 172.21.15.99:0/1352605873 >> 172.21.15.99:0/1352605873 conn(0x7fd50806acd0 msgr2=0x7fd50806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 -- 172.21.15.99:0/1352605873 shutdown_connections 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.269+0000 7fd50ff3b640 1 -- 172.21.15.99:0/1352605873 wait complete. 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 Processor -- start 2024-03-04T10:09:18.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 -- start start 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 0x7fd508150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5080ce9c0 0x7fd5081f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5080cfd60 0x7fd5081514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50815bf30 con 0x7fd50806d3b0 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080cf390 con 0x7fd5080ce9c0 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50ff3b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd508071350 con 0x7fd5080cfd60 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50dcd7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 0x7fd508150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50dcd7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 0x7fd508150320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33508/0 (socket says 172.21.15.99:33508) 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.271+0000 7fd50dcd7640 1 -- 172.21.15.99:0/2061259068 learned_addr learned my addr 172.21.15.99:0/2061259068 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50dcd7640 1 -- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5080cfd60 msgr2=0x7fd5081514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50dcd7640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5080cfd60 0x7fd5081514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50dcd7640 1 -- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5080ce9c0 msgr2=0x7fd5081f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50dcd7640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5080ce9c0 0x7fd5081f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50dcd7640 1 -- 172.21.15.99:0/2061259068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd508151de0 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50dcd7640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 0x7fd508150320 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8000c00 tx=0x7fd4f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f803ad60 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f80443f0 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5080af200 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80447e0 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.272+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080af7c0 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.273+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd508074f60 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.273+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd4f804b060 con 0x7fd50806d3b0 2024-03-04T10:09:18.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.274+0000 7fd4fe7fc640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd4ec06b4d0 0x7fd4ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.274+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4f80ac500 con 0x7fd50806d3b0 2024-03-04T10:09:18.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.274+0000 7fd50d4d6640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd4ec06b4d0 0x7fd4ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.275+0000 7fd50d4d6640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd4ec06b4d0 0x7fd4ec06d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd5080b0f60 tx=0x7fd504006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.278+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd4f803f070 con 0x7fd50806d3b0 2024-03-04T10:09:18.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.484+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5080ccf90 con 0x7fd50806d3b0 2024-03-04T10:09:18.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.484+0000 7fd4fe7fc640 1 -- 172.21.15.99:0/2061259068 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fd4f807b1f0 con 0x7fd50806d3b0 2024-03-04T10:09:18.486 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:18.486 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:17.472137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:18.486 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 64 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd4ec06b4d0 msgr2=0x7fd4ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd4ec06b4d0 0x7fd4ec06d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd5080b0f60 tx=0x7fd504006c90 comp rx=0 tx=0).stop 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 msgr2=0x7fd508150320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 0x7fd508150320 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8000c00 tx=0x7fd4f80046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 shutdown_connections 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd4ec06b4d0 0x7fd4ec06d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd5080cfd60 0x7fd5081514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5080ce9c0 0x7fd5081f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 --2- 172.21.15.99:0/2061259068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd50806d3b0 0x7fd508150320 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 >> 172.21.15.99:0/2061259068 conn(0x7fd50806acd0 msgr2=0x7fd50815a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:18.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 shutdown_connections 2024-03-04T10:09:18.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.486+0000 7fd50ff3b640 1 -- 172.21.15.99:0/2061259068 wait complete. 2024-03-04T10:09:18.499 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:18.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.597+0000 7fe7b1145640 1 Processor -- start 2024-03-04T10:09:18.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.597+0000 7fe7b1145640 1 -- start start 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7b1145640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe7ac15de00 0x7fe7ac0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7b1145640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe7ac0b5470 0x7fe7ac0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7b1145640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac15e1e0 con 0x7fe7ac0b9380 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7b1145640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ac0d2820 con 0x7fe7ac0b5470 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7b1145640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ac0d1c10 con 0x7fe7ac15de00 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7ab59e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe7ac0b5470 0x7fe7ac0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7aa59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe7ac0b9380 0x7fe7ac0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45942/0 (socket says 172.21.15.99:45942) 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.598+0000 7fe7aa59c640 1 -- 172.21.15.99:0/1306270295 learned_addr learned my addr 172.21.15.99:0/1306270295 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7ab59e640 1 -- 172.21.15.99:0/1306270295 >> v2:172.21.15.162:3300/0 conn(0x7fe7ac15de00 msgr2=0x7fe7ac0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7aad9d640 1 --2- 172.21.15.99:0/1306270295 >> v2:172.21.15.162:3300/0 conn(0x7fe7ac15de00 0x7fe7ac0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7ab59e640 1 --2- 172.21.15.99:0/1306270295 >> v2:172.21.15.162:3300/0 conn(0x7fe7ac15de00 0x7fe7ac0b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7ab59e640 1 -- 172.21.15.99:0/1306270295 >> v1:172.21.15.99:6789/0 conn(0x7fe7ac0b9380 legacy=0x7fe7ac0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7a9d9b640 1 -- 172.21.15.99:0/1306270295 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148518247 0 0) 0x7fe7ac15e1e0 con 0x7fe7ac0b9380 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7ab59e640 1 -- 172.21.15.99:0/1306270295 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac0d1ef0 con 0x7fe7ac0b5470 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7ab59e640 1 --2- 172.21.15.99:0/1306270295 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b5900 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0ce240 tx=0x7fe7a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2194db10e124bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7a9d9b640 1 -- 172.21.15.99:0/1306270295 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4035020 con 0x7fe7ac0b5470 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7a9d9b640 1 -- 172.21.15.99:0/1306270295 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a4031830 con 0x7fe7ac0b5470 2024-03-04T10:09:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.599+0000 7fe7a9d9b640 1 -- 172.21.15.99:0/1306270295 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4031c00 con 0x7fe7ac0b5470 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 -- 172.21.15.99:0/1306270295 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 msgr2=0x7fe7ac0b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1306270295 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b5900 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0ce240 tx=0x7fe7a402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 -- 172.21.15.99:0/1306270295 shutdown_connections 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1306270295 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b5900 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1306270295 >> v2:172.21.15.162:3300/0 conn(0x7fe7ac15de00 0x7fe7ac0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 -- 172.21.15.99:0/1306270295 >> 172.21.15.99:0/1306270295 conn(0x7fe7ac0d0470 msgr2=0x7fe7ac0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 -- 172.21.15.99:0/1306270295 shutdown_connections 2024-03-04T10:09:18.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.600+0000 7fe7b1145640 1 -- 172.21.15.99:0/1306270295 wait complete. 2024-03-04T10:09:18.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.601+0000 7fe7b1145640 1 Processor -- start 2024-03-04T10:09:18.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.601+0000 7fe7b1145640 1 -- start start 2024-03-04T10:09:18.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7b1145640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7b1145640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7ac0b9380 0x7fe7ac0aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7b1145640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe7ac15de00 0x7fe7ac15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7b1145640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b97e0 con 0x7fe7ac0b5470 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7b1145640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0d24b0 con 0x7fe7ac0b9380 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7b1145640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0b72a0 con 0x7fe7ac15de00 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33526/0 (socket says 172.21.15.99:33526) 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 -- 172.21.15.99:0/1979954165 learned_addr learned my addr 172.21.15.99:0/1979954165 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aa59c640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7ac0b9380 0x7fe7ac0aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7ab59e640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe7ac15de00 0x7fe7ac15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 -- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe7ac15de00 msgr2=0x7fe7ac15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe7ac15de00 0x7fe7ac15ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 -- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7ac0b9380 msgr2=0x7fe7ac0aa3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.602+0000 7fe7aad9d640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7ac0b9380 0x7fe7ac0aa3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7aad9d640 1 -- 172.21.15.99:0/1979954165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac0aa910 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7aa59c640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7ac0b9380 0x7fe7ac0aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7aad9d640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b1ec0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00ce10 tx=0x7fe79c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c003b90 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79c003d30 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79c004240 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac0cabf0 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.603+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac0cb1b0 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.604+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac074f60 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.604+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe79c0043e0 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.605+0000 7fe7937fe640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe79806b4d0 0x7fe79806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.605+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe79c08ae20 con 0x7fe7ac0b5470 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.605+0000 7fe7aa59c640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe79806b4d0 0x7fe79806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.606+0000 7fe7aa59c640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe79806b4d0 0x7fe79806d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0aad60 tx=0x7fe7a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.609+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe7ac0aa910 con 0x7fe7ac0b5470 2024-03-04T10:09:18.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.780+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7ac0cb9b0 con 0x7fe7ac0b5470 2024-03-04T10:09:18.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.781+0000 7fe7937fe640 1 -- 172.21.15.99:0/1979954165 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16909 (secure 0 0 0) 0x7fe79c01d340 con 0x7fe7ac0b5470 2024-03-04T10:09:18.783 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:18.783 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":81,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:17.120831+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe79806b4d0 msgr2=0x7fe79806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe79806b4d0 0x7fe79806d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac0aad60 tx=0x7fe7a0009040 comp rx=0 tx=0).stop 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 msgr2=0x7fe7ac0b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b1ec0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe79c00ce10 tx=0x7fe79c00a660 comp rx=0 tx=0).stop 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 shutdown_connections 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe79806b4d0 0x7fe79806d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe7ac15de00 0x7fe7ac15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe7ac0b9380 0x7fe7ac0aa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 --2- 172.21.15.99:0/1979954165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe7ac0b5470 0x7fe7ac0b1ec0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 >> 172.21.15.99:0/1979954165 conn(0x7fe7ac0d0470 msgr2=0x7fe7ac15cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:18.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 shutdown_connections 2024-03-04T10:09:18.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.783+0000 7fe7b1145640 1 -- 172.21.15.99:0/1979954165 wait complete. 2024-03-04T10:09:18.796 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T10:09:18.797 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:09:18.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.896+0000 7f164ab16640 1 Processor -- start 2024-03-04T10:09:18.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.896+0000 7f164ab16640 1 -- start start 2024-03-04T10:09:18.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f164ab16640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f164415ac40 0x7f16440d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f164ab16640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16440cc760 con 0x7f16440af7b0 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f164ab16640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16440ce720 con 0x7f16440cf080 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f164ab16640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f164415b020 con 0x7f164415ac40 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f16488b2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f164415ac40 0x7f16440d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f1643fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f16440af7b0 0x7f16440ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45948/0 (socket says 172.21.15.99:45948) 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f1643fff640 1 -- 172.21.15.99:0/3823008268 learned_addr learned my addr 172.21.15.99:0/3823008268 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f16488b2640 1 -- 172.21.15.99:0/3823008268 >> v1:172.21.15.162:6789/0 conn(0x7f16440cf080 legacy=0x7f16440ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f16488b2640 1 -- 172.21.15.99:0/3823008268 >> v1:172.21.15.99:6789/0 conn(0x7f16440af7b0 legacy=0x7f16440ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.897+0000 7f16488b2640 1 -- 172.21.15.99:0/3823008268 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16440cbe30 con 0x7f164415ac40 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.898+0000 7f16488b2640 1 --2- 172.21.15.99:0/3823008268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f16440d1290 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f16340097a0 tx=0x7f163402db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0e9a3529235b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.898+0000 7f16437fe640 1 -- 172.21.15.99:0/3823008268 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163403b070 con 0x7f164415ac40 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.898+0000 7f16437fe640 1 -- 172.21.15.99:0/3823008268 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163402e920 con 0x7f164415ac40 2024-03-04T10:09:18.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.898+0000 7f16437fe640 1 -- 172.21.15.99:0/3823008268 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163402ecf0 con 0x7f164415ac40 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 -- 172.21.15.99:0/3823008268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 msgr2=0x7f16440d1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 --2- 172.21.15.99:0/3823008268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f16440d1290 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f16340097a0 tx=0x7f163402db70 comp rx=0 tx=0).stop 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 -- 172.21.15.99:0/3823008268 shutdown_connections 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 --2- 172.21.15.99:0/3823008268 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f16440d1290 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 -- 172.21.15.99:0/3823008268 >> 172.21.15.99:0/3823008268 conn(0x7f16440d0770 msgr2=0x7f16440d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 -- 172.21.15.99:0/3823008268 shutdown_connections 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.899+0000 7f164ab16640 1 -- 172.21.15.99:0/3823008268 wait complete. 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.900+0000 7f164ab16640 1 Processor -- start 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.900+0000 7f164ab16640 1 -- start start 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f164ab16640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16440cf080 0x7f16440c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f164ab16640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f1644153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f164ab16640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16440c85a0 0x7f1644154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f164ab16640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16440ce340 con 0x7f164415ac40 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f164ab16640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16440cc3f0 con 0x7f16440c85a0 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f164ab16640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16440ce4c0 con 0x7f16440cf080 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f1644153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f1644153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33538/0 (socket says 172.21.15.99:33538) 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 -- 172.21.15.99:0/3832737316 learned_addr learned my addr 172.21.15.99:0/3832737316 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f16488b2640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16440cf080 0x7f16440c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f16490b3640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16440c85a0 0x7f1644154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 -- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16440cf080 msgr2=0x7f16440c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16440cf080 0x7f16440c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 -- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16440c85a0 msgr2=0x7f1644154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16440c85a0 0x7f1644154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 -- 172.21.15.99:0/3832737316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16440c42a0 con 0x7f164415ac40 2024-03-04T10:09:18.905 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f16490b3640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16440c85a0 0x7f1644154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f1643fff640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f1644153af0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f163800c400 tx=0x7f163800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.901+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1638017070 con 0x7f164415ac40 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.902+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1638005ce0 con 0x7f164415ac40 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.902+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16380072f0 con 0x7f164415ac40 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.902+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16440c4580 con 0x7f164415ac40 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.902+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16440c4b40 con 0x7f164415ac40 2024-03-04T10:09:18.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.903+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164406cd70 con 0x7f164415ac40 2024-03-04T10:09:18.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.903+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f163801a070 con 0x7f164415ac40 2024-03-04T10:09:18.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.903+0000 7f16417fa640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f163006b4d0 0x7f163006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:18.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.904+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16380897a0 con 0x7f164415ac40 2024-03-04T10:09:18.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.904+0000 7f16488b2640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f163006b4d0 0x7f163006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:18.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.904+0000 7f16488b2640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f163006b4d0 0x7f163006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1634038000 tx=0x7f1634032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:18.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:18.908+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1638058490 con 0x7f164415ac40 2024-03-04T10:09:19.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.074+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16440af450 con 0x7f163006b4d0 2024-03-04T10:09:19.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.081+0000 7f16417fa640 1 -- 172.21.15.99:0/3832737316 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252050 (secure 0 0 0) 0x7f16440af450 con 0x7f163006b4d0 2024-03-04T10:09:19.085 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:19.087 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:09:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.083+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f163006b4d0 msgr2=0x7f163006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.083+0000 7f164ab16640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f163006b4d0 0x7f163006d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1634038000 tx=0x7f1634032000 comp rx=0 tx=0).stop 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.083+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 msgr2=0x7f1644153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.083+0000 7f164ab16640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f1644153af0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f163800c400 tx=0x7f163800c8c0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 shutdown_connections 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f163006b4d0 0x7f163006d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16440c85a0 0x7f1644154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f164415ac40 0x7f1644153af0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 --2- 172.21.15.99:0/3832737316 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f16440cf080 0x7f16440c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 >> 172.21.15.99:0/3832737316 conn(0x7f16440d0770 msgr2=0x7f164406ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 shutdown_connections 2024-03-04T10:09:19.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.084+0000 7f164ab16640 1 -- 172.21.15.99:0/3832737316 wait complete. 2024-03-04T10:09:19.096 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-04T10:09:17.519519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229264,"kb_used_data":8884,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754745456,"statfs":{"total":773094113280,"available":772859346944,"internally_reserved":0,"allocated":9097216,"data_stored":4821296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000763"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126127+0000","last_change":"2024-03-04T10:09:16.135890+0000","last_active":"2024-03-04T10:09:17.126127+0000","last_peered":"2024-03-04T10:09:17.126127+0000","last_clean":"2024-03-04T10:09:17.126127+0000","last_became_active":"2024-03-04T10:09:16.135609+0000","last_became_peered":"2024-03-04T10:09:16.135609+0000","last_unstale":"2024-03-04T10:09:17.126127+0000","last_undegraded":"2024-03-04T10:09:17.126127+0000","last_fullsized":"2024-03-04T10:09:17.126127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:09:34.119832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142270+0000","last_change":"2024-03-04T10:09:16.137720+0000","last_active":"2024-03-04T10:09:17.142270+0000","last_peered":"2024-03-04T10:09:17.142270+0000","last_clean":"2024-03-04T10:09:17.142270+0000","last_became_active":"2024-03-04T10:09:16.137400+0000","last_became_peered":"2024-03-04T10:09:16.137400+0000","last_unstale":"2024-03-04T10:09:17.142270+0000","last_undegraded":"2024-03-04T10:09:17.142270+0000","last_fullsized":"2024-03-04T10:09:17.142270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:26.234049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126679+0000","last_change":"2024-03-04T10:09:16.138687+0000","last_active":"2024-03-04T10:09:17.126679+0000","last_peered":"2024-03-04T10:09:17.126679+0000","last_clean":"2024-03-04T10:09:17.126679+0000","last_became_active":"2024-03-04T10:09:16.138426+0000","last_became_peered":"2024-03-04T10:09:16.138426+0000","last_unstale":"2024-03-04T10:09:17.126679+0000","last_undegraded":"2024-03-04T10:09:17.126679+0000","last_fullsized":"2024-03-04T10:09:17.126679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:42:11.162194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141370+0000","last_change":"2024-03-04T10:09:16.138148+0000","last_active":"2024-03-04T10:09:16.141370+0000","last_peered":"2024-03-04T10:09:16.141370+0000","last_clean":"2024-03-04T10:09:16.141370+0000","last_became_active":"2024-03-04T10:09:16.137609+0000","last_became_peered":"2024-03-04T10:09:16.137609+0000","last_unstale":"2024-03-04T10:09:16.141370+0000","last_undegraded":"2024-03-04T10:09:16.141370+0000","last_fullsized":"2024-03-04T10:09:16.141370+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:14:48.132543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144716+0000","last_change":"2024-03-04T10:09:16.144013+0000","last_active":"2024-03-04T10:09:17.144716+0000","last_peered":"2024-03-04T10:09:17.144716+0000","last_clean":"2024-03-04T10:09:17.144716+0000","last_became_active":"2024-03-04T10:09:16.143691+0000","last_became_peered":"2024-03-04T10:09:16.143691+0000","last_unstale":"2024-03-04T10:09:17.144716+0000","last_undegraded":"2024-03-04T10:09:17.144716+0000","last_fullsized":"2024-03-04T10:09:17.144716+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:51.311633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.145036+0000","last_change":"2024-03-04T10:09:16.134941+0000","last_active":"2024-03-04T10:09:16.145036+0000","last_peered":"2024-03-04T10:09:16.145036+0000","last_clean":"2024-03-04T10:09:16.145036+0000","last_became_active":"2024-03-04T10:09:16.134703+0000","last_became_peered":"2024-03-04T10:09:16.134703+0000","last_unstale":"2024-03-04T10:09:16.145036+0000","last_undegraded":"2024-03-04T10:09:16.145036+0000","last_fullsized":"2024-03-04T10:09:16.145036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:50:29.042535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143667+0000","last_change":"2024-03-04T10:09:16.141909+0000","last_active":"2024-03-04T10:09:16.143667+0000","last_peered":"2024-03-04T10:09:16.143667+0000","last_clean":"2024-03-04T10:09:16.143667+0000","last_became_active":"2024-03-04T10:09:16.141664+0000","last_became_peered":"2024-03-04T10:09:16.141664+0000","last_unstale":"2024-03-04T10:09:16.143667+0000","last_undegraded":"2024-03-04T10:09:16.143667+0000","last_fullsized":"2024-03-04T10:09:16.143667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:52.847892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144492+0000","last_change":"2024-03-04T10:09:16.145950+0000","last_active":"2024-03-04T10:09:17.144492+0000","last_peered":"2024-03-04T10:09:17.144492+0000","last_clean":"2024-03-04T10:09:17.144492+0000","last_became_active":"2024-03-04T10:09:16.145700+0000","last_became_peered":"2024-03-04T10:09:16.145700+0000","last_unstale":"2024-03-04T10:09:17.144492+0000","last_undegraded":"2024-03-04T10:09:17.144492+0000","last_fullsized":"2024-03-04T10:09:17.144492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:12:56.186318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125662+0000","last_change":"2024-03-04T10:09:16.137250+0000","last_active":"2024-03-04T10:09:17.125662+0000","last_peered":"2024-03-04T10:09:17.125662+0000","last_clean":"2024-03-04T10:09:17.125662+0000","last_became_active":"2024-03-04T10:09:16.136946+0000","last_became_peered":"2024-03-04T10:09:16.136946+0000","last_unstale":"2024-03-04T10:09:17.125662+0000","last_undegraded":"2024-03-04T10:09:17.125662+0000","last_fullsized":"2024-03-04T10:09:17.125662+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:48.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141320+0000","last_change":"2024-03-04T10:09:16.138494+0000","last_active":"2024-03-04T10:09:16.141320+0000","last_peered":"2024-03-04T10:09:16.141320+0000","last_clean":"2024-03-04T10:09:16.141320+0000","last_became_active":"2024-03-04T10:09:16.138249+0000","last_became_peered":"2024-03-04T10:09:16.138249+0000","last_unstale":"2024-03-04T10:09:16.141320+0000","last_undegraded":"2024-03-04T10:09:16.141320+0000","last_fullsized":"2024-03-04T10:09:16.141320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:24.060713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142171+0000","last_change":"2024-03-04T10:09:16.137262+0000","last_active":"2024-03-04T10:09:17.142171+0000","last_peered":"2024-03-04T10:09:17.142171+0000","last_clean":"2024-03-04T10:09:17.142171+0000","last_became_active":"2024-03-04T10:09:16.136829+0000","last_became_peered":"2024-03-04T10:09:16.136829+0000","last_unstale":"2024-03-04T10:09:17.142171+0000","last_undegraded":"2024-03-04T10:09:17.142171+0000","last_fullsized":"2024-03-04T10:09:17.142171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:33:42.182525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.138898+0000","last_change":"2024-03-04T10:09:16.134474+0000","last_active":"2024-03-04T10:09:16.138898+0000","last_peered":"2024-03-04T10:09:16.138898+0000","last_clean":"2024-03-04T10:09:16.138898+0000","last_became_active":"2024-03-04T10:09:16.134188+0000","last_became_peered":"2024-03-04T10:09:16.134188+0000","last_unstale":"2024-03-04T10:09:16.138898+0000","last_undegraded":"2024-03-04T10:09:16.138898+0000","last_fullsized":"2024-03-04T10:09:16.138898+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:04.862839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144708+0000","last_change":"2024-03-04T10:09:16.137301+0000","last_active":"2024-03-04T10:09:17.144708+0000","last_peered":"2024-03-04T10:09:17.144708+0000","last_clean":"2024-03-04T10:09:17.144708+0000","last_became_active":"2024-03-04T10:09:16.137041+0000","last_became_peered":"2024-03-04T10:09:16.137041+0000","last_unstale":"2024-03-04T10:09:17.144708+0000","last_undegraded":"2024-03-04T10:09:17.144708+0000","last_fullsized":"2024-03-04T10:09:17.144708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:05.985342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.144964+0000","last_change":"2024-03-04T10:09:16.134878+0000","last_active":"2024-03-04T10:09:16.144964+0000","last_peered":"2024-03-04T10:09:16.144964+0000","last_clean":"2024-03-04T10:09:16.144964+0000","last_became_active":"2024-03-04T10:09:16.134612+0000","last_became_peered":"2024-03-04T10:09:16.134612+0000","last_unstale":"2024-03-04T10:09:16.144964+0000","last_undegraded":"2024-03-04T10:09:16.144964+0000","last_fullsized":"2024-03-04T10:09:16.144964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:51.306514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141890+0000","last_change":"2024-03-04T10:09:16.132703+0000","last_active":"2024-03-04T10:09:17.141890+0000","last_peered":"2024-03-04T10:09:17.141890+0000","last_clean":"2024-03-04T10:09:17.141890+0000","last_became_active":"2024-03-04T10:09:16.132424+0000","last_became_peered":"2024-03-04T10:09:16.132424+0000","last_unstale":"2024-03-04T10:09:17.141890+0000","last_undegraded":"2024-03-04T10:09:17.141890+0000","last_fullsized":"2024-03-04T10:09:17.141890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:59.089491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125404+0000","last_change":"2024-03-04T10:09:16.135262+0000","last_active":"2024-03-04T10:09:17.125404+0000","last_peered":"2024-03-04T10:09:17.125404+0000","last_clean":"2024-03-04T10:09:17.125404+0000","last_became_active":"2024-03-04T10:09:16.134221+0000","last_became_peered":"2024-03-04T10:09:16.134221+0000","last_unstale":"2024-03-04T10:09:17.125404+0000","last_undegraded":"2024-03-04T10:09:17.125404+0000","last_fullsized":"2024-03-04T10:09:17.125404+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:28.222882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145240+0000","last_change":"2024-03-04T10:09:16.140304+0000","last_active":"2024-03-04T10:09:17.145240+0000","last_peered":"2024-03-04T10:09:17.145240+0000","last_clean":"2024-03-04T10:09:17.145240+0000","last_became_active":"2024-03-04T10:09:16.140055+0000","last_became_peered":"2024-03-04T10:09:16.140055+0000","last_unstale":"2024-03-04T10:09:17.145240+0000","last_undegraded":"2024-03-04T10:09:17.145240+0000","last_fullsized":"2024-03-04T10:09:17.145240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:11.277585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126751+0000","last_change":"2024-03-04T10:09:16.128633+0000","last_active":"2024-03-04T10:09:17.126751+0000","last_peered":"2024-03-04T10:09:17.126751+0000","last_clean":"2024-03-04T10:09:17.126751+0000","last_became_active":"2024-03-04T10:09:16.128361+0000","last_became_peered":"2024-03-04T10:09:16.128361+0000","last_unstale":"2024-03-04T10:09:17.126751+0000","last_undegraded":"2024-03-04T10:09:17.126751+0000","last_fullsized":"2024-03-04T10:09:17.126751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:00.663611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147008+0000","last_change":"2024-03-04T10:09:16.141371+0000","last_active":"2024-03-04T10:09:16.147008+0000","last_peered":"2024-03-04T10:09:16.147008+0000","last_clean":"2024-03-04T10:09:16.147008+0000","last_became_active":"2024-03-04T10:09:16.139651+0000","last_became_peered":"2024-03-04T10:09:16.139651+0000","last_unstale":"2024-03-04T10:09:16.147008+0000","last_undegraded":"2024-03-04T10:09:16.147008+0000","last_fullsized":"2024-03-04T10:09:16.147008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:20:34.944328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144912+0000","last_change":"2024-03-04T10:09:16.144189+0000","last_active":"2024-03-04T10:09:17.144912+0000","last_peered":"2024-03-04T10:09:17.144912+0000","last_clean":"2024-03-04T10:09:17.144912+0000","last_became_active":"2024-03-04T10:09:16.143631+0000","last_became_peered":"2024-03-04T10:09:16.143631+0000","last_unstale":"2024-03-04T10:09:17.144912+0000","last_undegraded":"2024-03-04T10:09:17.144912+0000","last_fullsized":"2024-03-04T10:09:17.144912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:07:28.815258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125682+0000","last_change":"2024-03-04T10:09:16.132391+0000","last_active":"2024-03-04T10:09:17.125682+0000","last_peered":"2024-03-04T10:09:17.125682+0000","last_clean":"2024-03-04T10:09:17.125682+0000","last_became_active":"2024-03-04T10:09:16.131988+0000","last_became_peered":"2024-03-04T10:09:16.131988+0000","last_unstale":"2024-03-04T10:09:17.125682+0000","last_undegraded":"2024-03-04T10:09:17.125682+0000","last_fullsized":"2024-03-04T10:09:17.125682+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:30.718603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144783+0000","last_change":"2024-03-04T10:09:16.138851+0000","last_active":"2024-03-04T10:09:17.144783+0000","last_peered":"2024-03-04T10:09:17.144783+0000","last_clean":"2024-03-04T10:09:17.144783+0000","last_became_active":"2024-03-04T10:09:16.138606+0000","last_became_peered":"2024-03-04T10:09:16.138606+0000","last_unstale":"2024-03-04T10:09:17.144783+0000","last_undegraded":"2024-03-04T10:09:17.144783+0000","last_fullsized":"2024-03-04T10:09:17.144783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:32:05.969910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125141+0000","last_change":"2024-03-04T10:09:16.137092+0000","last_active":"2024-03-04T10:09:17.125141+0000","last_peered":"2024-03-04T10:09:17.125141+0000","last_clean":"2024-03-04T10:09:17.125141+0000","last_became_active":"2024-03-04T10:09:16.136709+0000","last_became_peered":"2024-03-04T10:09:16.136709+0000","last_unstale":"2024-03-04T10:09:17.125141+0000","last_undegraded":"2024-03-04T10:09:17.125141+0000","last_fullsized":"2024-03-04T10:09:17.125141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:58.743366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141988+0000","last_change":"2024-03-04T10:09:16.135716+0000","last_active":"2024-03-04T10:09:17.141988+0000","last_peered":"2024-03-04T10:09:17.141988+0000","last_clean":"2024-03-04T10:09:17.141988+0000","last_became_active":"2024-03-04T10:09:16.135437+0000","last_became_peered":"2024-03-04T10:09:16.135437+0000","last_unstale":"2024-03-04T10:09:17.141988+0000","last_undegraded":"2024-03-04T10:09:17.141988+0000","last_fullsized":"2024-03-04T10:09:17.141988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:07.839787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142298+0000","last_change":"2024-03-04T10:09:16.139810+0000","last_active":"2024-03-04T10:09:17.142298+0000","last_peered":"2024-03-04T10:09:17.142298+0000","last_clean":"2024-03-04T10:09:17.142298+0000","last_became_active":"2024-03-04T10:09:16.139438+0000","last_became_peered":"2024-03-04T10:09:16.139438+0000","last_unstale":"2024-03-04T10:09:17.142298+0000","last_undegraded":"2024-03-04T10:09:17.142298+0000","last_fullsized":"2024-03-04T10:09:17.142298+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:52.631492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141257+0000","last_change":"2024-03-04T10:09:16.135388+0000","last_active":"2024-03-04T10:09:16.141257+0000","last_peered":"2024-03-04T10:09:16.141257+0000","last_clean":"2024-03-04T10:09:16.141257+0000","last_became_active":"2024-03-04T10:09:16.135117+0000","last_became_peered":"2024-03-04T10:09:16.135117+0000","last_unstale":"2024-03-04T10:09:16.141257+0000","last_undegraded":"2024-03-04T10:09:16.141257+0000","last_fullsized":"2024-03-04T10:09:16.141257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:58.382819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125867+0000","last_change":"2024-03-04T10:09:16.135026+0000","last_active":"2024-03-04T10:09:17.125867+0000","last_peered":"2024-03-04T10:09:17.125867+0000","last_clean":"2024-03-04T10:09:17.125867+0000","last_became_active":"2024-03-04T10:09:16.134752+0000","last_became_peered":"2024-03-04T10:09:16.134752+0000","last_unstale":"2024-03-04T10:09:17.125867+0000","last_undegraded":"2024-03-04T10:09:17.125867+0000","last_fullsized":"2024-03-04T10:09:17.125867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:16.111850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144936+0000","last_change":"2024-03-04T10:09:16.136112+0000","last_active":"2024-03-04T10:09:17.144936+0000","last_peered":"2024-03-04T10:09:17.144936+0000","last_clean":"2024-03-04T10:09:17.144936+0000","last_became_active":"2024-03-04T10:09:16.135799+0000","last_became_peered":"2024-03-04T10:09:16.135799+0000","last_unstale":"2024-03-04T10:09:17.144936+0000","last_undegraded":"2024-03-04T10:09:17.144936+0000","last_fullsized":"2024-03-04T10:09:17.144936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:46.633538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142243+0000","last_change":"2024-03-04T10:09:16.137897+0000","last_active":"2024-03-04T10:09:17.142243+0000","last_peered":"2024-03-04T10:09:17.142243+0000","last_clean":"2024-03-04T10:09:17.142243+0000","last_became_active":"2024-03-04T10:09:16.137665+0000","last_became_peered":"2024-03-04T10:09:16.137665+0000","last_unstale":"2024-03-04T10:09:17.142243+0000","last_undegraded":"2024-03-04T10:09:17.142243+0000","last_fullsized":"2024-03-04T10:09:17.142243+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:53:31.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125229+0000","last_change":"2024-03-04T10:09:16.136338+0000","last_active":"2024-03-04T10:09:17.125229+0000","last_peered":"2024-03-04T10:09:17.125229+0000","last_clean":"2024-03-04T10:09:17.125229+0000","last_became_active":"2024-03-04T10:09:16.136082+0000","last_became_peered":"2024-03-04T10:09:16.136082+0000","last_unstale":"2024-03-04T10:09:17.125229+0000","last_undegraded":"2024-03-04T10:09:17.125229+0000","last_fullsized":"2024-03-04T10:09:17.125229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:23:54.224032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126558+0000","last_change":"2024-03-04T10:09:16.138190+0000","last_active":"2024-03-04T10:09:17.126558+0000","last_peered":"2024-03-04T10:09:17.126558+0000","last_clean":"2024-03-04T10:09:17.126558+0000","last_became_active":"2024-03-04T10:09:16.137946+0000","last_became_peered":"2024-03-04T10:09:16.137946+0000","last_unstale":"2024-03-04T10:09:17.126558+0000","last_undegraded":"2024-03-04T10:09:17.126558+0000","last_fullsized":"2024-03-04T10:09:17.126558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:24.457550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125560+0000","last_change":"2024-03-04T10:09:16.130287+0000","last_active":"2024-03-04T10:09:17.125560+0000","last_peered":"2024-03-04T10:09:17.125560+0000","last_clean":"2024-03-04T10:09:17.125560+0000","last_became_active":"2024-03-04T10:09:16.129700+0000","last_became_peered":"2024-03-04T10:09:16.129700+0000","last_unstale":"2024-03-04T10:09:17.125560+0000","last_undegraded":"2024-03-04T10:09:17.125560+0000","last_fullsized":"2024-03-04T10:09:17.125560+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:24.767855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126484+0000","last_change":"2024-03-04T10:09:16.128973+0000","last_active":"2024-03-04T10:09:17.126484+0000","last_peered":"2024-03-04T10:09:17.126484+0000","last_clean":"2024-03-04T10:09:17.126484+0000","last_became_active":"2024-03-04T10:09:16.128695+0000","last_became_peered":"2024-03-04T10:09:16.128695+0000","last_unstale":"2024-03-04T10:09:17.126484+0000","last_undegraded":"2024-03-04T10:09:17.126484+0000","last_fullsized":"2024-03-04T10:09:17.126484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:29.821425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"80'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145234+0000","last_change":"2024-03-04T10:09:15.127485+0000","last_active":"2024-03-04T10:09:17.145234+0000","last_peered":"2024-03-04T10:09:17.145234+0000","last_clean":"2024-03-04T10:09:17.145234+0000","last_became_active":"2024-03-04T10:09:15.127221+0000","last_became_peered":"2024-03-04T10:09:15.127221+0000","last_unstale":"2024-03-04T10:09:17.145234+0000","last_undegraded":"2024-03-04T10:09:17.145234+0000","last_fullsized":"2024-03-04T10:09:17.145234+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:35.998519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142028+0000","last_change":"2024-03-04T10:09:16.134441+0000","last_active":"2024-03-04T10:09:17.142028+0000","last_peered":"2024-03-04T10:09:17.142028+0000","last_clean":"2024-03-04T10:09:17.142028+0000","last_became_active":"2024-03-04T10:09:16.134184+0000","last_became_peered":"2024-03-04T10:09:16.134184+0000","last_unstale":"2024-03-04T10:09:17.142028+0000","last_undegraded":"2024-03-04T10:09:17.142028+0000","last_fullsized":"2024-03-04T10:09:17.142028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:36.763955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126103+0000","last_change":"2024-03-04T10:09:15.122940+0000","last_active":"2024-03-04T10:09:17.126103+0000","last_peered":"2024-03-04T10:09:17.126103+0000","last_clean":"2024-03-04T10:09:17.126103+0000","last_became_active":"2024-03-04T10:09:15.121273+0000","last_became_peered":"2024-03-04T10:09:15.121273+0000","last_unstale":"2024-03-04T10:09:17.126103+0000","last_undegraded":"2024-03-04T10:09:17.126103+0000","last_fullsized":"2024-03-04T10:09:17.126103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:23.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125256+0000","last_change":"2024-03-04T10:09:16.138687+0000","last_active":"2024-03-04T10:09:17.125256+0000","last_peered":"2024-03-04T10:09:17.125256+0000","last_clean":"2024-03-04T10:09:17.125256+0000","last_became_active":"2024-03-04T10:09:16.138380+0000","last_became_peered":"2024-03-04T10:09:16.138380+0000","last_unstale":"2024-03-04T10:09:17.125256+0000","last_undegraded":"2024-03-04T10:09:17.125256+0000","last_fullsized":"2024-03-04T10:09:17.125256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:52.847892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125854+0000","last_change":"2024-03-04T10:09:15.128491+0000","last_active":"2024-03-04T10:09:17.125854+0000","last_peered":"2024-03-04T10:09:17.125854+0000","last_clean":"2024-03-04T10:09:17.125854+0000","last_became_active":"2024-03-04T10:09:15.127634+0000","last_became_peered":"2024-03-04T10:09:15.127634+0000","last_unstale":"2024-03-04T10:09:17.125854+0000","last_undegraded":"2024-03-04T10:09:17.125854+0000","last_fullsized":"2024-03-04T10:09:17.125854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:36.947844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125928+0000","last_change":"2024-03-04T10:09:16.131461+0000","last_active":"2024-03-04T10:09:17.125928+0000","last_peered":"2024-03-04T10:09:17.125928+0000","last_clean":"2024-03-04T10:09:17.125928+0000","last_became_active":"2024-03-04T10:09:16.131215+0000","last_became_peered":"2024-03-04T10:09:16.131215+0000","last_unstale":"2024-03-04T10:09:17.125928+0000","last_undegraded":"2024-03-04T10:09:17.125928+0000","last_fullsized":"2024-03-04T10:09:17.125928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:36.993350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.151718+0000","last_change":"2024-03-04T10:09:15.129009+0000","last_active":"2024-03-04T10:09:16.151718+0000","last_peered":"2024-03-04T10:09:16.151718+0000","last_clean":"2024-03-04T10:09:16.151718+0000","last_became_active":"2024-03-04T10:09:15.128725+0000","last_became_peered":"2024-03-04T10:09:15.128725+0000","last_unstale":"2024-03-04T10:09:16.151718+0000","last_undegraded":"2024-03-04T10:09:16.151718+0000","last_fullsized":"2024-03-04T10:09:16.151718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:05.183540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142193+0000","last_change":"2024-03-04T10:09:16.136935+0000","last_active":"2024-03-04T10:09:17.142193+0000","last_peered":"2024-03-04T10:09:17.142193+0000","last_clean":"2024-03-04T10:09:17.142193+0000","last_became_active":"2024-03-04T10:09:16.136694+0000","last_became_peered":"2024-03-04T10:09:16.136694+0000","last_unstale":"2024-03-04T10:09:17.142193+0000","last_undegraded":"2024-03-04T10:09:17.142193+0000","last_fullsized":"2024-03-04T10:09:17.142193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:27:19.604502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125856+0000","last_change":"2024-03-04T10:09:15.124416+0000","last_active":"2024-03-04T10:09:17.125856+0000","last_peered":"2024-03-04T10:09:17.125856+0000","last_clean":"2024-03-04T10:09:17.125856+0000","last_became_active":"2024-03-04T10:09:15.124162+0000","last_became_peered":"2024-03-04T10:09:15.124162+0000","last_unstale":"2024-03-04T10:09:17.125856+0000","last_undegraded":"2024-03-04T10:09:17.125856+0000","last_fullsized":"2024-03-04T10:09:17.125856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:27:33.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142173+0000","last_change":"2024-03-04T10:09:16.137070+0000","last_active":"2024-03-04T10:09:17.142173+0000","last_peered":"2024-03-04T10:09:17.142173+0000","last_clean":"2024-03-04T10:09:17.142173+0000","last_became_active":"2024-03-04T10:09:16.136495+0000","last_became_peered":"2024-03-04T10:09:16.136495+0000","last_unstale":"2024-03-04T10:09:17.142173+0000","last_undegraded":"2024-03-04T10:09:17.142173+0000","last_fullsized":"2024-03-04T10:09:17.142173+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:42.956801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.149664+0000","last_change":"2024-03-04T10:09:15.130895+0000","last_active":"2024-03-04T10:09:16.149664+0000","last_peered":"2024-03-04T10:09:16.149664+0000","last_clean":"2024-03-04T10:09:16.149664+0000","last_became_active":"2024-03-04T10:09:15.130613+0000","last_became_peered":"2024-03-04T10:09:15.130613+0000","last_unstale":"2024-03-04T10:09:16.149664+0000","last_undegraded":"2024-03-04T10:09:16.149664+0000","last_fullsized":"2024-03-04T10:09:16.149664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:31.183254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126281+0000","last_change":"2024-03-04T10:09:16.136451+0000","last_active":"2024-03-04T10:09:17.126281+0000","last_peered":"2024-03-04T10:09:17.126281+0000","last_clean":"2024-03-04T10:09:17.126281+0000","last_became_active":"2024-03-04T10:09:16.135597+0000","last_became_peered":"2024-03-04T10:09:16.135597+0000","last_unstale":"2024-03-04T10:09:17.126281+0000","last_undegraded":"2024-03-04T10:09:17.126281+0000","last_fullsized":"2024-03-04T10:09:17.126281+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:29.821425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142002+0000","last_change":"2024-03-04T10:09:15.131823+0000","last_active":"2024-03-04T10:09:17.142002+0000","last_peered":"2024-03-04T10:09:17.142002+0000","last_clean":"2024-03-04T10:09:17.142002+0000","last_became_active":"2024-03-04T10:09:15.131531+0000","last_became_peered":"2024-03-04T10:09:15.131531+0000","last_unstale":"2024-03-04T10:09:17.142002+0000","last_undegraded":"2024-03-04T10:09:17.142002+0000","last_fullsized":"2024-03-04T10:09:17.142002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:05.910856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143772+0000","last_change":"2024-03-04T10:09:16.142474+0000","last_active":"2024-03-04T10:09:16.143772+0000","last_peered":"2024-03-04T10:09:16.143772+0000","last_clean":"2024-03-04T10:09:16.143772+0000","last_became_active":"2024-03-04T10:09:16.141870+0000","last_became_peered":"2024-03-04T10:09:16.141870+0000","last_unstale":"2024-03-04T10:09:16.143772+0000","last_undegraded":"2024-03-04T10:09:16.143772+0000","last_fullsized":"2024-03-04T10:09:16.143772+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:34.385257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144505+0000","last_change":"2024-03-04T10:09:15.123717+0000","last_active":"2024-03-04T10:09:17.144505+0000","last_peered":"2024-03-04T10:09:17.144505+0000","last_clean":"2024-03-04T10:09:17.144505+0000","last_became_active":"2024-03-04T10:09:15.123440+0000","last_became_peered":"2024-03-04T10:09:15.123440+0000","last_unstale":"2024-03-04T10:09:17.144505+0000","last_undegraded":"2024-03-04T10:09:17.144505+0000","last_fullsized":"2024-03-04T10:09:17.144505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:07:22.817059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141832+0000","last_change":"2024-03-04T10:09:16.132861+0000","last_active":"2024-03-04T10:09:17.141832+0000","last_peered":"2024-03-04T10:09:17.141832+0000","last_clean":"2024-03-04T10:09:17.141832+0000","last_became_active":"2024-03-04T10:09:16.132613+0000","last_became_peered":"2024-03-04T10:09:16.132613+0000","last_unstale":"2024-03-04T10:09:17.141832+0000","last_undegraded":"2024-03-04T10:09:17.141832+0000","last_fullsized":"2024-03-04T10:09:17.141832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:19.677891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126197+0000","last_change":"2024-03-04T10:09:15.123229+0000","last_active":"2024-03-04T10:09:17.126197+0000","last_peered":"2024-03-04T10:09:17.126197+0000","last_clean":"2024-03-04T10:09:17.126197+0000","last_became_active":"2024-03-04T10:09:15.122955+0000","last_became_peered":"2024-03-04T10:09:15.122955+0000","last_unstale":"2024-03-04T10:09:17.126197+0000","last_undegraded":"2024-03-04T10:09:17.126197+0000","last_fullsized":"2024-03-04T10:09:17.126197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:11.142337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141337+0000","last_change":"2024-03-04T10:09:16.138612+0000","last_active":"2024-03-04T10:09:16.141337+0000","last_peered":"2024-03-04T10:09:16.141337+0000","last_clean":"2024-03-04T10:09:16.141337+0000","last_became_active":"2024-03-04T10:09:16.137532+0000","last_became_peered":"2024-03-04T10:09:16.137532+0000","last_unstale":"2024-03-04T10:09:16.141337+0000","last_undegraded":"2024-03-04T10:09:16.141337+0000","last_fullsized":"2024-03-04T10:09:16.141337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:27:15.262941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.130467+0000","last_change":"2024-03-04T10:09:15.128123+0000","last_active":"2024-03-04T10:09:16.130467+0000","last_peered":"2024-03-04T10:09:16.130467+0000","last_clean":"2024-03-04T10:09:16.130467+0000","last_became_active":"2024-03-04T10:09:15.127846+0000","last_became_peered":"2024-03-04T10:09:15.127846+0000","last_unstale":"2024-03-04T10:09:16.130467+0000","last_undegraded":"2024-03-04T10:09:16.130467+0000","last_fullsized":"2024-03-04T10:09:16.130467+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:56.937737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147130+0000","last_change":"2024-03-04T10:09:16.143471+0000","last_active":"2024-03-04T10:09:16.147130+0000","last_peered":"2024-03-04T10:09:16.147130+0000","last_clean":"2024-03-04T10:09:16.147130+0000","last_became_active":"2024-03-04T10:09:16.143227+0000","last_became_peered":"2024-03-04T10:09:16.143227+0000","last_unstale":"2024-03-04T10:09:16.147130+0000","last_undegraded":"2024-03-04T10:09:16.147130+0000","last_fullsized":"2024-03-04T10:09:16.147130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:09.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"80'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145078+0000","last_change":"2024-03-04T10:09:15.130776+0000","last_active":"2024-03-04T10:09:17.145078+0000","last_peered":"2024-03-04T10:09:17.145078+0000","last_clean":"2024-03-04T10:09:17.145078+0000","last_became_active":"2024-03-04T10:09:15.130496+0000","last_became_peered":"2024-03-04T10:09:15.130496+0000","last_unstale":"2024-03-04T10:09:17.145078+0000","last_undegraded":"2024-03-04T10:09:17.145078+0000","last_fullsized":"2024-03-04T10:09:17.145078+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:42.989302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126728+0000","last_change":"2024-03-04T10:09:16.134170+0000","last_active":"2024-03-04T10:09:17.126728+0000","last_peered":"2024-03-04T10:09:17.126728+0000","last_clean":"2024-03-04T10:09:17.126728+0000","last_became_active":"2024-03-04T10:09:16.133921+0000","last_became_peered":"2024-03-04T10:09:16.133921+0000","last_unstale":"2024-03-04T10:09:17.126728+0000","last_undegraded":"2024-03-04T10:09:17.126728+0000","last_fullsized":"2024-03-04T10:09:17.126728+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:09:20.701215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.156334+0000","last_change":"2024-03-04T10:09:15.124775+0000","last_active":"2024-03-04T10:09:16.156334+0000","last_peered":"2024-03-04T10:09:16.156334+0000","last_clean":"2024-03-04T10:09:16.156334+0000","last_became_active":"2024-03-04T10:09:15.124512+0000","last_became_peered":"2024-03-04T10:09:15.124512+0000","last_unstale":"2024-03-04T10:09:16.156334+0000","last_undegraded":"2024-03-04T10:09:16.156334+0000","last_fullsized":"2024-03-04T10:09:16.156334+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:38.590456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143230+0000","last_change":"2024-03-04T10:09:16.139541+0000","last_active":"2024-03-04T10:09:16.143230+0000","last_peered":"2024-03-04T10:09:16.143230+0000","last_clean":"2024-03-04T10:09:16.143230+0000","last_became_active":"2024-03-04T10:09:16.139240+0000","last_became_peered":"2024-03-04T10:09:16.139240+0000","last_unstale":"2024-03-04T10:09:16.143230+0000","last_undegraded":"2024-03-04T10:09:16.143230+0000","last_fullsized":"2024-03-04T10:09:16.143230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:19:45.017069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.152162+0000","last_change":"2024-03-04T10:09:15.129717+0000","last_active":"2024-03-04T10:09:16.152162+0000","last_peered":"2024-03-04T10:09:16.152162+0000","last_clean":"2024-03-04T10:09:16.152162+0000","last_became_active":"2024-03-04T10:09:15.129440+0000","last_became_peered":"2024-03-04T10:09:15.129440+0000","last_unstale":"2024-03-04T10:09:16.152162+0000","last_undegraded":"2024-03-04T10:09:16.152162+0000","last_fullsized":"2024-03-04T10:09:16.152162+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:13:57.190223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143801+0000","last_change":"2024-03-04T10:09:16.142514+0000","last_active":"2024-03-04T10:09:16.143801+0000","last_peered":"2024-03-04T10:09:16.143801+0000","last_clean":"2024-03-04T10:09:16.143801+0000","last_became_active":"2024-03-04T10:09:16.142181+0000","last_became_peered":"2024-03-04T10:09:16.142181+0000","last_unstale":"2024-03-04T10:09:16.143801+0000","last_undegraded":"2024-03-04T10:09:16.143801+0000","last_fullsized":"2024-03-04T10:09:16.143801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:48.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125338+0000","last_change":"2024-03-04T10:09:15.130215+0000","last_active":"2024-03-04T10:09:17.125338+0000","last_peered":"2024-03-04T10:09:17.125338+0000","last_clean":"2024-03-04T10:09:17.125338+0000","last_became_active":"2024-03-04T10:09:15.129641+0000","last_became_peered":"2024-03-04T10:09:15.129641+0000","last_unstale":"2024-03-04T10:09:17.125338+0000","last_undegraded":"2024-03-04T10:09:17.125338+0000","last_fullsized":"2024-03-04T10:09:17.125338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:27:14.254579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141918+0000","last_change":"2024-03-04T10:09:16.141035+0000","last_active":"2024-03-04T10:09:16.141918+0000","last_peered":"2024-03-04T10:09:16.141918+0000","last_clean":"2024-03-04T10:09:16.141918+0000","last_became_active":"2024-03-04T10:09:16.140725+0000","last_became_peered":"2024-03-04T10:09:16.140725+0000","last_unstale":"2024-03-04T10:09:16.141918+0000","last_undegraded":"2024-03-04T10:09:16.141918+0000","last_fullsized":"2024-03-04T10:09:16.141918+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:16:40.650752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144543+0000","last_change":"2024-03-04T10:09:15.124919+0000","last_active":"2024-03-04T10:09:17.144543+0000","last_peered":"2024-03-04T10:09:17.144543+0000","last_clean":"2024-03-04T10:09:17.144543+0000","last_became_active":"2024-03-04T10:09:15.124651+0000","last_became_peered":"2024-03-04T10:09:15.124651+0000","last_unstale":"2024-03-04T10:09:17.144543+0000","last_undegraded":"2024-03-04T10:09:17.144543+0000","last_fullsized":"2024-03-04T10:09:17.144543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:26.630830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.151524+0000","last_change":"2024-03-04T10:09:16.148151+0000","last_active":"2024-03-04T10:09:16.151524+0000","last_peered":"2024-03-04T10:09:16.151524+0000","last_clean":"2024-03-04T10:09:16.151524+0000","last_became_active":"2024-03-04T10:09:16.147891+0000","last_became_peered":"2024-03-04T10:09:16.147891+0000","last_unstale":"2024-03-04T10:09:16.151524+0000","last_undegraded":"2024-03-04T10:09:16.151524+0000","last_fullsized":"2024-03-04T10:09:16.151524+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:24.457550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141947+0000","last_change":"2024-03-04T10:09:15.124980+0000","last_active":"2024-03-04T10:09:17.141947+0000","last_peered":"2024-03-04T10:09:17.141947+0000","last_clean":"2024-03-04T10:09:17.141947+0000","last_became_active":"2024-03-04T10:09:15.124544+0000","last_became_peered":"2024-03-04T10:09:15.124544+0000","last_unstale":"2024-03-04T10:09:17.141947+0000","last_undegraded":"2024-03-04T10:09:17.141947+0000","last_fullsized":"2024-03-04T10:09:17.141947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:36.947844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147166+0000","last_change":"2024-03-04T10:09:16.139831+0000","last_active":"2024-03-04T10:09:16.147166+0000","last_peered":"2024-03-04T10:09:16.147166+0000","last_clean":"2024-03-04T10:09:16.147166+0000","last_became_active":"2024-03-04T10:09:16.139553+0000","last_became_peered":"2024-03-04T10:09:16.139553+0000","last_unstale":"2024-03-04T10:09:16.147166+0000","last_undegraded":"2024-03-04T10:09:16.147166+0000","last_fullsized":"2024-03-04T10:09:16.147166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:31.412484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126230+0000","last_change":"2024-03-04T10:09:15.124284+0000","last_active":"2024-03-04T10:09:17.126230+0000","last_peered":"2024-03-04T10:09:17.126230+0000","last_clean":"2024-03-04T10:09:17.126230+0000","last_became_active":"2024-03-04T10:09:15.123547+0000","last_became_peered":"2024-03-04T10:09:15.123547+0000","last_unstale":"2024-03-04T10:09:17.126230+0000","last_undegraded":"2024-03-04T10:09:17.126230+0000","last_fullsized":"2024-03-04T10:09:17.126230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:28:53.036069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144512+0000","last_change":"2024-03-04T10:09:16.140894+0000","last_active":"2024-03-04T10:09:17.144512+0000","last_peered":"2024-03-04T10:09:17.144512+0000","last_clean":"2024-03-04T10:09:17.144512+0000","last_became_active":"2024-03-04T10:09:16.140665+0000","last_became_peered":"2024-03-04T10:09:16.140665+0000","last_unstale":"2024-03-04T10:09:17.144512+0000","last_undegraded":"2024-03-04T10:09:17.144512+0000","last_fullsized":"2024-03-04T10:09:17.144512+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:23:08.844499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126296+0000","last_change":"2024-03-04T10:09:15.123158+0000","last_active":"2024-03-04T10:09:17.126296+0000","last_peered":"2024-03-04T10:09:17.126296+0000","last_clean":"2024-03-04T10:09:17.126296+0000","last_became_active":"2024-03-04T10:09:15.122878+0000","last_became_peered":"2024-03-04T10:09:15.122878+0000","last_unstale":"2024-03-04T10:09:17.126296+0000","last_undegraded":"2024-03-04T10:09:17.126296+0000","last_fullsized":"2024-03-04T10:09:17.126296+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:04.283634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144689+0000","last_change":"2024-03-04T10:09:16.136749+0000","last_active":"2024-03-04T10:09:17.144689+0000","last_peered":"2024-03-04T10:09:17.144689+0000","last_clean":"2024-03-04T10:09:17.144689+0000","last_became_active":"2024-03-04T10:09:16.136518+0000","last_became_peered":"2024-03-04T10:09:16.136518+0000","last_unstale":"2024-03-04T10:09:17.144689+0000","last_undegraded":"2024-03-04T10:09:17.144689+0000","last_fullsized":"2024-03-04T10:09:17.144689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:05:49.274416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145102+0000","last_change":"2024-03-04T10:09:15.130358+0000","last_active":"2024-03-04T10:09:17.145102+0000","last_peered":"2024-03-04T10:09:17.145102+0000","last_clean":"2024-03-04T10:09:17.145102+0000","last_became_active":"2024-03-04T10:09:15.129963+0000","last_became_peered":"2024-03-04T10:09:15.129963+0000","last_unstale":"2024-03-04T10:09:17.145102+0000","last_undegraded":"2024-03-04T10:09:17.145102+0000","last_fullsized":"2024-03-04T10:09:17.145102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:10.151841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147106+0000","last_change":"2024-03-04T10:09:16.141675+0000","last_active":"2024-03-04T10:09:16.147106+0000","last_peered":"2024-03-04T10:09:16.147106+0000","last_clean":"2024-03-04T10:09:16.147106+0000","last_became_active":"2024-03-04T10:09:16.141234+0000","last_became_peered":"2024-03-04T10:09:16.141234+0000","last_unstale":"2024-03-04T10:09:16.147106+0000","last_undegraded":"2024-03-04T10:09:16.147106+0000","last_fullsized":"2024-03-04T10:09:16.147106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:57:54.587735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126527+0000","last_change":"2024-03-04T10:09:15.126009+0000","last_active":"2024-03-04T10:09:17.126527+0000","last_peered":"2024-03-04T10:09:17.126527+0000","last_clean":"2024-03-04T10:09:17.126527+0000","last_became_active":"2024-03-04T10:09:15.125752+0000","last_became_peered":"2024-03-04T10:09:15.125752+0000","last_unstale":"2024-03-04T10:09:17.126527+0000","last_undegraded":"2024-03-04T10:09:17.126527+0000","last_fullsized":"2024-03-04T10:09:17.126527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:05.183540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142302+0000","last_change":"2024-03-04T10:09:16.135091+0000","last_active":"2024-03-04T10:09:17.142302+0000","last_peered":"2024-03-04T10:09:17.142302+0000","last_clean":"2024-03-04T10:09:17.142302+0000","last_became_active":"2024-03-04T10:09:16.134821+0000","last_became_peered":"2024-03-04T10:09:16.134821+0000","last_unstale":"2024-03-04T10:09:17.142302+0000","last_undegraded":"2024-03-04T10:09:17.142302+0000","last_fullsized":"2024-03-04T10:09:17.142302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:40.894952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.151128+0000","last_change":"2024-03-04T10:09:15.126865+0000","last_active":"2024-03-04T10:09:16.151128+0000","last_peered":"2024-03-04T10:09:16.151128+0000","last_clean":"2024-03-04T10:09:16.151128+0000","last_became_active":"2024-03-04T10:09:15.126596+0000","last_became_peered":"2024-03-04T10:09:15.126596+0000","last_unstale":"2024-03-04T10:09:16.151128+0000","last_undegraded":"2024-03-04T10:09:16.151128+0000","last_fullsized":"2024-03-04T10:09:16.151128+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:36.947844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142332+0000","last_change":"2024-03-04T10:09:16.141939+0000","last_active":"2024-03-04T10:09:17.142332+0000","last_peered":"2024-03-04T10:09:17.142332+0000","last_clean":"2024-03-04T10:09:17.142332+0000","last_became_active":"2024-03-04T10:09:16.141672+0000","last_became_peered":"2024-03-04T10:09:16.141672+0000","last_unstale":"2024-03-04T10:09:17.142332+0000","last_undegraded":"2024-03-04T10:09:17.142332+0000","last_fullsized":"2024-03-04T10:09:17.142332+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:26.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.125941+0000","last_change":"2024-03-04T10:09:15.130978+0000","last_active":"2024-03-04T10:09:16.125941+0000","last_peered":"2024-03-04T10:09:16.125941+0000","last_clean":"2024-03-04T10:09:16.125941+0000","last_became_active":"2024-03-04T10:09:15.130697+0000","last_became_peered":"2024-03-04T10:09:15.130697+0000","last_unstale":"2024-03-04T10:09:16.125941+0000","last_undegraded":"2024-03-04T10:09:16.125941+0000","last_fullsized":"2024-03-04T10:09:16.125941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:28:53.036069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.140631+0000","last_change":"2024-03-04T10:09:16.136498+0000","last_active":"2024-03-04T10:09:16.140631+0000","last_peered":"2024-03-04T10:09:16.140631+0000","last_clean":"2024-03-04T10:09:16.140631+0000","last_became_active":"2024-03-04T10:09:16.136172+0000","last_became_peered":"2024-03-04T10:09:16.136172+0000","last_unstale":"2024-03-04T10:09:16.140631+0000","last_undegraded":"2024-03-04T10:09:16.140631+0000","last_fullsized":"2024-03-04T10:09:16.140631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:58.743366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.125840+0000","last_change":"2024-03-04T10:09:15.123560+0000","last_active":"2024-03-04T10:09:16.125840+0000","last_peered":"2024-03-04T10:09:16.125840+0000","last_clean":"2024-03-04T10:09:16.125840+0000","last_became_active":"2024-03-04T10:09:15.123310+0000","last_became_peered":"2024-03-04T10:09:15.123310+0000","last_unstale":"2024-03-04T10:09:16.125840+0000","last_undegraded":"2024-03-04T10:09:16.125840+0000","last_fullsized":"2024-03-04T10:09:16.125840+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:11.142337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125205+0000","last_change":"2024-03-04T10:09:16.136905+0000","last_active":"2024-03-04T10:09:17.125205+0000","last_peered":"2024-03-04T10:09:17.125205+0000","last_clean":"2024-03-04T10:09:17.125205+0000","last_became_active":"2024-03-04T10:09:16.135743+0000","last_became_peered":"2024-03-04T10:09:16.135743+0000","last_unstale":"2024-03-04T10:09:17.125205+0000","last_undegraded":"2024-03-04T10:09:17.125205+0000","last_fullsized":"2024-03-04T10:09:17.125205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:16:40.650752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.150619+0000","last_change":"2024-03-04T10:09:15.126337+0000","last_active":"2024-03-04T10:09:16.150619+0000","last_peered":"2024-03-04T10:09:16.150619+0000","last_clean":"2024-03-04T10:09:16.150619+0000","last_became_active":"2024-03-04T10:09:15.126056+0000","last_became_peered":"2024-03-04T10:09:15.126056+0000","last_unstale":"2024-03-04T10:09:16.150619+0000","last_undegraded":"2024-03-04T10:09:16.150619+0000","last_fullsized":"2024-03-04T10:09:16.150619+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:42:49.214205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.149420+0000","last_change":"2024-03-04T10:09:16.146918+0000","last_active":"2024-03-04T10:09:16.149420+0000","last_peered":"2024-03-04T10:09:16.149420+0000","last_clean":"2024-03-04T10:09:16.149420+0000","last_became_active":"2024-03-04T10:09:16.145785+0000","last_became_peered":"2024-03-04T10:09:16.145785+0000","last_unstale":"2024-03-04T10:09:16.149420+0000","last_undegraded":"2024-03-04T10:09:16.149420+0000","last_fullsized":"2024-03-04T10:09:16.149420+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:29.821425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.143478+0000","last_change":"2024-03-04T10:09:15.129061+0000","last_active":"2024-03-04T10:09:17.143478+0000","last_peered":"2024-03-04T10:09:17.143478+0000","last_clean":"2024-03-04T10:09:17.143478+0000","last_became_active":"2024-03-04T10:09:15.128761+0000","last_became_peered":"2024-03-04T10:09:15.128761+0000","last_unstale":"2024-03-04T10:09:17.143478+0000","last_undegraded":"2024-03-04T10:09:17.143478+0000","last_fullsized":"2024-03-04T10:09:17.143478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:50.298152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125426+0000","last_change":"2024-03-04T10:09:16.138463+0000","last_active":"2024-03-04T10:09:17.125426+0000","last_peered":"2024-03-04T10:09:17.125426+0000","last_clean":"2024-03-04T10:09:17.125426+0000","last_became_active":"2024-03-04T10:09:16.138003+0000","last_became_peered":"2024-03-04T10:09:16.138003+0000","last_unstale":"2024-03-04T10:09:17.125426+0000","last_undegraded":"2024-03-04T10:09:17.125426+0000","last_fullsized":"2024-03-04T10:09:17.125426+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:34.385257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126153+0000","last_change":"2024-03-04T10:09:15.128537+0000","last_active":"2024-03-04T10:09:17.126153+0000","last_peered":"2024-03-04T10:09:17.126153+0000","last_clean":"2024-03-04T10:09:17.126153+0000","last_became_active":"2024-03-04T10:09:15.128009+0000","last_became_peered":"2024-03-04T10:09:15.128009+0000","last_unstale":"2024-03-04T10:09:17.126153+0000","last_undegraded":"2024-03-04T10:09:17.126153+0000","last_fullsized":"2024-03-04T10:09:17.126153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:05.910856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142352+0000","last_change":"2024-03-04T10:09:16.144312+0000","last_active":"2024-03-04T10:09:17.142352+0000","last_peered":"2024-03-04T10:09:17.142352+0000","last_clean":"2024-03-04T10:09:17.142352+0000","last_became_active":"2024-03-04T10:09:16.143548+0000","last_became_peered":"2024-03-04T10:09:16.143548+0000","last_unstale":"2024-03-04T10:09:17.142352+0000","last_undegraded":"2024-03-04T10:09:17.142352+0000","last_fullsized":"2024-03-04T10:09:17.142352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:43:22.542989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125783+0000","last_change":"2024-03-04T10:09:15.122698+0000","last_active":"2024-03-04T10:09:17.125783+0000","last_peered":"2024-03-04T10:09:17.125783+0000","last_clean":"2024-03-04T10:09:17.125783+0000","last_became_active":"2024-03-04T10:09:15.122183+0000","last_became_peered":"2024-03-04T10:09:15.122183+0000","last_unstale":"2024-03-04T10:09:17.125783+0000","last_undegraded":"2024-03-04T10:09:17.125783+0000","last_fullsized":"2024-03-04T10:09:17.125783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:19:44.008707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.149335+0000","last_change":"2024-03-04T10:09:16.146099+0000","last_active":"2024-03-04T10:09:16.149335+0000","last_peered":"2024-03-04T10:09:16.149335+0000","last_clean":"2024-03-04T10:09:16.149335+0000","last_became_active":"2024-03-04T10:09:16.145802+0000","last_became_peered":"2024-03-04T10:09:16.145802+0000","last_unstale":"2024-03-04T10:09:16.149335+0000","last_undegraded":"2024-03-04T10:09:16.149335+0000","last_fullsized":"2024-03-04T10:09:16.149335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:36.993350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.132747+0000","last_change":"2024-03-04T10:09:15.127214+0000","last_active":"2024-03-04T10:09:16.132747+0000","last_peered":"2024-03-04T10:09:16.132747+0000","last_clean":"2024-03-04T10:09:16.132747+0000","last_became_active":"2024-03-04T10:09:15.126943+0000","last_became_peered":"2024-03-04T10:09:15.126943+0000","last_unstale":"2024-03-04T10:09:16.132747+0000","last_undegraded":"2024-03-04T10:09:16.132747+0000","last_fullsized":"2024-03-04T10:09:16.132747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:08.865710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143821+0000","last_change":"2024-03-04T10:09:16.142557+0000","last_active":"2024-03-04T10:09:16.143821+0000","last_peered":"2024-03-04T10:09:16.143821+0000","last_clean":"2024-03-04T10:09:16.143821+0000","last_became_active":"2024-03-04T10:09:16.142198+0000","last_became_peered":"2024-03-04T10:09:16.142198+0000","last_unstale":"2024-03-04T10:09:16.143821+0000","last_undegraded":"2024-03-04T10:09:16.143821+0000","last_fullsized":"2024-03-04T10:09:16.143821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:30.718603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144774+0000","last_change":"2024-03-04T10:09:15.124503+0000","last_active":"2024-03-04T10:09:17.144774+0000","last_peered":"2024-03-04T10:09:17.144774+0000","last_clean":"2024-03-04T10:09:17.144774+0000","last_became_active":"2024-03-04T10:09:15.124077+0000","last_became_peered":"2024-03-04T10:09:15.124077+0000","last_unstale":"2024-03-04T10:09:17.144774+0000","last_undegraded":"2024-03-04T10:09:17.144774+0000","last_fullsized":"2024-03-04T10:09:17.144774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:05:20.795272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126758+0000","last_change":"2024-03-04T10:09:16.138400+0000","last_active":"2024-03-04T10:09:17.126758+0000","last_peered":"2024-03-04T10:09:17.126758+0000","last_clean":"2024-03-04T10:09:17.126758+0000","last_became_active":"2024-03-04T10:09:16.137926+0000","last_became_peered":"2024-03-04T10:09:16.137926+0000","last_unstale":"2024-03-04T10:09:17.126758+0000","last_undegraded":"2024-03-04T10:09:17.126758+0000","last_fullsized":"2024-03-04T10:09:17.126758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:05:02.026552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.133310+0000","last_change":"2024-03-04T10:09:15.131615+0000","last_active":"2024-03-04T10:09:16.133310+0000","last_peered":"2024-03-04T10:09:16.133310+0000","last_clean":"2024-03-04T10:09:16.133310+0000","last_became_active":"2024-03-04T10:09:15.131360+0000","last_became_peered":"2024-03-04T10:09:15.131360+0000","last_unstale":"2024-03-04T10:09:16.133310+0000","last_undegraded":"2024-03-04T10:09:16.133310+0000","last_fullsized":"2024-03-04T10:09:16.133310+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:11.359273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126767+0000","last_change":"2024-03-04T10:09:16.128352+0000","last_active":"2024-03-04T10:09:17.126767+0000","last_peered":"2024-03-04T10:09:17.126767+0000","last_clean":"2024-03-04T10:09:17.126767+0000","last_became_active":"2024-03-04T10:09:16.128046+0000","last_became_peered":"2024-03-04T10:09:16.128046+0000","last_unstale":"2024-03-04T10:09:17.126767+0000","last_undegraded":"2024-03-04T10:09:17.126767+0000","last_fullsized":"2024-03-04T10:09:17.126767+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:34.410941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126499+0000","last_change":"2024-03-04T10:09:15.125545+0000","last_active":"2024-03-04T10:09:17.126499+0000","last_peered":"2024-03-04T10:09:17.126499+0000","last_clean":"2024-03-04T10:09:17.126499+0000","last_became_active":"2024-03-04T10:09:15.125304+0000","last_became_peered":"2024-03-04T10:09:15.125304+0000","last_unstale":"2024-03-04T10:09:17.126499+0000","last_undegraded":"2024-03-04T10:09:17.126499+0000","last_fullsized":"2024-03-04T10:09:17.126499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:11.359273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126044+0000","last_change":"2024-03-04T10:09:16.130150+0000","last_active":"2024-03-04T10:09:17.126044+0000","last_peered":"2024-03-04T10:09:17.126044+0000","last_clean":"2024-03-04T10:09:17.126044+0000","last_became_active":"2024-03-04T10:09:16.129868+0000","last_became_peered":"2024-03-04T10:09:16.129868+0000","last_unstale":"2024-03-04T10:09:17.126044+0000","last_undegraded":"2024-03-04T10:09:17.126044+0000","last_fullsized":"2024-03-04T10:09:17.126044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:34:42.938735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.128800+0000","last_change":"2024-03-04T10:09:15.130632+0000","last_active":"2024-03-04T10:09:16.128800+0000","last_peered":"2024-03-04T10:09:16.128800+0000","last_clean":"2024-03-04T10:09:16.128800+0000","last_became_active":"2024-03-04T10:09:15.130215+0000","last_became_peered":"2024-03-04T10:09:15.130215+0000","last_unstale":"2024-03-04T10:09:16.128800+0000","last_undegraded":"2024-03-04T10:09:16.128800+0000","last_fullsized":"2024-03-04T10:09:16.128800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:05.910856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":294,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125617+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:09:17.125617+0000","last_peered":"2024-03-04T10:09:17.125617+0000","last_clean":"2024-03-04T10:09:17.125617+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:09:17.125617+0000","last_undegraded":"2024-03-04T10:09:17.125617+0000","last_fullsized":"2024-03-04T10:09:17.125617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1044021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":991232,"data_stored":455243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":453665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":999424,"data_stored":455347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":454213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":1003520,"data_stored":461065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":987136,"data_stored":453743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1572864,"data_stored":1043999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:09:19.101 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:09:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.235+0000 7fb92064b640 1 Processor -- start 2024-03-04T10:09:19.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.235+0000 7fb92064b640 1 -- start start 2024-03-04T10:09:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.236+0000 7fb92064b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb91815dcd0 0x7fb9180ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.236+0000 7fb92064b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9180cfeb0 0x7fb9181618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.236+0000 7fb92064b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb918161de0 0x7fb9180d18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.236+0000 7fb92064b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9180cc0c0 con 0x7fb91815dcd0 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.236+0000 7fb92064b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb918162230 con 0x7fb918161de0 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.236+0000 7fb92064b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180cf200 con 0x7fb9180cfeb0 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb91815dcd0 0x7fb9180ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb91815dcd0 0x7fb9180ce0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33554/0 (socket says 172.21.15.99:33554) 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 -- 172.21.15.99:0/3265050669 learned_addr learned my addr 172.21.15.99:0/3265050669 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91ebe8640 1 --2- 172.21.15.99:0/3265050669 >> v2:172.21.15.162:3300/0 conn(0x7fb918161de0 0x7fb9180d18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91dbe6640 1 --2- 172.21.15.99:0/3265050669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9180cfeb0 0x7fb9181618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 -- 172.21.15.99:0/3265050669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9180cfeb0 msgr2=0x7fb9181618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 --2- 172.21.15.99:0/3265050669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9180cfeb0 0x7fb9181618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 -- 172.21.15.99:0/3265050669 >> v2:172.21.15.162:3300/0 conn(0x7fb918161de0 msgr2=0x7fb9180d18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:19.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 --2- 172.21.15.99:0/3265050669 >> v2:172.21.15.162:3300/0 conn(0x7fb918161de0 0x7fb9180d18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.237+0000 7fb91e3e7640 1 -- 172.21.15.99:0/3265050669 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9180ce680 con 0x7fb91815dcd0 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb91e3e7640 1 --2- 172.21.15.99:0/3265050669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb91815dcd0 0x7fb9180ce0a0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb9180b7860 tx=0x7fb91402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c912aea6af6b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb91d3e5640 1 -- 172.21.15.99:0/3265050669 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91402ca70 con 0x7fb91815dcd0 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb91d3e5640 1 -- 172.21.15.99:0/3265050669 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91402cc10 con 0x7fb91815dcd0 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb91d3e5640 1 -- 172.21.15.99:0/3265050669 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9140343f0 con 0x7fb91815dcd0 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb92064b640 1 -- 172.21.15.99:0/3265050669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb91815dcd0 msgr2=0x7fb9180ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb92064b640 1 --2- 172.21.15.99:0/3265050669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb91815dcd0 0x7fb9180ce0a0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb9180b7860 tx=0x7fb91402bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb92064b640 1 -- 172.21.15.99:0/3265050669 shutdown_connections 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb92064b640 1 --2- 172.21.15.99:0/3265050669 >> v2:172.21.15.162:3300/0 conn(0x7fb918161de0 0x7fb9180d18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.238+0000 7fb92064b640 1 --2- 172.21.15.99:0/3265050669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9180cfeb0 0x7fb9181618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.239+0000 7fb92064b640 1 --2- 172.21.15.99:0/3265050669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb91815dcd0 0x7fb9180ce0a0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.239+0000 7fb92064b640 1 -- 172.21.15.99:0/3265050669 >> 172.21.15.99:0/3265050669 conn(0x7fb9180b3480 msgr2=0x7fb9180b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.239+0000 7fb92064b640 1 -- 172.21.15.99:0/3265050669 shutdown_connections 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.239+0000 7fb92064b640 1 -- 172.21.15.99:0/3265050669 wait complete. 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 Processor -- start 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 -- start start 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 0x7fb91815c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb91815dcd0 0x7fb9180b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb918161de0 0x7fb9180c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180cef20 con 0x7fb9180cfeb0 2024-03-04T10:09:19.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9180cec40 con 0x7fb91815dcd0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb92064b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91806bff0 con 0x7fb918161de0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.240+0000 7fb91e3e7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 0x7fb91815c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 0x7fb91815c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33558/0 (socket says 172.21.15.99:33558) 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 -- 172.21.15.99:0/2575309330 learned_addr learned my addr 172.21.15.99:0/2575309330 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 -- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb918161de0 msgr2=0x7fb9180c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb918161de0 0x7fb9180c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 -- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb91815dcd0 msgr2=0x7fb9180b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb91815dcd0 0x7fb9180b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 -- 172.21.15.99:0/2575309330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9180b5840 con 0x7fb9180cfeb0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb91e3e7640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 0x7fb91815c4c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb914000f80 tx=0x7fb914004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb914043020 con 0x7fb9180cfeb0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91403e450 con 0x7fb9180cfeb0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9180b5b20 con 0x7fb9180cfeb0 2024-03-04T10:09:19.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91403e840 con 0x7fb9180cfeb0 2024-03-04T10:09:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.241+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9180b60e0 con 0x7fb9180cfeb0 2024-03-04T10:09:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.242+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb918074f60 con 0x7fb9180cfeb0 2024-03-04T10:09:19.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.243+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb914046020 con 0x7fb9180cfeb0 2024-03-04T10:09:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.243+0000 7fb90effd640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.243+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb9140acc60 con 0x7fb9180cfeb0 2024-03-04T10:09:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.243+0000 7fb91dbe6640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.244+0000 7fb91dbe6640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb9180b7610 tx=0x7fb908008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.247+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb91404d030 con 0x7fb9180cfeb0 2024-03-04T10:09:19.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.415+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9180b7640 con 0x7fb8fc06b4d0 2024-03-04T10:09:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.421+0000 7fb90effd640 1 -- 172.21.15.99:0/2575309330 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252050 (secure 0 0 0) 0x7fb9180b7640 con 0x7fb8fc06b4d0 2024-03-04T10:09:19.425 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:19.426 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:09:19.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8fc06b4d0 msgr2=0x7fb8fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb9180b7610 tx=0x7fb908008040 comp rx=0 tx=0).stop 2024-03-04T10:09:19.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 msgr2=0x7fb91815c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 0x7fb91815c4c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb914000f80 tx=0x7fb914004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:19.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 shutdown_connections 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb918161de0 0x7fb9180c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb91815dcd0 0x7fb9180b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 --2- 172.21.15.99:0/2575309330 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9180cfeb0 0x7fb91815c4c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 >> 172.21.15.99:0/2575309330 conn(0x7fb9180b3480 msgr2=0x7fb9180cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 shutdown_connections 2024-03-04T10:09:19.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.424+0000 7fb92064b640 1 -- 172.21.15.99:0/2575309330 wait complete. 2024-03-04T10:09:19.436 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-03-04T10:09:17.519519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":131,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":229264,"kb_used_data":8884,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754745456,"statfs":{"total":773094113280,"available":772859346944,"internally_reserved":0,"allocated":9097216,"data_stored":4821296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":225496626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000763"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126127+0000","last_change":"2024-03-04T10:09:16.135890+0000","last_active":"2024-03-04T10:09:17.126127+0000","last_peered":"2024-03-04T10:09:17.126127+0000","last_clean":"2024-03-04T10:09:17.126127+0000","last_became_active":"2024-03-04T10:09:16.135609+0000","last_became_peered":"2024-03-04T10:09:16.135609+0000","last_unstale":"2024-03-04T10:09:17.126127+0000","last_undegraded":"2024-03-04T10:09:17.126127+0000","last_fullsized":"2024-03-04T10:09:17.126127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:09:34.119832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142270+0000","last_change":"2024-03-04T10:09:16.137720+0000","last_active":"2024-03-04T10:09:17.142270+0000","last_peered":"2024-03-04T10:09:17.142270+0000","last_clean":"2024-03-04T10:09:17.142270+0000","last_became_active":"2024-03-04T10:09:16.137400+0000","last_became_peered":"2024-03-04T10:09:16.137400+0000","last_unstale":"2024-03-04T10:09:17.142270+0000","last_undegraded":"2024-03-04T10:09:17.142270+0000","last_fullsized":"2024-03-04T10:09:17.142270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:26.234049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126679+0000","last_change":"2024-03-04T10:09:16.138687+0000","last_active":"2024-03-04T10:09:17.126679+0000","last_peered":"2024-03-04T10:09:17.126679+0000","last_clean":"2024-03-04T10:09:17.126679+0000","last_became_active":"2024-03-04T10:09:16.138426+0000","last_became_peered":"2024-03-04T10:09:16.138426+0000","last_unstale":"2024-03-04T10:09:17.126679+0000","last_undegraded":"2024-03-04T10:09:17.126679+0000","last_fullsized":"2024-03-04T10:09:17.126679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:42:11.162194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141370+0000","last_change":"2024-03-04T10:09:16.138148+0000","last_active":"2024-03-04T10:09:16.141370+0000","last_peered":"2024-03-04T10:09:16.141370+0000","last_clean":"2024-03-04T10:09:16.141370+0000","last_became_active":"2024-03-04T10:09:16.137609+0000","last_became_peered":"2024-03-04T10:09:16.137609+0000","last_unstale":"2024-03-04T10:09:16.141370+0000","last_undegraded":"2024-03-04T10:09:16.141370+0000","last_fullsized":"2024-03-04T10:09:16.141370+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:14:48.132543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144716+0000","last_change":"2024-03-04T10:09:16.144013+0000","last_active":"2024-03-04T10:09:17.144716+0000","last_peered":"2024-03-04T10:09:17.144716+0000","last_clean":"2024-03-04T10:09:17.144716+0000","last_became_active":"2024-03-04T10:09:16.143691+0000","last_became_peered":"2024-03-04T10:09:16.143691+0000","last_unstale":"2024-03-04T10:09:17.144716+0000","last_undegraded":"2024-03-04T10:09:17.144716+0000","last_fullsized":"2024-03-04T10:09:17.144716+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:47:51.311633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.145036+0000","last_change":"2024-03-04T10:09:16.134941+0000","last_active":"2024-03-04T10:09:16.145036+0000","last_peered":"2024-03-04T10:09:16.145036+0000","last_clean":"2024-03-04T10:09:16.145036+0000","last_became_active":"2024-03-04T10:09:16.134703+0000","last_became_peered":"2024-03-04T10:09:16.134703+0000","last_unstale":"2024-03-04T10:09:16.145036+0000","last_undegraded":"2024-03-04T10:09:16.145036+0000","last_fullsized":"2024-03-04T10:09:16.145036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:50:29.042535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143667+0000","last_change":"2024-03-04T10:09:16.141909+0000","last_active":"2024-03-04T10:09:16.143667+0000","last_peered":"2024-03-04T10:09:16.143667+0000","last_clean":"2024-03-04T10:09:16.143667+0000","last_became_active":"2024-03-04T10:09:16.141664+0000","last_became_peered":"2024-03-04T10:09:16.141664+0000","last_unstale":"2024-03-04T10:09:16.143667+0000","last_undegraded":"2024-03-04T10:09:16.143667+0000","last_fullsized":"2024-03-04T10:09:16.143667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:52.847892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144492+0000","last_change":"2024-03-04T10:09:16.145950+0000","last_active":"2024-03-04T10:09:17.144492+0000","last_peered":"2024-03-04T10:09:17.144492+0000","last_clean":"2024-03-04T10:09:17.144492+0000","last_became_active":"2024-03-04T10:09:16.145700+0000","last_became_peered":"2024-03-04T10:09:16.145700+0000","last_unstale":"2024-03-04T10:09:17.144492+0000","last_undegraded":"2024-03-04T10:09:17.144492+0000","last_fullsized":"2024-03-04T10:09:17.144492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:12:56.186318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125662+0000","last_change":"2024-03-04T10:09:16.137250+0000","last_active":"2024-03-04T10:09:17.125662+0000","last_peered":"2024-03-04T10:09:17.125662+0000","last_clean":"2024-03-04T10:09:17.125662+0000","last_became_active":"2024-03-04T10:09:16.136946+0000","last_became_peered":"2024-03-04T10:09:16.136946+0000","last_unstale":"2024-03-04T10:09:17.125662+0000","last_undegraded":"2024-03-04T10:09:17.125662+0000","last_fullsized":"2024-03-04T10:09:17.125662+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:48.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141320+0000","last_change":"2024-03-04T10:09:16.138494+0000","last_active":"2024-03-04T10:09:16.141320+0000","last_peered":"2024-03-04T10:09:16.141320+0000","last_clean":"2024-03-04T10:09:16.141320+0000","last_became_active":"2024-03-04T10:09:16.138249+0000","last_became_peered":"2024-03-04T10:09:16.138249+0000","last_unstale":"2024-03-04T10:09:16.141320+0000","last_undegraded":"2024-03-04T10:09:16.141320+0000","last_fullsized":"2024-03-04T10:09:16.141320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:24.060713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142171+0000","last_change":"2024-03-04T10:09:16.137262+0000","last_active":"2024-03-04T10:09:17.142171+0000","last_peered":"2024-03-04T10:09:17.142171+0000","last_clean":"2024-03-04T10:09:17.142171+0000","last_became_active":"2024-03-04T10:09:16.136829+0000","last_became_peered":"2024-03-04T10:09:16.136829+0000","last_unstale":"2024-03-04T10:09:17.142171+0000","last_undegraded":"2024-03-04T10:09:17.142171+0000","last_fullsized":"2024-03-04T10:09:17.142171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:33:42.182525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.138898+0000","last_change":"2024-03-04T10:09:16.134474+0000","last_active":"2024-03-04T10:09:16.138898+0000","last_peered":"2024-03-04T10:09:16.138898+0000","last_clean":"2024-03-04T10:09:16.138898+0000","last_became_active":"2024-03-04T10:09:16.134188+0000","last_became_peered":"2024-03-04T10:09:16.134188+0000","last_unstale":"2024-03-04T10:09:16.138898+0000","last_undegraded":"2024-03-04T10:09:16.138898+0000","last_fullsized":"2024-03-04T10:09:16.138898+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:04.862839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144708+0000","last_change":"2024-03-04T10:09:16.137301+0000","last_active":"2024-03-04T10:09:17.144708+0000","last_peered":"2024-03-04T10:09:17.144708+0000","last_clean":"2024-03-04T10:09:17.144708+0000","last_became_active":"2024-03-04T10:09:16.137041+0000","last_became_peered":"2024-03-04T10:09:16.137041+0000","last_unstale":"2024-03-04T10:09:17.144708+0000","last_undegraded":"2024-03-04T10:09:17.144708+0000","last_fullsized":"2024-03-04T10:09:17.144708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:19:05.985342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.144964+0000","last_change":"2024-03-04T10:09:16.134878+0000","last_active":"2024-03-04T10:09:16.144964+0000","last_peered":"2024-03-04T10:09:16.144964+0000","last_clean":"2024-03-04T10:09:16.144964+0000","last_became_active":"2024-03-04T10:09:16.134612+0000","last_became_peered":"2024-03-04T10:09:16.134612+0000","last_unstale":"2024-03-04T10:09:16.144964+0000","last_undegraded":"2024-03-04T10:09:16.144964+0000","last_fullsized":"2024-03-04T10:09:16.144964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:51.306514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141890+0000","last_change":"2024-03-04T10:09:16.132703+0000","last_active":"2024-03-04T10:09:17.141890+0000","last_peered":"2024-03-04T10:09:17.141890+0000","last_clean":"2024-03-04T10:09:17.141890+0000","last_became_active":"2024-03-04T10:09:16.132424+0000","last_became_peered":"2024-03-04T10:09:16.132424+0000","last_unstale":"2024-03-04T10:09:17.141890+0000","last_undegraded":"2024-03-04T10:09:17.141890+0000","last_fullsized":"2024-03-04T10:09:17.141890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:28:59.089491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125404+0000","last_change":"2024-03-04T10:09:16.135262+0000","last_active":"2024-03-04T10:09:17.125404+0000","last_peered":"2024-03-04T10:09:17.125404+0000","last_clean":"2024-03-04T10:09:17.125404+0000","last_became_active":"2024-03-04T10:09:16.134221+0000","last_became_peered":"2024-03-04T10:09:16.134221+0000","last_unstale":"2024-03-04T10:09:17.125404+0000","last_undegraded":"2024-03-04T10:09:17.125404+0000","last_fullsized":"2024-03-04T10:09:17.125404+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:44:28.222882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145240+0000","last_change":"2024-03-04T10:09:16.140304+0000","last_active":"2024-03-04T10:09:17.145240+0000","last_peered":"2024-03-04T10:09:17.145240+0000","last_clean":"2024-03-04T10:09:17.145240+0000","last_became_active":"2024-03-04T10:09:16.140055+0000","last_became_peered":"2024-03-04T10:09:16.140055+0000","last_unstale":"2024-03-04T10:09:17.145240+0000","last_undegraded":"2024-03-04T10:09:17.145240+0000","last_fullsized":"2024-03-04T10:09:17.145240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:18:11.277585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126751+0000","last_change":"2024-03-04T10:09:16.128633+0000","last_active":"2024-03-04T10:09:17.126751+0000","last_peered":"2024-03-04T10:09:17.126751+0000","last_clean":"2024-03-04T10:09:17.126751+0000","last_became_active":"2024-03-04T10:09:16.128361+0000","last_became_peered":"2024-03-04T10:09:16.128361+0000","last_unstale":"2024-03-04T10:09:17.126751+0000","last_undegraded":"2024-03-04T10:09:17.126751+0000","last_fullsized":"2024-03-04T10:09:17.126751+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:00.663611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147008+0000","last_change":"2024-03-04T10:09:16.141371+0000","last_active":"2024-03-04T10:09:16.147008+0000","last_peered":"2024-03-04T10:09:16.147008+0000","last_clean":"2024-03-04T10:09:16.147008+0000","last_became_active":"2024-03-04T10:09:16.139651+0000","last_became_peered":"2024-03-04T10:09:16.139651+0000","last_unstale":"2024-03-04T10:09:16.147008+0000","last_undegraded":"2024-03-04T10:09:16.147008+0000","last_fullsized":"2024-03-04T10:09:16.147008+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:20:34.944328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144912+0000","last_change":"2024-03-04T10:09:16.144189+0000","last_active":"2024-03-04T10:09:17.144912+0000","last_peered":"2024-03-04T10:09:17.144912+0000","last_clean":"2024-03-04T10:09:17.144912+0000","last_became_active":"2024-03-04T10:09:16.143631+0000","last_became_peered":"2024-03-04T10:09:16.143631+0000","last_unstale":"2024-03-04T10:09:17.144912+0000","last_undegraded":"2024-03-04T10:09:17.144912+0000","last_fullsized":"2024-03-04T10:09:17.144912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:07:28.815258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125682+0000","last_change":"2024-03-04T10:09:16.132391+0000","last_active":"2024-03-04T10:09:17.125682+0000","last_peered":"2024-03-04T10:09:17.125682+0000","last_clean":"2024-03-04T10:09:17.125682+0000","last_became_active":"2024-03-04T10:09:16.131988+0000","last_became_peered":"2024-03-04T10:09:16.131988+0000","last_unstale":"2024-03-04T10:09:17.125682+0000","last_undegraded":"2024-03-04T10:09:17.125682+0000","last_fullsized":"2024-03-04T10:09:17.125682+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:30.718603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144783+0000","last_change":"2024-03-04T10:09:16.138851+0000","last_active":"2024-03-04T10:09:17.144783+0000","last_peered":"2024-03-04T10:09:17.144783+0000","last_clean":"2024-03-04T10:09:17.144783+0000","last_became_active":"2024-03-04T10:09:16.138606+0000","last_became_peered":"2024-03-04T10:09:16.138606+0000","last_unstale":"2024-03-04T10:09:17.144783+0000","last_undegraded":"2024-03-04T10:09:17.144783+0000","last_fullsized":"2024-03-04T10:09:17.144783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:32:05.969910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125141+0000","last_change":"2024-03-04T10:09:16.137092+0000","last_active":"2024-03-04T10:09:17.125141+0000","last_peered":"2024-03-04T10:09:17.125141+0000","last_clean":"2024-03-04T10:09:17.125141+0000","last_became_active":"2024-03-04T10:09:16.136709+0000","last_became_peered":"2024-03-04T10:09:16.136709+0000","last_unstale":"2024-03-04T10:09:17.125141+0000","last_undegraded":"2024-03-04T10:09:17.125141+0000","last_fullsized":"2024-03-04T10:09:17.125141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:58.743366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141988+0000","last_change":"2024-03-04T10:09:16.135716+0000","last_active":"2024-03-04T10:09:17.141988+0000","last_peered":"2024-03-04T10:09:17.141988+0000","last_clean":"2024-03-04T10:09:17.141988+0000","last_became_active":"2024-03-04T10:09:16.135437+0000","last_became_peered":"2024-03-04T10:09:16.135437+0000","last_unstale":"2024-03-04T10:09:17.141988+0000","last_undegraded":"2024-03-04T10:09:17.141988+0000","last_fullsized":"2024-03-04T10:09:17.141988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:07.839787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142298+0000","last_change":"2024-03-04T10:09:16.139810+0000","last_active":"2024-03-04T10:09:17.142298+0000","last_peered":"2024-03-04T10:09:17.142298+0000","last_clean":"2024-03-04T10:09:17.142298+0000","last_became_active":"2024-03-04T10:09:16.139438+0000","last_became_peered":"2024-03-04T10:09:16.139438+0000","last_unstale":"2024-03-04T10:09:17.142298+0000","last_undegraded":"2024-03-04T10:09:17.142298+0000","last_fullsized":"2024-03-04T10:09:17.142298+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:32:52.631492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141257+0000","last_change":"2024-03-04T10:09:16.135388+0000","last_active":"2024-03-04T10:09:16.141257+0000","last_peered":"2024-03-04T10:09:16.141257+0000","last_clean":"2024-03-04T10:09:16.141257+0000","last_became_active":"2024-03-04T10:09:16.135117+0000","last_became_peered":"2024-03-04T10:09:16.135117+0000","last_unstale":"2024-03-04T10:09:16.141257+0000","last_undegraded":"2024-03-04T10:09:16.141257+0000","last_fullsized":"2024-03-04T10:09:16.141257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:05:58.382819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125867+0000","last_change":"2024-03-04T10:09:16.135026+0000","last_active":"2024-03-04T10:09:17.125867+0000","last_peered":"2024-03-04T10:09:17.125867+0000","last_clean":"2024-03-04T10:09:17.125867+0000","last_became_active":"2024-03-04T10:09:16.134752+0000","last_became_peered":"2024-03-04T10:09:16.134752+0000","last_unstale":"2024-03-04T10:09:17.125867+0000","last_undegraded":"2024-03-04T10:09:17.125867+0000","last_fullsized":"2024-03-04T10:09:17.125867+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:00:16.111850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144936+0000","last_change":"2024-03-04T10:09:16.136112+0000","last_active":"2024-03-04T10:09:17.144936+0000","last_peered":"2024-03-04T10:09:17.144936+0000","last_clean":"2024-03-04T10:09:17.144936+0000","last_became_active":"2024-03-04T10:09:16.135799+0000","last_became_peered":"2024-03-04T10:09:16.135799+0000","last_unstale":"2024-03-04T10:09:17.144936+0000","last_undegraded":"2024-03-04T10:09:17.144936+0000","last_fullsized":"2024-03-04T10:09:17.144936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:36:46.633538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142243+0000","last_change":"2024-03-04T10:09:16.137897+0000","last_active":"2024-03-04T10:09:17.142243+0000","last_peered":"2024-03-04T10:09:17.142243+0000","last_clean":"2024-03-04T10:09:17.142243+0000","last_became_active":"2024-03-04T10:09:16.137665+0000","last_became_peered":"2024-03-04T10:09:16.137665+0000","last_unstale":"2024-03-04T10:09:17.142243+0000","last_undegraded":"2024-03-04T10:09:17.142243+0000","last_fullsized":"2024-03-04T10:09:17.142243+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:53:31.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125229+0000","last_change":"2024-03-04T10:09:16.136338+0000","last_active":"2024-03-04T10:09:17.125229+0000","last_peered":"2024-03-04T10:09:17.125229+0000","last_clean":"2024-03-04T10:09:17.125229+0000","last_became_active":"2024-03-04T10:09:16.136082+0000","last_became_peered":"2024-03-04T10:09:16.136082+0000","last_unstale":"2024-03-04T10:09:17.125229+0000","last_undegraded":"2024-03-04T10:09:17.125229+0000","last_fullsized":"2024-03-04T10:09:17.125229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:23:54.224032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126558+0000","last_change":"2024-03-04T10:09:16.138190+0000","last_active":"2024-03-04T10:09:17.126558+0000","last_peered":"2024-03-04T10:09:17.126558+0000","last_clean":"2024-03-04T10:09:17.126558+0000","last_became_active":"2024-03-04T10:09:16.137946+0000","last_became_peered":"2024-03-04T10:09:16.137946+0000","last_unstale":"2024-03-04T10:09:17.126558+0000","last_undegraded":"2024-03-04T10:09:17.126558+0000","last_fullsized":"2024-03-04T10:09:17.126558+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:24.457550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125560+0000","last_change":"2024-03-04T10:09:16.130287+0000","last_active":"2024-03-04T10:09:17.125560+0000","last_peered":"2024-03-04T10:09:17.125560+0000","last_clean":"2024-03-04T10:09:17.125560+0000","last_became_active":"2024-03-04T10:09:16.129700+0000","last_became_peered":"2024-03-04T10:09:16.129700+0000","last_unstale":"2024-03-04T10:09:17.125560+0000","last_undegraded":"2024-03-04T10:09:17.125560+0000","last_fullsized":"2024-03-04T10:09:17.125560+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:51:24.767855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126484+0000","last_change":"2024-03-04T10:09:16.128973+0000","last_active":"2024-03-04T10:09:17.126484+0000","last_peered":"2024-03-04T10:09:17.126484+0000","last_clean":"2024-03-04T10:09:17.126484+0000","last_became_active":"2024-03-04T10:09:16.128695+0000","last_became_peered":"2024-03-04T10:09:16.128695+0000","last_unstale":"2024-03-04T10:09:17.126484+0000","last_undegraded":"2024-03-04T10:09:17.126484+0000","last_fullsized":"2024-03-04T10:09:17.126484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:29.821425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"80'4","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145234+0000","last_change":"2024-03-04T10:09:15.127485+0000","last_active":"2024-03-04T10:09:17.145234+0000","last_peered":"2024-03-04T10:09:17.145234+0000","last_clean":"2024-03-04T10:09:17.145234+0000","last_became_active":"2024-03-04T10:09:15.127221+0000","last_became_peered":"2024-03-04T10:09:15.127221+0000","last_unstale":"2024-03-04T10:09:17.145234+0000","last_undegraded":"2024-03-04T10:09:17.145234+0000","last_fullsized":"2024-03-04T10:09:17.145234+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:49:35.998519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142028+0000","last_change":"2024-03-04T10:09:16.134441+0000","last_active":"2024-03-04T10:09:17.142028+0000","last_peered":"2024-03-04T10:09:17.142028+0000","last_clean":"2024-03-04T10:09:17.142028+0000","last_became_active":"2024-03-04T10:09:16.134184+0000","last_became_peered":"2024-03-04T10:09:16.134184+0000","last_unstale":"2024-03-04T10:09:17.142028+0000","last_undegraded":"2024-03-04T10:09:17.142028+0000","last_fullsized":"2024-03-04T10:09:17.142028+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:36.763955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126103+0000","last_change":"2024-03-04T10:09:15.122940+0000","last_active":"2024-03-04T10:09:17.126103+0000","last_peered":"2024-03-04T10:09:17.126103+0000","last_clean":"2024-03-04T10:09:17.126103+0000","last_became_active":"2024-03-04T10:09:15.121273+0000","last_became_peered":"2024-03-04T10:09:15.121273+0000","last_unstale":"2024-03-04T10:09:17.126103+0000","last_undegraded":"2024-03-04T10:09:17.126103+0000","last_fullsized":"2024-03-04T10:09:17.126103+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:47:23.052351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125256+0000","last_change":"2024-03-04T10:09:16.138687+0000","last_active":"2024-03-04T10:09:17.125256+0000","last_peered":"2024-03-04T10:09:17.125256+0000","last_clean":"2024-03-04T10:09:17.125256+0000","last_became_active":"2024-03-04T10:09:16.138380+0000","last_became_peered":"2024-03-04T10:09:16.138380+0000","last_unstale":"2024-03-04T10:09:17.125256+0000","last_undegraded":"2024-03-04T10:09:17.125256+0000","last_fullsized":"2024-03-04T10:09:17.125256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:45:52.847892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125854+0000","last_change":"2024-03-04T10:09:15.128491+0000","last_active":"2024-03-04T10:09:17.125854+0000","last_peered":"2024-03-04T10:09:17.125854+0000","last_clean":"2024-03-04T10:09:17.125854+0000","last_became_active":"2024-03-04T10:09:15.127634+0000","last_became_peered":"2024-03-04T10:09:15.127634+0000","last_unstale":"2024-03-04T10:09:17.125854+0000","last_undegraded":"2024-03-04T10:09:17.125854+0000","last_fullsized":"2024-03-04T10:09:17.125854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:36.947844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125928+0000","last_change":"2024-03-04T10:09:16.131461+0000","last_active":"2024-03-04T10:09:17.125928+0000","last_peered":"2024-03-04T10:09:17.125928+0000","last_clean":"2024-03-04T10:09:17.125928+0000","last_became_active":"2024-03-04T10:09:16.131215+0000","last_became_peered":"2024-03-04T10:09:16.131215+0000","last_unstale":"2024-03-04T10:09:17.125928+0000","last_undegraded":"2024-03-04T10:09:17.125928+0000","last_fullsized":"2024-03-04T10:09:17.125928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:36.993350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.151718+0000","last_change":"2024-03-04T10:09:15.129009+0000","last_active":"2024-03-04T10:09:16.151718+0000","last_peered":"2024-03-04T10:09:16.151718+0000","last_clean":"2024-03-04T10:09:16.151718+0000","last_became_active":"2024-03-04T10:09:15.128725+0000","last_became_peered":"2024-03-04T10:09:15.128725+0000","last_unstale":"2024-03-04T10:09:16.151718+0000","last_undegraded":"2024-03-04T10:09:16.151718+0000","last_fullsized":"2024-03-04T10:09:16.151718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:05.183540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142193+0000","last_change":"2024-03-04T10:09:16.136935+0000","last_active":"2024-03-04T10:09:17.142193+0000","last_peered":"2024-03-04T10:09:17.142193+0000","last_clean":"2024-03-04T10:09:17.142193+0000","last_became_active":"2024-03-04T10:09:16.136694+0000","last_became_peered":"2024-03-04T10:09:16.136694+0000","last_unstale":"2024-03-04T10:09:17.142193+0000","last_undegraded":"2024-03-04T10:09:17.142193+0000","last_fullsized":"2024-03-04T10:09:17.142193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:27:19.604502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125856+0000","last_change":"2024-03-04T10:09:15.124416+0000","last_active":"2024-03-04T10:09:17.125856+0000","last_peered":"2024-03-04T10:09:17.125856+0000","last_clean":"2024-03-04T10:09:17.125856+0000","last_became_active":"2024-03-04T10:09:15.124162+0000","last_became_peered":"2024-03-04T10:09:15.124162+0000","last_unstale":"2024-03-04T10:09:17.125856+0000","last_undegraded":"2024-03-04T10:09:17.125856+0000","last_fullsized":"2024-03-04T10:09:17.125856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:27:33.295064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142173+0000","last_change":"2024-03-04T10:09:16.137070+0000","last_active":"2024-03-04T10:09:17.142173+0000","last_peered":"2024-03-04T10:09:17.142173+0000","last_clean":"2024-03-04T10:09:17.142173+0000","last_became_active":"2024-03-04T10:09:16.136495+0000","last_became_peered":"2024-03-04T10:09:16.136495+0000","last_unstale":"2024-03-04T10:09:17.142173+0000","last_undegraded":"2024-03-04T10:09:17.142173+0000","last_fullsized":"2024-03-04T10:09:17.142173+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:06:42.956801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"80'1","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.149664+0000","last_change":"2024-03-04T10:09:15.130895+0000","last_active":"2024-03-04T10:09:16.149664+0000","last_peered":"2024-03-04T10:09:16.149664+0000","last_clean":"2024-03-04T10:09:16.149664+0000","last_became_active":"2024-03-04T10:09:15.130613+0000","last_became_peered":"2024-03-04T10:09:15.130613+0000","last_unstale":"2024-03-04T10:09:16.149664+0000","last_undegraded":"2024-03-04T10:09:16.149664+0000","last_fullsized":"2024-03-04T10:09:16.149664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:33:31.183254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126281+0000","last_change":"2024-03-04T10:09:16.136451+0000","last_active":"2024-03-04T10:09:17.126281+0000","last_peered":"2024-03-04T10:09:17.126281+0000","last_clean":"2024-03-04T10:09:17.126281+0000","last_became_active":"2024-03-04T10:09:16.135597+0000","last_became_peered":"2024-03-04T10:09:16.135597+0000","last_unstale":"2024-03-04T10:09:17.126281+0000","last_undegraded":"2024-03-04T10:09:17.126281+0000","last_fullsized":"2024-03-04T10:09:17.126281+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:29.821425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142002+0000","last_change":"2024-03-04T10:09:15.131823+0000","last_active":"2024-03-04T10:09:17.142002+0000","last_peered":"2024-03-04T10:09:17.142002+0000","last_clean":"2024-03-04T10:09:17.142002+0000","last_became_active":"2024-03-04T10:09:15.131531+0000","last_became_peered":"2024-03-04T10:09:15.131531+0000","last_unstale":"2024-03-04T10:09:17.142002+0000","last_undegraded":"2024-03-04T10:09:17.142002+0000","last_fullsized":"2024-03-04T10:09:17.142002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:05.910856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143772+0000","last_change":"2024-03-04T10:09:16.142474+0000","last_active":"2024-03-04T10:09:16.143772+0000","last_peered":"2024-03-04T10:09:16.143772+0000","last_clean":"2024-03-04T10:09:16.143772+0000","last_became_active":"2024-03-04T10:09:16.141870+0000","last_became_peered":"2024-03-04T10:09:16.141870+0000","last_unstale":"2024-03-04T10:09:16.143772+0000","last_undegraded":"2024-03-04T10:09:16.143772+0000","last_fullsized":"2024-03-04T10:09:16.143772+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:34.385257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144505+0000","last_change":"2024-03-04T10:09:15.123717+0000","last_active":"2024-03-04T10:09:17.144505+0000","last_peered":"2024-03-04T10:09:17.144505+0000","last_clean":"2024-03-04T10:09:17.144505+0000","last_became_active":"2024-03-04T10:09:15.123440+0000","last_became_peered":"2024-03-04T10:09:15.123440+0000","last_unstale":"2024-03-04T10:09:17.144505+0000","last_undegraded":"2024-03-04T10:09:17.144505+0000","last_fullsized":"2024-03-04T10:09:17.144505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:07:22.817059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141832+0000","last_change":"2024-03-04T10:09:16.132861+0000","last_active":"2024-03-04T10:09:17.141832+0000","last_peered":"2024-03-04T10:09:17.141832+0000","last_clean":"2024-03-04T10:09:17.141832+0000","last_became_active":"2024-03-04T10:09:16.132613+0000","last_became_peered":"2024-03-04T10:09:16.132613+0000","last_unstale":"2024-03-04T10:09:17.141832+0000","last_undegraded":"2024-03-04T10:09:17.141832+0000","last_fullsized":"2024-03-04T10:09:17.141832+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:19.677891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126197+0000","last_change":"2024-03-04T10:09:15.123229+0000","last_active":"2024-03-04T10:09:17.126197+0000","last_peered":"2024-03-04T10:09:17.126197+0000","last_clean":"2024-03-04T10:09:17.126197+0000","last_became_active":"2024-03-04T10:09:15.122955+0000","last_became_peered":"2024-03-04T10:09:15.122955+0000","last_unstale":"2024-03-04T10:09:17.126197+0000","last_undegraded":"2024-03-04T10:09:17.126197+0000","last_fullsized":"2024-03-04T10:09:17.126197+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:11.142337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141337+0000","last_change":"2024-03-04T10:09:16.138612+0000","last_active":"2024-03-04T10:09:16.141337+0000","last_peered":"2024-03-04T10:09:16.141337+0000","last_clean":"2024-03-04T10:09:16.141337+0000","last_became_active":"2024-03-04T10:09:16.137532+0000","last_became_peered":"2024-03-04T10:09:16.137532+0000","last_unstale":"2024-03-04T10:09:16.141337+0000","last_undegraded":"2024-03-04T10:09:16.141337+0000","last_fullsized":"2024-03-04T10:09:16.141337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:27:15.262941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.130467+0000","last_change":"2024-03-04T10:09:15.128123+0000","last_active":"2024-03-04T10:09:16.130467+0000","last_peered":"2024-03-04T10:09:16.130467+0000","last_clean":"2024-03-04T10:09:16.130467+0000","last_became_active":"2024-03-04T10:09:15.127846+0000","last_became_peered":"2024-03-04T10:09:15.127846+0000","last_unstale":"2024-03-04T10:09:16.130467+0000","last_undegraded":"2024-03-04T10:09:16.130467+0000","last_fullsized":"2024-03-04T10:09:16.130467+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:05:56.937737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147130+0000","last_change":"2024-03-04T10:09:16.143471+0000","last_active":"2024-03-04T10:09:16.147130+0000","last_peered":"2024-03-04T10:09:16.147130+0000","last_clean":"2024-03-04T10:09:16.147130+0000","last_became_active":"2024-03-04T10:09:16.143227+0000","last_became_peered":"2024-03-04T10:09:16.143227+0000","last_unstale":"2024-03-04T10:09:16.147130+0000","last_undegraded":"2024-03-04T10:09:16.147130+0000","last_fullsized":"2024-03-04T10:09:16.147130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:09.154804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"80'3","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145078+0000","last_change":"2024-03-04T10:09:15.130776+0000","last_active":"2024-03-04T10:09:17.145078+0000","last_peered":"2024-03-04T10:09:17.145078+0000","last_clean":"2024-03-04T10:09:17.145078+0000","last_became_active":"2024-03-04T10:09:15.130496+0000","last_became_peered":"2024-03-04T10:09:15.130496+0000","last_unstale":"2024-03-04T10:09:17.145078+0000","last_undegraded":"2024-03-04T10:09:17.145078+0000","last_fullsized":"2024-03-04T10:09:17.145078+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:42.989302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126728+0000","last_change":"2024-03-04T10:09:16.134170+0000","last_active":"2024-03-04T10:09:17.126728+0000","last_peered":"2024-03-04T10:09:17.126728+0000","last_clean":"2024-03-04T10:09:17.126728+0000","last_became_active":"2024-03-04T10:09:16.133921+0000","last_became_peered":"2024-03-04T10:09:16.133921+0000","last_unstale":"2024-03-04T10:09:17.126728+0000","last_undegraded":"2024-03-04T10:09:17.126728+0000","last_fullsized":"2024-03-04T10:09:17.126728+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:09:20.701215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"80'6","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.156334+0000","last_change":"2024-03-04T10:09:15.124775+0000","last_active":"2024-03-04T10:09:16.156334+0000","last_peered":"2024-03-04T10:09:16.156334+0000","last_clean":"2024-03-04T10:09:16.156334+0000","last_became_active":"2024-03-04T10:09:15.124512+0000","last_became_peered":"2024-03-04T10:09:15.124512+0000","last_unstale":"2024-03-04T10:09:16.156334+0000","last_undegraded":"2024-03-04T10:09:16.156334+0000","last_fullsized":"2024-03-04T10:09:16.156334+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:38.590456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143230+0000","last_change":"2024-03-04T10:09:16.139541+0000","last_active":"2024-03-04T10:09:16.143230+0000","last_peered":"2024-03-04T10:09:16.143230+0000","last_clean":"2024-03-04T10:09:16.143230+0000","last_became_active":"2024-03-04T10:09:16.139240+0000","last_became_peered":"2024-03-04T10:09:16.139240+0000","last_unstale":"2024-03-04T10:09:16.143230+0000","last_undegraded":"2024-03-04T10:09:16.143230+0000","last_fullsized":"2024-03-04T10:09:16.143230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:19:45.017069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.152162+0000","last_change":"2024-03-04T10:09:15.129717+0000","last_active":"2024-03-04T10:09:16.152162+0000","last_peered":"2024-03-04T10:09:16.152162+0000","last_clean":"2024-03-04T10:09:16.152162+0000","last_became_active":"2024-03-04T10:09:15.129440+0000","last_became_peered":"2024-03-04T10:09:15.129440+0000","last_unstale":"2024-03-04T10:09:16.152162+0000","last_undegraded":"2024-03-04T10:09:16.152162+0000","last_fullsized":"2024-03-04T10:09:16.152162+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:13:57.190223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143801+0000","last_change":"2024-03-04T10:09:16.142514+0000","last_active":"2024-03-04T10:09:16.143801+0000","last_peered":"2024-03-04T10:09:16.143801+0000","last_clean":"2024-03-04T10:09:16.143801+0000","last_became_active":"2024-03-04T10:09:16.142181+0000","last_became_peered":"2024-03-04T10:09:16.142181+0000","last_unstale":"2024-03-04T10:09:16.143801+0000","last_undegraded":"2024-03-04T10:09:16.143801+0000","last_fullsized":"2024-03-04T10:09:16.143801+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:55:48.740679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125338+0000","last_change":"2024-03-04T10:09:15.130215+0000","last_active":"2024-03-04T10:09:17.125338+0000","last_peered":"2024-03-04T10:09:17.125338+0000","last_clean":"2024-03-04T10:09:17.125338+0000","last_became_active":"2024-03-04T10:09:15.129641+0000","last_became_peered":"2024-03-04T10:09:15.129641+0000","last_unstale":"2024-03-04T10:09:17.125338+0000","last_undegraded":"2024-03-04T10:09:17.125338+0000","last_fullsized":"2024-03-04T10:09:17.125338+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:27:14.254579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.141918+0000","last_change":"2024-03-04T10:09:16.141035+0000","last_active":"2024-03-04T10:09:16.141918+0000","last_peered":"2024-03-04T10:09:16.141918+0000","last_clean":"2024-03-04T10:09:16.141918+0000","last_became_active":"2024-03-04T10:09:16.140725+0000","last_became_peered":"2024-03-04T10:09:16.140725+0000","last_unstale":"2024-03-04T10:09:16.141918+0000","last_undegraded":"2024-03-04T10:09:16.141918+0000","last_fullsized":"2024-03-04T10:09:16.141918+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:16:40.650752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144543+0000","last_change":"2024-03-04T10:09:15.124919+0000","last_active":"2024-03-04T10:09:17.144543+0000","last_peered":"2024-03-04T10:09:17.144543+0000","last_clean":"2024-03-04T10:09:17.144543+0000","last_became_active":"2024-03-04T10:09:15.124651+0000","last_became_peered":"2024-03-04T10:09:15.124651+0000","last_unstale":"2024-03-04T10:09:17.144543+0000","last_undegraded":"2024-03-04T10:09:17.144543+0000","last_fullsized":"2024-03-04T10:09:17.144543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:26.630830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.151524+0000","last_change":"2024-03-04T10:09:16.148151+0000","last_active":"2024-03-04T10:09:16.151524+0000","last_peered":"2024-03-04T10:09:16.151524+0000","last_clean":"2024-03-04T10:09:16.151524+0000","last_became_active":"2024-03-04T10:09:16.147891+0000","last_became_peered":"2024-03-04T10:09:16.147891+0000","last_unstale":"2024-03-04T10:09:16.151524+0000","last_undegraded":"2024-03-04T10:09:16.151524+0000","last_fullsized":"2024-03-04T10:09:16.151524+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:27:24.457550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"80'8","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.141947+0000","last_change":"2024-03-04T10:09:15.124980+0000","last_active":"2024-03-04T10:09:17.141947+0000","last_peered":"2024-03-04T10:09:17.141947+0000","last_clean":"2024-03-04T10:09:17.141947+0000","last_became_active":"2024-03-04T10:09:15.124544+0000","last_became_peered":"2024-03-04T10:09:15.124544+0000","last_unstale":"2024-03-04T10:09:17.141947+0000","last_undegraded":"2024-03-04T10:09:17.141947+0000","last_fullsized":"2024-03-04T10:09:17.141947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:36.947844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147166+0000","last_change":"2024-03-04T10:09:16.139831+0000","last_active":"2024-03-04T10:09:16.147166+0000","last_peered":"2024-03-04T10:09:16.147166+0000","last_clean":"2024-03-04T10:09:16.147166+0000","last_became_active":"2024-03-04T10:09:16.139553+0000","last_became_peered":"2024-03-04T10:09:16.139553+0000","last_unstale":"2024-03-04T10:09:16.147166+0000","last_undegraded":"2024-03-04T10:09:16.147166+0000","last_fullsized":"2024-03-04T10:09:16.147166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:10:31.412484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126230+0000","last_change":"2024-03-04T10:09:15.124284+0000","last_active":"2024-03-04T10:09:17.126230+0000","last_peered":"2024-03-04T10:09:17.126230+0000","last_clean":"2024-03-04T10:09:17.126230+0000","last_became_active":"2024-03-04T10:09:15.123547+0000","last_became_peered":"2024-03-04T10:09:15.123547+0000","last_unstale":"2024-03-04T10:09:17.126230+0000","last_undegraded":"2024-03-04T10:09:17.126230+0000","last_fullsized":"2024-03-04T10:09:17.126230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:28:53.036069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144512+0000","last_change":"2024-03-04T10:09:16.140894+0000","last_active":"2024-03-04T10:09:17.144512+0000","last_peered":"2024-03-04T10:09:17.144512+0000","last_clean":"2024-03-04T10:09:17.144512+0000","last_became_active":"2024-03-04T10:09:16.140665+0000","last_became_peered":"2024-03-04T10:09:16.140665+0000","last_unstale":"2024-03-04T10:09:17.144512+0000","last_undegraded":"2024-03-04T10:09:17.144512+0000","last_fullsized":"2024-03-04T10:09:17.144512+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:23:08.844499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"80'2","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126296+0000","last_change":"2024-03-04T10:09:15.123158+0000","last_active":"2024-03-04T10:09:17.126296+0000","last_peered":"2024-03-04T10:09:17.126296+0000","last_clean":"2024-03-04T10:09:17.126296+0000","last_became_active":"2024-03-04T10:09:15.122878+0000","last_became_peered":"2024-03-04T10:09:15.122878+0000","last_unstale":"2024-03-04T10:09:17.126296+0000","last_undegraded":"2024-03-04T10:09:17.126296+0000","last_fullsized":"2024-03-04T10:09:17.126296+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:18:04.283634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144689+0000","last_change":"2024-03-04T10:09:16.136749+0000","last_active":"2024-03-04T10:09:17.144689+0000","last_peered":"2024-03-04T10:09:17.144689+0000","last_clean":"2024-03-04T10:09:17.144689+0000","last_became_active":"2024-03-04T10:09:16.136518+0000","last_became_peered":"2024-03-04T10:09:16.136518+0000","last_unstale":"2024-03-04T10:09:17.144689+0000","last_undegraded":"2024-03-04T10:09:17.144689+0000","last_fullsized":"2024-03-04T10:09:17.144689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:05:49.274416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"80'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.145102+0000","last_change":"2024-03-04T10:09:15.130358+0000","last_active":"2024-03-04T10:09:17.145102+0000","last_peered":"2024-03-04T10:09:17.145102+0000","last_clean":"2024-03-04T10:09:17.145102+0000","last_became_active":"2024-03-04T10:09:15.129963+0000","last_became_peered":"2024-03-04T10:09:15.129963+0000","last_unstale":"2024-03-04T10:09:17.145102+0000","last_undegraded":"2024-03-04T10:09:17.145102+0000","last_fullsized":"2024-03-04T10:09:17.145102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:10.151841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.147106+0000","last_change":"2024-03-04T10:09:16.141675+0000","last_active":"2024-03-04T10:09:16.147106+0000","last_peered":"2024-03-04T10:09:16.147106+0000","last_clean":"2024-03-04T10:09:16.147106+0000","last_became_active":"2024-03-04T10:09:16.141234+0000","last_became_peered":"2024-03-04T10:09:16.141234+0000","last_unstale":"2024-03-04T10:09:16.147106+0000","last_undegraded":"2024-03-04T10:09:16.147106+0000","last_fullsized":"2024-03-04T10:09:16.147106+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:57:54.587735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126527+0000","last_change":"2024-03-04T10:09:15.126009+0000","last_active":"2024-03-04T10:09:17.126527+0000","last_peered":"2024-03-04T10:09:17.126527+0000","last_clean":"2024-03-04T10:09:17.126527+0000","last_became_active":"2024-03-04T10:09:15.125752+0000","last_became_peered":"2024-03-04T10:09:15.125752+0000","last_unstale":"2024-03-04T10:09:17.126527+0000","last_undegraded":"2024-03-04T10:09:17.126527+0000","last_fullsized":"2024-03-04T10:09:17.126527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:05.183540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142302+0000","last_change":"2024-03-04T10:09:16.135091+0000","last_active":"2024-03-04T10:09:17.142302+0000","last_peered":"2024-03-04T10:09:17.142302+0000","last_clean":"2024-03-04T10:09:17.142302+0000","last_became_active":"2024-03-04T10:09:16.134821+0000","last_became_peered":"2024-03-04T10:09:16.134821+0000","last_unstale":"2024-03-04T10:09:17.142302+0000","last_undegraded":"2024-03-04T10:09:17.142302+0000","last_fullsized":"2024-03-04T10:09:17.142302+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:31:40.894952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.151128+0000","last_change":"2024-03-04T10:09:15.126865+0000","last_active":"2024-03-04T10:09:16.151128+0000","last_peered":"2024-03-04T10:09:16.151128+0000","last_clean":"2024-03-04T10:09:16.151128+0000","last_became_active":"2024-03-04T10:09:15.126596+0000","last_became_peered":"2024-03-04T10:09:15.126596+0000","last_unstale":"2024-03-04T10:09:16.151128+0000","last_undegraded":"2024-03-04T10:09:16.151128+0000","last_fullsized":"2024-03-04T10:09:16.151128+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:36.947844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142332+0000","last_change":"2024-03-04T10:09:16.141939+0000","last_active":"2024-03-04T10:09:17.142332+0000","last_peered":"2024-03-04T10:09:17.142332+0000","last_clean":"2024-03-04T10:09:17.142332+0000","last_became_active":"2024-03-04T10:09:16.141672+0000","last_became_peered":"2024-03-04T10:09:16.141672+0000","last_unstale":"2024-03-04T10:09:17.142332+0000","last_undegraded":"2024-03-04T10:09:17.142332+0000","last_fullsized":"2024-03-04T10:09:17.142332+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:26.391457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.125941+0000","last_change":"2024-03-04T10:09:15.130978+0000","last_active":"2024-03-04T10:09:16.125941+0000","last_peered":"2024-03-04T10:09:16.125941+0000","last_clean":"2024-03-04T10:09:16.125941+0000","last_became_active":"2024-03-04T10:09:15.130697+0000","last_became_peered":"2024-03-04T10:09:15.130697+0000","last_unstale":"2024-03-04T10:09:16.125941+0000","last_undegraded":"2024-03-04T10:09:16.125941+0000","last_fullsized":"2024-03-04T10:09:16.125941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:28:53.036069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.140631+0000","last_change":"2024-03-04T10:09:16.136498+0000","last_active":"2024-03-04T10:09:16.140631+0000","last_peered":"2024-03-04T10:09:16.140631+0000","last_clean":"2024-03-04T10:09:16.140631+0000","last_became_active":"2024-03-04T10:09:16.136172+0000","last_became_peered":"2024-03-04T10:09:16.136172+0000","last_unstale":"2024-03-04T10:09:16.140631+0000","last_undegraded":"2024-03-04T10:09:16.140631+0000","last_fullsized":"2024-03-04T10:09:16.140631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:58.743366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.125840+0000","last_change":"2024-03-04T10:09:15.123560+0000","last_active":"2024-03-04T10:09:16.125840+0000","last_peered":"2024-03-04T10:09:16.125840+0000","last_clean":"2024-03-04T10:09:16.125840+0000","last_became_active":"2024-03-04T10:09:15.123310+0000","last_became_peered":"2024-03-04T10:09:15.123310+0000","last_unstale":"2024-03-04T10:09:16.125840+0000","last_undegraded":"2024-03-04T10:09:16.125840+0000","last_fullsized":"2024-03-04T10:09:16.125840+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:11.142337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125205+0000","last_change":"2024-03-04T10:09:16.136905+0000","last_active":"2024-03-04T10:09:17.125205+0000","last_peered":"2024-03-04T10:09:17.125205+0000","last_clean":"2024-03-04T10:09:17.125205+0000","last_became_active":"2024-03-04T10:09:16.135743+0000","last_became_peered":"2024-03-04T10:09:16.135743+0000","last_unstale":"2024-03-04T10:09:17.125205+0000","last_undegraded":"2024-03-04T10:09:17.125205+0000","last_fullsized":"2024-03-04T10:09:17.125205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:16:40.650752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"80'2","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.150619+0000","last_change":"2024-03-04T10:09:15.126337+0000","last_active":"2024-03-04T10:09:16.150619+0000","last_peered":"2024-03-04T10:09:16.150619+0000","last_clean":"2024-03-04T10:09:16.150619+0000","last_became_active":"2024-03-04T10:09:15.126056+0000","last_became_peered":"2024-03-04T10:09:15.126056+0000","last_unstale":"2024-03-04T10:09:16.150619+0000","last_undegraded":"2024-03-04T10:09:16.150619+0000","last_fullsized":"2024-03-04T10:09:16.150619+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:42:49.214205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.149420+0000","last_change":"2024-03-04T10:09:16.146918+0000","last_active":"2024-03-04T10:09:16.149420+0000","last_peered":"2024-03-04T10:09:16.149420+0000","last_clean":"2024-03-04T10:09:16.149420+0000","last_became_active":"2024-03-04T10:09:16.145785+0000","last_became_peered":"2024-03-04T10:09:16.145785+0000","last_unstale":"2024-03-04T10:09:16.149420+0000","last_undegraded":"2024-03-04T10:09:16.149420+0000","last_fullsized":"2024-03-04T10:09:16.149420+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:59:29.821425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.143478+0000","last_change":"2024-03-04T10:09:15.129061+0000","last_active":"2024-03-04T10:09:17.143478+0000","last_peered":"2024-03-04T10:09:17.143478+0000","last_clean":"2024-03-04T10:09:17.143478+0000","last_became_active":"2024-03-04T10:09:15.128761+0000","last_became_peered":"2024-03-04T10:09:15.128761+0000","last_unstale":"2024-03-04T10:09:17.143478+0000","last_undegraded":"2024-03-04T10:09:17.143478+0000","last_fullsized":"2024-03-04T10:09:17.143478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:50.298152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125426+0000","last_change":"2024-03-04T10:09:16.138463+0000","last_active":"2024-03-04T10:09:17.125426+0000","last_peered":"2024-03-04T10:09:17.125426+0000","last_clean":"2024-03-04T10:09:17.125426+0000","last_became_active":"2024-03-04T10:09:16.138003+0000","last_became_peered":"2024-03-04T10:09:16.138003+0000","last_unstale":"2024-03-04T10:09:17.125426+0000","last_undegraded":"2024-03-04T10:09:17.125426+0000","last_fullsized":"2024-03-04T10:09:17.125426+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:12:34.385257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126153+0000","last_change":"2024-03-04T10:09:15.128537+0000","last_active":"2024-03-04T10:09:17.126153+0000","last_peered":"2024-03-04T10:09:17.126153+0000","last_clean":"2024-03-04T10:09:17.126153+0000","last_became_active":"2024-03-04T10:09:15.128009+0000","last_became_peered":"2024-03-04T10:09:15.128009+0000","last_unstale":"2024-03-04T10:09:17.126153+0000","last_undegraded":"2024-03-04T10:09:17.126153+0000","last_fullsized":"2024-03-04T10:09:17.126153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:05.910856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.142352+0000","last_change":"2024-03-04T10:09:16.144312+0000","last_active":"2024-03-04T10:09:17.142352+0000","last_peered":"2024-03-04T10:09:17.142352+0000","last_clean":"2024-03-04T10:09:17.142352+0000","last_became_active":"2024-03-04T10:09:16.143548+0000","last_became_peered":"2024-03-04T10:09:16.143548+0000","last_unstale":"2024-03-04T10:09:17.142352+0000","last_undegraded":"2024-03-04T10:09:17.142352+0000","last_fullsized":"2024-03-04T10:09:17.142352+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:43:22.542989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125783+0000","last_change":"2024-03-04T10:09:15.122698+0000","last_active":"2024-03-04T10:09:17.125783+0000","last_peered":"2024-03-04T10:09:17.125783+0000","last_clean":"2024-03-04T10:09:17.125783+0000","last_became_active":"2024-03-04T10:09:15.122183+0000","last_became_peered":"2024-03-04T10:09:15.122183+0000","last_unstale":"2024-03-04T10:09:17.125783+0000","last_undegraded":"2024-03-04T10:09:17.125783+0000","last_fullsized":"2024-03-04T10:09:17.125783+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:19:44.008707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.149335+0000","last_change":"2024-03-04T10:09:16.146099+0000","last_active":"2024-03-04T10:09:16.149335+0000","last_peered":"2024-03-04T10:09:16.149335+0000","last_clean":"2024-03-04T10:09:16.149335+0000","last_became_active":"2024-03-04T10:09:16.145802+0000","last_became_peered":"2024-03-04T10:09:16.145802+0000","last_unstale":"2024-03-04T10:09:16.149335+0000","last_undegraded":"2024-03-04T10:09:16.149335+0000","last_fullsized":"2024-03-04T10:09:16.149335+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:36.993350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.132747+0000","last_change":"2024-03-04T10:09:15.127214+0000","last_active":"2024-03-04T10:09:16.132747+0000","last_peered":"2024-03-04T10:09:16.132747+0000","last_clean":"2024-03-04T10:09:16.132747+0000","last_became_active":"2024-03-04T10:09:15.126943+0000","last_became_peered":"2024-03-04T10:09:15.126943+0000","last_unstale":"2024-03-04T10:09:16.132747+0000","last_undegraded":"2024-03-04T10:09:16.132747+0000","last_fullsized":"2024-03-04T10:09:16.132747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:08.865710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.143821+0000","last_change":"2024-03-04T10:09:16.142557+0000","last_active":"2024-03-04T10:09:16.143821+0000","last_peered":"2024-03-04T10:09:16.143821+0000","last_clean":"2024-03-04T10:09:16.143821+0000","last_became_active":"2024-03-04T10:09:16.142198+0000","last_became_peered":"2024-03-04T10:09:16.142198+0000","last_unstale":"2024-03-04T10:09:16.143821+0000","last_undegraded":"2024-03-04T10:09:16.143821+0000","last_fullsized":"2024-03-04T10:09:16.143821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:30.718603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.144774+0000","last_change":"2024-03-04T10:09:15.124503+0000","last_active":"2024-03-04T10:09:17.144774+0000","last_peered":"2024-03-04T10:09:17.144774+0000","last_clean":"2024-03-04T10:09:17.144774+0000","last_became_active":"2024-03-04T10:09:15.124077+0000","last_became_peered":"2024-03-04T10:09:15.124077+0000","last_unstale":"2024-03-04T10:09:17.144774+0000","last_undegraded":"2024-03-04T10:09:17.144774+0000","last_fullsized":"2024-03-04T10:09:17.144774+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:05:20.795272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126758+0000","last_change":"2024-03-04T10:09:16.138400+0000","last_active":"2024-03-04T10:09:17.126758+0000","last_peered":"2024-03-04T10:09:17.126758+0000","last_clean":"2024-03-04T10:09:17.126758+0000","last_became_active":"2024-03-04T10:09:16.137926+0000","last_became_peered":"2024-03-04T10:09:16.137926+0000","last_unstale":"2024-03-04T10:09:17.126758+0000","last_undegraded":"2024-03-04T10:09:17.126758+0000","last_fullsized":"2024-03-04T10:09:17.126758+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:05:02.026552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.133310+0000","last_change":"2024-03-04T10:09:15.131615+0000","last_active":"2024-03-04T10:09:16.133310+0000","last_peered":"2024-03-04T10:09:16.133310+0000","last_clean":"2024-03-04T10:09:16.133310+0000","last_became_active":"2024-03-04T10:09:15.131360+0000","last_became_peered":"2024-03-04T10:09:15.131360+0000","last_unstale":"2024-03-04T10:09:16.133310+0000","last_undegraded":"2024-03-04T10:09:16.133310+0000","last_fullsized":"2024-03-04T10:09:16.133310+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:11.359273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126767+0000","last_change":"2024-03-04T10:09:16.128352+0000","last_active":"2024-03-04T10:09:17.126767+0000","last_peered":"2024-03-04T10:09:17.126767+0000","last_clean":"2024-03-04T10:09:17.126767+0000","last_became_active":"2024-03-04T10:09:16.128046+0000","last_became_peered":"2024-03-04T10:09:16.128046+0000","last_unstale":"2024-03-04T10:09:17.126767+0000","last_undegraded":"2024-03-04T10:09:17.126767+0000","last_fullsized":"2024-03-04T10:09:17.126767+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:07:34.410941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126499+0000","last_change":"2024-03-04T10:09:15.125545+0000","last_active":"2024-03-04T10:09:17.126499+0000","last_peered":"2024-03-04T10:09:17.126499+0000","last_clean":"2024-03-04T10:09:17.126499+0000","last_became_active":"2024-03-04T10:09:15.125304+0000","last_became_peered":"2024-03-04T10:09:15.125304+0000","last_unstale":"2024-03-04T10:09:17.126499+0000","last_undegraded":"2024-03-04T10:09:17.126499+0000","last_fullsized":"2024-03-04T10:09:17.126499+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:25:11.359273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.126044+0000","last_change":"2024-03-04T10:09:16.130150+0000","last_active":"2024-03-04T10:09:17.126044+0000","last_peered":"2024-03-04T10:09:17.126044+0000","last_clean":"2024-03-04T10:09:17.126044+0000","last_became_active":"2024-03-04T10:09:16.129868+0000","last_became_peered":"2024-03-04T10:09:16.129868+0000","last_unstale":"2024-03-04T10:09:17.126044+0000","last_undegraded":"2024-03-04T10:09:17.126044+0000","last_fullsized":"2024-03-04T10:09:17.126044+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:15.105552+0000","last_clean_scrub_stamp":"2024-03-04T10:09:15.105552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:34:42.938735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-04T10:09:16.128800+0000","last_change":"2024-03-04T10:09:15.130632+0000","last_active":"2024-03-04T10:09:16.128800+0000","last_peered":"2024-03-04T10:09:16.128800+0000","last_clean":"2024-03-04T10:09:16.128800+0000","last_became_active":"2024-03-04T10:09:15.130215+0000","last_became_peered":"2024-03-04T10:09:15.130215+0000","last_unstale":"2024-03-04T10:09:16.128800+0000","last_undegraded":"2024-03-04T10:09:16.128800+0000","last_fullsized":"2024-03-04T10:09:16.128800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:09:14.097190+0000","last_clean_scrub_stamp":"2024-03-04T10:09:14.097190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:05.910856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":294,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-04T10:09:17.125617+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:09:17.125617+0000","last_peered":"2024-03-04T10:09:17.125617+0000","last_clean":"2024-03-04T10:09:17.125617+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:09:17.125617+0000","last_undegraded":"2024-03-04T10:09:17.125617+0000","last_fullsized":"2024-03-04T10:09:17.125617+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1044021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":991232,"data_stored":455243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574922,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":453665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":999424,"data_stored":455347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":454213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":1003520,"data_stored":461065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":987136,"data_stored":453743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28568,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343272,"statfs":{"total":96636764160,"available":96607510528,"internally_reserved":0,"allocated":1572864,"data_stored":1043999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:09:19.440 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T10:09:19.441 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T10:09:19.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.584+0000 7f72bb627640 1 Processor -- start 2024-03-04T10:09:19.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.584+0000 7f72bb627640 1 -- start start 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72bb627640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f72b40cdd70 0x7f72b40cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72bb627640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f72b40ccfe0 0x7f72b40cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72bb627640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b406ae00 con 0x7f72b40cc900 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72bb627640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40ccde0 con 0x7f72b40ccfe0 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72bb627640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40d0480 con 0x7f72b40cdd70 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f72b40ccfe0 0x7f72b40cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f72b40ccfe0 0x7f72b40cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33562/0 (socket says 172.21.15.99:33562) 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 -- 172.21.15.99:0/696475042 learned_addr learned my addr 172.21.15.99:0/696475042 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b93c3640 1 --2- 172.21.15.99:0/696475042 >> v2:172.21.15.162:3300/0 conn(0x7f72b40cdd70 0x7f72b40cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 -- 172.21.15.99:0/696475042 >> v2:172.21.15.162:3300/0 conn(0x7f72b40cdd70 msgr2=0x7f72b40cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:19.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 --2- 172.21.15.99:0/696475042 >> v2:172.21.15.162:3300/0 conn(0x7f72b40cdd70 0x7f72b40cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 -- 172.21.15.99:0/696475042 >> v1:172.21.15.162:6789/0 conn(0x7f72b40cc900 legacy=0x7f72b415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.585+0000 7f72b9bc4640 1 -- 172.21.15.99:0/696475042 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b406b0e0 con 0x7f72b40ccfe0 2024-03-04T10:09:19.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.586+0000 7f72b9bc4640 1 --2- 172.21.15.99:0/696475042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40ccfe0 0x7f72b40cff40 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f72b406e420 tx=0x7f72ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1a02649ac12c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.586+0000 7f72abfff640 1 -- 172.21.15.99:0/696475042 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac035020 con 0x7f72b40ccfe0 2024-03-04T10:09:19.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.586+0000 7f72abfff640 1 -- 172.21.15.99:0/696475042 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac031870 con 0x7f72b40ccfe0 2024-03-04T10:09:19.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.586+0000 7f72abfff640 1 -- 172.21.15.99:0/696475042 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac031c40 con 0x7f72b40ccfe0 2024-03-04T10:09:19.589 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 -- 172.21.15.99:0/696475042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40ccfe0 msgr2=0x7f72b40cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 --2- 172.21.15.99:0/696475042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40ccfe0 0x7f72b40cff40 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f72b406e420 tx=0x7f72ac02b910 comp rx=0 tx=0).stop 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 -- 172.21.15.99:0/696475042 shutdown_connections 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 --2- 172.21.15.99:0/696475042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40ccfe0 0x7f72b40cff40 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 --2- 172.21.15.99:0/696475042 >> v2:172.21.15.162:3300/0 conn(0x7f72b40cdd70 0x7f72b40cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 -- 172.21.15.99:0/696475042 >> 172.21.15.99:0/696475042 conn(0x7f72b40b22e0 msgr2=0x7f72b40b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 -- 172.21.15.99:0/696475042 shutdown_connections 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.587+0000 7f72bb627640 1 -- 172.21.15.99:0/696475042 wait complete. 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.588+0000 7f72bb627640 1 Processor -- start 2024-03-04T10:09:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.588+0000 7f72bb627640 1 -- start start 2024-03-04T10:09:19.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72bb627640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 0x7f72b40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72bb627640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72b40ccfe0 0x7f72b415c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72bb627640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72b40cdd70 0x7f72b415cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72bb627640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40b2e90 con 0x7f72b40cc900 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72bb627640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b406b6a0 con 0x7f72b40ccfe0 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72bb627640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b40b3010 con 0x7f72b40cdd70 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 0x7f72b40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 0x7f72b40c7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33566/0 (socket says 172.21.15.99:33566) 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 -- 172.21.15.99:0/2003032720 learned_addr learned my addr 172.21.15.99:0/2003032720 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b8bc2640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72b40ccfe0 0x7f72b415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b9bc4640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72b40cdd70 0x7f72b415cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 -- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72b40cdd70 msgr2=0x7f72b415cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72b40cdd70 0x7f72b415cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 -- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72b40ccfe0 msgr2=0x7f72b415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72b40ccfe0 0x7f72b415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 -- 172.21.15.99:0/2003032720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b41f42a0 con 0x7f72b40cc900 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b8bc2640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72b40ccfe0 0x7f72b415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:19.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b9bc4640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72b40cdd70 0x7f72b415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.589+0000 7f72b93c3640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 0x7f72b40c7080 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f72b0007e40 tx=0x7f72b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.590+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0014620 con 0x7f72b40cc900 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.590+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b0019070 con 0x7f72b40cc900 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.590+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b40c75c0 con 0x7f72b40cc900 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.590+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0014a00 con 0x7f72b40cc900 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.590+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b41f47f0 con 0x7f72b40cc900 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.591+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4074f60 con 0x7f72b40cc900 2024-03-04T10:09:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.591+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f72b0014ba0 con 0x7f72b40cc900 2024-03-04T10:09:19.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.591+0000 7f72a9ffb640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f729806b500 0x7f729806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.592+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72b008ac20 con 0x7f72b40cc900 2024-03-04T10:09:19.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.592+0000 7f72b8bc2640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f729806b500 0x7f729806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.592+0000 7f72b8bc2640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f729806b500 0x7f729806d9c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f72a4007fd0 tx=0x7f72a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.596+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f72b0059910 con 0x7f72b40cc900 2024-03-04T10:09:19.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.832+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f72b40b6710 con 0x7f72b40cc900 2024-03-04T10:09:19.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.835+0000 7f72a9ffb640 1 -- 172.21.15.99:0/2003032720 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f72b0059210 con 0x7f72b40cc900 2024-03-04T10:09:19.836 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-04T10:09:19.836 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:09:19.836 INFO:teuthology.orchestra.run.smithi099.stdout: caps mds = "allow" 2024-03-04T10:09:19.836 INFO:teuthology.orchestra.run.smithi099.stdout: caps mon = "allow r" 2024-03-04T10:09:19.836 INFO:teuthology.orchestra.run.smithi099.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T10:09:19.837 INFO:teuthology.orchestra.run.smithi099.stderr:updated caps for client.0 2024-03-04T10:09:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f729806b500 msgr2=0x7f729806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f729806b500 0x7f729806d9c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f72a4007fd0 tx=0x7f72a4009040 comp rx=0 tx=0).stop 2024-03-04T10:09:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 msgr2=0x7f72b40c7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 0x7f72b40c7080 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f72b0007e40 tx=0x7f72b000e4b0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 shutdown_connections 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f729806b500 0x7f729806d9c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f72b40cdd70 0x7f72b415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72b40ccfe0 0x7f72b415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 --2- 172.21.15.99:0/2003032720 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f72b40cc900 0x7f72b40c7080 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 >> 172.21.15.99:0/2003032720 conn(0x7f72b40b22e0 msgr2=0x7f72b4160c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 shutdown_connections 2024-03-04T10:09:19.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.837+0000 7f72bb627640 1 -- 172.21.15.99:0/2003032720 wait complete. 2024-03-04T10:09:19.849 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:19.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.947+0000 7ffb27353640 1 Processor -- start 2024-03-04T10:09:19.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.947+0000 7ffb27353640 1 -- start start 2024-03-04T10:09:19.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb27353640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb2006d3b0 0x7ffb2015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb27353640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2006d8b0 con 0x7ffb2015b410 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb27353640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2006db90 con 0x7ffb2015b030 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb27353640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb200b2720 con 0x7ffb2006d3b0 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb258f0640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ffb2015b410 0x7ffb2015b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45964/0 (socket says 172.21.15.99:45964) 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb248ee640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb2006d3b0 0x7ffb2015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb258f0640 1 -- 172.21.15.99:0/2251229531 learned_addr learned my addr 172.21.15.99:0/2251229531 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb248ee640 1 -- 172.21.15.99:0/2251229531 >> v1:172.21.15.162:6789/0 conn(0x7ffb2015b030 legacy=0x7ffb2015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb248ee640 1 -- 172.21.15.99:0/2251229531 >> v1:172.21.15.99:6789/0 conn(0x7ffb2015b410 legacy=0x7ffb2015b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb17fff640 1 -- 172.21.15.99:0/2251229531 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464853489 0 0) 0x7ffb2006d8b0 con 0x7ffb2015b410 2024-03-04T10:09:19.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.948+0000 7ffb248ee640 1 -- 172.21.15.99:0/2251229531 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2006de70 con 0x7ffb2006d3b0 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb248ee640 1 --2- 172.21.15.99:0/2251229531 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb2015c120 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ffb100097a0 tx=0x7ffb1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985173c835cd793e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb17fff640 1 -- 172.21.15.99:0/2251229531 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10035020 con 0x7ffb2006d3b0 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb17fff640 1 -- 172.21.15.99:0/2251229531 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb10031830 con 0x7ffb2006d3b0 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb17fff640 1 -- 172.21.15.99:0/2251229531 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10031c00 con 0x7ffb2006d3b0 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb27353640 1 -- 172.21.15.99:0/2251229531 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 msgr2=0x7ffb2015c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb27353640 1 --2- 172.21.15.99:0/2251229531 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb2015c120 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ffb100097a0 tx=0x7ffb1002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb27353640 1 -- 172.21.15.99:0/2251229531 shutdown_connections 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.949+0000 7ffb27353640 1 --2- 172.21.15.99:0/2251229531 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb2015c120 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.950+0000 7ffb27353640 1 -- 172.21.15.99:0/2251229531 >> 172.21.15.99:0/2251229531 conn(0x7ffb2006acd0 msgr2=0x7ffb2006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.950+0000 7ffb27353640 1 -- 172.21.15.99:0/2251229531 shutdown_connections 2024-03-04T10:09:19.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.950+0000 7ffb27353640 1 -- 172.21.15.99:0/2251229531 wait complete. 2024-03-04T10:09:19.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 Processor -- start 2024-03-04T10:09:19.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 -- start start 2024-03-04T10:09:19.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb200ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb2015b030 0x7ffb2014d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb2015b410 0x7ffb201525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2006e810 con 0x7ffb2006d3b0 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2006e690 con 0x7ffb2015b030 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb27353640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200b2bb0 con 0x7ffb2015b410 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb250ef640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb200ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb250ef640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb200ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33578/0 (socket says 172.21.15.99:33578) 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.951+0000 7ffb250ef640 1 -- 172.21.15.99:0/1133343162 learned_addr learned my addr 172.21.15.99:0/1133343162 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb250ef640 1 -- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb2015b410 msgr2=0x7ffb201525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb248ee640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb2015b030 0x7ffb2014d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb258f0640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb2015b410 0x7ffb201525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb250ef640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb2015b410 0x7ffb201525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb250ef640 1 -- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb2015b030 msgr2=0x7ffb2014d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb250ef640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb2015b030 0x7ffb2014d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb250ef640 1 -- 172.21.15.99:0/1133343162 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb200ce7c0 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb250ef640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb200ce280 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c00c450 tx=0x7ffb1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c0052d0 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1c005470 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2014dda0 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c012870 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.952+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb201e2ca0 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.953+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb20074f60 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.953+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffb1c012a10 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.954+0000 7ffb15ffb640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb0406b4d0 0x7ffb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.954+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb1c08a050 con 0x7ffb2006d3b0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.954+0000 7ffb248ee640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb0406b4d0 0x7ffb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:19.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.955+0000 7ffb248ee640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb0406b4d0 0x7ffb0406d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ffb10036000 tx=0x7ffb10008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:19.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:19.958+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffb1c058cb0 con 0x7ffb2006d3b0 2024-03-04T10:09:20.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.162+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb200ae9b0 con 0x7ffb2006d3b0 2024-03-04T10:09:20.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.163+0000 7ffb15ffb640 1 -- 172.21.15.99:0/1133343162 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4946 (secure 0 0 0) 0x7ffb1c0585b0 con 0x7ffb2006d3b0 2024-03-04T10:09:20.165 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:20.165 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:17.472137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:20.165 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 64 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb0406b4d0 msgr2=0x7ffb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb0406b4d0 0x7ffb0406d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ffb10036000 tx=0x7ffb10008030 comp rx=0 tx=0).stop 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 msgr2=0x7ffb200ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb200ce280 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c00c450 tx=0x7ffb1c00c910 comp rx=0 tx=0).stop 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 shutdown_connections 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb0406b4d0 0x7ffb0406d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb2015b410 0x7ffb201525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb2015b030 0x7ffb2014d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 --2- 172.21.15.99:0/1133343162 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb2006d3b0 0x7ffb200ce280 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 >> 172.21.15.99:0/1133343162 conn(0x7ffb2006acd0 msgr2=0x7ffb20159480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 shutdown_connections 2024-03-04T10:09:20.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.165+0000 7ffb27353640 1 -- 172.21.15.99:0/1133343162 wait complete. 2024-03-04T10:09:20.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:09:15.461841+0000', 'modified': '2024-03-04T10:09:17.472137+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7137}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7137': {'gid': 7137, 'name': 'a', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6835/3857383686', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6833', 'nonce': 3857383686}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 3857383686}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:09:20.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:09:20.580 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-04T10:09:20.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.684+0000 7fc59f4fc640 1 Processor -- start 2024-03-04T10:09:20.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.685+0000 7fc59f4fc640 1 -- start start 2024-03-04T10:09:20.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.685+0000 7fc59f4fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc59815dea0 0x7fc5980cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.685+0000 7fc59f4fc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc5980cc8d0 0x7fc59806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59f4fc640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59806fd20 con 0x7fc5980d1200 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59f4fc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59806fad0 con 0x7fc59815dea0 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59f4fc640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc598070000 con 0x7fc5980cc8d0 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59d298640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc59815dea0 0x7fc5980cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59da99640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc5980d1200 0x7fc5980cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45966/0 (socket says 172.21.15.99:45966) 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59da99640 1 -- 172.21.15.99:0/2209570996 learned_addr learned my addr 172.21.15.99:0/2209570996 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59ca97640 1 --2- 172.21.15.99:0/2209570996 >> v2:172.21.15.162:3300/0 conn(0x7fc5980cc8d0 0x7fc59806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59d298640 1 -- 172.21.15.99:0/2209570996 >> v2:172.21.15.162:3300/0 conn(0x7fc5980cc8d0 msgr2=0x7fc59806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59d298640 1 --2- 172.21.15.99:0/2209570996 >> v2:172.21.15.162:3300/0 conn(0x7fc5980cc8d0 0x7fc59806acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc58ffff640 1 -- 172.21.15.99:0/2209570996 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320748708 0 0) 0x7fc59806fd20 con 0x7fc5980d1200 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59d298640 1 -- 172.21.15.99:0/2209570996 >> v1:172.21.15.99:6789/0 conn(0x7fc5980d1200 legacy=0x7fc5980cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.686+0000 7fc59d298640 1 -- 172.21.15.99:0/2209570996 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980702e0 con 0x7fc59815dea0 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59d298640 1 --2- 172.21.15.99:0/2209570996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc59815dea0 0x7fc5980cad60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc59806e310 tx=0x7fc58802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59158965c58a4b76 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc58ffff640 1 -- 172.21.15.99:0/2209570996 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588035020 con 0x7fc59815dea0 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc58ffff640 1 -- 172.21.15.99:0/2209570996 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc588031830 con 0x7fc59815dea0 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc58ffff640 1 -- 172.21.15.99:0/2209570996 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588031c00 con 0x7fc59815dea0 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59f4fc640 1 -- 172.21.15.99:0/2209570996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc59815dea0 msgr2=0x7fc5980cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/2209570996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc59815dea0 0x7fc5980cad60 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc59806e310 tx=0x7fc58802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59f4fc640 1 -- 172.21.15.99:0/2209570996 shutdown_connections 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/2209570996 >> v2:172.21.15.162:3300/0 conn(0x7fc5980cc8d0 0x7fc59806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/2209570996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc59815dea0 0x7fc5980cad60 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.687+0000 7fc59f4fc640 1 -- 172.21.15.99:0/2209570996 >> 172.21.15.99:0/2209570996 conn(0x7fc5980d0710 msgr2=0x7fc5980d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.688+0000 7fc59f4fc640 1 -- 172.21.15.99:0/2209570996 shutdown_connections 2024-03-04T10:09:20.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.688+0000 7fc59f4fc640 1 -- 172.21.15.99:0/2209570996 wait complete. 2024-03-04T10:09:20.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 Processor -- start 2024-03-04T10:09:20.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 -- start start 2024-03-04T10:09:20.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 0x7fc5980c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5980d1200 0x7fc5980b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc59815dea0 0x7fc5981f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806b390 con 0x7fc5980cc8d0 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806b210 con 0x7fc5980d1200 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59f4fc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59806b510 con 0x7fc59815dea0 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59d298640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 0x7fc5980c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59d298640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 0x7fc5980c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33596/0 (socket says 172.21.15.99:33596) 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.689+0000 7fc59d298640 1 -- 172.21.15.99:0/120566654 learned_addr learned my addr 172.21.15.99:0/120566654 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59da99640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc59815dea0 0x7fc5981f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59ca97640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5980d1200 0x7fc5980b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59d298640 1 -- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc59815dea0 msgr2=0x7fc5981f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59d298640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc59815dea0 0x7fc5981f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59d298640 1 -- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5980d1200 msgr2=0x7fc5980b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59d298640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5980d1200 0x7fc5980b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59d298640 1 -- 172.21.15.99:0/120566654 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5980c9b50 con 0x7fc5980cc8d0 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59ca97640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5980d1200 0x7fc5980b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59d298640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 0x7fc5980c94c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc588009890 tx=0x7fc5880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc58dffb640 1 -- 172.21.15.99:0/120566654 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588035050 con 0x7fc5980cc8d0 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc58dffb640 1 -- 172.21.15.99:0/120566654 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc588004250 con 0x7fc5980cc8d0 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59815e3b0 con 0x7fc5980cc8d0 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc58dffb640 1 -- 172.21.15.99:0/120566654 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58803b6c0 con 0x7fc5980cc8d0 2024-03-04T10:09:20.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.690+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59815e970 con 0x7fc5980cc8d0 2024-03-04T10:09:20.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.692+0000 7fc58dffb640 1 -- 172.21.15.99:0/120566654 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc588038020 con 0x7fc5980cc8d0 2024-03-04T10:09:20.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.692+0000 7fc58dffb640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc57c06b4d0 0x7fc57c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.692+0000 7fc58dffb640 1 -- 172.21.15.99:0/120566654 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc5880ac0c0 con 0x7fc5980cc8d0 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.693+0000 7fc59ca97640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc57c06b4d0 0x7fc57c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.693+0000 7fc5a21b5740 1 Processor -- start 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.693+0000 7fc5a21b5740 1 -- start start 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.693+0000 7fc59ca97640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc57c06b4d0 0x7fc57c06d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc594006fd0 tx=0x7fc594008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef1909e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef1937280 0x559ef1939c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef193a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ef18747c0 con 0x559ef1936ea0 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ef181db50 con 0x559ef190a380 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ef1870860 con 0x559ef1937280 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef1909e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef1909e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33606/0 (socket says 172.21.15.99:33606) 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 -- 172.21.15.99:0/2725523397 learned_addr learned my addr 172.21.15.99:0/2725523397 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59da99640 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef193a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59ca97640 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef1937280 0x559ef1939c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 -- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef1937280 msgr2=0x559ef1939c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef1937280 0x559ef1939c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 -- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 msgr2=0x559ef193a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef193a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 -- 172.21.15.99:0/2725523397 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ef18319a0 con 0x559ef1936ea0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59da99640 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef193a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc59d298640 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef1909e40 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc588082920 tx=0x7fc588009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03f1132d80b25c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.694+0000 7fc582ffd640 1 -- 172.21.15.99:0/2725523397 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588092030 con 0x559ef1936ea0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc582ffd640 1 -- 172.21.15.99:0/2725523397 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5880a3e50 con 0x559ef1936ea0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc582ffd640 1 -- 172.21.15.99:0/2725523397 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58807a740 con 0x559ef1936ea0 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2725523397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 msgr2=0x559ef1909e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef1909e40 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc588082920 tx=0x7fc588009190 comp rx=0 tx=0).stop 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2725523397 shutdown_connections 2024-03-04T10:09:20.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef193a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef1937280 0x559ef1939c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2725523397 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef1909e40 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2725523397 >> 172.21.15.99:0/2725523397 conn(0x559ef190c0b0 msgr2=0x559ef1954c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2725523397 shutdown_connections 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.695+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2725523397 wait complete. 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.696+0000 7fc5a21b5740 1 -- start start 2024-03-04T10:09:20.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.696+0000 7fc5a21b5740 1 Processor -- start 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef1936ea0 0x559ef192f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 0x559ef1965ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef192fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ef1870860 con 0x559ef1937280 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ef1830f90 con 0x559ef1936ea0 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ef18747c0 con 0x559ef192fa10 2024-03-04T10:09:20.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 10 client.0 _ll_register_callbacks cb 0x559ef1852d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 0x559ef1965ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 0x559ef1965ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33618/0 (socket says 172.21.15.99:33618) 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 -- 172.21.15.99:0/2161774140 learned_addr learned my addr 172.21.15.99:0/2161774140 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59d298640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef1936ea0 0x559ef192f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59da99640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef192fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 -- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 msgr2=0x559ef192fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef192fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 -- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef1936ea0 msgr2=0x559ef192f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef1936ea0 0x559ef192f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 -- 172.21.15.99:0/2161774140 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ef15892d0 con 0x559ef1937280 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc59ca97640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 0x559ef1965ae0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc5980b0c90 tx=0x7fc594004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6cc597c08fffe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc577fff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc577fff640 1 -- 172.21.15.99:0/2161774140 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc59400f040 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5880a3e50 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc577fff640 1 -- 172.21.15.99:0/2161774140 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc594015bd0 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc577fff640 1 -- 172.21.15.99:0/2161774140 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc59400ee60 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.697+0000 7fc5a21b5740 10 client.7278 fetch_fsmap learned FSMap version 64 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x559ef16536b0 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc577fff640 1 -- 172.21.15.99:0/2161774140 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5880a3e50 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc577fff640 1 -- 172.21.15.99:0/2161774140 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc594031080 con 0x559ef1937280 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 10 client.7278 fetch_fsmap finished waiting for FSMap version 64 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 10 client.7278 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 20 client.7278 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1938860 0x559ef1938c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 4 client.7278 mds_command: new command op to 7137 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 --> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ef1881d00 con 0x559ef1938860 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc59d298640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1938860 0x559ef1938c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc577fff640 10 client.7278 ms_handle_connect on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.698+0000 7fc59d298640 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1938860 0x559ef1938c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65611793fd233d04 server_cookie=bb18f249d934b47f in_seq=0 out_seq=0 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc577fff640 1 -- 172.21.15.99:0/2161774140 <== mds.0 v2:172.21.15.99:6833/3857383686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559ef1881d00 con 0x559ef1938860 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc577fff640 10 client.7278 handle_command_reply: tid=0 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 client.7278 shutdown 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 msgr2=0x559ef1965ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 0x559ef1965ae0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fc5980b0c90 tx=0x7fc594004730 comp rx=0 tx=0).stop 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 shutdown_connections 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1938860 0x559ef1938c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef192fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1937280 0x559ef1965ae0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/2161774140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef1936ea0 0x559ef192f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 >> 172.21.15.99:0/2161774140 conn(0x559ef190a380 msgr2=0x559ef190dd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc577fff640 0 client.7278 ms_handle_reset on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:20.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 shutdown_connections 2024-03-04T10:09:20.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 1 -- 172.21.15.99:0/2161774140 wait complete. 2024-03-04T10:09:20.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.699+0000 7fc5a21b5740 20 client.7278 trim_cache size 0 max 16384 2024-03-04T10:09:20.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 Processor -- start 2024-03-04T10:09:20.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 -- start start 2024-03-04T10:09:20.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef1930060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef1935080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.718 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5982004a0 con 0x559ef1936ea0 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981ff680 con 0x559ef190a380 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981fea80 con 0x559ef192fa10 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc59da99640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc59da99640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19355c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33628/0 (socket says 172.21.15.99:33628) 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.715+0000 7fc59da99640 1 -- 172.21.15.99:0/320395049 learned_addr learned my addr 172.21.15.99:0/320395049 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59da99640 1 -- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 msgr2=0x559ef1935080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59da99640 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef1935080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59da99640 1 -- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 msgr2=0x559ef1930060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59d298640 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef1930060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59da99640 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef1930060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59da99640 1 -- 172.21.15.99:0/320395049 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ef1909cf0 con 0x559ef1936ea0 2024-03-04T10:09:20.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59d298640 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef1930060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc59da99640 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19355c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc590007290 tx=0x7fc590007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00af1c33ce41141 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5827fc640 1 -- 172.21.15.99:0/320395049 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5900079a0 con 0x559ef1936ea0 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5827fc640 1 -- 172.21.15.99:0/320395049 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59001b070 con 0x559ef1936ea0 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5827fc640 1 -- 172.21.15.99:0/320395049 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc590001dd0 con 0x559ef1936ea0 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 -- 172.21.15.99:0/320395049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 msgr2=0x559ef19355c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19355c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc590007290 tx=0x7fc590007750 comp rx=0 tx=0).stop 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 -- 172.21.15.99:0/320395049 shutdown_connections 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19355c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef192fa10 0x559ef1935080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/320395049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef190a380 0x559ef1930060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.716+0000 7fc5a21b5740 1 -- 172.21.15.99:0/320395049 >> 172.21.15.99:0/320395049 conn(0x559ef193dc50 msgr2=0x559ef1937940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.717+0000 7fc5a21b5740 1 -- 172.21.15.99:0/320395049 shutdown_connections 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.717+0000 7fc5a21b5740 1 -- 172.21.15.99:0/320395049 wait complete. 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.717+0000 7fc5a21b5740 1 -- start start 2024-03-04T10:09:20.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.717+0000 7fc5a21b5740 1 Processor -- start 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef190a380 0x559ef1962a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef192fa10 0x559ef1962f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ef181db50 con 0x559ef1936ea0 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5981ff080 con 0x559ef192fa10 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ef19324e0 con 0x559ef190a380 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc5a21b5740 10 client.0 _ll_register_callbacks cb 0x7fc5880769e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33642/0 (socket says 172.21.15.99:33642) 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 -- 172.21.15.99:0/3940543974 learned_addr learned my addr 172.21.15.99:0/3940543974 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59ca97640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef192fa10 0x559ef1962f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59d298640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef190a380 0x559ef1962a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 -- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef190a380 msgr2=0x559ef1962a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef190a380 0x559ef1962a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 -- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef192fa10 msgr2=0x559ef1962f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef192fa10 0x559ef1962f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.718+0000 7fc59da99640 1 -- 172.21.15.99:0/3940543974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559ef1930570 con 0x559ef1936ea0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc59da99640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19bf9b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x559ef190fe20 tx=0x7fc590007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52aebd3c515a0466 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc577fff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc577fff640 1 -- 172.21.15.99:0/3940543974 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5900169c0 con 0x559ef1936ea0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc577fff640 1 -- 172.21.15.99:0/3940543974 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc590016b60 con 0x559ef1936ea0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc59001b070 con 0x559ef1936ea0 2024-03-04T10:09:20.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc577fff640 1 -- 172.21.15.99:0/3940543974 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc590031050 con 0x559ef1936ea0 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 10 client.7284 fetch_fsmap learned FSMap version 64 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x559ef19bf4c0 con 0x559ef1936ea0 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc577fff640 1 -- 172.21.15.99:0/3940543974 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc59001b070 con 0x559ef1936ea0 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc577fff640 1 -- 172.21.15.99:0/3940543974 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc590034020 con 0x559ef1936ea0 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 10 client.7284 fetch_fsmap finished waiting for FSMap version 64 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 10 client.7284 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 20 client.7284 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1937280 0x559ef19bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 4 client.7284 mds_command: new command op to 7137 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 --> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559ef19319b0 con 0x559ef1937280 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.719+0000 7fc59d298640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1937280 0x559ef19bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc577fff640 10 client.7284 ms_handle_connect on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc59d298640 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1937280 0x559ef19bfef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7baa418e5ec4a5c9 server_cookie=5e12fc27e886c2a8 in_seq=0 out_seq=0 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc577fff640 1 -- 172.21.15.99:0/3940543974 <== mds.0 v2:172.21.15.99:6833/3857383686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559ef19319b0 con 0x559ef1937280 2024-03-04T10:09:20.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc577fff640 10 client.7284 handle_command_reply: tid=0 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 client.7284 shutdown 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 msgr2=0x559ef19bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19bf9b0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x559ef190fe20 tx=0x7fc590007d90 comp rx=0 tx=0).stop 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 shutdown_connections 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x559ef1937280 0x559ef19bfef0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559ef1936ea0 0x559ef19bf9b0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc577fff640 0 client.7284 ms_handle_reset on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ef192fa10 0x559ef1962f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 --2- 172.21.15.99:0/3940543974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559ef190a380 0x559ef1962a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 >> 172.21.15.99:0/3940543974 conn(0x559ef193dc50 msgr2=0x559ef193ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 shutdown_connections 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 1 -- 172.21.15.99:0/3940543974 wait complete. 2024-03-04T10:09:20.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.720+0000 7fc5a21b5740 20 client.7284 trim_cache size 0 max 16384 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 7137, 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.5869524449999997, 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 64, 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 81, 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 81, 2024-03-04T10:09:20.728 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 8.9436965219999998 2024-03-04T10:09:20.729 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:09:20.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc57c06b4d0 msgr2=0x7fc57c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc57c06b4d0 0x7fc57c06d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc594006fd0 tx=0x7fc594008040 comp rx=0 tx=0).stop 2024-03-04T10:09:20.729 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 msgr2=0x7fc5980c94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 0x7fc5980c94c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fc588009890 tx=0x7fc5880046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 shutdown_connections 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc57c06b4d0 0x7fc57c06d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc59815dea0 0x7fc5981f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc5980d1200 0x7fc5980b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 --2- 172.21.15.99:0/120566654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc5980cc8d0 0x7fc5980c94c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 >> 172.21.15.99:0/120566654 conn(0x7fc5980d0710 msgr2=0x7fc5980af4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 shutdown_connections 2024-03-04T10:09:20.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.721+0000 7fc59f4fc640 1 -- 172.21.15.99:0/120566654 wait complete. 2024-03-04T10:09:20.733 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:09:20.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.831+0000 7f5a7b6ca640 1 Processor -- start 2024-03-04T10:09:20.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.831+0000 7f5a7b6ca640 1 -- start start 2024-03-04T10:09:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a7b6ca640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5a7415b030 0x7f5a7415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a7b6ca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 0x7f5a7415dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a7b6ca640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7415e2d0 con 0x7f5a7415b410 2024-03-04T10:09:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a7b6ca640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a740b2720 con 0x7f5a7415b030 2024-03-04T10:09:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a7b6ca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406d8b0 con 0x7f5a7406d3b0 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a79c67640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5a7415b410 0x7f5a7415b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45982/0 (socket says 172.21.15.99:45982) 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a79c67640 1 -- 172.21.15.99:0/2631670097 learned_addr learned my addr 172.21.15.99:0/2631670097 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a79466640 1 --2- 172.21.15.99:0/2631670097 >> v2:172.21.15.162:3300/0 conn(0x7f5a7415b030 0x7f5a7415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a78c65640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 0x7f5a7415dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804617946 0 0) 0x7f5a7415e2d0 con 0x7f5a7415b410 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.832+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a60003520 con 0x7f5a7415b410 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039238457 0 0) 0x7f5a60003520 con 0x7f5a7415b410 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 msgr2=0x7f5a7415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 0x7f5a7415dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 >> v2:172.21.15.162:3300/0 conn(0x7f5a7415b030 msgr2=0x7f5a7415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 --2- 172.21.15.99:0/2631670097 >> v2:172.21.15.162:3300/0 conn(0x7f5a7415b030 0x7f5a7415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7406db90 con 0x7f5a7415b410 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5a6c001670 con 0x7f5a7415b410 2024-03-04T10:09:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79466640 1 --2- 172.21.15.99:0/2631670097 >> v2:172.21.15.162:3300/0 conn(0x7f5a7415b030 0x7f5a7415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 >> v1:172.21.15.99:6789/0 conn(0x7f5a7415b410 legacy=0x7f5a7415b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 0x7f5a600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a60003f30 0x7f5a60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a60004d40 0x7f5a600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740b2720 con 0x7f5a60004d40 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406d8b0 con 0x7f5a60003f30 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6000ba10 con 0x7f5a7406d3b0 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79c67640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a60004d40 0x7f5a600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a78c65640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a60003f30 0x7f5a60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79466640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 0x7f5a600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79c67640 1 -- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 msgr2=0x7f5a600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79c67640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a7406d3b0 0x7f5a600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79c67640 1 -- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a60003f30 msgr2=0x7f5a60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79c67640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a60003f30 0x7f5a60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.833+0000 7f5a79c67640 1 -- 172.21.15.99:0/2631670097 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a60003520 con 0x7f5a60004d40 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a79c67640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a60004d40 0x7f5a600051c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c240 tx=0x7f5a60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7e395b73d92513 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c007d30 con 0x7f5a60004d40 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c030070 con 0x7f5a60004d40 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a6bfff640 1 -- 172.21.15.99:0/2631670097 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c005d70 con 0x7f5a60004d40 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/2631670097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a60004d40 msgr2=0x7f5a600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/2631670097 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a60004d40 0x7f5a600051c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c240 tx=0x7f5a60005830 comp rx=0 tx=0).stop 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a79c67640 1 -- 172.21.15.99:0/2631670097 reap_dead start 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/2631670097 shutdown_connections 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.834+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/2631670097 >> 172.21.15.99:0/2631670097 conn(0x7f5a7406acd0 msgr2=0x7f5a7406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.835+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/2631670097 shutdown_connections 2024-03-04T10:09:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.835+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/2631670097 wait complete. 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 Processor -- start 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 -- start start 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a741503d0 0x7f5a741f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a7415e4e0 0x7f5a74151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 0x7f5a7415d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406e690 con 0x7f5a7415d510 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406e150 con 0x7f5a7415e4e0 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.836+0000 7f5a7b6ca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7406e810 con 0x7f5a741503d0 2024-03-04T10:09:20.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 0x7f5a7415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 0x7f5a7415d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33668/0 (socket says 172.21.15.99:33668) 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 -- 172.21.15.99:0/3290585031 learned_addr learned my addr 172.21.15.99:0/3290585031 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79466640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a741503d0 0x7f5a741f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a78c65640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a7415e4e0 0x7f5a74151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 -- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a741503d0 msgr2=0x7f5a741f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a741503d0 0x7f5a741f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 -- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a7415e4e0 msgr2=0x7f5a74151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a7415e4e0 0x7f5a74151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 -- 172.21.15.99:0/3290585031 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7415ec80 con 0x7f5a7415d510 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79466640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a741503d0 0x7f5a741f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a79c67640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 0x7f5a7415d990 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c360 tx=0x7f5a7406dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c007d30 con 0x7f5a7415d510 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c004cb0 con 0x7f5a7415d510 2024-03-04T10:09:20.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c02de60 con 0x7f5a7415d510 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a741542c0 con 0x7f5a7415d510 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.837+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74152080 con 0x7f5a7415d510 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.838+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7415ded0 con 0x7f5a7415d510 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.839+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5a6c0072b0 con 0x7f5a7415d510 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.839+0000 7f5a69ffb640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5a5806b4d0 0x7f5a5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.839+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a6c0abab0 con 0x7f5a7415d510 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.839+0000 7f5a79466640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5a5806b4d0 0x7f5a5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.840+0000 7f5a79466640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5a5806b4d0 0x7f5a5806d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007660 tx=0x7f5a7000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:20.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:20.843+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5a6c07a830 con 0x7f5a7415d510 2024-03-04T10:09:21.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.049+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5a7415c480 con 0x7f5a7415d510 2024-03-04T10:09:21.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.050+0000 7f5a69ffb640 1 -- 172.21.15.99:0/3290585031 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5a6c07a130 con 0x7f5a7415d510 2024-03-04T10:09:21.052 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:09:21.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.051+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5a5806b4d0 msgr2=0x7f5a5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:21.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.051+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5a5806b4d0 0x7f5a5806d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007660 tx=0x7f5a7000b040 comp rx=0 tx=0).stop 2024-03-04T10:09:21.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 msgr2=0x7f5a7415d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:21.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 0x7f5a7415d990 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00c360 tx=0x7f5a7406dd60 comp rx=0 tx=0).stop 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 shutdown_connections 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5a5806b4d0 0x7f5a5806d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5a7415d510 0x7f5a7415d990 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a7415e4e0 0x7f5a74151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 --2- 172.21.15.99:0/3290585031 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5a741503d0 0x7f5a741f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 >> 172.21.15.99:0/3290585031 conn(0x7f5a7406acd0 msgr2=0x7f5a740b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 shutdown_connections 2024-03-04T10:09:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.052+0000 7f5a7b6ca640 1 -- 172.21.15.99:0/3290585031 wait complete. 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:09:21.064 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:09:21.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.065 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:09:21.080 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:09:21.080 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:09:21.081 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42424sec preferred_lft 42424sec 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:09:21.082 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:09:21.083 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:09:21.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.084 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:09:21.146 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:09:21.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.148 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:09:21.204 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:09:21.204 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:09:21.204 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:09:21.204 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:09:21.204 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:09:21.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.204 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:09:21.205 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:09:21.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:09:21.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:09:21.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:09:21.205 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:09:21.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.300 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:09:21.314 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:09:21.314 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:09:21.314 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:09:21.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.315 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:09:21.381 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:09:21.382 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:21.437 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:09:21.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:09:21.437 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:21.491 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:09:21.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.491 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:09:21.576 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:09:21.576 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:09:21.584 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:09:21.585 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:09:21.585 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:09:21.585 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:09:21.586 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:09:21.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.586 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:21.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:21.613 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:09:21.673 INFO:teuthology.orchestra.run.smithi099.stdout:client7293 2024-03-04T10:09:21.674 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:21.674 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:21.729 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.867+0000 7f24a5d69640 1 Processor -- start 2024-03-04T10:09:21.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.867+0000 7f24a5d69640 1 -- start start 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.868+0000 7f24a5d69640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f24a00cfd60 0x7f24a00cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.868+0000 7f24a5d69640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a00d0140 con 0x7f24a006d3b0 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.868+0000 7f24a5d69640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a015c280 con 0x7f24a00cf0d0 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.868+0000 7f24a5d69640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a015cbb0 con 0x7f24a00cfd60 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.868+0000 7f249f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f24a00cfd60 0x7f24a00cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.868+0000 7f249effd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f24a006d3b0 0x7f24a00ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45984/0 (socket says 172.21.15.99:45984) 2024-03-04T10:09:21.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.869+0000 7f249effd640 1 -- 172.21.15.99:0/3559062649 learned_addr learned my addr 172.21.15.99:0/3559062649 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:21.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.869+0000 7f249f7fe640 1 -- 172.21.15.99:0/3559062649 >> v1:172.21.15.162:6789/0 conn(0x7f24a00cf0d0 legacy=0x7f24a015f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:21.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.869+0000 7f249f7fe640 1 -- 172.21.15.99:0/3559062649 >> v1:172.21.15.99:6789/0 conn(0x7f24a006d3b0 legacy=0x7f24a00ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:21.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.869+0000 7f249f7fe640 1 -- 172.21.15.99:0/3559062649 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a015c560 con 0x7f24a00cfd60 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.869+0000 7f249f7fe640 1 --2- 172.21.15.99:0/3559062649 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cfd60 0x7f24a00cbdf0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f24a015cd30 tx=0x7f249002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3807d3a034d05d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f249e7fc640 1 -- 172.21.15.99:0/3559062649 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490035020 con 0x7f24a00cfd60 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f249e7fc640 1 -- 172.21.15.99:0/3559062649 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490031830 con 0x7f24a00cfd60 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f249e7fc640 1 -- 172.21.15.99:0/3559062649 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490031c00 con 0x7f24a00cfd60 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 -- 172.21.15.99:0/3559062649 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cfd60 msgr2=0x7f24a00cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3559062649 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cfd60 0x7f24a00cbdf0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f24a015cd30 tx=0x7f249002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 -- 172.21.15.99:0/3559062649 shutdown_connections 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3559062649 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cfd60 0x7f24a00cbdf0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 -- 172.21.15.99:0/3559062649 >> 172.21.15.99:0/3559062649 conn(0x7f24a006acd0 msgr2=0x7f24a006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:21.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 -- 172.21.15.99:0/3559062649 shutdown_connections 2024-03-04T10:09:21.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 -- 172.21.15.99:0/3559062649 wait complete. 2024-03-04T10:09:21.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.870+0000 7f24a5d69640 1 Processor -- start 2024-03-04T10:09:21.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 -- start start 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a006d3b0 0x7f24a01f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 0x7f24a01e4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a00cfd60 0x7f24a006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00cc330 con 0x7f24a00cf0d0 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a015bf30 con 0x7f24a00cfd60 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f24a5d69640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00cc4b0 con 0x7f24a006d3b0 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f249effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 0x7f24a01e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:21.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f249effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 0x7f24a01e4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33686/0 (socket says 172.21.15.99:33686) 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.871+0000 7f249effd640 1 -- 172.21.15.99:0/3312900666 learned_addr learned my addr 172.21.15.99:0/3312900666 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249f7fe640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a006d3b0 0x7f24a01f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249ffff640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a00cfd60 0x7f24a006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249effd640 1 -- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a006d3b0 msgr2=0x7f24a01f6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249effd640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a006d3b0 0x7f24a01f6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249effd640 1 -- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a00cfd60 msgr2=0x7f24a006ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249effd640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a00cfd60 0x7f24a006ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249effd640 1 -- 172.21.15.99:0/3312900666 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a00b28c0 con 0x7f24a00cf0d0 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249f7fe640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a006d3b0 0x7f24a01f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249ffff640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a00cfd60 0x7f24a006ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f249effd640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 0x7f24a01e4280 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f249400c440 tx=0x7f2494001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2494017070 con 0x7f24a00cf0d0 2024-03-04T10:09:21.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2494005ce0 con 0x7f24a00cf0d0 2024-03-04T10:09:21.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a01f7080 con 0x7f24a00cf0d0 2024-03-04T10:09:21.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24940072f0 con 0x7f24a00cf0d0 2024-03-04T10:09:21.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.872+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a01f7920 con 0x7f24a00cf0d0 2024-03-04T10:09:21.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.873+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a01f7470 con 0x7f24a00cf0d0 2024-03-04T10:09:21.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.874+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f249401a070 con 0x7f24a00cf0d0 2024-03-04T10:09:21.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.874+0000 7f2487fff640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f248c06b4d0 0x7f248c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:21.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.874+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2494089760 con 0x7f24a00cf0d0 2024-03-04T10:09:21.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.874+0000 7f249f7fe640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f248c06b4d0 0x7f248c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:21.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.875+0000 7f249f7fe640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f248c06b4d0 0x7f248c06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2490036000 tx=0x7f249003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:21.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:21.878+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2494058450 con 0x7f24a00cf0d0 2024-03-04T10:09:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.084+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a0074f60 con 0x7f24a00cf0d0 2024-03-04T10:09:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.084+0000 7f2487fff640 1 -- 172.21.15.99:0/3312900666 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f2494057d50 con 0x7f24a00cf0d0 2024-03-04T10:09:22.087 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:22.087 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:17.472137+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 64 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f248c06b4d0 msgr2=0x7f248c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f248c06b4d0 0x7f248c06d990 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2490036000 tx=0x7f249003c040 comp rx=0 tx=0).stop 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 msgr2=0x7f24a01e4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 0x7f24a01e4280 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f249400c440 tx=0x7f2494001e70 comp rx=0 tx=0).stop 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 shutdown_connections 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f248c06b4d0 0x7f248c06d990 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f24a00cfd60 0x7f24a006ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f24a00cf0d0 0x7f24a01e4280 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 --2- 172.21.15.99:0/3312900666 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f24a006d3b0 0x7f24a01f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 >> 172.21.15.99:0/3312900666 conn(0x7f24a006acd0 msgr2=0x7f24a0159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 shutdown_connections 2024-03-04T10:09:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.086+0000 7f24a5d69640 1 -- 172.21.15.99:0/3312900666 wait complete. 2024-03-04T10:09:22.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 true 2024-03-04T10:09:22.100 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:09:22.098+0000 7f9aeb87b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:09:22.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.199+0000 7f61ab5c4640 1 Processor -- start 2024-03-04T10:09:22.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.200+0000 7f61ab5c4640 1 -- start start 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.200+0000 7f61ab5c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f61a40b70d0 0x7f61a40cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.200+0000 7f61ab5c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61a40d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.200+0000 7f61ab5c4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a40d1520 con 0x7f61a415bbf0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.200+0000 7f61ab5c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a40cfa30 con 0x7f61a40b70d0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.200+0000 7f61ab5c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40cfd10 con 0x7f61a40d1770 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9360640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f61a415bbf0 0x7f61a40d0ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45992/0 (socket says 172.21.15.99:45992) 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9360640 1 -- 172.21.15.99:0/3053487573 learned_addr learned my addr 172.21.15.99:0/3053487573 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923495221 0 0) 0x7f61a40d1520 con 0x7f61a415bbf0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6190003520 con 0x7f61a415bbf0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a8b5f640 1 --2- 172.21.15.99:0/3053487573 >> v2:172.21.15.162:3300/0 conn(0x7f61a40b70d0 0x7f61a40cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9b61640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61a40d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2111726458 0 0) 0x7f6190003520 con 0x7f61a415bbf0 2024-03-04T10:09:22.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 msgr2=0x7f61a40d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61a40d1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 >> v2:172.21.15.162:3300/0 conn(0x7f61a40b70d0 msgr2=0x7f61a40cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 --2- 172.21.15.99:0/3053487573 >> v2:172.21.15.162:3300/0 conn(0x7f61a40b70d0 0x7f61a40cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40cfff0 con 0x7f61a415bbf0 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6198001690 con 0x7f61a415bbf0 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 >> v1:172.21.15.99:6789/0 conn(0x7f61a415bbf0 legacy=0x7f61a40d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190003f30 0x7f6190004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6190004d40 0x7f61900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40cfa30 con 0x7f6190004d40 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40cfd10 con 0x7f6190003f30 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619000ba10 con 0x7f61a40d1770 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9b61640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6190004d40 0x7f61900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9360640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a8b5f640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190003f30 0x7f6190004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9b61640 1 -- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 msgr2=0x7f61900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.201+0000 7f61a9b61640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f61a9b61640 1 -- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190003f30 msgr2=0x7f6190004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f61a9b61640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190003f30 0x7f6190004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f61a9b61640 1 -- 172.21.15.99:0/3053487573 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6190003520 con 0x7f6190004d40 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f61a9360640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61a40d1770 0x7f61900039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f61a8b5f640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6190003f30 0x7f6190004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f61a9b61640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6190004d40 0x7f61900051c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f619c00e170 tx=0x7f6190005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd6f28b4c3f984e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c039070 con 0x7f6190004d40 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c02cbf0 con 0x7f6190004d40 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.202+0000 7f6197fff640 1 -- 172.21.15.99:0/3053487573 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c009c10 con 0x7f6190004d40 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61ab5c4640 1 -- 172.21.15.99:0/3053487573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6190004d40 msgr2=0x7f61900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/3053487573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6190004d40 0x7f61900051c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f619c00e170 tx=0x7f6190005830 comp rx=0 tx=0).stop 2024-03-04T10:09:22.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61a9b61640 1 -- 172.21.15.99:0/3053487573 reap_dead start 2024-03-04T10:09:22.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61ab5c4640 1 -- 172.21.15.99:0/3053487573 shutdown_connections 2024-03-04T10:09:22.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61ab5c4640 1 -- 172.21.15.99:0/3053487573 >> 172.21.15.99:0/3053487573 conn(0x7f61a40b3330 msgr2=0x7f61a40b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:22.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61ab5c4640 1 -- 172.21.15.99:0/3053487573 shutdown_connections 2024-03-04T10:09:22.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.203+0000 7f61ab5c4640 1 -- 172.21.15.99:0/3053487573 wait complete. 2024-03-04T10:09:22.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.204+0000 7f61ab5c4640 1 Processor -- start 2024-03-04T10:09:22.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.204+0000 7f61ab5c4640 1 -- start start 2024-03-04T10:09:22.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6190004d40 0x7f61a415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61a406ca80 0x7f61a415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 0x7f61a4152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a415bfd0 con 0x7f61a4152840 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40d05b0 con 0x7f61a406ca80 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40d1c20 con 0x7f6190004d40 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 0x7f61a4152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 0x7f61a4152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33712/0 (socket says 172.21.15.99:33712) 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 -- 172.21.15.99:0/1941291712 learned_addr learned my addr 172.21.15.99:0/1941291712 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 -- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6190004d40 msgr2=0x7f61a415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6190004d40 0x7f61a415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 -- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61a406ca80 msgr2=0x7f61a415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61a406ca80 0x7f61a415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 -- 172.21.15.99:0/1941291712 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a406d040 con 0x7f61a4152840 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61a9b61640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 0x7f61a4152cc0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f619c00e170 tx=0x7f619c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c039070 con 0x7f61a4152840 2024-03-04T10:09:22.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c03a040 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.205+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a406d320 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.206+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c003990 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.206+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a40c9d80 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.207+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a4074f60 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.207+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f619c046070 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.207+0000 7f6195ffb640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f618806b4d0 0x7f618806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.207+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f619c0abe90 con 0x7f61a4152840 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.208+0000 7f61a9360640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f618806b4d0 0x7f618806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:22.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.208+0000 7f61a9360640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f618806b4d0 0x7f618806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6198001d30 tx=0x7f6198001be0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:22.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.212+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f619c07a530 con 0x7f61a4152840 2024-03-04T10:09:22.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:22.451+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"} v 0) v1 -- 0x7f61a4073e40 con 0x7f61a4152840 2024-03-04T10:09:23.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.161+0000 7f6195ffb640 1 -- 172.21.15.99:0/1941291712 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "true"}]=0 freezing mds.7137 v65) v1 ==== 112+0+0 (secure 0 0 0) 0x7f619c07e5d0 con 0x7f61a4152840 2024-03-04T10:09:23.164 INFO:teuthology.orchestra.run.smithi099.stderr:freezing mds.7137 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f618806b4d0 msgr2=0x7f618806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f618806b4d0 0x7f618806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6198001d30 tx=0x7f6198001be0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 msgr2=0x7f61a4152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 0x7f61a4152cc0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f619c00e170 tx=0x7f619c009750 comp rx=0 tx=0).stop 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 shutdown_connections 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f618806b4d0 0x7f618806d990 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61a4152840 0x7f61a4152cc0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61a406ca80 0x7f61a415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 --2- 172.21.15.99:0/1941291712 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6190004d40 0x7f61a415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 >> 172.21.15.99:0/1941291712 conn(0x7f61a40b3330 msgr2=0x7f61a40b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 shutdown_connections 2024-03-04T10:09:23.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.164+0000 7f61ab5c4640 1 -- 172.21.15.99:0/1941291712 wait complete. 2024-03-04T10:09:23.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:23.179 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'mkdir -p a/{b,c}/d') 2024-03-04T10:09:23.202 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T10:09:23.202 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-03-04T10:09:23.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.345+0000 7fb7cf51b640 1 Processor -- start 2024-03-04T10:09:23.348 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.346+0000 7fb7cf51b640 1 -- start start 2024-03-04T10:09:23.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.346+0000 7fb7cf51b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb7c80cdd70 0x7fb7c80cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.346+0000 7fb7cf51b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80ccfe0 0x7fb7c80cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.346+0000 7fb7cf51b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c806ae00 con 0x7fb7c80cc900 2024-03-04T10:09:23.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cf51b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80ccde0 con 0x7fb7c80cdd70 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cf51b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80d0480 con 0x7fb7c80ccfe0 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb7c80cdd70 0x7fb7c80cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb7c80cdd70 0x7fb7c80cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33728/0 (socket says 172.21.15.99:33728) 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7ccab6640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb7c80cc900 0x7fb7c815d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:46000/0 (socket says 172.21.15.99:46000) 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2616914599 learned_addr learned my addr 172.21.15.99:0/2616914599 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2616914599 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80ccfe0 msgr2=0x7fb7c80cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/2616914599 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80ccfe0 0x7fb7c80cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2616914599 >> v1:172.21.15.99:6789/0 conn(0x7fb7c80cc900 legacy=0x7fb7c815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7bbfff640 1 -- 172.21.15.99:0/2616914599 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636694445 0 0) 0x7fb7c806ae00 con 0x7fb7c80cc900 2024-03-04T10:09:23.350 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.347+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2616914599 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c806b0e0 con 0x7fb7c80cdd70 2024-03-04T10:09:23.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/2616914599 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80cfa00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009870 tx=0x7fb7bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ec82d925569fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7bbfff640 1 -- 172.21.15.99:0/2616914599 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc035020 con 0x7fb7c80cdd70 2024-03-04T10:09:23.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7bbfff640 1 -- 172.21.15.99:0/2616914599 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc031640 con 0x7fb7c80cdd70 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7bbfff640 1 -- 172.21.15.99:0/2616914599 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc02d040 con 0x7fb7c80cdd70 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cf51b640 1 -- 172.21.15.99:0/2616914599 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 msgr2=0x7fb7c80cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/2616914599 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80cfa00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009870 tx=0x7fb7bc02b8f0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cf51b640 1 -- 172.21.15.99:0/2616914599 shutdown_connections 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/2616914599 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80ccfe0 0x7fb7c80cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/2616914599 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80cfa00 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.348+0000 7fb7cf51b640 1 -- 172.21.15.99:0/2616914599 >> 172.21.15.99:0/2616914599 conn(0x7fb7c80b22e0 msgr2=0x7fb7c80b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.349+0000 7fb7cf51b640 1 -- 172.21.15.99:0/2616914599 shutdown_connections 2024-03-04T10:09:23.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.349+0000 7fb7cf51b640 1 -- 172.21.15.99:0/2616914599 wait complete. 2024-03-04T10:09:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 Processor -- start 2024-03-04T10:09:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 -- start start 2024-03-04T10:09:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80cc900 0x7fb7c81525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7c80ccfe0 0x7fb7c80b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80c6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b2e90 con 0x7fb7c80cdd70 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c806b6a0 con 0x7fb7c80ccfe0 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cf51b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b3010 con 0x7fb7c80cc900 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cdab8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80c6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cdab8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80c6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33738/0 (socket says 172.21.15.99:33738) 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.350+0000 7fb7cdab8640 1 -- 172.21.15.99:0/3055058855 learned_addr learned my addr 172.21.15.99:0/3055058855 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cdab8640 1 -- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80cc900 msgr2=0x7fb7c81525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80cc900 0x7fb7c81525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7c80ccfe0 0x7fb7c80b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80cc900 0x7fb7c81525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cdab8640 1 -- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7c80ccfe0 msgr2=0x7fb7c80b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7c80ccfe0 0x7fb7c80b5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cdab8640 1 -- 172.21.15.99:0/3055058855 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80c56b0 con 0x7fb7c80cdd70 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80c6750 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000b970 tx=0x7fb7c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7b9ffb640 1 -- 172.21.15.99:0/3055058855 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0019070 con 0x7fb7c80cdd70 2024-03-04T10:09:23.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7b9ffb640 1 -- 172.21.15.99:0/3055058855 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c000cb60 con 0x7fb7c80cdd70 2024-03-04T10:09:23.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c80c5990 con 0x7fb7c80cdd70 2024-03-04T10:09:23.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7b9ffb640 1 -- 172.21.15.99:0/3055058855 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0014440 con 0x7fb7c80cdd70 2024-03-04T10:09:23.356 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.351+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c80c5f50 con 0x7fb7c80cdd70 2024-03-04T10:09:23.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.353+0000 7fb7b9ffb640 1 -- 172.21.15.99:0/3055058855 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb7c001c070 con 0x7fb7c80cdd70 2024-03-04T10:09:23.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.353+0000 7fb7b9ffb640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb7ac06b4d0 0x7fb7ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.353+0000 7fb7b9ffb640 1 -- 172.21.15.99:0/3055058855 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7c008a0c0 con 0x7fb7c80cdd70 2024-03-04T10:09:23.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.353+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb7ac06b4d0 0x7fb7ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.354+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb7ac06b4d0 0x7fb7ac06d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc036000 tx=0x7fb7bc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.354+0000 7fb7d21d4740 1 Processor -- start 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.354+0000 7fb7d21d4740 1 -- start start 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693da8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 0x55e693dd85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693da8d40 0x55e693dd8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e693d11e00 con 0x55e693dd5c40 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e693cbab10 con 0x55e693dd5860 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e693d0dea0 con 0x55e693da8d40 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 0x55e693dd85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 0x55e693dd85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33742/0 (socket says 172.21.15.99:33742) 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 -- 172.21.15.99:0/3403518185 learned_addr learned my addr 172.21.15.99:0/3403518185 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693da8d40 0x55e693dd8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693da8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 -- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693da8d40 msgr2=0x55e693dd8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693da8d40 0x55e693dd8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 -- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 msgr2=0x55e693da8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693da8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 -- 172.21.15.99:0/3403518185 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e693af07d0 con 0x55e693dd5c40 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 0x55e693dd85f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4001690 tx=0x7fb7c4001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588ea47f976a2cca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.355+0000 7fb7b2ffd640 1 -- 172.21.15.99:0/3403518185 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40074e0 con 0x55e693dd5c40 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7b2ffd640 1 -- 172.21.15.99:0/3403518185 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c40047e0 con 0x55e693dd5c40 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7b2ffd640 1 -- 172.21.15.99:0/3403518185 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4004b90 con 0x55e693dd5c40 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3403518185 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 msgr2=0x55e693dd85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 0x55e693dd85f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4001690 tx=0x7fb7c4001b50 comp rx=0 tx=0).stop 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3403518185 shutdown_connections 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693da8d40 0x55e693dd8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5c40 0x55e693dd85f0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3403518185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693da8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3403518185 >> 172.21.15.99:0/3403518185 conn(0x55e693daaa70 msgr2=0x55e693df35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3403518185 shutdown_connections 2024-03-04T10:09:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.356+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3403518185 wait complete. 2024-03-04T10:09:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.357+0000 7fb7d21d4740 1 -- start start 2024-03-04T10:09:23.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.357+0000 7fb7d21d4740 1 Processor -- start 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693dc2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693ddaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693decb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e693dad010 con 0x55e693dd5860 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e693cce0a0 con 0x55e693dd5c40 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e693dac400 con 0x55e693dd4c60 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 10 client.0 _ll_register_callbacks cb 0x55e693ccda70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693dc2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693dc2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33750/0 (socket says 172.21.15.99:33750) 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2805104504 learned_addr learned my addr 172.21.15.99:0/2805104504 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693ddaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693decb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 msgr2=0x55e693decb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693decb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 msgr2=0x55e693ddaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693ddaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/2805104504 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e693a263f0 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693dc2f70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009990 tx=0x7fb7bc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c176a5b4a31a9a9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7a7fff640 1 -- 172.21.15.99:0/2805104504 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc054060 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7a7fff640 1 -- 172.21.15.99:0/2805104504 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7bc03adc0 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7c40047e0 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7a7fff640 1 -- 172.21.15.99:0/2805104504 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7bc04f470 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.358+0000 7fb7d21d4740 10 client.7323 fetch_fsmap learned FSMap version 65 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) v3 -- 0x55e6939e60f0 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7a7fff640 1 -- 172.21.15.99:0/2805104504 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c40047e0 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7a7fff640 1 -- 172.21.15.99:0/2805104504 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 65) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb7bc04f610 con 0x55e693dd5860 2024-03-04T10:09:23.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 10 client.7323 fetch_fsmap finished waiting for FSMap version 65 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 10 client.7323 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 20 client.7323 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693dc3e60 0x55e693dc42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 4 client.7323 mds_command: new command op to 7137 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 --> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e693d1f340 con 0x55e693dc3e60 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693dc3e60 0x55e693dc42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693dc3e60 0x55e693dc42a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52ed02a940e0dd28 server_cookie=20fc4ac2a33217c in_seq=0 out_seq=0 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.359+0000 7fb7a7fff640 10 client.7323 ms_handle_connect on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7a7fff640 1 -- 172.21.15.99:0/2805104504 <== mds.0 v2:172.21.15.99:6833/3857383686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e693d1f340 con 0x55e693dc3e60 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7a7fff640 10 client.7323 handle_command_reply: tid=0 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 client.7323 shutdown 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 msgr2=0x55e693dc2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693dc2f70 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009990 tx=0x7fb7bc02e000 comp rx=0 tx=0).stop 2024-03-04T10:09:23.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 shutdown_connections 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693dc3e60 0x55e693dc42a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693decb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693ddaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2805104504 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693dc2f70 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7a7fff640 0 client.7323 ms_handle_reset on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 >> 172.21.15.99:0/2805104504 conn(0x55e693da8d40 msgr2=0x55e693dad500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 shutdown_connections 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2805104504 wait complete. 2024-03-04T10:09:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.360+0000 7fb7d21d4740 20 client.7323 trim_cache size 0 max 16384 2024-03-04T10:09:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.373+0000 7fb7d21d4740 1 Processor -- start 2024-03-04T10:09:23.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.373+0000 7fb7d21d4740 1 -- start start 2024-03-04T10:09:23.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693dadad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693da76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693da7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e693dee8d0 con 0x55e693dd5860 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c8200590 con 0x55e693dd5c40 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c81ff7c0 con 0x55e693dd4c60 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693da76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693da76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33754/0 (socket says 172.21.15.99:33754) 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 -- 172.21.15.99:0/2103687125 learned_addr learned my addr 172.21.15.99:0/2103687125 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 -- 172.21.15.99:0/2103687125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 msgr2=0x55e693dadad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693dadad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 -- 172.21.15.99:0/2103687125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 msgr2=0x55e693da7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693da7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 -- 172.21.15.99:0/2103687125 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e693ded9e0 con 0x55e693dd5860 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693da76d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4016040 tx=0x7fb7c4006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d6a6a84f58598 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7b27fc640 1 -- 172.21.15.99:0/2103687125 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c40079e0 con 0x55e693dd5860 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7b27fc640 1 -- 172.21.15.99:0/2103687125 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c402c070 con 0x55e693dd5860 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.374+0000 7fb7b27fc640 1 -- 172.21.15.99:0/2103687125 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4027b00 con 0x55e693dd5860 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2103687125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 msgr2=0x55e693da76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693da76d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4016040 tx=0x7fb7c4006000 comp rx=0 tx=0).stop 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2103687125 shutdown_connections 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5c40 0x55e693da7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd5860 0x55e693da76d0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/2103687125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd4c60 0x55e693dadad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2103687125 >> 172.21.15.99:0/2103687125 conn(0x55e693da8d40 msgr2=0x55e693dc20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2103687125 shutdown_connections 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 -- 172.21.15.99:0/2103687125 wait complete. 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 -- start start 2024-03-04T10:09:23.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.375+0000 7fb7d21d4740 1 Processor -- start 2024-03-04T10:09:23.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 0x55e693df2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693e60c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd5c40 0x55e693e61180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c815e3b0 con 0x55e693dd4c60 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c815ddb0 con 0x55e693dd5860 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e693dac740 con 0x55e693dd5c40 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7d21d4740 10 client.0 _ll_register_callbacks cb 0x7fb7bc031c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7cd2b7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 0x55e693df2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7cd2b7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 0x55e693df2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33764/0 (socket says 172.21.15.99:33764) 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.376+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/3374791158 learned_addr learned my addr 172.21.15.99:0/3374791158 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd5c40 0x55e693e61180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693e60c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd5c40 msgr2=0x55e693e61180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd5c40 0x55e693e61180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 msgr2=0x55e693e60c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693e60c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cd2b7640 1 -- 172.21.15.99:0/3374791158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e693df61e0 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cdab8640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd5c40 0x55e693e61180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7cd2b7640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 0x55e693df2b30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc030650 tx=0x7fb7bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d1583dd0af0721 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7a7fff640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7a7fff640 1 -- 172.21.15.99:0/3374791158 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc030830 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7a7fff640 1 -- 172.21.15.99:0/3374791158 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7bc05fc30 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7c402c070 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7a7fff640 1 -- 172.21.15.99:0/3374791158 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7bc05fe30 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 10 client.7329 fetch_fsmap learned FSMap version 65 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) v3 -- 0x55e693df4390 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7a7fff640 1 -- 172.21.15.99:0/3374791158 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c402c070 con 0x55e693dd4c60 2024-03-04T10:09:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7a7fff640 1 -- 172.21.15.99:0/3374791158 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 65) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb7bc066050 con 0x55e693dd4c60 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 10 client.7329 fetch_fsmap finished waiting for FSMap version 65 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 10 client.7329 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 20 client.7329 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693df1760 0x55e693e5d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 4 client.7329 mds_command: new command op to 7137 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.377+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 --> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e693d19690 con 0x55e693df1760 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.378+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693df1760 0x55e693e5d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.378+0000 7fb7a7fff640 10 client.7329 ms_handle_connect on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.378+0000 7fb7ccab6640 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693df1760 0x55e693e5d650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55a479bcebd84614 server_cookie=1edcf4adfd14259b in_seq=0 out_seq=0 2024-03-04T10:09:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7a7fff640 1 -- 172.21.15.99:0/3374791158 <== mds.0 v2:172.21.15.99:6833/3857383686 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e693d19690 con 0x55e693df1760 2024-03-04T10:09:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7a7fff640 10 client.7329 handle_command_reply: tid=0 2024-03-04T10:09:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 client.7329 shutdown 2024-03-04T10:09:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 msgr2=0x55e693df2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 0x55e693df2b30 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc030650 tx=0x7fb7bc0046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 shutdown_connections 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:6833/3857383686,v1:172.21.15.99:6835/3857383686] conn(0x55e693df1760 0x55e693e5d650 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e693dd5c40 0x55e693e61180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e693dd5860 0x55e693e60c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7a7fff640 0 client.7329 ms_handle_reset on v2:172.21.15.99:6833/3857383686 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 --2- 172.21.15.99:0/3374791158 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e693dd4c60 0x55e693df2b30 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 >> 172.21.15.99:0/3374791158 conn(0x55e693da8d40 msgr2=0x55e693dc20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 shutdown_connections 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.393+0000 7fb7d21d4740 1 -- 172.21.15.99:0/3374791158 wait complete. 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7d21d4740 20 client.7329 trim_cache size 0 max 16384 2024-03-04T10:09:23.397 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:09:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "message": "", 2024-03-04T10:09:23.398 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0 2024-03-04T10:09:23.398 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb7ac06b4d0 msgr2=0x7fb7ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb7ac06b4d0 0x7fb7ac06d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc036000 tx=0x7fb7bc03c040 comp rx=0 tx=0).stop 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 msgr2=0x7fb7c80c6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80c6750 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000b970 tx=0x7fb7c000be30 comp rx=0 tx=0).stop 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 shutdown_connections 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb7ac06b4d0 0x7fb7ac06d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb7c80cdd70 0x7fb7c80c6750 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb7c80ccfe0 0x7fb7c80b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 --2- 172.21.15.99:0/3055058855 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb7c80cc900 0x7fb7c81525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 >> 172.21.15.99:0/3055058855 conn(0x7fb7c80b22e0 msgr2=0x7fb7c806e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 shutdown_connections 2024-03-04T10:09:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.394+0000 7fb7cf51b640 1 -- 172.21.15.99:0/3055058855 wait complete. 2024-03-04T10:09:23.407 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_inject_rename_corrupt_dentry_first 1.0 2024-03-04T10:09:23.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.508+0000 7f1c99059640 1 Processor -- start 2024-03-04T10:09:23.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.508+0000 7f1c99059640 1 -- start start 2024-03-04T10:09:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c99059640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 0x7f1c940ad530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c99059640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940cc7a0 con 0x7f1c940ada70 2024-03-04T10:09:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c99059640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c940cca80 con 0x7f1c9415bc50 2024-03-04T10:09:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c99059640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940ccd60 con 0x7f1c9406acd0 2024-03-04T10:09:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c9359e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1c940ada70 0x7f1c940cf580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:38256/0 (socket says 172.21.15.99:38256) 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c9359e640 1 -- 172.21.15.99:0/3227110413 learned_addr learned my addr 172.21.15.99:0/3227110413 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c9259c640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 0x7f1c940ad530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203034231 0 0) 0x7f1c940cc7a0 con 0x7f1c940ada70 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c7c003520 con 0x7f1c940ada70 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3846368471 0 0) 0x7f1c7c003520 con 0x7f1c940ada70 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 msgr2=0x7f1c940ad530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c91d9b640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 0x7f1c940ad530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.509+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 >> v1:172.21.15.162:6789/0 conn(0x7f1c9415bc50 legacy=0x7f1c940cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c940cd040 con 0x7f1c940ada70 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f1c8c0019f0 con 0x7f1c940ada70 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 >> v1:172.21.15.99:6789/0 conn(0x7f1c940ada70 legacy=0x7f1c940cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c7c0039e0 0x7f1c7c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c004800 0x7f1c7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 0x7f1c7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940ccd60 con 0x7f1c7c0039e0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c00ba70 con 0x7f1c7c004800 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c00bc70 con 0x7f1c9406acd0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c92d9d640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c7c0039e0 0x7f1c7c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c9259c640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c004800 0x7f1c7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c9359e640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 0x7f1c7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c92d9d640 1 -- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 msgr2=0x7f1c7c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c92d9d640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c9406acd0 0x7f1c7c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c92d9d640 1 -- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c004800 msgr2=0x7f1c7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c92d9d640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c004800 0x7f1c7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c92d9d640 1 -- 172.21.15.99:0/3227110413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c003520 con 0x7f1c7c0039e0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.510+0000 7f1c9259c640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c004800 0x7f1c7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c92d9d640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c7c0039e0 0x7f1c7c003e00 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f1c84001720 tx=0x7f1c7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11192d8ed3ce4ab server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84001370 con 0x7f1c7c0039e0 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c840052d0 con 0x7f1c7c0039e0 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c91d9b640 1 -- 172.21.15.99:0/3227110413 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84003830 con 0x7f1c7c0039e0 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c99059640 1 -- 172.21.15.99:0/3227110413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c7c0039e0 msgr2=0x7f1c7c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c99059640 1 --2- 172.21.15.99:0/3227110413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c7c0039e0 0x7f1c7c003e00 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f1c84001720 tx=0x7f1c7c005830 comp rx=0 tx=0).stop 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c9359e640 1 -- 172.21.15.99:0/3227110413 reap_dead start 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c99059640 1 -- 172.21.15.99:0/3227110413 shutdown_connections 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c99059640 1 -- 172.21.15.99:0/3227110413 >> 172.21.15.99:0/3227110413 conn(0x7f1c940b2250 msgr2=0x7f1c940b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c99059640 1 -- 172.21.15.99:0/3227110413 shutdown_connections 2024-03-04T10:09:23.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.511+0000 7f1c99059640 1 -- 172.21.15.99:0/3227110413 wait complete. 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 Processor -- start 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 -- start start 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c0039e0 0x7f1c9415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 0x7f1c940c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c940ae600 0x7f1c9414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940cc330 con 0x7f1c940c8530 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940cdf50 con 0x7f1c7c0039e0 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.513+0000 7f1c99059640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c940cc4b0 con 0x7f1c940ae600 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 0x7f1c940c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 0x7f1c940c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41380/0 (socket says 172.21.15.99:41380) 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 -- 172.21.15.99:0/1707973717 learned_addr learned my addr 172.21.15.99:0/1707973717 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 -- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c940ae600 msgr2=0x7f1c9414b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9359e640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c940ae600 0x7f1c9414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c940ae600 0x7f1c9414b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 -- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c0039e0 msgr2=0x7f1c9415e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c0039e0 0x7f1c9415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 -- 172.21.15.99:0/1707973717 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9415e9f0 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c9259c640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 0x7f1c940c8a00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f1c880116d0 tx=0x7f1c88011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800f9c0 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8800fb60 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c940c9320 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c88015460 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.514+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9415dcb0 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.515+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9406cd70 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.516+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1c88015600 con 0x7f1c940c8530 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.516+0000 7f1c7b7fe640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1c8006b4d0 0x7f1c8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.516+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c8808ac50 con 0x7f1c940c8530 2024-03-04T10:09:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.517+0000 7f1c92d9d640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1c8006b4d0 0x7f1c8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.517+0000 7f1c92d9d640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1c8006b4d0 0x7f1c8006d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1c84033000 tx=0x7f1c8402f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.521+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1c88090020 con 0x7f1c940c8530 2024-03-04T10:09:23.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.691+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}] v 0) v1 -- 0x7f1c94074f60 con 0x7f1c940c8530 2024-03-04T10:09:23.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.695+0000 7f1c7b7fe640 1 -- 172.21.15.99:0/1707973717 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_inject_rename_corrupt_dentry_first}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1c88019030 con 0x7f1c940c8530 2024-03-04T10:09:23.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1c8006b4d0 msgr2=0x7f1c8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1c8006b4d0 0x7f1c8006d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f1c84033000 tx=0x7f1c8402f000 comp rx=0 tx=0).stop 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 msgr2=0x7f1c940c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 0x7f1c940c8a00 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f1c880116d0 tx=0x7f1c88011b90 comp rx=0 tx=0).stop 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 shutdown_connections 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1c8006b4d0 0x7f1c8006d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1c940ae600 0x7f1c9414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1c940c8530 0x7f1c940c8a00 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 --2- 172.21.15.99:0/1707973717 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c7c0039e0 0x7f1c9415e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 >> 172.21.15.99:0/1707973717 conn(0x7f1c940b2250 msgr2=0x7f1c940cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 shutdown_connections 2024-03-04T10:09:23.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:23.697+0000 7f1c99059640 1 -- 172.21.15.99:0/1707973717 wait complete. 2024-03-04T10:09:28.711 DEBUG:teuthology.orchestra.run.smithi099:> sudo daemon-helper kill ceph --cluster ceph -w 2024-03-04T10:09:28.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:28.714 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'timeout 60 mv a/b a/z') 2024-03-04T10:09:28.717 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-03-04T10:09:28.735 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:09:28.733+0000 7f9aee080640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:28.735 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:09:28.733+0000 7f9aee080640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f9aee080640 time 2024-03-04T10:09:28.735121+0000 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f9af3b81517] 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.737 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr:*** Caught signal (Aborted) ** 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7f9aee080640 thread_name:ms_dispatch 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:09:28.734+0000 7f9aee080640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f9aee080640 time 2024-03-04T10:09:28.735121+0000 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.738 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f9af3b81517] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.739 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9af3254db0] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9af32a154c] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: raise() 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: abort() 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f9af3b815d4] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.740 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:09:28.735+0000 7f9aee080640 -1 *** Caught signal (Aborted) ** 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7f9aee080640 thread_name:ms_dispatch 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9af3254db0] 2024-03-04T10:09:28.741 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9af32a154c] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: raise() 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: abort() 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f9af3b815d4] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.742 INFO:tasks.ceph.mds.a.smithi099.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.743 INFO:tasks.ceph.mds.a.smithi099.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.743 INFO:tasks.ceph.mds.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:09:28.743 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.744 INFO:tasks.ceph.mds.a.smithi099.stderr: -2462> 2024-03-04T10:09:22.098+0000 7f9aeb87b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:09:28.747 INFO:tasks.ceph.mds.a.smithi099.stderr: -8> 2024-03-04T10:09:28.733+0000 7f9aee080640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:28.747 INFO:tasks.ceph.mds.a.smithi099.stderr: -6> 2024-03-04T10:09:28.733+0000 7f9aee080640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: -1> 2024-03-04T10:09:28.734+0000 7f9aee080640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f9aee080640 time 2024-03-04T10:09:28.735121+0000 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f9af3b81517] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.748 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 0> 2024-03-04T10:09:28.735+0000 7f9aee080640 -1 *** Caught signal (Aborted) ** 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7f9aee080640 thread_name:ms_dispatch 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9af3254db0] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9af32a154c] 2024-03-04T10:09:28.749 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: raise() 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: abort() 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f9af3b815d4] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.750 INFO:tasks.ceph.mds.a.smithi099.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.751 INFO:tasks.ceph.mds.a.smithi099.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.751 INFO:tasks.ceph.mds.a.smithi099.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.751 INFO:tasks.ceph.mds.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:09:28.751 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.751 INFO:tasks.ceph.mds.a.smithi099.stderr: -2462> 2024-03-04T10:09:22.098+0000 7f9aeb87b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:09:28.753 INFO:tasks.ceph.mds.a.smithi099.stderr: -8> 2024-03-04T10:09:28.733+0000 7f9aee080640 -1 mds.0.cache.den(0x10000000000 z) newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:28.753 INFO:tasks.ceph.mds.a.smithi099.stderr: -6> 2024-03-04T10:09:28.733+0000 7f9aee080640 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:28.753 INFO:tasks.ceph.mds.a.smithi099.stderr: -1> 2024-03-04T10:09:28.734+0000 7f9aee080640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7f9aee080640 time 2024-03-04T10:09:28.735121+0000 2024-03-04T10:09:28.753 INFO:tasks.ceph.mds.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1845-g717ce3cc/rpm/el9/BUILD/ceph-19.0.0-1845-g717ce3cc/src/mds/MDSRank.cc: 939: ceph_abort_msg("abort() called") 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd3) [0x7f9af3b81517] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.754 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 0> 2024-03-04T10:09:28.735+0000 7f9aee080640 -1 *** Caught signal (Aborted) ** 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7f9aee080640 thread_name:ms_dispatch 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:09:28.755 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f9af3254db0] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f9af32a154c] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: raise() 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: abort() 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f9af3b815d4] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDSRank::abort(std::basic_string_view >)+0x61) [0x5576082174f1] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: (CDentry::check_corruption(bool)+0x48d) [0x557608426d7d] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: ceph-mds(+0x19d4a9) [0x55760825a4a9] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: (Server::_rename_prepare(boost::intrusive_ptr const&, EMetaBlob*, ceph::buffer::v15_2_0::list*, CDentry*, CDentry*, std::basic_string_view >, CDentry*)+0x208b) [0x5576082dbceb] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 10: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x2449) [0x5576082ded49] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 11: (Server::handle_client_request(boost::intrusive_ptr const&)+0x3e3) [0x5576082a7fc3] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x13c) [0x557608295a9c] 2024-03-04T10:09:28.756 INFO:tasks.ceph.mds.a.smithi099.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4eb) [0x55760821c74b] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55760821cc5c] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x557608203695] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x53a) [0x7f9af3d7c1fa] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 17: /usr/lib64/ceph/libceph-common.so.2(+0x410141) [0x7f9af3e10141] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 18: /lib64/libc.so.6(+0x9f802) [0x7f9af329f802] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 19: /lib64/libc.so.6(+0x3f450) [0x7f9af323f450] 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:09:28.757 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:09:28.852 INFO:teuthology.orchestra.run.smithi099.stdout:{"mds_beacon_grace":"15.000000"} 2024-03-04T10:09:28.853 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-03-04T10:09:28.853 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:28.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.853+0000 7f925fb0a640 1 Processor -- start 2024-03-04T10:09:28.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.853+0000 7f925fb0a640 1 -- start start 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925fb0a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580ce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925fb0a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580ceb80 con 0x7f9258070240 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925fb0a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580b1090 con 0x7f92580b0cb0 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925fb0a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925806fdb0 con 0x7f92580ce1c0 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925e0a7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f9258070240 0x7f92580706d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:38262/0 (socket says 172.21.15.99:38262) 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925e0a7640 1 -- 172.21.15.99:0/1202053675 learned_addr learned my addr 172.21.15.99:0/1202053675 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925d0a5640 1 --2- 172.21.15.99:0/1202053675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.854+0000 7f925c8a4640 1 -- 172.21.15.99:0/1202053675 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139638099 0 0) 0x7f92580ceb80 con 0x7f9258070240 2024-03-04T10:09:28.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.855+0000 7f925c8a4640 1 -- 172.21.15.99:0/1202053675 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9244003520 con 0x7f9258070240 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.855+0000 7f925c8a4640 1 -- 172.21.15.99:0/1202053675 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212766388 0 0) 0x7f92580b1090 con 0x7f92580b0cb0 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.855+0000 7f925d0a5640 1 -- 172.21.15.99:0/1202053675 >> v1:172.21.15.162:6789/0 conn(0x7f92580b0cb0 legacy=0x7f92580d0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.855+0000 7f925d0a5640 1 -- 172.21.15.99:0/1202053675 >> v1:172.21.15.99:6789/0 conn(0x7f9258070240 legacy=0x7f92580706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.855+0000 7f925d0a5640 1 -- 172.21.15.99:0/1202053675 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580d11a0 con 0x7f92580ce1c0 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.855+0000 7f925d0a5640 1 --2- 172.21.15.99:0/1202053675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580ce640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92480097d0 tx=0x7f924802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=961baaee8c79d9f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925c8a4640 1 -- 172.21.15.99:0/1202053675 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248035020 con 0x7f92580ce1c0 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925c8a4640 1 -- 172.21.15.99:0/1202053675 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9248031890 con 0x7f92580ce1c0 2024-03-04T10:09:28.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925c8a4640 1 -- 172.21.15.99:0/1202053675 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248031c40 con 0x7f92580ce1c0 2024-03-04T10:09:28.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 -- 172.21.15.99:0/1202053675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 msgr2=0x7f92580ce640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 --2- 172.21.15.99:0/1202053675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580ce640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92480097d0 tx=0x7f924802b930 comp rx=0 tx=0).stop 2024-03-04T10:09:28.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 -- 172.21.15.99:0/1202053675 shutdown_connections 2024-03-04T10:09:28.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 --2- 172.21.15.99:0/1202053675 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580ce640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 -- 172.21.15.99:0/1202053675 >> 172.21.15.99:0/1202053675 conn(0x7f92580d1840 msgr2=0x7f92580d1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 -- 172.21.15.99:0/1202053675 shutdown_connections 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.856+0000 7f925fb0a640 1 -- 172.21.15.99:0/1202053675 wait complete. 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 Processor -- start 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 -- start start 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9258070240 0x7f92581f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92580b0cb0 0x7f925815fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925806f760 con 0x7f92580b0cb0 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925806f480 con 0x7f9258070240 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925fb0a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925806fa40 con 0x7f92580ce1c0 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925d0a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92580b0cb0 0x7f925815fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925d0a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92580b0cb0 0x7f925815fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41396/0 (socket says 172.21.15.99:41396) 2024-03-04T10:09:28.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.858+0000 7f925d0a5640 1 -- 172.21.15.99:0/128889669 learned_addr learned my addr 172.21.15.99:0/128889669 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d8a6640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9258070240 0x7f92581f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925e0a7640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d0a5640 1 -- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 msgr2=0x7f92580c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d0a5640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f92580ce1c0 0x7f92580c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d0a5640 1 -- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9258070240 msgr2=0x7f92581f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d0a5640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9258070240 0x7f92581f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d0a5640 1 -- 172.21.15.99:0/128889669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92581f6d90 con 0x7f92580b0cb0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925d0a5640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f92580b0cb0 0x7f925815fd20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f924802b970 tx=0x7f92480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248035040 con 0x7f92580b0cb0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9248003820 con 0x7f92580b0cb0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248003bd0 con 0x7f92580b0cb0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925fb0a640 1 -- 172.21.15.99:0/128889669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92581f7030 con 0x7f92580b0cb0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.859+0000 7f925fb0a640 1 -- 172.21.15.99:0/128889669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92581f75f0 con 0x7f92580b0cb0 2024-03-04T10:09:28.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.860+0000 7f925fb0a640 1 -- 172.21.15.99:0/128889669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f9258074f60 con 0x7f92580b0cb0 2024-03-04T10:09:28.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.861+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9248038020 con 0x7f92580b0cb0 2024-03-04T10:09:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.861+0000 7f924e7fc640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f923c06b4d0 0x7f923c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.861+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92480aca80 con 0x7f92580b0cb0 2024-03-04T10:09:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.861+0000 7f925d8a6640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f923c06b4d0 0x7f923c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.861+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+511 (secure 0 0 0) 0x7f924802d040 con 0x7f92580b0cb0 2024-03-04T10:09:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.862+0000 7f925fb0a640 1 -- 172.21.15.99:0/128889669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f92581f77c0 con 0x7f92580b0cb0 2024-03-04T10:09:28.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.862+0000 7f925d8a6640 1 --2- 172.21.15.99:0/128889669 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f923c06b4d0 0x7f923c06d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9254007fd0 tx=0x7f9254009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:28.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.952+0000 7fea8f9b3640 1 Processor -- start 2024-03-04T10:09:28.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.952+0000 7fea8f9b3640 1 -- start start 2024-03-04T10:09:28.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8f9b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8f9b3640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fea880d1ca0 0x7fea880d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8f9b3640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea880cfb40 con 0x7fea8815bbb0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8f9b3640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea880d2880 con 0x7fea880d1ca0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8f9b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880cc900 con 0x7fea880cbf80 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8d74f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fea8815bbb0 0x7fea880d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:38276/0 (socket says 172.21.15.99:38276) 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.953+0000 7fea8d74f640 1 -- 172.21.15.99:0/2703717134 learned_addr learned my addr 172.21.15.99:0/2703717134 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea8df50640 1 --2- 172.21.15.99:0/2703717134 >> v2:172.21.15.162:3300/0 conn(0x7fea880d1ca0 0x7fea880d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea8cf4e640 1 --2- 172.21.15.99:0/2703717134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea7ffff640 1 -- 172.21.15.99:0/2703717134 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061316224 0 0) 0x7fea880cfb40 con 0x7fea8815bbb0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea7ffff640 1 -- 172.21.15.99:0/2703717134 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74003520 con 0x7fea8815bbb0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea7ffff640 1 -- 172.21.15.99:0/2703717134 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2281315960 0 0) 0x7fea74003520 con 0x7fea8815bbb0 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea8cf4e640 1 -- 172.21.15.99:0/2703717134 >> v2:172.21.15.162:3300/0 conn(0x7fea880d1ca0 msgr2=0x7fea880d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:28.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea8cf4e640 1 --2- 172.21.15.99:0/2703717134 >> v2:172.21.15.162:3300/0 conn(0x7fea880d1ca0 0x7fea880d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea8cf4e640 1 -- 172.21.15.99:0/2703717134 >> v1:172.21.15.99:6789/0 conn(0x7fea8815bbb0 legacy=0x7fea880d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.954+0000 7fea8cf4e640 1 -- 172.21.15.99:0/2703717134 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880cfe20 con 0x7fea880cbf80 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8cf4e640 1 --2- 172.21.15.99:0/2703717134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880cc3c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fea780097a0 tx=0x7fea7802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16dca86585f4f6f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea7ffff640 1 -- 172.21.15.99:0/2703717134 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78037070 con 0x7fea880cbf80 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea7ffff640 1 -- 172.21.15.99:0/2703717134 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea78030510 con 0x7fea880cbf80 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea7ffff640 1 -- 172.21.15.99:0/2703717134 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea840019f0 con 0x7fea880cbf80 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8f9b3640 1 -- 172.21.15.99:0/2703717134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 msgr2=0x7fea880cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/2703717134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880cc3c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fea780097a0 tx=0x7fea7802b790 comp rx=0 tx=0).stop 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8f9b3640 1 -- 172.21.15.99:0/2703717134 shutdown_connections 2024-03-04T10:09:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/2703717134 >> v2:172.21.15.162:3300/0 conn(0x7fea880d1ca0 0x7fea880d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/2703717134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880cc3c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.955+0000 7fea8f9b3640 1 -- 172.21.15.99:0/2703717134 >> 172.21.15.99:0/2703717134 conn(0x7fea880b3480 msgr2=0x7fea880b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.956+0000 7fea8f9b3640 1 -- 172.21.15.99:0/2703717134 shutdown_connections 2024-03-04T10:09:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.956+0000 7fea8f9b3640 1 -- 172.21.15.99:0/2703717134 wait complete. 2024-03-04T10:09:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 Processor -- start 2024-03-04T10:09:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 -- start start 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 0x7fea880a9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea8815bbb0 0x7fea880a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880d1010 con 0x7fea880d1ca0 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880d03e0 con 0x7fea8815bbb0 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.957+0000 7fea8f9b3640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea8815bf90 con 0x7fea880cbf80 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 0x7fea880a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 0x7fea880a9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41404/0 (socket says 172.21.15.99:41404) 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 -- 172.21.15.99:0/4066229934 learned_addr learned my addr 172.21.15.99:0/4066229934 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 -- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 msgr2=0x7fea880b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 -- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea8815bbb0 msgr2=0x7fea880a9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea8815bbb0 0x7fea880a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 -- 172.21.15.99:0/4066229934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880c9a10 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8cf4e640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 0x7fea880a9830 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fea78000c00 tx=0x7fea780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78041030 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7803c670 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea880c8c50 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78037040 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.958+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea880c9210 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.959+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea880c89e0 con 0x7fea880d1ca0 2024-03-04T10:09:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.960+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fea78044070 con 0x7fea880d1ca0 2024-03-04T10:09:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.960+0000 7fea7dffb640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.960+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fea780abd30 con 0x7fea880d1ca0 2024-03-04T10:09:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.960+0000 7fea8d74f640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.961+0000 7fea8d74f640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea6c06b4d0 0x7fea6c06d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fea84002230 tx=0x7fea84001dc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:28.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:28.964+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fea78045e50 con 0x7fea880d1ca0 2024-03-04T10:09:29.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.173+0000 7f924e7fc640 1 -- 172.21.15.99:0/128889669 <== mon.0 v2:172.21.15.99:3300/0 7 ==== log(1 entries from seq 1 at 2024-03-04T10:09:28.735082+0000) v1 ==== 434+0+0 (secure 0 0 0) 0x7f92480aaf30 con 0x7f92580b0cb0 2024-03-04T10:09:29.176 INFO:teuthology.orchestra.run.smithi099.stdout: cluster: 2024-03-04T10:09:29.176 INFO:teuthology.orchestra.run.smithi099.stdout: id: 8fdf3d83-d8a6-45f1-acf1-cb71ab619f47 2024-03-04T10:09:29.176 INFO:teuthology.orchestra.run.smithi099.stdout: health: HEALTH_OK 2024-03-04T10:09:29.176 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: services: 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: mgr: x(active, since 6m), standbys: y 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: mds: 1/1 daemons up, 3 standby 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: osd: 8 osds: 8 up (since 6m), 8 in (since 6m) 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: data: 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: volumes: 1/1 healthy 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: pools: 3 pools, 97 pgs 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: objects: 29 objects, 596 KiB 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: usage: 224 MiB used, 720 GiB / 720 GiB avail 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: pgs: 97 active+clean 2024-03-04T10:09:29.177 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:29.178 INFO:teuthology.orchestra.run.smithi099.stdout: io: 2024-03-04T10:09:29.178 INFO:teuthology.orchestra.run.smithi099.stdout: client: 0 B/s rd, 4.6 KiB/s wr, 0 op/s rd, 7 op/s wr 2024-03-04T10:09:29.178 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:29.178 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:29.178 INFO:teuthology.orchestra.run.smithi099.stdout:2024-03-04T10:09:28.735082+0000 mds.a [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a/z [fffffffffffffff6,head] auth NULL (dn xlock x=1 by 0x55760a688d80) (dversion lock w=1 last_client=7293) pv=11 v=10 ap=2 ino=(nil) state=1073741824 | request=1 lock=2 authpin=1 0x55760a56e280] 2024-03-04T10:09:29.178 INFO:tasks.ceph.mds.a.smithi099.stderr:daemon-helper: command crashed with signal 6 2024-03-04T10:09:29.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.175+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea88074f60 con 0x7fea880d1ca0 2024-03-04T10:09:29.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.175+0000 7fea7dffb640 1 -- 172.21.15.99:0/4066229934 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4946 (secure 0 0 0) 0x7fea7807a990 con 0x7fea880d1ca0 2024-03-04T10:09:29.179 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:29.179 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":65,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:22.453470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:29.179 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 65 2024-03-04T10:09:29.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea6c06b4d0 msgr2=0x7fea6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea6c06b4d0 0x7fea6c06d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fea84002230 tx=0x7fea84001dc0 comp rx=0 tx=0).stop 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 msgr2=0x7fea880a9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 0x7fea880a9830 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fea78000c00 tx=0x7fea780046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 shutdown_connections 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea6c06b4d0 0x7fea6c06d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea8815bbb0 0x7fea880a9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea880d1ca0 0x7fea880a9830 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 --2- 172.21.15.99:0/4066229934 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea880cbf80 0x7fea880b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 >> 172.21.15.99:0/4066229934 conn(0x7fea880b3480 msgr2=0x7fea8815ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 shutdown_connections 2024-03-04T10:09:29.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:29.178+0000 7fea8f9b3640 1 -- 172.21.15.99:0/4066229934 wait complete. 2024-03-04T10:09:29.191 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-04T10:09:33.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-04T10:09:34.192 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:34.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.301+0000 7f1e60177640 1 Processor -- start 2024-03-04T10:09:34.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.301+0000 7f1e60177640 1 -- start start 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e60177640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1e58158410 0x7f1e580b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e60177640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e60177640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5806b500 con 0x7f1e58161ca0 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e60177640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e580b3f00 con 0x7f1e58158410 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e60177640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5806b7e0 con 0x7f1e5806acd0 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e5d712640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e5df13640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1e58158410 0x7f1e580b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e5d712640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:32878/0 (socket says 172.21.15.99:32878) 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e5d712640 1 -- 172.21.15.99:0/913790127 learned_addr learned my addr 172.21.15.99:0/913790127 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:34.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.302+0000 7f1e5e714640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f1e58161ca0 0x7f1e5815e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:51550/0 (socket says 172.21.15.99:51550) 2024-03-04T10:09:34.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5d712640 1 -- 172.21.15.99:0/913790127 >> v2:172.21.15.162:3300/0 conn(0x7f1e58158410 msgr2=0x7f1e580b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:34.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5d712640 1 --2- 172.21.15.99:0/913790127 >> v2:172.21.15.162:3300/0 conn(0x7f1e58158410 0x7f1e580b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5d712640 1 -- 172.21.15.99:0/913790127 >> v1:172.21.15.162:6789/0 conn(0x7f1e58161ca0 legacy=0x7f1e5815e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:34.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5d712640 1 -- 172.21.15.99:0/913790127 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5806bac0 con 0x7f1e5806acd0 2024-03-04T10:09:34.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5df13640 1 --2- 172.21.15.99:0/913790127 >> v2:172.21.15.162:3300/0 conn(0x7f1e58158410 0x7f1e580b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:34.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5d712640 1 --2- 172.21.15.99:0/913790127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58161760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009870 tx=0x7f1e4802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1992fa9a9aba1db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:34.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.303+0000 7f1e5cf11640 1 -- 172.21.15.99:0/913790127 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4802f840 con 0x7f1e5806acd0 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e5cf11640 1 -- 172.21.15.99:0/913790127 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4802f9e0 con 0x7f1e5806acd0 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e5cf11640 1 -- 172.21.15.99:0/913790127 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4802fdb0 con 0x7f1e5806acd0 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 -- 172.21.15.99:0/913790127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 msgr2=0x7f1e58161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 --2- 172.21.15.99:0/913790127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58161760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009870 tx=0x7f1e4802ba20 comp rx=0 tx=0).stop 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 -- 172.21.15.99:0/913790127 shutdown_connections 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 --2- 172.21.15.99:0/913790127 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58161760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 --2- 172.21.15.99:0/913790127 >> v2:172.21.15.162:3300/0 conn(0x7f1e58158410 0x7f1e580b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 -- 172.21.15.99:0/913790127 >> 172.21.15.99:0/913790127 conn(0x7f1e580cfef0 msgr2=0x7f1e580d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 -- 172.21.15.99:0/913790127 shutdown_connections 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.304+0000 7f1e60177640 1 -- 172.21.15.99:0/913790127 wait complete. 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.305+0000 7f1e60177640 1 Processor -- start 2024-03-04T10:09:34.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.305+0000 7f1e60177640 1 -- start start 2024-03-04T10:09:34.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e60177640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:34.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e60177640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e58158410 0x7f1e58159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:34.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e60177640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 0x7f1e580b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:34.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e60177640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5806c5a0 con 0x7f1e58161ca0 2024-03-04T10:09:34.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e60177640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5806c080 con 0x7f1e58158410 2024-03-04T10:09:34.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e60177640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5806ee60 con 0x7f1e5806acd0 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e5e714640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 0x7f1e580b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e5e714640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 0x7f1e580b0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54898/0 (socket says 172.21.15.99:54898) 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e5e714640 1 -- 172.21.15.99:0/2876987794 learned_addr learned my addr 172.21.15.99:0/2876987794 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e5df13640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.306+0000 7f1e5d712640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e58158410 0x7f1e58159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e5e714640 1 -- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 msgr2=0x7f1e58155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e5e714640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e5e714640 1 -- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e58158410 msgr2=0x7f1e58159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e5e714640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e58158410 0x7f1e58159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e5e714640 1 -- 172.21.15.99:0/2876987794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e580ce310 con 0x7f1e58161ca0 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e5e714640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 0x7f1e580b0720 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000c730 tx=0x7f1e5000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:34.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50017070 con 0x7f1e58161ca0 2024-03-04T10:09:34.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5000ce40 con 0x7f1e58161ca0 2024-03-04T10:09:34.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e580ce030 con 0x7f1e58161ca0 2024-03-04T10:09:34.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5000a840 con 0x7f1e58161ca0 2024-03-04T10:09:34.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.307+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e58158b70 con 0x7f1e58161ca0 2024-03-04T10:09:34.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.308+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e58074f60 con 0x7f1e58161ca0 2024-03-04T10:09:34.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.308+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1e5001a070 con 0x7f1e58161ca0 2024-03-04T10:09:34.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.309+0000 7f1e4e7fc640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e3c06b4d0 0x7f1e3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:34.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.309+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1e50089760 con 0x7f1e58161ca0 2024-03-04T10:09:34.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.309+0000 7f1e5df13640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e3c06b4d0 0x7f1e3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:34.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.310+0000 7f1e5df13640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e3c06b4d0 0x7f1e3c06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009ac0 tx=0x7f1e54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:34.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.313+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e50057e90 con 0x7f1e58161ca0 2024-03-04T10:09:34.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.517+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e58066e10 con 0x7f1e58161ca0 2024-03-04T10:09:34.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.518+0000 7f1e4e7fc640 1 -- 172.21.15.99:0/2876987794 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f1e5005bf30 con 0x7f1e58161ca0 2024-03-04T10:09:34.520 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:34.520 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":65,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:22.453470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 65 2024-03-04T10:09:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e3c06b4d0 msgr2=0x7f1e3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e3c06b4d0 0x7f1e3c06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009ac0 tx=0x7f1e54008040 comp rx=0 tx=0).stop 2024-03-04T10:09:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 msgr2=0x7f1e580b0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 0x7f1e580b0720 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1e5000c730 tx=0x7f1e5000cbf0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 shutdown_connections 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e3c06b4d0 0x7f1e3c06d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e58161ca0 0x7f1e580b0720 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e58158410 0x7f1e58159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 --2- 172.21.15.99:0/2876987794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e5806acd0 0x7f1e58155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 >> 172.21.15.99:0/2876987794 conn(0x7f1e580cfef0 msgr2=0x7f1e580aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 shutdown_connections 2024-03-04T10:09:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:34.520+0000 7f1e60177640 1 -- 172.21.15.99:0/2876987794 wait complete. 2024-03-04T10:09:34.533 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-04T10:09:39.534 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:39.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.644+0000 7f7317509640 1 Processor -- start 2024-03-04T10:09:39.646 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.644+0000 7f7317509640 1 -- start start 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7317509640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f73100b2e60 0x7f73100b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7317509640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73100b5470 0x7f73100b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7317509640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731015e1e0 con 0x7f731015de00 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7317509640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100b98c0 con 0x7f73100b2e60 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7317509640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73100b32c0 con 0x7f73100b5470 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7314aa4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f73100b2e60 0x7f73100b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7314aa4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f73100b2e60 0x7f73100b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54910/0 (socket says 172.21.15.99:54910) 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7314aa4640 1 -- 172.21.15.99:0/3348766573 learned_addr learned my addr 172.21.15.99:0/3348766573 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f73152a5640 1 --1- 172.21.15.99:0/3348766573 >> v1:172.21.15.99:6789/0 conn(0x7f731015de00 0x7f73100b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56736/0 (socket says 172.21.15.99:56736) 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.645+0000 7f7315aa6640 1 --2- 172.21.15.99:0/3348766573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73100b5470 0x7f73100b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7314aa4640 1 -- 172.21.15.99:0/3348766573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73100b5470 msgr2=0x7f73100b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7314aa4640 1 --2- 172.21.15.99:0/3348766573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73100b5470 0x7f73100b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7314aa4640 1 -- 172.21.15.99:0/3348766573 >> v1:172.21.15.99:6789/0 conn(0x7f731015de00 legacy=0x7f73100b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7307fff640 1 -- 172.21.15.99:0/3348766573 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497259963 0 0) 0x7f731015e1e0 con 0x7f731015de00 2024-03-04T10:09:39.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7314aa4640 1 -- 172.21.15.99:0/3348766573 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100d1c10 con 0x7f73100b2e60 2024-03-04T10:09:39.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7314aa4640 1 --2- 172.21.15.99:0/3348766573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100b9380 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7300009870 tx=0x7f730002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2137aa8f169ba4c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7307fff640 1 -- 172.21.15.99:0/3348766573 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300034070 con 0x7f73100b2e60 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7307fff640 1 -- 172.21.15.99:0/3348766573 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7300030840 con 0x7f73100b2e60 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.646+0000 7f7307fff640 1 -- 172.21.15.99:0/3348766573 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300030c10 con 0x7f73100b2e60 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 -- 172.21.15.99:0/3348766573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 msgr2=0x7f73100b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 --2- 172.21.15.99:0/3348766573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100b9380 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7300009870 tx=0x7f730002b980 comp rx=0 tx=0).stop 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 -- 172.21.15.99:0/3348766573 shutdown_connections 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 --2- 172.21.15.99:0/3348766573 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f73100b5470 0x7f73100b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 --2- 172.21.15.99:0/3348766573 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100b9380 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 -- 172.21.15.99:0/3348766573 >> 172.21.15.99:0/3348766573 conn(0x7f73100d0470 msgr2=0x7f73100d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 -- 172.21.15.99:0/3348766573 shutdown_connections 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 -- 172.21.15.99:0/3348766573 wait complete. 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 Processor -- start 2024-03-04T10:09:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.648+0000 7f7317509640 1 -- start start 2024-03-04T10:09:39.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7317509640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7317509640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73100b5470 0x7f73100cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7317509640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f731015de00 0x7f73101f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7317509640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100d21d0 con 0x7f73100b2e60 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7317509640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100d2820 con 0x7f73100b5470 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7317509640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73100d24b0 con 0x7f731015de00 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54916/0 (socket says 172.21.15.99:54916) 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 -- 172.21.15.99:0/29082149 learned_addr learned my addr 172.21.15.99:0/29082149 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7314aa4640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73100b5470 0x7f73100cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7315aa6640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f731015de00 0x7f73101f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 -- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f731015de00 msgr2=0x7f73101f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f731015de00 0x7f73101f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 -- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73100b5470 msgr2=0x7f73100cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73100b5470 0x7f73100cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f73152a5640 1 -- 172.21.15.99:0/29082149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100c8aa0 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.649+0000 7f7314aa4640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73100b5470 0x7f73100cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.650+0000 7f73152a5640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100cab30 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f730c00c890 tx=0x7f730c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.650+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c018070 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.650+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00a5e0 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.650+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00a9d0 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.650+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73100c8d80 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.650+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73100c9340 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.652+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7310074f60 con 0x7f73100b2e60 2024-03-04T10:09:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.652+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f730c01b070 con 0x7f73100b2e60 2024-03-04T10:09:39.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.652+0000 7f7305ffb640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f72f406b4d0 0x7f72f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:39.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.652+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f730c089980 con 0x7f73100b2e60 2024-03-04T10:09:39.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.652+0000 7f7314aa4640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f72f406b4d0 0x7f72f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:39.656 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.652+0000 7f7314aa4640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f72f406b4d0 0x7f72f406d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7300036000 tx=0x7f730003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:39.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.656+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f730c058700 con 0x7f73100b2e60 2024-03-04T10:09:39.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.865+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73100b3710 con 0x7f73100b2e60 2024-03-04T10:09:39.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.866+0000 7f7305ffb640 1 -- 172.21.15.99:0/29082149 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4946 (secure 0 0 0) 0x7f730c058000 con 0x7f73100b2e60 2024-03-04T10:09:39.868 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:39.868 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":65,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:22.453470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:39.868 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 65 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f72f406b4d0 msgr2=0x7f72f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f72f406b4d0 0x7f72f406d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f7300036000 tx=0x7f730003b040 comp rx=0 tx=0).stop 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 msgr2=0x7f73100cab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100cab30 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f730c00c890 tx=0x7f730c00cd50 comp rx=0 tx=0).stop 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 shutdown_connections 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f72f406b4d0 0x7f72f406d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f731015de00 0x7f73101f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f73100b5470 0x7f73100cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 --2- 172.21.15.99:0/29082149 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f73100b2e60 0x7f73100cab30 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 >> 172.21.15.99:0/29082149 conn(0x7f73100d0470 msgr2=0x7f73100b4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 shutdown_connections 2024-03-04T10:09:39.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:39.868+0000 7f7317509640 1 -- 172.21.15.99:0/29082149 wait complete. 2024-03-04T10:09:39.880 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-03-04T10:09:40.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-03-04T10:09:40.357 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:40.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:09:44.881 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:44.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.992+0000 7f71d3174640 1 Processor -- start 2024-03-04T10:09:44.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.992+0000 7f71d3174640 1 -- start start 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d3174640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f71cc0cfd60 0x7f71cc0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d3174640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71cc0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d3174640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71cc0d0140 con 0x7f71cc06d3b0 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d3174640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71cc15cbb0 con 0x7f71cc0cfd60 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d3174640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc15c280 con 0x7f71cc0cf0d0 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cbfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f71cc06d3b0 0x7f71cc0ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49156/0 (socket says 172.21.15.99:49156) 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cbfff640 1 -- 172.21.15.99:0/778219517 learned_addr learned my addr 172.21.15.99:0/778219517 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:44.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d0f10640 1 --2- 172.21.15.99:0/778219517 >> v2:172.21.15.162:3300/0 conn(0x7f71cc0cfd60 0x7f71cc0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71d1711640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71cc0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071723508 0 0) 0x7f71cc0d0140 con 0x7f71cc06d3b0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc003520 con 0x7f71cc06d3b0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737830604 0 0) 0x7f71bc003520 con 0x7f71cc06d3b0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 msgr2=0x7f71cc0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71cc0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 >> v2:172.21.15.162:3300/0 conn(0x7f71cc0cfd60 msgr2=0x7f71cc0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 --2- 172.21.15.99:0/778219517 >> v2:172.21.15.162:3300/0 conn(0x7f71cc0cfd60 0x7f71cc0cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.993+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc15c560 con 0x7f71cc06d3b0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f71b8001670 con 0x7f71cc06d3b0 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71d0f10640 1 --2- 172.21.15.99:0/778219517 >> v2:172.21.15.162:3300/0 conn(0x7f71cc0cfd60 0x7f71cc0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 >> v1:172.21.15.99:6789/0 conn(0x7f71cc06d3b0 legacy=0x7f71cc0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:44.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71bc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71bc003f30 0x7f71bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71bc004d40 0x7f71bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc15cbb0 con 0x7f71bc003f30 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc15c280 con 0x7f71bc004d40 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71bc00ba10 con 0x7f71cc0cf0d0 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cbfff640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71bc003f30 0x7f71bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71d1711640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71bc004d40 0x7f71bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71d0f10640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71bc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cbfff640 1 -- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 msgr2=0x7f71bc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cbfff640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71bc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cbfff640 1 -- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71bc004d40 msgr2=0x7f71bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cbfff640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71bc004d40 0x7f71bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71cbfff640 1 -- 172.21.15.99:0/778219517 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc003520 con 0x7f71bc003f30 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.994+0000 7f71d0f10640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71cc0cf0d0 0x7f71bc0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71cbfff640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71bc003f30 0x7f71bc004800 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f71b8007fd0 tx=0x7f71bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27d438cb1ca36fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:44.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8007d50 con 0x7f71bc003f30 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b8030070 con 0x7f71bc003f30 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71cb7fe640 1 -- 172.21.15.99:0/778219517 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8005d10 con 0x7f71bc003f30 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71d3174640 1 -- 172.21.15.99:0/778219517 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71bc003f30 msgr2=0x7f71bc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71d3174640 1 --2- 172.21.15.99:0/778219517 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71bc003f30 0x7f71bc004800 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f71b8007fd0 tx=0x7f71bc005830 comp rx=0 tx=0).stop 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71d1711640 1 -- 172.21.15.99:0/778219517 reap_dead start 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71d3174640 1 -- 172.21.15.99:0/778219517 shutdown_connections 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.995+0000 7f71d3174640 1 -- 172.21.15.99:0/778219517 >> 172.21.15.99:0/778219517 conn(0x7f71cc06acd0 msgr2=0x7f71cc06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.996+0000 7f71d3174640 1 -- 172.21.15.99:0/778219517 shutdown_connections 2024-03-04T10:09:45.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.996+0000 7f71d3174640 1 -- 172.21.15.99:0/778219517 wait complete. 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.997+0000 7f71d3174640 1 Processor -- start 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.997+0000 7f71d3174640 1 -- start start 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71bc003f30 0x7f71cc154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 0x7f71cc15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71cc1f70e0 0x7f71cc1f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0cc330 con 0x7f71cc15af50 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc15bf30 con 0x7f71cc1f70e0 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71cc0cc4b0 con 0x7f71bc003f30 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 0x7f71cc15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 0x7f71cc15b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53412/0 (socket says 172.21.15.99:53412) 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 -- 172.21.15.99:0/810497974 learned_addr learned my addr 172.21.15.99:0/810497974 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d1711640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71cc1f70e0 0x7f71cc1f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d0f10640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71bc003f30 0x7f71cc154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 -- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71bc003f30 msgr2=0x7f71cc154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71bc003f30 0x7f71cc154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 -- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71cc1f70e0 msgr2=0x7f71cc1f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71cc1f70e0 0x7f71cc1f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 -- 172.21.15.99:0/810497974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71cc1f7e40 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71cbfff640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 0x7f71cc15b3d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f71b802c520 tx=0x7f71b8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8001cb0 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b8004bf0 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8007070 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71cc155150 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.998+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71cc0ad8a0 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:44.999+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71cc074f60 con 0x7f71cc15af50 2024-03-04T10:09:45.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.000+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f71b8007210 con 0x7f71cc15af50 2024-03-04T10:09:45.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.000+0000 7f71c97fa640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f71b006b4d0 0x7f71b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.000+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f71b80ab7f0 con 0x7f71cc15af50 2024-03-04T10:09:45.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.000+0000 7f71d0f10640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f71b006b4d0 0x7f71b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.001+0000 7f71d0f10640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f71b006b4d0 0x7f71b006d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f71cc1598b0 tx=0x7f71c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.005+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f71cc155150 con 0x7f71cc15af50 2024-03-04T10:09:45.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.209+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71cc155a40 con 0x7f71cc15af50 2024-03-04T10:09:45.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.210+0000 7f71c97fa640 1 -- 172.21.15.99:0/810497974 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4994 (secure 0 0 0) 0x7f71b807a710 con 0x7f71cc15af50 2024-03-04T10:09:45.212 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:45.212 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":66,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7134,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:44.966809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7137},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7137":{"gid":7137,"name":"a","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3857383686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":3857383686},{"type":"v1","addr":"172.21.15.99:6835","nonce":3857383686}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:09:44.966809+0000","export_targets":[],"features":4540701547738038271,"flags":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:45.212 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 66 2024-03-04T10:09:45.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.212+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f71b006b4d0 msgr2=0x7f71b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.212+0000 7f71d3174640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f71b006b4d0 0x7f71b006d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f71cc1598b0 tx=0x7f71c0008040 comp rx=0 tx=0).stop 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.212+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 msgr2=0x7f71cc15b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.212+0000 7f71d3174640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 0x7f71cc15b3d0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f71b802c520 tx=0x7f71b8000fe0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 shutdown_connections 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f71b006b4d0 0x7f71b006d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71cc1f70e0 0x7f71cc1f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71cc15af50 0x7f71cc15b3d0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 --2- 172.21.15.99:0/810497974 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f71bc003f30 0x7f71cc154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 >> 172.21.15.99:0/810497974 conn(0x7f71cc06acd0 msgr2=0x7f71cc0ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 shutdown_connections 2024-03-04T10:09:45.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.213+0000 7f71d3174640 1 -- 172.21.15.99:0/810497974 wait complete. 2024-03-04T10:09:45.225 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-03-04T10:09:45.326 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-03-04T10:09:45.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.428+0000 7f0610843640 1 Processor -- start 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.428+0000 7f0610843640 1 -- start start 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0610843640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0cdd70 0x7f060c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0610843640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f060c0cc900 0x7f060c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0610843640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f060c06ae00 con 0x7f060c0ccfe0 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0610843640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060c0ccde0 con 0x7f060c0cc900 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0610843640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c0d0480 con 0x7f060c0cdd70 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f060c0cc900 0x7f060c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f060c0cc900 0x7f060c0cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53424/0 (socket says 172.21.15.99:53424) 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f060ad9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f060c0ccfe0 0x7f060c15d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49164/0 (socket says 172.21.15.99:49164) 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 -- 172.21.15.99:0/1159209478 learned_addr learned my addr 172.21.15.99:0/1159209478 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 -- 172.21.15.99:0/1159209478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0cdd70 msgr2=0x7f060c0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 --2- 172.21.15.99:0/1159209478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0cdd70 0x7f060c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 -- 172.21.15.99:0/1159209478 >> v1:172.21.15.99:6789/0 conn(0x7f060c0ccfe0 legacy=0x7f060c15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f060959a640 1 -- 172.21.15.99:0/1159209478 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227328363 0 0) 0x7f060c06ae00 con 0x7f060c0ccfe0 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.429+0000 7f0609d9b640 1 -- 172.21.15.99:0/1159209478 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c06b0e0 con 0x7f060c0cc900 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.430+0000 7f0609d9b640 1 --2- 172.21.15.99:0/1159209478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cc900 0x7f060c0cff40 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f0600009870 tx=0x7f060002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e9e6463a751ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.430+0000 7f060959a640 1 -- 172.21.15.99:0/1159209478 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600035020 con 0x7f060c0cc900 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.430+0000 7f060959a640 1 -- 172.21.15.99:0/1159209478 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0600031640 con 0x7f060c0cc900 2024-03-04T10:09:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.430+0000 7f060959a640 1 -- 172.21.15.99:0/1159209478 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060002d040 con 0x7f060c0cc900 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 -- 172.21.15.99:0/1159209478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cc900 msgr2=0x7f060c0cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 --2- 172.21.15.99:0/1159209478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cc900 0x7f060c0cff40 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f0600009870 tx=0x7f060002b8f0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 -- 172.21.15.99:0/1159209478 shutdown_connections 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 --2- 172.21.15.99:0/1159209478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cc900 0x7f060c0cff40 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 --2- 172.21.15.99:0/1159209478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0cdd70 0x7f060c0cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 -- 172.21.15.99:0/1159209478 >> 172.21.15.99:0/1159209478 conn(0x7f060c0b22e0 msgr2=0x7f060c0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 -- 172.21.15.99:0/1159209478 shutdown_connections 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.431+0000 7f0610843640 1 -- 172.21.15.99:0/1159209478 wait complete. 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.432+0000 7f0610843640 1 Processor -- start 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.432+0000 7f0610843640 1 -- start start 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0cc900 0x7f060c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0ccfe0 0x7f060c14b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 0x7f060c14dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c0b2e90 con 0x7f060c0cdd70 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c06b6a0 con 0x7f060c0cc900 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060c0b3010 con 0x7f060c0ccfe0 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 0x7f060c14dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 0x7f060c14dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53436/0 (socket says 172.21.15.99:53436) 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 -- 172.21.15.99:0/677055174 learned_addr learned my addr 172.21.15.99:0/677055174 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0609d9b640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0ccfe0 0x7f060c14b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060a59c640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0cc900 0x7f060c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 -- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0ccfe0 msgr2=0x7f060c14b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0ccfe0 0x7f060c14b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 -- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0cc900 msgr2=0x7f060c0b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0cc900 0x7f060c0b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 -- 172.21.15.99:0/677055174 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c15af80 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0609d9b640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0ccfe0 0x7f060c14b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060a59c640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0cc900 0x7f060c0b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f060ad9d640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 0x7f060c14dbe0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f060400c700 tx=0x7f060400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0604018070 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060400ce10 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c0c7dd0 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060400a860 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.433+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c0b46e0 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.435+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c074f60 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.435+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f060401b070 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.435+0000 7f05faffd640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05ec06b4d0 0x7f05ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.435+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0604089980 con 0x7f060c0cdd70 2024-03-04T10:09:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.435+0000 7f060a59c640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05ec06b4d0 0x7f05ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.436+0000 7f060a59c640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05ec06b4d0 0x7f05ec06d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f05fc006fd0 tx=0x7f05fc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.439+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f060c15af80 con 0x7f060c0cdd70 2024-03-04T10:09:45.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.610+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) v1 -- 0x7f060c0ce480 con 0x7f060c0cdd70 2024-03-04T10:09:45.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.613+0000 7f05faffd640 1 -- 172.21.15.99:0/677055174 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v24) v1 ==== 121+0+0 (secure 0 0 0) 0x7f06040585e0 con 0x7f060c0cdd70 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05ec06b4d0 msgr2=0x7f05ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05ec06b4d0 0x7f05ec06d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f05fc006fd0 tx=0x7f05fc008040 comp rx=0 tx=0).stop 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 msgr2=0x7f060c14dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 0x7f060c14dbe0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f060400c700 tx=0x7f060400cbc0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 shutdown_connections 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f05ec06b4d0 0x7f05ec06d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f060c0cdd70 0x7f060c14dbe0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f060c0ccfe0 0x7f060c14b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 --2- 172.21.15.99:0/677055174 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f060c0cc900 0x7f060c0b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 >> 172.21.15.99:0/677055174 conn(0x7f060c0b22e0 msgr2=0x7f060c0af560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 shutdown_connections 2024-03-04T10:09:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.615+0000 7f0610843640 1 -- 172.21.15.99:0/677055174 wait complete. 2024-03-04T10:09:45.628 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds freeze 5:0 false 2024-03-04T10:09:45.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.729+0000 7fba1d285640 1 Processor -- start 2024-03-04T10:09:45.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.729+0000 7fba1d285640 1 -- start start 2024-03-04T10:09:45.733 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba1d285640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba1815a810 0x7fba180b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba1d285640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba180cbfb0 0x7fba180b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba1d285640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba1806d8a0 con 0x7fba1806d3c0 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba1d285640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba1806db80 con 0x7fba1815a810 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba1d285640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1806de60 con 0x7fba180cbfb0 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba1815a810 0x7fba180b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fba1815a810 0x7fba180b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53444/0 (socket says 172.21.15.99:53444) 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 -- 172.21.15.99:0/1062023059 learned_addr learned my addr 172.21.15.99:0/1062023059 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba177fe640 1 --2- 172.21.15.99:0/1062023059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba180cbfb0 0x7fba180b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 -- 172.21.15.99:0/1062023059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba180cbfb0 msgr2=0x7fba180b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 --2- 172.21.15.99:0/1062023059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba180cbfb0 0x7fba180b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 -- 172.21.15.99:0/1062023059 >> v1:172.21.15.162:6789/0 conn(0x7fba1806d3c0 legacy=0x7fba1815abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.730+0000 7fba16ffd640 1 -- 172.21.15.99:0/1062023059 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1806e140 con 0x7fba1815a810 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.731+0000 7fba16ffd640 1 --2- 172.21.15.99:0/1062023059 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1815a810 0x7fba180b4190 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fba08009870 tx=0x7fba0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7ff9f3f7e981dd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.731+0000 7fba15ffb640 1 -- 172.21.15.99:0/1062023059 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08034070 con 0x7fba1815a810 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.731+0000 7fba15ffb640 1 -- 172.21.15.99:0/1062023059 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba08030840 con 0x7fba1815a810 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.731+0000 7fba15ffb640 1 -- 172.21.15.99:0/1062023059 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08030c10 con 0x7fba1815a810 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 -- 172.21.15.99:0/1062023059 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1815a810 msgr2=0x7fba180b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 --2- 172.21.15.99:0/1062023059 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1815a810 0x7fba180b4190 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fba08009870 tx=0x7fba0802b980 comp rx=0 tx=0).stop 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 -- 172.21.15.99:0/1062023059 shutdown_connections 2024-03-04T10:09:45.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 --2- 172.21.15.99:0/1062023059 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba180cbfb0 0x7fba180b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 --2- 172.21.15.99:0/1062023059 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1815a810 0x7fba180b4190 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 -- 172.21.15.99:0/1062023059 >> 172.21.15.99:0/1062023059 conn(0x7fba1806acd0 msgr2=0x7fba1806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:45.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 -- 172.21.15.99:0/1062023059 shutdown_connections 2024-03-04T10:09:45.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.732+0000 7fba1d285640 1 -- 172.21.15.99:0/1062023059 wait complete. 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.733+0000 7fba1d285640 1 Processor -- start 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.733+0000 7fba1d285640 1 -- start start 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba1d285640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 0x7fba181584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba1d285640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba180cbfb0 0x7fba180c5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba1d285640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba1815a810 0x7fba180b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba1d285640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1815e3c0 con 0x7fba1806d3c0 2024-03-04T10:09:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba1d285640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180b3c90 con 0x7fba180cbfb0 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba1d285640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1815e540 con 0x7fba1815a810 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 0x7fba181584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 0x7fba181584a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53454/0 (socket says 172.21.15.99:53454) 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 -- 172.21.15.99:0/301974537 learned_addr learned my addr 172.21.15.99:0/301974537 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba167fc640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba180cbfb0 0x7fba180c5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba177fe640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba1815a810 0x7fba180b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 -- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba1815a810 msgr2=0x7fba180b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba1815a810 0x7fba180b0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 -- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba180cbfb0 msgr2=0x7fba180c5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba180cbfb0 0x7fba180c5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 -- 172.21.15.99:0/301974537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba180c5bd0 con 0x7fba1806d3c0 2024-03-04T10:09:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.734+0000 7fba16ffd640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 0x7fba181584a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fba0802b9c0 tx=0x7fba080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.735+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08034040 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.735+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba08039d90 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.735+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba180ab710 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.735+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08043610 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.735+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba180abcd0 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.736+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba18074f60 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.736+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fba0804a060 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.736+0000 7fb9ff7fe640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba0406b4d0 0x7fba0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.736+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fba080ac370 con 0x7fba1806d3c0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.737+0000 7fba167fc640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba0406b4d0 0x7fba0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.737+0000 7fba167fc640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba0406b4d0 0x7fba0406d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fba0c001640 tx=0x7fba0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:45.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.741+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fba0803e070 con 0x7fba1806d3c0 2024-03-04T10:09:45.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:45.979+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"} v 0) v1 -- 0x7fba1815baf0 con 0x7fba1806d3c0 2024-03-04T10:09:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.613+0000 7fb9ff7fe640 1 -- 172.21.15.99:0/301974537 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds freeze", "role_or_gid": "5:0", "val": "false"}]=0 unfreezing mds.7137 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7fba0802d330 con 0x7fba1806d3c0 2024-03-04T10:09:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:unfreezing mds.7137 2024-03-04T10:09:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.615+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba0406b4d0 msgr2=0x7fba0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.615+0000 7fba1d285640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba0406b4d0 0x7fba0406d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fba0c001640 tx=0x7fba0c009040 comp rx=0 tx=0).stop 2024-03-04T10:09:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 msgr2=0x7fba181584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 0x7fba181584a0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fba0802b9c0 tx=0x7fba080046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 shutdown_connections 2024-03-04T10:09:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fba0406b4d0 0x7fba0406d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fba1815a810 0x7fba180b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fba180cbfb0 0x7fba180c5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 --2- 172.21.15.99:0/301974537 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fba1806d3c0 0x7fba181584a0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 >> 172.21.15.99:0/301974537 conn(0x7fba1806acd0 msgr2=0x7fba181596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 shutdown_connections 2024-03-04T10:09:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.616+0000 7fba1d285640 1 -- 172.21.15.99:0/301974537 wait complete. 2024-03-04T10:09:46.634 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -n kernel.core_pattern 2024-03-04T10:09:46.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-03-04T10:09:46.661 INFO:teuthology.orchestra.run.smithi099.stdout:/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T10:09:46.663 INFO:tasks.cephfs.cephfs_test_case:Clearing core from directory: /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:09:46.663 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest/archive/coredump && sudo ls | sudo xargs file 2024-03-04T10:09:46.734 INFO:teuthology.orchestra.run.smithi099.stdout:1709546968.65199.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-03-04T10:09:46.736 INFO:tasks.cephfs.cephfs_test_case:Enumerated cores: ['1709546968.65199.core'] 2024-03-04T10:09:46.736 INFO:tasks.cephfs.cephfs_test_case:Found core file 1709546968.65199.core, deleting it 2024-03-04T10:09:46.737 DEBUG:teuthology.orchestra.run.smithi099:> cd /home/ubuntu/cephtest/archive/coredump && sudo rm -f 1709546968.65199.core 2024-03-04T10:09:46.807 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:09:46.807 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T10:09:46.807 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:09:46.807 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:09:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:09:46.808 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-03-04T10:09:46.808 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:09:46.808 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:09:46.850 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:09:46.850 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:46.926 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:09:46.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.950+0000 7f604640b640 1 Processor -- start 2024-03-04T10:09:46.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 -- start start 2024-03-04T10:09:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f604015de00 0x7f60400b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f60400b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f60400b70d0 0x7f60400b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60400d2820 con 0x7f60400b70d0 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604015e1e0 con 0x7f604015de00 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.951+0000 7f604640b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400d1c10 con 0x7f60400b9380 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f60400b70d0 0x7f60400b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f60400b70d0 0x7f60400b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53486/0 (socket says 172.21.15.99:53486) 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 -- 172.21.15.99:0/375031002 learned_addr learned my addr 172.21.15.99:0/375031002 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f603ffff640 1 --2- 172.21.15.99:0/375031002 >> v2:172.21.15.162:3300/0 conn(0x7f604015de00 0x7f60400b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f603f7fe640 1 --2- 172.21.15.99:0/375031002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f60400b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 -- 172.21.15.99:0/375031002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 msgr2=0x7f60400b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 --2- 172.21.15.99:0/375031002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f60400b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 -- 172.21.15.99:0/375031002 >> v2:172.21.15.162:3300/0 conn(0x7f604015de00 msgr2=0x7f60400b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 --2- 172.21.15.99:0/375031002 >> v2:172.21.15.162:3300/0 conn(0x7f604015de00 0x7f60400b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.952+0000 7f60449a8640 1 -- 172.21.15.99:0/375031002 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400d1ef0 con 0x7f60400b70d0 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f60449a8640 1 --2- 172.21.15.99:0/375031002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b5470 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f60400ce240 tx=0x7f603802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814508b23169e998 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f603effd640 1 -- 172.21.15.99:0/375031002 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603802ca70 con 0x7f60400b70d0 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f603effd640 1 -- 172.21.15.99:0/375031002 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603802cc10 con 0x7f60400b70d0 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f603effd640 1 -- 172.21.15.99:0/375031002 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60380343f0 con 0x7f60400b70d0 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 -- 172.21.15.99:0/375031002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 msgr2=0x7f60400b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 --2- 172.21.15.99:0/375031002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b5470 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f60400ce240 tx=0x7f603802bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 -- 172.21.15.99:0/375031002 shutdown_connections 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 --2- 172.21.15.99:0/375031002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b5470 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 --2- 172.21.15.99:0/375031002 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f60400b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 --2- 172.21.15.99:0/375031002 >> v2:172.21.15.162:3300/0 conn(0x7f604015de00 0x7f60400b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 -- 172.21.15.99:0/375031002 >> 172.21.15.99:0/375031002 conn(0x7f60400d0470 msgr2=0x7f60400d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 -- 172.21.15.99:0/375031002 shutdown_connections 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.953+0000 7f604640b640 1 -- 172.21.15.99:0/375031002 wait complete. 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.954+0000 7f604640b640 1 Processor -- start 2024-03-04T10:09:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 -- start start 2024-03-04T10:09:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f604015b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604015de00 0x7f60400ab530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400b97e0 con 0x7f60400b70d0 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400d24b0 con 0x7f604015de00 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f604640b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60400b59b0 con 0x7f60400b9380 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f603ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f603ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53498/0 (socket says 172.21.15.99:53498) 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.955+0000 7f603ffff640 1 -- 172.21.15.99:0/2741191939 learned_addr learned my addr 172.21.15.99:0/2741191939 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f60449a8640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604015de00 0x7f60400ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603f7fe640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f604015b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603ffff640 1 -- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 msgr2=0x7f604015b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603ffff640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f604015b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603ffff640 1 -- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604015de00 msgr2=0x7f60400ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603ffff640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604015de00 0x7f60400ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603ffff640 1 -- 172.21.15.99:0/2741191939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400b2af0 con 0x7f60400b70d0 2024-03-04T10:09:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f60449a8640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604015de00 0x7f60400ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603ffff640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b3b90 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f603000c7f0 tx=0x7f603000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030015070 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6030004220 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030007620 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60400b2dd0 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.956+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60400b3390 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.957+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6040074f60 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.958+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f60300079d0 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.958+0000 7f603cff9640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f602c06b4d0 0x7f602c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.958+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f603008a680 con 0x7f60400b70d0 2024-03-04T10:09:46.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.958+0000 7f603f7fe640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f602c06b4d0 0x7f602c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:46.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.959+0000 7f603f7fe640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f602c06b4d0 0x7f602c06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f60400d0970 tx=0x7f6034008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:46.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:46.962+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f60300593b0 con 0x7f60400b70d0 2024-03-04T10:09:47.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.168+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60400ced90 con 0x7f60400b70d0 2024-03-04T10:09:47.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.169+0000 7f603cff9640 1 -- 172.21.15.99:0/2741191939 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4156 (secure 0 0 0) 0x7f6030058cb0 con 0x7f60400b70d0 2024-03-04T10:09:47.170 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:47.171 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":68,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:46.616150+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:47.171 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 68 2024-03-04T10:09:47.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.170+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f602c06b4d0 msgr2=0x7f602c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.170+0000 7f604640b640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f602c06b4d0 0x7f602c06d990 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f60400d0970 tx=0x7f6034008040 comp rx=0 tx=0).stop 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.170+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 msgr2=0x7f60400b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.170+0000 7f604640b640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b3b90 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f603000c7f0 tx=0x7f603000ccb0 comp rx=0 tx=0).stop 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 shutdown_connections 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f602c06b4d0 0x7f602c06d990 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f604015de00 0x7f60400ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f60400b9380 0x7f604015b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 --2- 172.21.15.99:0/2741191939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f60400b70d0 0x7f60400b3b90 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 >> 172.21.15.99:0/2741191939 conn(0x7f60400d0470 msgr2=0x7f604015ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 shutdown_connections 2024-03-04T10:09:47.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:47.171+0000 7f604640b640 1 -- 172.21.15.99:0/2741191939 wait complete. 2024-03-04T10:09:47.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:09:15.461841+0000', 'modified': '2024-03-04T10:09:46.616150+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 82, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7134': {'gid': 7134, 'name': 'c', 'rank': 0, 'incarnation': 68, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.99:6834/3121450352', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 3121450352}, {'type': 'v1', 'addr': '172.21.15.99:6834', 'nonce': 3121450352}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:09:47.586 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7134:up:replay 2024-03-04T10:09:48.586 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.698+0000 7f576af59640 1 Processor -- start 2024-03-04T10:09:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.698+0000 7f576af59640 1 -- start start 2024-03-04T10:09:48.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.698+0000 7f576af59640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5764071a60 0x7f5764150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.698+0000 7f576af59640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576415da40 con 0x7f576415d660 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f576af59640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57640af090 con 0x7f57640aeb20 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f576af59640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576415cbc0 con 0x7f5764071a60 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f57694f6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5764071a60 0x7f5764150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f5768cf5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f576415d660 0x7f576415e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49188/0 (socket says 172.21.15.99:49188) 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f5768cf5640 1 -- 172.21.15.99:0/638555684 learned_addr learned my addr 172.21.15.99:0/638555684 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f57694f6640 1 -- 172.21.15.99:0/638555684 >> v1:172.21.15.162:6789/0 conn(0x7f57640aeb20 legacy=0x7f5764071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f57694f6640 1 -- 172.21.15.99:0/638555684 >> v1:172.21.15.99:6789/0 conn(0x7f576415d660 legacy=0x7f576415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f57637fe640 1 -- 172.21.15.99:0/638555684 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971871974 0 0) 0x7f576415da40 con 0x7f576415d660 2024-03-04T10:09:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.699+0000 7f57694f6640 1 -- 172.21.15.99:0/638555684 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576415a9f0 con 0x7f5764071a60 2024-03-04T10:09:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f57694f6640 1 --2- 172.21.15.99:0/638555684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764150960 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f575c00aea0 tx=0x7f575c02ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1099b6acf4e9a7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f57637fe640 1 -- 172.21.15.99:0/638555684 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c02da30 con 0x7f5764071a60 2024-03-04T10:09:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f57637fe640 1 -- 172.21.15.99:0/638555684 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c02dbd0 con 0x7f5764071a60 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f57637fe640 1 -- 172.21.15.99:0/638555684 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c008440 con 0x7f5764071a60 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f576af59640 1 -- 172.21.15.99:0/638555684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 msgr2=0x7f5764150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f576af59640 1 --2- 172.21.15.99:0/638555684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764150960 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f575c00aea0 tx=0x7f575c02ce90 comp rx=0 tx=0).stop 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f576af59640 1 -- 172.21.15.99:0/638555684 shutdown_connections 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f576af59640 1 --2- 172.21.15.99:0/638555684 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764150960 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.700+0000 7f576af59640 1 -- 172.21.15.99:0/638555684 >> 172.21.15.99:0/638555684 conn(0x7f576406acd0 msgr2=0x7f576406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.701+0000 7f576af59640 1 -- 172.21.15.99:0/638555684 shutdown_connections 2024-03-04T10:09:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.701+0000 7f576af59640 1 -- 172.21.15.99:0/638555684 wait complete. 2024-03-04T10:09:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.702+0000 7f576af59640 1 Processor -- start 2024-03-04T10:09:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.702+0000 7f576af59640 1 -- start start 2024-03-04T10:09:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57640aeb20 0x7f576414a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f576415d660 0x7f576414dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576415a660 con 0x7f5764071a60 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576415afb0 con 0x7f576415d660 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576415b4d0 con 0x7f57640aeb20 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53524/0 (socket says 172.21.15.99:53524) 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 -- 172.21.15.99:0/1797613858 learned_addr learned my addr 172.21.15.99:0/1797613858 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f57694f6640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f576415d660 0x7f576414dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5763fff640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57640aeb20 0x7f576414a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 -- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57640aeb20 msgr2=0x7f576414a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57640aeb20 0x7f576414a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 -- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f576415d660 msgr2=0x7f576414dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f576415d660 0x7f576414dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 -- 172.21.15.99:0/1797613858 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576414ac20 con 0x7f5764071a60 2024-03-04T10:09:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5763fff640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57640aeb20 0x7f576414a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f5768cf5640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764159260 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f575800c4a0 tx=0x7f575800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758017070 con 0x7f5764071a60 2024-03-04T10:09:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758005ce0 con 0x7f5764071a60 2024-03-04T10:09:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576414d1c0 con 0x7f5764071a60 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.703+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57580072b0 con 0x7f5764071a60 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.704+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576414d780 con 0x7f5764071a60 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.704+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576414d980 con 0x7f5764071a60 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.705+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f575801a070 con 0x7f5764071a60 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.705+0000 7f57617fa640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f574806b4d0 0x7f574806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.705+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f57580896b0 con 0x7f5764071a60 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.706+0000 7f5763fff640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f574806b4d0 0x7f574806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.706+0000 7f5763fff640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f574806b4d0 0x7f574806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f57640b0dc0 tx=0x7f5750009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:48.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.710+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f576414d980 con 0x7f5764071a60 2024-03-04T10:09:48.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.922+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5764074f60 con 0x7f5764071a60 2024-03-04T10:09:48.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.922+0000 7f57617fa640 1 -- 172.21.15.99:0/1797613858 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4948 (secure 0 0 0) 0x7f5758087e20 con 0x7f5764071a60 2024-03-04T10:09:48.924 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:48.924 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:47.625332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:48.925 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 70 2024-03-04T10:09:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f574806b4d0 msgr2=0x7f574806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f574806b4d0 0x7f574806d990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f57640b0dc0 tx=0x7f5750009040 comp rx=0 tx=0).stop 2024-03-04T10:09:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 msgr2=0x7f5764159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764159260 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f575800c4a0 tx=0x7f575800c960 comp rx=0 tx=0).stop 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 shutdown_connections 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f574806b4d0 0x7f574806d990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f576415d660 0x7f576414dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f57640aeb20 0x7f576414a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 --2- 172.21.15.99:0/1797613858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5764071a60 0x7f5764159260 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 >> 172.21.15.99:0/1797613858 conn(0x7f576406acd0 msgr2=0x7f57640af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 shutdown_connections 2024-03-04T10:09:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:48.925+0000 7f576af59640 1 -- 172.21.15.99:0/1797613858 wait complete. 2024-03-04T10:09:49.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:09:15.461841+0000', 'modified': '2024-03-04T10:09:47.625332+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 82, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7134': {'gid': 7134, 'name': 'c', 'rank': 0, 'incarnation': 68, 'state': 'up:rejoin', 'state_seq': 11, 'addr': '172.21.15.99:6834/3121450352', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 3121450352}, {'type': 'v1', 'addr': '172.21.15.99:6834', 'nonce': 3121450352}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:09:49.340 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7134:up:rejoin 2024-03-04T10:09:50.341 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.451+0000 7f104acc0640 1 Processor -- start 2024-03-04T10:09:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.451+0000 7f104acc0640 1 -- start start 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f104acc0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f104415bbb0 0x7f10440d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f104acc0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440cfb40 con 0x7f10440cc690 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f104acc0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440cfe20 con 0x7f10440d0e40 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f104acc0640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10440d2880 con 0x7f104415bbb0 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f1048a5c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f104415bbb0 0x7f10440d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f104925d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f10440cc690 0x7f10440d1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49192/0 (socket says 172.21.15.99:49192) 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f104925d640 1 -- 172.21.15.99:0/1950174019 learned_addr learned my addr 172.21.15.99:0/1950174019 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f1048a5c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f104415bbb0 0x7f10440d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53532/0 (socket says 172.21.15.99:53532) 2024-03-04T10:09:50.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f1048a5c640 1 -- 172.21.15.99:0/1950174019 >> v1:172.21.15.162:6789/0 conn(0x7f10440d0e40 legacy=0x7f10440cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:50.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f1048a5c640 1 -- 172.21.15.99:0/1950174019 >> v1:172.21.15.99:6789/0 conn(0x7f10440cc690 legacy=0x7f10440d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:50.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.452+0000 7f1048a5c640 1 -- 172.21.15.99:0/1950174019 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440d0100 con 0x7f104415bbb0 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.453+0000 7f1048a5c640 1 --2- 172.21.15.99:0/1950174019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104415bbb0 0x7f10440d0900 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f10340097a0 tx=0x7f103402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e68843ac5e0cae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.453+0000 7f10437fe640 1 -- 172.21.15.99:0/1950174019 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034035020 con 0x7f104415bbb0 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.453+0000 7f10437fe640 1 -- 172.21.15.99:0/1950174019 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1034031830 con 0x7f104415bbb0 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.453+0000 7f10437fe640 1 -- 172.21.15.99:0/1950174019 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1034031c00 con 0x7f104415bbb0 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 -- 172.21.15.99:0/1950174019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104415bbb0 msgr2=0x7f10440d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 --2- 172.21.15.99:0/1950174019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104415bbb0 0x7f10440d0900 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f10340097a0 tx=0x7f103402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 -- 172.21.15.99:0/1950174019 shutdown_connections 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 --2- 172.21.15.99:0/1950174019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104415bbb0 0x7f10440d0900 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 -- 172.21.15.99:0/1950174019 >> 172.21.15.99:0/1950174019 conn(0x7f10440b3480 msgr2=0x7f10440b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 -- 172.21.15.99:0/1950174019 shutdown_connections 2024-03-04T10:09:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.454+0000 7f104acc0640 1 -- 172.21.15.99:0/1950174019 wait complete. 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.455+0000 7f104acc0640 1 Processor -- start 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.455+0000 7f104acc0640 1 -- start start 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104acc0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10440cc690 0x7f10441e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104acc0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 0x7f104414db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104acc0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104415bbb0 0x7f10440c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104acc0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104415bf90 con 0x7f10440d0e40 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104acc0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10440d23b0 con 0x7f10440cc690 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104acc0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10440c9ab0 con 0x7f104415bbb0 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 0x7f104414db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 0x7f104414db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53540/0 (socket says 172.21.15.99:53540) 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 -- 172.21.15.99:0/3021132895 learned_addr learned my addr 172.21.15.99:0/3021132895 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f104925d640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104415bbb0 0x7f10440c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1048a5c640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10440cc690 0x7f10441e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 -- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104415bbb0 msgr2=0x7f10440c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104415bbb0 0x7f10440c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 -- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10440cc690 msgr2=0x7f10441e4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10440cc690 0x7f10441e4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1043fff640 1 -- 172.21.15.99:0/3021132895 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440c9d90 con 0x7f10440d0e40 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.456+0000 7f1048a5c640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10440cc690 0x7f10441e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.457+0000 7f1043fff640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 0x7f104414db40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f103800c400 tx=0x7f103800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.457+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038017070 con 0x7f10440d0e40 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.457+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10380035f0 con 0x7f10440d0e40 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.457+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10440ca070 con 0x7f10440d0e40 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.457+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038003a20 con 0x7f10440d0e40 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.457+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10440ca630 con 0x7f10440d0e40 2024-03-04T10:09:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.458+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104406b6f0 con 0x7f10440d0e40 2024-03-04T10:09:50.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.458+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f103801a070 con 0x7f10440d0e40 2024-03-04T10:09:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.459+0000 7f10417fa640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.459+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f10380896b0 con 0x7f10440d0e40 2024-03-04T10:09:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.459+0000 7f1048a5c640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.459+0000 7f1048a5c640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1034036000 tx=0x7f103403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.463+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f104406b6f0 con 0x7f10440d0e40 2024-03-04T10:09:50.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.670+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10440ca830 con 0x7f10440d0e40 2024-03-04T10:09:50.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.670+0000 7f10417fa640 1 -- 172.21.15.99:0/3021132895 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4948 (secure 0 0 0) 0x7f10440ca830 con 0x7f10440d0e40 2024-03-04T10:09:50.672 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:50.672 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:48.638042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":12,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:50.672 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 71 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 msgr2=0x7f103006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f1034036000 tx=0x7f103403c040 comp rx=0 tx=0).stop 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 msgr2=0x7f104414db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 0x7f104414db40 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f103800c400 tx=0x7f103800c8c0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 shutdown_connections 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104415bbb0 0x7f10440c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10440d0e40 0x7f104414db40 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 --2- 172.21.15.99:0/3021132895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10440cc690 0x7f10441e4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:50.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 >> 172.21.15.99:0/3021132895 conn(0x7f10440b3480 msgr2=0x7f10440ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:50.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 shutdown_connections 2024-03-04T10:09:50.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:50.672+0000 7f104acc0640 1 -- 172.21.15.99:0/3021132895 wait complete. 2024-03-04T10:09:51.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:09:15.461841+0000', 'modified': '2024-03-04T10:09:48.638042+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 82, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7134}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7134': {'gid': 7134, 'name': 'c', 'rank': 0, 'incarnation': 68, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.99:6834/3121450352', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6832', 'nonce': 3121450352}, {'type': 'v1', 'addr': '172.21.15.99:6834', 'nonce': 3121450352}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:09:51.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:09:51.088 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:09:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.191+0000 7f8c34b90640 1 Processor -- start 2024-03-04T10:09:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.191+0000 7f8c34b90640 1 -- start start 2024-03-04T10:09:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c34b90640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8c3006d3b0 0x7f8c300ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c34b90640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cc500 0x7f8c300cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c34b90640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c300d0140 con 0x7f8c300cfd60 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c34b90640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c300cf440 con 0x7f8c3006d3b0 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c34b90640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3015cbb0 con 0x7f8c300cc500 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8c3006d3b0 0x7f8c300ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8c3006d3b0 0x7f8c300ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53546/0 (socket says 172.21.15.99:53546) 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 -- 172.21.15.99:0/477522473 learned_addr learned my addr 172.21.15.99:0/477522473 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c2ed9d640 1 --2- 172.21.15.99:0/477522473 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cc500 0x7f8c300cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 -- 172.21.15.99:0/477522473 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cc500 msgr2=0x7f8c300cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 --2- 172.21.15.99:0/477522473 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cc500 0x7f8c300cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 -- 172.21.15.99:0/477522473 >> v1:172.21.15.162:6789/0 conn(0x7f8c300cfd60 legacy=0x7f8c300cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.192+0000 7f8c27fff640 1 -- 172.21.15.99:0/477522473 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3015c280 con 0x7f8c3006d3b0 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.193+0000 7f8c27fff640 1 --2- 172.21.15.99:0/477522473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c3006d3b0 0x7f8c300ce9c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f8c300d0810 tx=0x7f8c1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c85926b98978d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.193+0000 7f8c2dd9b640 1 -- 172.21.15.99:0/477522473 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18035020 con 0x7f8c3006d3b0 2024-03-04T10:09:51.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.193+0000 7f8c2dd9b640 1 -- 172.21.15.99:0/477522473 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18031830 con 0x7f8c3006d3b0 2024-03-04T10:09:51.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.193+0000 7f8c2dd9b640 1 -- 172.21.15.99:0/477522473 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18031c00 con 0x7f8c3006d3b0 2024-03-04T10:09:51.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 -- 172.21.15.99:0/477522473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c3006d3b0 msgr2=0x7f8c300ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 --2- 172.21.15.99:0/477522473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c3006d3b0 0x7f8c300ce9c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f8c300d0810 tx=0x7f8c1802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 -- 172.21.15.99:0/477522473 shutdown_connections 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 --2- 172.21.15.99:0/477522473 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cc500 0x7f8c300cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 --2- 172.21.15.99:0/477522473 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c3006d3b0 0x7f8c300ce9c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 -- 172.21.15.99:0/477522473 >> 172.21.15.99:0/477522473 conn(0x7f8c3006acd0 msgr2=0x7f8c3006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 -- 172.21.15.99:0/477522473 shutdown_connections 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.194+0000 7f8c34b90640 1 -- 172.21.15.99:0/477522473 wait complete. 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.195+0000 7f8c34b90640 1 Processor -- start 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.195+0000 7f8c34b90640 1 -- start start 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c3006d3b0 0x7f8c300a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 0x7f8c301f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cfd60 0x7f8c300aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3015bf30 con 0x7f8c300cc500 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3015c840 con 0x7f8c3006d3b0 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3015f520 con 0x7f8c300cfd60 2024-03-04T10:09:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 0x7f8c301f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 0x7f8c301f0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53554/0 (socket says 172.21.15.99:53554) 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 -- 172.21.15.99:0/1144398594 learned_addr learned my addr 172.21.15.99:0/1144398594 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 -- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cfd60 msgr2=0x7f8c300aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cfd60 0x7f8c300aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 -- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c3006d3b0 msgr2=0x7f8c300a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c3006d3b0 0x7f8c300a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 -- 172.21.15.99:0/1144398594 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300a89a0 con 0x7f8c300cc500 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c27fff640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 0x7f8c301f0e20 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8c18000c00 tx=0x7f8c180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c26ffd640 1 -- 172.21.15.99:0/1144398594 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18040070 con 0x7f8c300cc500 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c26ffd640 1 -- 172.21.15.99:0/1144398594 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c18039940 con 0x7f8c300cc500 2024-03-04T10:09:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c300a8c70 con 0x7f8c300cc500 2024-03-04T10:09:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c26ffd640 1 -- 172.21.15.99:0/1144398594 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18035040 con 0x7f8c300cc500 2024-03-04T10:09:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.196+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c300a9230 con 0x7f8c300cc500 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.198+0000 7f8c26ffd640 1 -- 172.21.15.99:0/1144398594 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8c1804b060 con 0x7f8c300cc500 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.198+0000 7f8c26ffd640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c1006b4d0 0x7f8c1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.198+0000 7f8c26ffd640 1 -- 172.21.15.99:0/1144398594 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8c180abcf0 con 0x7f8c300cc500 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.198+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c1006b4d0 0x7f8c1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.199+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c1006b4d0 0x7f8c1006d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8c20006fd0 tx=0x7f8c20006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.199+0000 7f8c37772740 1 Processor -- start 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 -- start start 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5e920 0x55df3be5c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be15f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be3e260 0x55df3be16490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3bd7b7c0 con 0x55df3be5ed00 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3bd24b50 con 0x55df3be3e260 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df3bd77860 con 0x55df3be5e920 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be15f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be15f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53560/0 (socket says 172.21.15.99:53560) 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.200+0000 7f8c27fff640 1 -- 172.21.15.99:0/2016922080 learned_addr learned my addr 172.21.15.99:0/2016922080 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c2ed9d640 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be3e260 0x55df3be16490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5e920 0x55df3be5c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c27fff640 1 -- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5e920 msgr2=0x55df3be5c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c27fff640 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5e920 0x55df3be5c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c27fff640 1 -- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be3e260 msgr2=0x55df3be16490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c27fff640 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be3e260 0x55df3be16490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c27fff640 1 -- 172.21.15.99:0/2016922080 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df3bd389a0 con 0x55df3be5ed00 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5e920 0x55df3be5c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c27fff640 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be15f50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8c18082470 tx=0x7f8c18079f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7d952bd5a66669 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c0ffff640 1 -- 172.21.15.99:0/2016922080 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18078640 con 0x55df3be5ed00 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c0ffff640 1 -- 172.21.15.99:0/2016922080 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1807e0d0 con 0x55df3be5ed00 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.201+0000 7f8c0ffff640 1 -- 172.21.15.99:0/2016922080 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c180769b0 con 0x55df3be5ed00 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 -- 172.21.15.99:0/2016922080 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 msgr2=0x55df3be15f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be15f50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f8c18082470 tx=0x7f8c18079f90 comp rx=0 tx=0).stop 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 -- 172.21.15.99:0/2016922080 shutdown_connections 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be3e260 0x55df3be16490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be15f50 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 --2- 172.21.15.99:0/2016922080 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5e920 0x55df3be5c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 -- 172.21.15.99:0/2016922080 >> 172.21.15.99:0/2016922080 conn(0x55df3be130b0 msgr2=0x55df3be5bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 -- 172.21.15.99:0/2016922080 shutdown_connections 2024-03-04T10:09:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 -- 172.21.15.99:0/2016922080 wait complete. 2024-03-04T10:09:51.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 -- start start 2024-03-04T10:09:51.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.202+0000 7f8c37772740 1 Processor -- start 2024-03-04T10:09:51.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 0x55df3be3d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be3b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be3b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3bd7b7c0 con 0x55df3be5ed00 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3bd37f90 con 0x55df3be5e920 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df3be116e0 con 0x55df3be5aaa0 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c37772740 10 client.0 _ll_register_callbacks cb 0x55df3bd59d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be3b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be3b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53572/0 (socket says 172.21.15.99:53572) 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.203+0000 7f8c27fff640 1 -- 172.21.15.99:0/3938185695 learned_addr learned my addr 172.21.15.99:0/3938185695 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c27fff640 1 -- 172.21.15.99:0/3938185695 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 msgr2=0x55df3be3b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c27fff640 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be3b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c27fff640 1 -- 172.21.15.99:0/3938185695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 msgr2=0x55df3be3d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c27fff640 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 0x55df3be3d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c27fff640 1 -- 172.21.15.99:0/3938185695 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df3ba902d0 con 0x55df3be5ed00 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c27fff640 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be3b110 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8c18076750 tx=0x7f8c18031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e1cab1d1febc23 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:51.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/3938185695 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c180a7030 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/3938185695 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8c180786c0 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c1807e0d0 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/3938185695 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c180769b0 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 10 client.7425 fetch_fsmap learned FSMap version 71 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x55df3bb5a6b0 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/3938185695 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1807e0d0 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/3938185695 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 71) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8c18090030 con 0x55df3be5ed00 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 10 client.7425 fetch_fsmap finished waiting for FSMap version 71 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 10 client.7425 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 20 client.7425 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3be70480 0x55df3be70860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 4 client.7425 mds_command: new command op to 7134 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.204+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 --> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df3bd88d00 con 0x55df3be70480 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3be70480 0x55df3be70860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c0d7fa640 10 client.7425 ms_handle_connect on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3be70480 0x55df3be70860 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31b7b369dfe9d0f0 server_cookie=395f071436198fe in_seq=0 out_seq=0 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/3938185695 <== mds.0 v2:172.21.15.99:6832/3121450352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55df3bd88d00 con 0x55df3be70480 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c0d7fa640 10 client.7425 handle_command_reply: tid=0 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c37772740 1 client.7425 shutdown 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 msgr2=0x55df3be3b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.205+0000 7f8c37772740 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be3b110 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f8c18076750 tx=0x7f8c18031f90 comp rx=0 tx=0).stop 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 shutdown_connections 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be3b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3be70480 0x55df3be70860 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be3b110 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 --2- 172.21.15.99:0/3938185695 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 0x55df3be3d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c0d7fa640 0 client.7425 ms_handle_reset on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 >> 172.21.15.99:0/3938185695 conn(0x55df3be3e260 msgr2=0x55df3be0fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 shutdown_connections 2024-03-04T10:09:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 1 -- 172.21.15.99:0/3938185695 wait complete. 2024-03-04T10:09:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.206+0000 7f8c37772740 20 client.7425 trim_cache size 0 max 16384 2024-03-04T10:09:51.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.221+0000 7f8c37772740 1 Processor -- start 2024-03-04T10:09:51.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.221+0000 7f8c37772740 1 -- start start 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be3ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c37772740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 0x55df3be3b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c37772740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300b40b0 con 0x55df3be5e920 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30157c00 con 0x55df3be5ed00 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300b3ab0 con 0x55df3be5aaa0 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 0x55df3be3b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 0x55df3be3b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53576/0 (socket says 172.21.15.99:53576) 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 -- 172.21.15.99:0/1862044402 learned_addr learned my addr 172.21.15.99:0/1862044402 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 -- 172.21.15.99:0/1862044402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 msgr2=0x55df3be3ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be3ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 -- 172.21.15.99:0/1862044402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5ed00 msgr2=0x55df3be37000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.222+0000 7f8c27fff640 1 -- 172.21.15.99:0/1862044402 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df3be5a8b0 con 0x55df3be5e920 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c27fff640 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 0x55df3be3b0a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f8c1807a040 tx=0x7f8c1807d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f57504a1d5c5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c0f7fe640 1 -- 172.21.15.99:0/1862044402 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c180a7070 con 0x55df3be5e920 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c0f7fe640 1 -- 172.21.15.99:0/1862044402 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c180a9070 con 0x55df3be5e920 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c0f7fe640 1 -- 172.21.15.99:0/1862044402 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18078640 con 0x55df3be5e920 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 -- 172.21.15.99:0/1862044402 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 msgr2=0x55df3be3b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 0x55df3be3b0a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f8c1807a040 tx=0x7f8c1807d070 comp rx=0 tx=0).stop 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 -- 172.21.15.99:0/1862044402 shutdown_connections 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5e920 0x55df3be3b0a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 --2- 172.21.15.99:0/1862044402 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be3ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 -- 172.21.15.99:0/1862044402 >> 172.21.15.99:0/1862044402 conn(0x55df3be3e260 msgr2=0x55df3be70860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 -- 172.21.15.99:0/1862044402 shutdown_connections 2024-03-04T10:09:51.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.223+0000 7f8c37772740 1 -- 172.21.15.99:0/1862044402 wait complete. 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.224+0000 7f8c37772740 1 -- start start 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.224+0000 7f8c37772740 1 Processor -- start 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be15410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 0x55df3be15950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3bd7b7c0 con 0x55df3be5ed00 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3be116e0 con 0x55df3be5e920 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df3be34250 con 0x55df3be5aaa0 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 10 client.0 _ll_register_callbacks cb 0x7f8c180765e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53584/0 (socket says 172.21.15.99:53584) 2024-03-04T10:09:51.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 -- 172.21.15.99:0/1629112773 learned_addr learned my addr 172.21.15.99:0/1629112773 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 -- 172.21.15.99:0/1629112773 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 msgr2=0x55df3be15410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be15410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 -- 172.21.15.99:0/1629112773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 msgr2=0x55df3be15950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 0x55df3be15950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 -- 172.21.15.99:0/1629112773 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df3be10190 con 0x55df3be5ed00 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c2ed9d640 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be37000 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8c28007950 tx=0x7f8c28007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2fe990565f3441 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/1629112773 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800fac0 con 0x55df3be5ed00 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/1629112773 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8c2800fc60 con 0x55df3be5ed00 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.225+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c180a9070 con 0x55df3be5ed00 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/1629112773 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c28023050 con 0x55df3be5ed00 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 10 client.7431 fetch_fsmap learned FSMap version 71 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x55df3becbfe0 con 0x55df3be5ed00 2024-03-04T10:09:51.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/1629112773 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c180a9070 con 0x55df3be5ed00 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/1629112773 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 71) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8c28026020 con 0x55df3be5ed00 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 10 client.7431 fetch_fsmap finished waiting for FSMap version 71 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 10 client.7431 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 20 client.7431 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3becabd0 0x55df3becafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 4 client.7431 mds_command: new command op to 7134 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 --> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55df3bd83050 con 0x55df3becabd0 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3becabd0 0x55df3becafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c0d7fa640 10 client.7431 ms_handle_connect on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c2e59c640 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3becabd0 0x55df3becafb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9851180486bda695 server_cookie=fb7d2bb83f91e0ef in_seq=0 out_seq=0 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.226+0000 7f8c0d7fa640 1 -- 172.21.15.99:0/1629112773 <== mds.0 v2:172.21.15.99:6832/3121450352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55df3bd83050 con 0x55df3becabd0 2024-03-04T10:09:51.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c0d7fa640 10 client.7431 handle_command_reply: tid=0 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 client.7431 shutdown 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 msgr2=0x55df3be37000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be37000 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8c28007950 tx=0x7f8c28007e10 comp rx=0 tx=0).stop 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 shutdown_connections 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x55df3becabd0 0x55df3becafb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55df3be5ed00 0x55df3be37000 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c0d7fa640 0 client.7431 ms_handle_reset on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55df3be5e920 0x55df3be15950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 --2- 172.21.15.99:0/1629112773 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55df3be5aaa0 0x55df3be15410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 >> 172.21.15.99:0/1629112773 conn(0x55df3be3e260 msgr2=0x55df3be59f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 shutdown_connections 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 1 -- 172.21.15.99:0/1629112773 wait complete. 2024-03-04T10:09:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c37772740 20 client.7431 trim_cache size 0 max 16384 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 7134, 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 4.6074878799999999, 2024-03-04T10:09:51.236 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 71, 2024-03-04T10:09:51.237 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 82, 2024-03-04T10:09:51.237 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 82, 2024-03-04T10:09:51.237 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 39.450943594999998 2024-03-04T10:09:51.237 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c1006b4d0 msgr2=0x7f8c1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c1006b4d0 0x7f8c1006d990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8c20006fd0 tx=0x7f8c20006bc0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 msgr2=0x7f8c301f0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 0x7f8c301f0e20 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f8c18000c00 tx=0x7f8c180046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 shutdown_connections 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c1006b4d0 0x7f8c1006d990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c300cfd60 0x7f8c300aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c300cc500 0x7f8c301f0e20 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 --2- 172.21.15.99:0/1144398594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c3006d3b0 0x7f8c300a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 >> 172.21.15.99:0/1144398594 conn(0x7f8c3006acd0 msgr2=0x7f8c3015b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 shutdown_connections 2024-03-04T10:09:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.227+0000 7f8c34b90640 1 -- 172.21.15.99:0/1144398594 wait complete. 2024-03-04T10:09:51.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:51.240 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat a/ && find a/') 2024-03-04T10:09:51.255 INFO:teuthology.orchestra.run.smithi099.stdout: File: a/ 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 2 Blocks: 0 IO Block: 65536 directory 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 29h/41d Inode: 1099511627776 Links: 4 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-03-04 10:09:23.194968267 +0000 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-03-04 10:09:28.732835596 +0000 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-03-04 10:09:28.732835596 +0000 2024-03-04T10:09:51.256 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-03-04 10:09:23.194968267 +0000 2024-03-04T10:09:51.262 INFO:teuthology.orchestra.run.smithi099.stdout:a/ 2024-03-04T10:09:51.262 INFO:teuthology.orchestra.run.smithi099.stdout:a/z 2024-03-04T10:09:51.263 INFO:teuthology.orchestra.run.smithi099.stdout:a/z/d 2024-03-04T10:09:51.263 INFO:teuthology.orchestra.run.smithi099.stdout:a/c 2024-03-04T10:09:51.263 INFO:teuthology.orchestra.run.smithi099.stdout:a/c/d 2024-03-04T10:09:51.263 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T10:09:51.263 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 flush journal 2024-03-04T10:09:51.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.403+0000 7fa2ba1d4640 1 Processor -- start 2024-03-04T10:09:51.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.404+0000 7fa2ba1d4640 1 -- start start 2024-03-04T10:09:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.404+0000 7fa2ba1d4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa2b40ceec0 0x7fa2b40b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.404+0000 7fa2ba1d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b415c2e0 0x7fa2b40b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.404+0000 7fa2ba1d4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b415c790 con 0x7fa2b40ccbd0 2024-03-04T10:09:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.404+0000 7fa2ba1d4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b415d2f0 con 0x7fa2b40ceec0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.404+0000 7fa2ba1d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b415d6d0 con 0x7fa2b415c2e0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b2ffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa2b40ceec0 0x7fa2b40b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b415c2e0 0x7fa2b40b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b37fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa2b40ccbd0 0x7fa2b40ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:52356/0 (socket says 172.21.15.99:52356) 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b37fe640 1 -- 172.21.15.99:0/944441659 learned_addr learned my addr 172.21.15.99:0/944441659 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b27fc640 1 -- 172.21.15.99:0/944441659 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6476200 0 0) 0x7fa2b415c790 con 0x7fa2b40ccbd0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b27fc640 1 -- 172.21.15.99:0/944441659 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c003520 con 0x7fa2b40ccbd0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b3fff640 1 -- 172.21.15.99:0/944441659 >> v2:172.21.15.162:3300/0 conn(0x7fa2b40ceec0 msgr2=0x7fa2b40b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/944441659 >> v2:172.21.15.162:3300/0 conn(0x7fa2b40ceec0 0x7fa2b40b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b3fff640 1 -- 172.21.15.99:0/944441659 >> v1:172.21.15.162:6789/0 conn(0x7fa2b40ccbd0 legacy=0x7fa2b40ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b3fff640 1 -- 172.21.15.99:0/944441659 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b415ca50 con 0x7fa2b415c2e0 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.405+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/944441659 >> v2:172.21.15.162:3300/0 conn(0x7fa2b40ceec0 0x7fa2b40b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.406+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/944441659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b415c2e0 0x7fa2b40b0560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009d90 tx=0x7fa2ac02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e1e682b842b573 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.406+0000 7fa2b27fc640 1 -- 172.21.15.99:0/944441659 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac039070 con 0x7fa2b415c2e0 2024-03-04T10:09:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.406+0000 7fa2b27fc640 1 -- 172.21.15.99:0/944441659 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac02cc50 con 0x7fa2b415c2e0 2024-03-04T10:09:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.406+0000 7fa2b27fc640 1 -- 172.21.15.99:0/944441659 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac034440 con 0x7fa2b415c2e0 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/944441659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b415c2e0 msgr2=0x7fa2b40b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/944441659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b415c2e0 0x7fa2b40b0560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009d90 tx=0x7fa2ac02bea0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/944441659 shutdown_connections 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/944441659 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b415c2e0 0x7fa2b40b0560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/944441659 >> v2:172.21.15.162:3300/0 conn(0x7fa2b40ceec0 0x7fa2b40b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/944441659 >> 172.21.15.99:0/944441659 conn(0x7fa2b4066dc0 msgr2=0x7fa2b40671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/944441659 shutdown_connections 2024-03-04T10:09:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.407+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/944441659 wait complete. 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.408+0000 7fa2ba1d4640 1 Processor -- start 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.408+0000 7fa2ba1d4640 1 -- start start 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2ba1d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b40ccbd0 0x7fa2b4149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2ba1d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40ceec0 0x7fa2b4158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2ba1d4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 0x7fa2b40d1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2ba1d4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40ccfb0 con 0x7fa2b415c2e0 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2ba1d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b415d010 con 0x7fa2b40ccbd0 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2ba1d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40b0aa0 con 0x7fa2b40ceec0 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 0x7fa2b40d1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 0x7fa2b40d1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53600/0 (socket says 172.21.15.99:53600) 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 -- 172.21.15.99:0/3289721750 learned_addr learned my addr 172.21.15.99:0/3289721750 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40ceec0 0x7fa2b4158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b40ccbd0 0x7fa2b4149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 -- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40ceec0 msgr2=0x7fa2b4158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40ceec0 0x7fa2b4158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 -- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b40ccbd0 msgr2=0x7fa2b4149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b40ccbd0 0x7fa2b4149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 -- 172.21.15.99:0/3289721750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b40c54a0 con 0x7fa2b415c2e0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40ceec0 0x7fa2b4158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 0x7fa2b40d1c60 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009d90 tx=0x7fa2ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa29bfff640 1 -- 172.21.15.99:0/3289721750 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac039050 con 0x7fa2b415c2e0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.409+0000 7fa29bfff640 1 -- 172.21.15.99:0/3289721750 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac045070 con 0x7fa2b415c2e0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.410+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b40c5740 con 0x7fa2b415c2e0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.410+0000 7fa29bfff640 1 -- 172.21.15.99:0/3289721750 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac009450 con 0x7fa2b415c2e0 2024-03-04T10:09:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.410+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b40c5d00 con 0x7fa2b415c2e0 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.411+0000 7fa29bfff640 1 -- 172.21.15.99:0/3289721750 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa2ac03e610 con 0x7fa2b415c2e0 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.411+0000 7fa29bfff640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.412+0000 7fa29bfff640 1 -- 172.21.15.99:0/3289721750 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa2ac0acd00 con 0x7fa2b415c2e0 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.412+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.412+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4005880 tx=0x7fa2a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.412+0000 7fa2bcdbc740 1 Processor -- start 2024-03-04T10:09:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.412+0000 7fa2bcdbc740 1 -- start start 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a25860 0x5651f59f8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a285f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f59f8d40 0x5651f5a28b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651f5961e00 con 0x5651f5a25c40 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651f590ab10 con 0x5651f59f8d40 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651f595dea0 con 0x5651f5a25860 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a285f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53612/0 (socket says 172.21.15.99:53612) 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1487332830 learned_addr learned my addr 172.21.15.99:0/1487332830 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a25860 0x5651f59f8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.413+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f59f8d40 0x5651f5a28b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a25860 msgr2=0x5651f59f8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a25860 0x5651f59f8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f59f8d40 msgr2=0x5651f5a28b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f59f8d40 0x5651f5a28b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1487332830 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651f57407d0 con 0x5651f5a25c40 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a25860 0x5651f59f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f59f8d40 0x5651f5a28b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a285f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009c10 tx=0x7fa2a8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fc235c80fb3880 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2997fa640 1 -- 172.21.15.99:0/1487332830 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8009e20 con 0x5651f5a25c40 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2997fa640 1 -- 172.21.15.99:0/1487332830 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8004030 con 0x5651f5a25c40 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2997fa640 1 -- 172.21.15.99:0/1487332830 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a80043e0 con 0x5651f5a25c40 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1487332830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 msgr2=0x5651f5a285f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.414+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a285f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009c10 tx=0x7fa2a8009190 comp rx=0 tx=0).stop 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1487332830 shutdown_connections 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f59f8d40 0x5651f5a28b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a285f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1487332830 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a25860 0x5651f59f8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1487332830 >> 172.21.15.99:0/1487332830 conn(0x5651f59faa70 msgr2=0x5651f5a435f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1487332830 shutdown_connections 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1487332830 wait complete. 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 -- start start 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.415+0000 7fa2bcdbc740 1 Processor -- start 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a0f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a3a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a0f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651f59fd010 con 0x5651f5a25c40 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651f591e0a0 con 0x5651f5a25860 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651f59fc400 con 0x5651f5a27810 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2bcdbc740 10 client.0 _ll_register_callbacks cb 0x5651f591da70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a3a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a3a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53618/0 (socket says 172.21.15.99:53618) 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/4055067263 learned_addr learned my addr 172.21.15.99:0/4055067263 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a0f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a0f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 msgr2=0x5651f5a0f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a0f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 msgr2=0x5651f5a0f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a0f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/4055067263 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651f56763f0 con 0x5651f5a25c40 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a0f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.416+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a3a6a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8004000 tx=0x7fa2a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d64014d4b9fc84 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2927fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2927fc640 1 -- 172.21.15.99:0/4055067263 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a802c060 con 0x5651f5a25c40 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2927fc640 1 -- 172.21.15.99:0/4055067263 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa2a8003820 con 0x5651f5a25c40 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2a8004030 con 0x5651f5a25c40 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2927fc640 1 -- 172.21.15.99:0/4055067263 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2a8032050 con 0x5651f5a25c40 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 10 client.7440 fetch_fsmap learned FSMap version 71 2024-03-04T10:09:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x5651f56360f0 con 0x5651f5a25c40 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2927fc640 1 -- 172.21.15.99:0/4055067263 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a8004030 con 0x5651f5a25c40 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2927fc640 1 -- 172.21.15.99:0/4055067263 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 71) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa2a8035020 con 0x5651f5a25c40 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 10 client.7440 fetch_fsmap finished waiting for FSMap version 71 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 10 client.7440 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 20 client.7440 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a27c90 0x5651f5a11f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 4 client.7440 mds_command: new command op to 7134 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 --> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651f596f340 con 0x5651f5a27c90 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.417+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a27c90 0x5651f5a11f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2927fc640 10 client.7440 ms_handle_connect on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a27c90 0x5651f5a11f30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6c0709a2d4212c0 server_cookie=c9567d66752f1224 in_seq=0 out_seq=0 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2927fc640 1 -- 172.21.15.99:0/4055067263 <== mds.0 v2:172.21.15.99:6832/3121450352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5651f596f340 con 0x5651f5a27c90 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2927fc640 10 client.7440 handle_command_reply: tid=0 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 client.7440 shutdown 2024-03-04T10:09:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 msgr2=0x5651f5a3a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a3a6a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8004000 tx=0x7fa2a8004730 comp rx=0 tx=0).stop 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 shutdown_connections 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a0f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a27c90 0x5651f5a11f30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a3a6a0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/4055067263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a0f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2927fc640 0 client.7440 ms_handle_reset on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 >> 172.21.15.99:0/4055067263 conn(0x5651f59f8d40 msgr2=0x5651f59f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 shutdown_connections 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/4055067263 wait complete. 2024-03-04T10:09:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.418+0000 7fa2bcdbc740 20 client.7440 trim_cache size 0 max 16384 2024-03-04T10:09:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.431+0000 7fa2bcdbc740 1 Processor -- start 2024-03-04T10:09:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.431+0000 7fa2bcdbc740 1 -- start start 2024-03-04T10:09:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 0x5651f5a3a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25c40 0x5651f5a28360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a262d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b41fe240 con 0x5651f5a25860 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40ca2e0 con 0x5651f5a25c40 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b41572c0 con 0x5651f5a27810 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2b37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 0x5651f5a3a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2b37fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 0x5651f5a3a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53626/0 (socket says 172.21.15.99:53626) 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.432+0000 7fa2b37fe640 1 -- 172.21.15.99:0/1057440901 learned_addr learned my addr 172.21.15.99:0/1057440901 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2b37fe640 1 -- 172.21.15.99:0/1057440901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 msgr2=0x5651f5a262d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a262d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2b37fe640 1 -- 172.21.15.99:0/1057440901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25c40 msgr2=0x5651f5a28360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25c40 0x5651f5a28360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2b37fe640 1 -- 172.21.15.99:0/1057440901 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651f5a0f690 con 0x5651f5a25860 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 0x5651f5a3a670 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4000f60 tx=0x7fa2a4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6758e16e5b39fff server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa298ff9640 1 -- 172.21.15.99:0/1057440901 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4014a10 con 0x5651f5a25860 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa298ff9640 1 -- 172.21.15.99:0/1057440901 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4014bb0 con 0x5651f5a25860 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa298ff9640 1 -- 172.21.15.99:0/1057440901 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4014da0 con 0x5651f5a25860 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1057440901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 msgr2=0x5651f5a3a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.433+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 0x5651f5a3a670 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4000f60 tx=0x7fa2a4000ed0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1057440901 shutdown_connections 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a262d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25c40 0x5651f5a28360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1057440901 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25860 0x5651f5a3a670 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1057440901 >> 172.21.15.99:0/1057440901 conn(0x5651f59f8d40 msgr2=0x5651f59ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1057440901 shutdown_connections 2024-03-04T10:09:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1057440901 wait complete. 2024-03-04T10:09:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 -- start start 2024-03-04T10:09:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.434+0000 7fa2bcdbc740 1 Processor -- start 2024-03-04T10:09:51.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a39990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a5c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a5cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651f5a20d90 con 0x5651f5a25c40 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40c9ce0 con 0x5651f5a25860 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5651f59fe9c0 con 0x5651f5a27810 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2bcdbc740 10 client.0 _ll_register_callbacks cb 0x7fa2a80096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a5c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a5c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53642/0 (socket says 172.21.15.99:53642) 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1888587413 learned_addr learned my addr 172.21.15.99:0/1888587413 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a39990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b3fff640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a5cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 msgr2=0x5651f5a5cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a5cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 msgr2=0x5651f5a39990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a39990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 -- 172.21.15.99:0/1888587413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5651f5a10560 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2b2ffd640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a5c710 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009b00 tx=0x7fa2a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9a641d0fdf7cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.435+0000 7fa2927fc640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2927fc640 1 -- 172.21.15.99:0/1888587413 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8000e60 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2927fc640 1 -- 172.21.15.99:0/1888587413 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa2a8003bb0 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2a4014bb0 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2927fc640 1 -- 172.21.15.99:0/1888587413 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2a8045050 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 10 client.7446 fetch_fsmap learned FSMap version 71 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) v3 -- 0x5651f5aae0d0 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2927fc640 1 -- 172.21.15.99:0/1888587413 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a4014bb0 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2927fc640 1 -- 172.21.15.99:0/1888587413 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 71) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa2a8048020 con 0x5651f5a25c40 2024-03-04T10:09:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 10 client.7446 fetch_fsmap finished waiting for FSMap version 71 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 10 client.7446 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 20 client.7446 populate_metadata read hostname 'smithi099' 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a54ef0 0x5651f5a55300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 4 client.7446 mds_command: new command op to 7134 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 --> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5651f5969690 con 0x5651f5a54ef0 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a54ef0 0x5651f5a55300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2927fc640 10 client.7446 ms_handle_connect on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.436+0000 7fa2b37fe640 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a54ef0 0x5651f5a55300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0a7c8c56c3fdbfc server_cookie=568ed1b79a9ae4ed in_seq=0 out_seq=0 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2927fc640 1 -- 172.21.15.99:0/1888587413 <== mds.0 v2:172.21.15.99:6832/3121450352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5651f5969690 con 0x5651f5a54ef0 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2927fc640 10 client.7446 handle_command_reply: tid=0 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 client.7446 shutdown 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 msgr2=0x5651f5a5c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a5c710 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8009b00 tx=0x7fa2a80046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 shutdown_connections 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5651f5a27810 0x5651f5a5cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:6832/3121450352,v1:172.21.15.99:6834/3121450352] conn(0x5651f5a54ef0 0x5651f5a55300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5651f5a25c40 0x5651f5a5c710 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 --2- 172.21.15.99:0/1888587413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5651f5a25860 0x5651f5a39990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2927fc640 0 client.7446 ms_handle_reset on v2:172.21.15.99:6832/3121450352 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 >> 172.21.15.99:0/1888587413 conn(0x5651f59f8d40 msgr2=0x5651f5a28360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 shutdown_connections 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 1 -- 172.21.15.99:0/1888587413 wait complete. 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.455+0000 7fa2bcdbc740 20 client.7446 trim_cache size 0 max 16384 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "message": "", 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stdout: "return_code": 0 2024-03-04T10:09:51.458 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:09:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a006b4d0 msgr2=0x7fa2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4005880 tx=0x7fa2a4009040 comp rx=0 tx=0).stop 2024-03-04T10:09:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 msgr2=0x7fa2b40d1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 0x7fa2b40d1c60 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009d90 tx=0x7fa2ac004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 shutdown_connections 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa2b415c2e0 0x7fa2b40d1c60 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa2b40ceec0 0x7fa2b4158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 --2- 172.21.15.99:0/3289721750 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa2b40ccbd0 0x7fa2b4149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 >> 172.21.15.99:0/3289721750 conn(0x7fa2b4066dc0 msgr2=0x7fa2b40b5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 shutdown_connections 2024-03-04T10:09:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.456+0000 7fa2ba1d4640 1 -- 172.21.15.99:0/3289721750 wait complete. 2024-03-04T10:09:51.468 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:09:51.468 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:09:51.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:09:51.572 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:09:51.575 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:51.575 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:51.591 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:51.591 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:51.646 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.785+0000 7fc8a7d59640 1 Processor -- start 2024-03-04T10:09:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.785+0000 7fc8a7d59640 1 -- start start 2024-03-04T10:09:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a7d59640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc8a006d3b0 0x7fc8a015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a7d59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc8a015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a7d59640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a015be00 con 0x7fc8a015b030 2024-03-04T10:09:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a7d59640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a00b2720 con 0x7fc8a006d3b0 2024-03-04T10:09:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a7d59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a006d8b0 con 0x7fc8a015b410 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a5af5640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc8a015b030 0x7fc8a015dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49204/0 (socket says 172.21.15.99:49204) 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a5af5640 1 -- 172.21.15.99:0/3272753647 learned_addr learned my addr 172.21.15.99:0/3272753647 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725726 0 0) 0x7fc8a015be00 con 0x7fc8a015b030 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc88c003520 con 0x7fc8a015b030 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc8a015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a52f4640 1 --2- 172.21.15.99:0/3272753647 >> v2:172.21.15.162:3300/0 conn(0x7fc8a006d3b0 0x7fc8a015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3474057818 0 0) 0x7fc88c003520 con 0x7fc8a015b030 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 msgr2=0x7fc8a015b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc8a015b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 >> v2:172.21.15.162:3300/0 conn(0x7fc8a006d3b0 msgr2=0x7fc8a015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 --2- 172.21.15.99:0/3272753647 >> v2:172.21.15.162:3300/0 conn(0x7fc8a006d3b0 0x7fc8a015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a006db90 con 0x7fc8a015b030 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fc8980019f0 con 0x7fc8a015b030 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc8a015b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a52f4640 1 --2- 172.21.15.99:0/3272753647 >> v2:172.21.15.162:3300/0 conn(0x7fc8a006d3b0 0x7fc8a015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 >> v1:172.21.15.99:6789/0 conn(0x7fc8a015b030 legacy=0x7fc8a015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc88c0039b0 0x7fc88c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc88c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc88c004d40 0x7fc88c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00b2720 con 0x7fc88c004d40 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a006d8b0 con 0x7fc88c0039b0 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c00baf0 con 0x7fc8a015b410 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.786+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc88c004d40 0x7fc88c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a5af5640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc88c0039b0 0x7fc88c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a52f4640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc88c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a62f6640 1 -- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 msgr2=0x7fc88c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a015b410 0x7fc88c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a62f6640 1 -- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc88c0039b0 msgr2=0x7fc88c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc88c0039b0 0x7fc88c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a62f6640 1 -- 172.21.15.99:0/3272753647 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c003520 con 0x7fc88c004d40 2024-03-04T10:09:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.787+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc88c004d40 0x7fc88c0051c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098a0 tx=0x7fc88c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd75ff8b45e3626b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c00f040 con 0x7fc88c004d40 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89c004230 con 0x7fc88c004d40 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a4af3640 1 -- 172.21.15.99:0/3272753647 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c003610 con 0x7fc88c004d40 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a7d59640 1 -- 172.21.15.99:0/3272753647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc88c004d40 msgr2=0x7fc88c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/3272753647 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc88c004d40 0x7fc88c0051c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098a0 tx=0x7fc88c005830 comp rx=0 tx=0).stop 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a62f6640 1 -- 172.21.15.99:0/3272753647 reap_dead start 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a7d59640 1 -- 172.21.15.99:0/3272753647 shutdown_connections 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a7d59640 1 -- 172.21.15.99:0/3272753647 >> 172.21.15.99:0/3272753647 conn(0x7fc8a006acd0 msgr2=0x7fc8a006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a7d59640 1 -- 172.21.15.99:0/3272753647 shutdown_connections 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.788+0000 7fc8a7d59640 1 -- 172.21.15.99:0/3272753647 wait complete. 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.789+0000 7fc8a7d59640 1 Processor -- start 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.789+0000 7fc8a7d59640 1 -- start start 2024-03-04T10:09:51.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a7d59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8a0145b20 0x7fc8a0148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a7d59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 0x7fc8a0071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a7d59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a0071890 0x7fc8a00af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a7d59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a006e690 con 0x7fc8a0148960 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a7d59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a006e150 con 0x7fc8a0145b20 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a7d59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a006e810 con 0x7fc8a0071890 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 0x7fc8a0071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 0x7fc8a0071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53666/0 (socket says 172.21.15.99:53666) 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 -- 172.21.15.99:0/1384031623 learned_addr learned my addr 172.21.15.99:0/1384031623 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a5af5640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8a0145b20 0x7fc8a0148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a0071890 0x7fc8a00af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 -- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a0071890 msgr2=0x7fc8a00af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a0071890 0x7fc8a00af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 -- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8a0145b20 msgr2=0x7fc8a0148420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8a0145b20 0x7fc8a0148420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a52f4640 1 -- 172.21.15.99:0/1384031623 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a0148e00 con 0x7fc8a0148960 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.790+0000 7fc8a62f6640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a0071890 0x7fc8a00af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.791+0000 7fc8a52f4640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 0x7fc8a0071350 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc8900116d0 tx=0x7fc890011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.791+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890019070 con 0x7fc8a0148960 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.791+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc890004220 con 0x7fc8a0148960 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.791+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a01490e0 con 0x7fc8a0148960 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.791+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89000f040 con 0x7fc8a0148960 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.791+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a00ab860 con 0x7fc8a0148960 2024-03-04T10:09:51.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.792+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a0071ca0 con 0x7fc8a0148960 2024-03-04T10:09:51.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.792+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc89000f7c0 con 0x7fc8a0148960 2024-03-04T10:09:51.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.793+0000 7fc8967fc640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88406b4d0 0x7fc88406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:51.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.793+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc89008a710 con 0x7fc8a0148960 2024-03-04T10:09:51.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.793+0000 7fc8a5af5640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88406b4d0 0x7fc88406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:51.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.793+0000 7fc8a5af5640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88406b4d0 0x7fc88406d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc8980013d0 tx=0x7fc898006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:51.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.797+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc8a0071ca0 con 0x7fc8a0148960 2024-03-04T10:09:51.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.967+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a0074f60 con 0x7fc8a0148960 2024-03-04T10:09:51.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.968+0000 7fc8967fc640 1 -- 172.21.15.99:0/1384031623 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+17038 (secure 0 0 0) 0x7fc8900593b0 con 0x7fc8a0148960 2024-03-04T10:09:51.969 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:51.970 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":82,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:46.616184+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3857383686":"2024-03-05T10:09:46.616133+0000","172.21.15.99:6833/3857383686":"2024-03-05T10:09:46.616133+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:51.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88406b4d0 msgr2=0x7fc88406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88406b4d0 0x7fc88406d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc8980013d0 tx=0x7fc898006040 comp rx=0 tx=0).stop 2024-03-04T10:09:51.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 msgr2=0x7fc8a0071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 0x7fc8a0071350 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc8900116d0 tx=0x7fc890011b90 comp rx=0 tx=0).stop 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 shutdown_connections 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88406b4d0 0x7fc88406d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8a0071890 0x7fc8a00af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc8a0148960 0x7fc8a0071350 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 --2- 172.21.15.99:0/1384031623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8a0145b20 0x7fc8a0148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 >> 172.21.15.99:0/1384031623 conn(0x7fc8a006acd0 msgr2=0x7fc8a00b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 shutdown_connections 2024-03-04T10:09:51.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:51.970+0000 7fc8a7d59640 1 -- 172.21.15.99:0/1384031623 wait complete. 2024-03-04T10:09:51.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:09:13.676364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:09:14.431767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:09:51.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:09:52.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.084+0000 7f3c62b7d640 1 Processor -- start 2024-03-04T10:09:52.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c62b7d640 1 -- start start 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c62b7d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3c5c15df00 0x7f3c5c06b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c62b7d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0cfa40 0x7f3c5c0cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c62b7d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0701d0 con 0x7f3c5c06fa80 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c62b7d640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c0704b0 con 0x7f3c5c15df00 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c62b7d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c0cc8e0 con 0x7f3c5c0cfa40 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c60919640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3c5c15df00 0x7f3c5c06b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c60919640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3c5c15df00 0x7f3c5c06b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53672/0 (socket says 172.21.15.99:53672) 2024-03-04T10:09:52.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.085+0000 7f3c60919640 1 -- 172.21.15.99:0/1740643425 learned_addr learned my addr 172.21.15.99:0/1740643425 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c6111a640 1 --2- 172.21.15.99:0/1740643425 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0cfa40 0x7f3c5c0cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c60919640 1 -- 172.21.15.99:0/1740643425 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0cfa40 msgr2=0x7f3c5c0cfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c60919640 1 --2- 172.21.15.99:0/1740643425 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0cfa40 0x7f3c5c0cfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c60919640 1 -- 172.21.15.99:0/1740643425 >> v1:172.21.15.162:6789/0 conn(0x7f3c5c06fa80 legacy=0x7f3c5c0cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c60919640 1 -- 172.21.15.99:0/1740643425 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0ccbc0 con 0x7f3c5c15df00 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c6111a640 1 --2- 172.21.15.99:0/1740643425 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0cfa40 0x7f3c5c0cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.086+0000 7f3c60919640 1 --2- 172.21.15.99:0/1740643425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c15df00 0x7f3c5c06b700 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0ce230 tx=0x7f3c4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dcf32e964f2f6b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c5b7fe640 1 -- 172.21.15.99:0/1740643425 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c034070 con 0x7f3c5c15df00 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c5b7fe640 1 -- 172.21.15.99:0/1740643425 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4c030860 con 0x7f3c5c15df00 2024-03-04T10:09:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c5b7fe640 1 -- 172.21.15.99:0/1740643425 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c030c30 con 0x7f3c5c15df00 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 -- 172.21.15.99:0/1740643425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c15df00 msgr2=0x7f3c5c06b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/1740643425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c15df00 0x7f3c5c06b700 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0ce230 tx=0x7f3c4c02b9a0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 -- 172.21.15.99:0/1740643425 shutdown_connections 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/1740643425 >> v2:172.21.15.162:3300/0 conn(0x7f3c5c0cfa40 0x7f3c5c0cfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/1740643425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c15df00 0x7f3c5c06b700 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 -- 172.21.15.99:0/1740643425 >> 172.21.15.99:0/1740643425 conn(0x7f3c5c0b2250 msgr2=0x7f3c5c0b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 -- 172.21.15.99:0/1740643425 shutdown_connections 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.087+0000 7f3c62b7d640 1 -- 172.21.15.99:0/1740643425 wait complete. 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.088+0000 7f3c62b7d640 1 Processor -- start 2024-03-04T10:09:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.088+0000 7f3c62b7d640 1 -- start start 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c62b7d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 0x7f3c5c147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c62b7d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0cfa40 0x7f3c5c157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c62b7d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c15df00 0x7f3c5c1e7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c62b7d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0d0410 con 0x7f3c5c06fa80 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c62b7d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c06fee0 con 0x7f3c5c0cfa40 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c62b7d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0cd3c0 con 0x7f3c5c15df00 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c60919640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 0x7f3c5c147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c60919640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 0x7f3c5c147f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53678/0 (socket says 172.21.15.99:53678) 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.089+0000 7f3c60919640 1 -- 172.21.15.99:0/3521253264 learned_addr learned my addr 172.21.15.99:0/3521253264 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c60919640 1 -- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c15df00 msgr2=0x7f3c5c1e7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c60919640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c15df00 0x7f3c5c1e7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c60919640 1 -- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0cfa40 msgr2=0x7f3c5c157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c60919640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0cfa40 0x7f3c5c157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c60919640 1 -- 172.21.15.99:0/3521253264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0c4290 con 0x7f3c5c06fa80 2024-03-04T10:09:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c60919640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 0x7f3c5c147f60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c000c00 tx=0x7f3c4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c034040 con 0x7f3c5c06fa80 2024-03-04T10:09:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4c039d90 con 0x7f3c5c06fa80 2024-03-04T10:09:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c157a40 con 0x7f3c5c06fa80 2024-03-04T10:09:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c043610 con 0x7f3c5c06fa80 2024-03-04T10:09:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.090+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c148b40 con 0x7f3c5c06fa80 2024-03-04T10:09:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.091+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c074f60 con 0x7f3c5c06fa80 2024-03-04T10:09:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.091+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3c4c037070 con 0x7f3c5c06fa80 2024-03-04T10:09:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.092+0000 7f3c597fa640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4806b4d0 0x7f3c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.092+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3c4c0ac620 con 0x7f3c5c06fa80 2024-03-04T10:09:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.092+0000 7f3c5bfff640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4806b4d0 0x7f3c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.093+0000 7f3c5bfff640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4806b4d0 0x7f3c4806d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3c50000f60 tx=0x7f3c500079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.096+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3c4c03e070 con 0x7f3c5c06fa80 2024-03-04T10:09:52.101 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:09:52.100+0000 7f07d4dfd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:09:52.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.266+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3c5c0b0bd0 con 0x7f3c5c06fa80 2024-03-04T10:09:52.268 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.267+0000 7f3c597fa640 1 -- 172.21.15.99:0/3521253264 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3c4c07b230 con 0x7f3c5c06fa80 2024-03-04T10:09:52.268 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:09:52.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4806b4d0 msgr2=0x7f3c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4806b4d0 0x7f3c4806d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3c50000f60 tx=0x7f3c500079d0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 msgr2=0x7f3c5c147f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 0x7f3c5c147f60 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c000c00 tx=0x7f3c4c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 shutdown_connections 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c4806b4d0 0x7f3c4806d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3c5c15df00 0x7f3c5c1e7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c5c0cfa40 0x7f3c5c157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 --2- 172.21.15.99:0/3521253264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3c5c06fa80 0x7f3c5c147f60 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 >> 172.21.15.99:0/3521253264 conn(0x7f3c5c0b2250 msgr2=0x7f3c5c0b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 shutdown_connections 2024-03-04T10:09:52.271 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.269+0000 7f3c62b7d640 1 -- 172.21.15.99:0/3521253264 wait complete. 2024-03-04T10:09:52.281 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:09:52.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.382+0000 7f5c117d4640 1 Processor -- start 2024-03-04T10:09:52.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.382+0000 7f5c117d4640 1 -- start start 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c117d4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5c0c15da20 0x7f5c0c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c117d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c117d4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0c162450 con 0x7f5c0c0cf660 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c117d4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c0c0d0000 con 0x7f5c0c15da20 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c117d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c0cc920 con 0x7f5c0c161df0 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c0affd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5c0c15da20 0x7f5c0c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c0b7fe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c0a7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5c0c0cf660 0x7f5c0c1616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:52372/0 (socket says 172.21.15.99:52372) 2024-03-04T10:09:52.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.383+0000 7f5c0a7fc640 1 -- 172.21.15.99:0/1751263535 learned_addr learned my addr 172.21.15.99:0/1751263535 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:52.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.384+0000 7f5c0affd640 1 -- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 msgr2=0x7f5c0c0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.384+0000 7f5c0affd640 1 --2- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.384+0000 7f5c0affd640 1 -- 172.21.15.99:0/1751263535 >> v1:172.21.15.162:6789/0 conn(0x7f5c0c0cf660 legacy=0x7f5c0c1616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.384+0000 7f5c0affd640 1 -- 172.21.15.99:0/1751263535 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c0cc2d0 con 0x7f5c0c15da20 2024-03-04T10:09:52.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.384+0000 7f5c0b7fe640 1 --2- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:52.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.384+0000 7f5c0affd640 1 --2- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c15da20 0x7f5c0c0ccdb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009d90 tx=0x7f5bfc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5780b63528736c52 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c09ffb640 1 -- 172.21.15.99:0/1751263535 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc039070 con 0x7f5c0c15da20 2024-03-04T10:09:52.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c09ffb640 1 -- 172.21.15.99:0/1751263535 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bfc02cc50 con 0x7f5c0c15da20 2024-03-04T10:09:52.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c09ffb640 1 -- 172.21.15.99:0/1751263535 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc034440 con 0x7f5c0c15da20 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 -- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c15da20 msgr2=0x7f5c0c0ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c15da20 0x7f5c0c0ccdb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009d90 tx=0x7f5bfc02bea0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 -- 172.21.15.99:0/1751263535 shutdown_connections 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1751263535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c15da20 0x7f5c0c0ccdb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 -- 172.21.15.99:0/1751263535 >> 172.21.15.99:0/1751263535 conn(0x7f5c0c0b33a0 msgr2=0x7f5c0c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 -- 172.21.15.99:0/1751263535 shutdown_connections 2024-03-04T10:09:52.388 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.385+0000 7f5c117d4640 1 -- 172.21.15.99:0/1751263535 wait complete. 2024-03-04T10:09:52.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.386+0000 7f5c117d4640 1 Processor -- start 2024-03-04T10:09:52.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.386+0000 7f5c117d4640 1 -- start start 2024-03-04T10:09:52.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c117d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c0cf660 0x7f5c0c0ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.390 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c117d4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 0x7f5c0c14f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c117d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c1e7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c117d4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c0cbf80 con 0x7f5c0c15da20 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c117d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c0cd2f0 con 0x7f5c0c0cf660 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c117d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0c15de00 con 0x7f5c0c161df0 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c0a7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 0x7f5c0c14f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c0a7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 0x7f5c0c14f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53680/0 (socket says 172.21.15.99:53680) 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c0a7fc640 1 -- 172.21.15.99:0/1839282233 learned_addr learned my addr 172.21.15.99:0/1839282233 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c0affd640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c0cf660 0x7f5c0c0ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.387+0000 7f5c0b7fe640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c1e7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c0a7fc640 1 -- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 msgr2=0x7f5c0c1e7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c0a7fc640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c1e7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c0a7fc640 1 -- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c0cf660 msgr2=0x7f5c0c0ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c0a7fc640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c0cf660 0x7f5c0c0ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.391 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c0a7fc640 1 -- 172.21.15.99:0/1839282233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c15ef00 con 0x7f5c0c15da20 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c0a7fc640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 0x7f5c0c14f380 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c430 tx=0x7f5bf400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4017070 con 0x7f5c0c15da20 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf4005ce0 con 0x7f5c0c15da20 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0c0ca900 con 0x7f5c0c15da20 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40072f0 con 0x7f5c0c15da20 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.388+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0c0caec0 con 0x7f5c0c15da20 2024-03-04T10:09:52.392 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.389+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0c074f60 con 0x7f5c0c15da20 2024-03-04T10:09:52.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.390+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5bf401a070 con 0x7f5c0c15da20 2024-03-04T10:09:52.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.390+0000 7f5c037fe640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bec06b4d0 0x7f5bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.390+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5bf4089680 con 0x7f5c0c15da20 2024-03-04T10:09:52.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.390+0000 7f5c0affd640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bec06b4d0 0x7f5bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.391+0000 7f5c0affd640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bec06b4d0 0x7f5bec06d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc036000 tx=0x7f5bfc030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.394+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5bf4057cd0 con 0x7f5c0c15da20 2024-03-04T10:09:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.563+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c0c0d1c40 con 0x7f5c0c15da20 2024-03-04T10:09:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.564+0000 7f5c037fe640 1 -- 172.21.15.99:0/1839282233 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5bf405bd70 con 0x7f5c0c15da20 2024-03-04T10:09:52.566 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:09:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bec06b4d0 msgr2=0x7f5bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bec06b4d0 0x7f5bec06d990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc036000 tx=0x7f5bfc030000 comp rx=0 tx=0).stop 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 msgr2=0x7f5c0c14f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 0x7f5c0c14f380 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c430 tx=0x7f5bf400c830 comp rx=0 tx=0).stop 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 shutdown_connections 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bec06b4d0 0x7f5bec06d990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c0c161df0 0x7f5c0c1e7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c0c15da20 0x7f5c0c14f380 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 --2- 172.21.15.99:0/1839282233 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c0c0cf660 0x7f5c0c0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 >> 172.21.15.99:0/1839282233 conn(0x7f5c0c0b33a0 msgr2=0x7f5c0c0b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 shutdown_connections 2024-03-04T10:09:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.566+0000 7f5c117d4640 1 -- 172.21.15.99:0/1839282233 wait complete. 2024-03-04T10:09:52.578 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:09:52.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.681+0000 7fed8ee8e640 1 Processor -- start 2024-03-04T10:09:52.682 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.681+0000 7fed8ee8e640 1 -- start start 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8ee8e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fed8806bce0 0x7fed880728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8ee8e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8815cec0 con 0x7fed880700a0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8ee8e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8815c100 con 0x7fed8815bba0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8ee8e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8815c3e0 con 0x7fed8806bce0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8d42b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fed880700a0 0x7fed8815d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49208/0 (socket says 172.21.15.99:49208) 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8d42b640 1 -- 172.21.15.99:0/3484856957 learned_addr learned my addr 172.21.15.99:0/3484856957 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed8cc2a640 1 --2- 172.21.15.99:0/3484856957 >> v2:172.21.15.162:3300/0 conn(0x7fed8806bce0 0x7fed880728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989780788 0 0) 0x7fed8815cec0 con 0x7fed880700a0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed78003520 con 0x7fed880700a0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 526139476 0 0) 0x7fed78003520 con 0x7fed880700a0 2024-03-04T10:09:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 >> v2:172.21.15.162:3300/0 conn(0x7fed8806bce0 msgr2=0x7fed880728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 --2- 172.21.15.99:0/3484856957 >> v2:172.21.15.162:3300/0 conn(0x7fed8806bce0 0x7fed880728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 >> v1:172.21.15.162:6789/0 conn(0x7fed8815bba0 legacy=0x7fed8806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8815c6c0 con 0x7fed880700a0 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fed800019f0 con 0x7fed880700a0 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 >> v1:172.21.15.99:6789/0 conn(0x7fed880700a0 legacy=0x7fed8815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed780039b0 0x7fed78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed78004800 0x7fed78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.682+0000 7fed877fe640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed780051c0 0x7fed7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8815c3e0 con 0x7fed78004800 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7800c160 con 0x7fed780051c0 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7800c360 con 0x7fed780039b0 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed87fff640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed78004800 0x7fed78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed8d42b640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed780051c0 0x7fed7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed8cc2a640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed780039b0 0x7fed78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed87fff640 1 -- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed780039b0 msgr2=0x7fed78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed87fff640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed780039b0 0x7fed78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed87fff640 1 -- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed780051c0 msgr2=0x7fed7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed87fff640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed780051c0 0x7fed7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed8d42b640 1 -- 172.21.15.99:0/3484856957 reap_dead start 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.683+0000 7fed87fff640 1 -- 172.21.15.99:0/3484856957 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed78003520 con 0x7fed78004800 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed87fff640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed78004800 0x7fed78004c80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fed74001750 tx=0x7fed78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fe2926688745fa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74001370 con 0x7fed78004800 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed740052d0 con 0x7fed78004800 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed877fe640 1 -- 172.21.15.99:0/3484856957 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74003830 con 0x7fed78004800 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3484856957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed78004800 msgr2=0x7fed78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed78004800 0x7fed78004c80 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fed74001750 tx=0x7fed78005830 comp rx=0 tx=0).stop 2024-03-04T10:09:52.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3484856957 shutdown_connections 2024-03-04T10:09:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3484856957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed78004800 0x7fed78004c80 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3484856957 >> 172.21.15.99:0/3484856957 conn(0x7fed88071bc0 msgr2=0x7fed88071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3484856957 shutdown_connections 2024-03-04T10:09:52.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.684+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3484856957 wait complete. 2024-03-04T10:09:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.685+0000 7fed8ee8e640 1 Processor -- start 2024-03-04T10:09:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.685+0000 7fed8ee8e640 1 -- start start 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8ee8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed78004800 0x7fed881f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8ee8e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8814f060 0x7fed881f5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8ee8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 0x7fed881f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8ee8e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880728f0 con 0x7fed881f56d0 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8ee8e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed880cc3e0 con 0x7fed78004800 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8ee8e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed88072a70 con 0x7fed8814f060 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8d42b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 0x7fed881f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8d42b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 0x7fed881f5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53690/0 (socket says 172.21.15.99:53690) 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.686+0000 7fed8d42b640 1 -- 172.21.15.99:0/3792370280 learned_addr learned my addr 172.21.15.99:0/3792370280 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed87fff640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8814f060 0x7fed881f5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8cc2a640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed78004800 0x7fed881f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8d42b640 1 -- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8814f060 msgr2=0x7fed881f5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8d42b640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8814f060 0x7fed881f5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8d42b640 1 -- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed78004800 msgr2=0x7fed881f4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8d42b640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed78004800 0x7fed881f4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8d42b640 1 -- 172.21.15.99:0/3792370280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed880ce710 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed87fff640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8814f060 0x7fed881f5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8d42b640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 0x7fed881f5b80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fed88071980 tx=0x7fed80007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000ab60 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80019070 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed880ce9f0 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800133f0 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.687+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed881f4e80 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.688+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed88074f60 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.688+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fed80013590 con 0x7fed881f56d0 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.689+0000 7fed857fa640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fed6c06b500 0x7fed6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.689+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed8008a920 con 0x7fed881f56d0 2024-03-04T10:09:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.689+0000 7fed8cc2a640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fed6c06b500 0x7fed6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.690+0000 7fed8cc2a640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fed6c06b500 0x7fed6c06d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009b00 tx=0x7fed7c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.693+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fed80059650 con 0x7fed881f56d0 2024-03-04T10:09:52.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.862+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed880cbd80 con 0x7fed881f56d0 2024-03-04T10:09:52.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.862+0000 7fed857fa640 1 -- 172.21.15.99:0/3792370280 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed80058f50 con 0x7fed881f56d0 2024-03-04T10:09:52.864 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:09:52.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.864+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fed6c06b500 msgr2=0x7fed6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.864+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fed6c06b500 0x7fed6c06d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fed7c009b00 tx=0x7fed7c008040 comp rx=0 tx=0).stop 2024-03-04T10:09:52.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.864+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 msgr2=0x7fed881f5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.864+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 0x7fed881f5b80 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fed88071980 tx=0x7fed80007b00 comp rx=0 tx=0).stop 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 shutdown_connections 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fed6c06b500 0x7fed6c06d9c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fed881f56d0 0x7fed881f5b80 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fed8814f060 0x7fed881f5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 --2- 172.21.15.99:0/3792370280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fed78004800 0x7fed881f4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 >> 172.21.15.99:0/3792370280 conn(0x7fed88071bc0 msgr2=0x7fed88071fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 shutdown_connections 2024-03-04T10:09:52.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.865+0000 7fed8ee8e640 1 -- 172.21.15.99:0/3792370280 wait complete. 2024-03-04T10:09:52.877 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:09:52.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.976+0000 7f4344309640 1 Processor -- start 2024-03-04T10:09:52.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.976+0000 7f4344309640 1 -- start start 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.977+0000 7f4344309640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f433c0cfd60 0x7f433c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.977+0000 7f4344309640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c0d0140 con 0x7f433c0cf0d0 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.977+0000 7f4344309640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433c15c280 con 0x7f433c06d3b0 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.977+0000 7f4344309640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f433c15cbb0 con 0x7f433c0cfd60 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.977+0000 7f43428a6640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f433c0cf0d0 0x7f433c15f520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:49220/0 (socket says 172.21.15.99:49220) 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.977+0000 7f43428a6640 1 -- 172.21.15.99:0/958344727 learned_addr learned my addr 172.21.15.99:0/958344727 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43420a5640 1 --2- 172.21.15.99:0/958344727 >> v2:172.21.15.162:3300/0 conn(0x7f433c0cfd60 0x7f433c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852210516 0 0) 0x7f433c0d0140 con 0x7f433c0cf0d0 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4324003520 con 0x7f433c0cf0d0 2024-03-04T10:09:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427998438 0 0) 0x7f433c15c280 con 0x7f433c06d3b0 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433c0d0140 con 0x7f433c06d3b0 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104413902 0 0) 0x7f4324003520 con 0x7f433c0cf0d0 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 >> v2:172.21.15.162:3300/0 conn(0x7f433c0cfd60 msgr2=0x7f433c0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 --2- 172.21.15.99:0/958344727 >> v2:172.21.15.162:3300/0 conn(0x7f433c0cfd60 0x7f433c0cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 >> v1:172.21.15.162:6789/0 conn(0x7f433c06d3b0 legacy=0x7f433c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c15c560 con 0x7f433c0cf0d0 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f432c001670 con 0x7f433c0cf0d0 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 >> v1:172.21.15.99:6789/0 conn(0x7f433c0cf0d0 legacy=0x7f433c15f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4324003b90 0x7f4324003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4324004800 0x7f4324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43240051c0 0x7f432400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c15cbb0 con 0x7f4324004800 2024-03-04T10:09:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432400c0c0 con 0x7f43240051c0 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432400c2c0 con 0x7f4324003b90 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43420a5640 1 --2- 172.21.15.99:0/958344727 >> v2:172.21.15.162:3300/0 conn(0x7f433c0cfd60 0x7f433c0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43418a4640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4324004800 0x7f4324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43428a6640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43240051c0 0x7f432400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43420a5640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4324003b90 0x7f4324003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43418a4640 1 -- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4324003b90 msgr2=0x7f4324003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43418a4640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4324003b90 0x7f4324003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43418a4640 1 -- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43240051c0 msgr2=0x7f432400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43418a4640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43240051c0 0x7f432400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43418a4640 1 -- 172.21.15.99:0/958344727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4324003520 con 0x7f4324004800 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43428a6640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43240051c0 0x7f432400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43420a5640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4324003b90 0x7f4324003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.978+0000 7f43428a6640 1 -- 172.21.15.99:0/958344727 reap_dead start 2024-03-04T10:09:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.979+0000 7f43418a4640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4324004800 0x7f4324004c80 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f4330001890 tx=0x7f4324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93f8ffbf311543c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.979+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300014e0 con 0x7f4324004800 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.979+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433002f050 con 0x7f4324004800 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.979+0000 7f43410a3640 1 -- 172.21.15.99:0/958344727 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300035f0 con 0x7f4324004800 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 -- 172.21.15.99:0/958344727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4324004800 msgr2=0x7f4324004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4324004800 0x7f4324004c80 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f4330001890 tx=0x7f4324005830 comp rx=0 tx=0).stop 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 -- 172.21.15.99:0/958344727 shutdown_connections 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 --2- 172.21.15.99:0/958344727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4324004800 0x7f4324004c80 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 -- 172.21.15.99:0/958344727 >> 172.21.15.99:0/958344727 conn(0x7f433c06acd0 msgr2=0x7f433c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 -- 172.21.15.99:0/958344727 shutdown_connections 2024-03-04T10:09:52.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.980+0000 7f4344309640 1 -- 172.21.15.99:0/958344727 wait complete. 2024-03-04T10:09:52.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.981+0000 7f4344309640 1 Processor -- start 2024-03-04T10:09:52.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.981+0000 7f4344309640 1 -- start start 2024-03-04T10:09:52.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f4344309640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4324004800 0x7f433c157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f4344309640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 0x7f433c154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f4344309640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f433c155400 0x7f433c155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f4344309640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0ce9c0 con 0x7f433c157ed0 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f4344309640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433c15bf30 con 0x7f4324004800 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f4344309640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433c0ceb40 con 0x7f433c155400 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 0x7f433c154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 0x7f433c154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:53698/0 (socket says 172.21.15.99:53698) 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 -- 172.21.15.99:0/850139326 learned_addr learned my addr 172.21.15.99:0/850139326 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43420a5640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4324004800 0x7f433c157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43428a6640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f433c155400 0x7f433c155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 -- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f433c155400 msgr2=0x7f433c155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f433c155400 0x7f433c155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 -- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4324004800 msgr2=0x7f433c157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4324004800 0x7f433c157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 -- 172.21.15.99:0/850139326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f433c147890 con 0x7f433c157ed0 2024-03-04T10:09:52.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43428a6640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f433c155400 0x7f433c155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43420a5640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4324004800 0x7f433c157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f43418a4640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 0x7f433c154ec0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f4330005040 tx=0x7f43300012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330041070 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330007250 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.982+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330007a50 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.983+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f433c1475b0 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.983+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433c0a8850 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.983+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f433c074f60 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.984+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4330044070 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.984+0000 7f433affd640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f432806b4d0 0x7f432806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.984+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f43300abec0 con 0x7f433c157ed0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.985+0000 7f43420a5640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f432806b4d0 0x7f432806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:52.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.985+0000 7f43420a5640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f432806b4d0 0x7f432806d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f433c069760 tx=0x7f433400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:52.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:52.989+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f433007ab60 con 0x7f433c157ed0 2024-03-04T10:09:53.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.170+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f433c06d7c0 con 0x7f433c157ed0 2024-03-04T10:09:53.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.170+0000 7f433affd640 1 -- 172.21.15.99:0/850139326 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v82) v1 ==== 80+0+122 (secure 0 0 0) 0x7f433007a460 con 0x7f433c157ed0 2024-03-04T10:09:53.172 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6835/3857383686 2024-03-05T10:09:46.616133+0000 2024-03-04T10:09:53.172 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6833/3857383686 2024-03-05T10:09:46.616133+0000 2024-03-04T10:09:53.172 INFO:teuthology.orchestra.run.smithi099.stderr:listed 2 entries 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f432806b4d0 msgr2=0x7f432806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f432806b4d0 0x7f432806d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f433c069760 tx=0x7f433400b040 comp rx=0 tx=0).stop 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 msgr2=0x7f433c154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 0x7f433c154ec0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f4330005040 tx=0x7f43300012d0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 shutdown_connections 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f432806b4d0 0x7f432806d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f433c155400 0x7f433c155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f433c157ed0 0x7f433c154ec0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 --2- 172.21.15.99:0/850139326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4324004800 0x7f433c157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 >> 172.21.15.99:0/850139326 conn(0x7f433c06acd0 msgr2=0x7f433c15e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:53.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 shutdown_connections 2024-03-04T10:09:53.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.172+0000 7f4344309640 1 -- 172.21.15.99:0/850139326 wait complete. 2024-03-04T10:09:53.184 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:09:53.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:53.184 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:09:53.214 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:09:53.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:09:53.215 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:09:53.270 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:09:53.270 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:53.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:09:53.289 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:09:53.289 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:53.325 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:09:53.325 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:09:53.325 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:09:53.347 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:09:53.347 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:53.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-04T10:09:53.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.464+0000 7f0e997a9640 1 Processor -- start 2024-03-04T10:09:53.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.464+0000 7f0e997a9640 1 -- start start 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e997a9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0e94158410 0x7f0e940b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e997a9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e9406acd0 0x7f0e94161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e997a9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e9415e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e997a9640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e940b3f00 con 0x7f0e9406acd0 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e997a9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9406b500 con 0x7f0e94158410 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e997a9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9406b7e0 con 0x7f0e94161ca0 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e9406acd0 0x7f0e94161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e9406acd0 0x7f0e94161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50494/0 (socket says 172.21.15.99:50494) 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 -- 172.21.15.99:0/4292669499 learned_addr learned my addr 172.21.15.99:0/4292669499 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 -- 172.21.15.99:0/4292669499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 msgr2=0x7f0e9415e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/4292669499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e9415e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 -- 172.21.15.99:0/4292669499 >> v2:172.21.15.162:3300/0 conn(0x7f0e94158410 msgr2=0x7f0e940b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/4292669499 >> v2:172.21.15.162:3300/0 conn(0x7f0e94158410 0x7f0e940b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.465+0000 7f0e8a7fc640 1 -- 172.21.15.99:0/4292669499 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9406bac0 con 0x7f0e9406acd0 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.466+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/4292669499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e94161760 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f0e9406c830 tx=0x7f0e7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746a475e093f7ad4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.466+0000 7f0e927fc640 1 -- 172.21.15.99:0/4292669499 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c034070 con 0x7f0e9406acd0 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.466+0000 7f0e927fc640 1 -- 172.21.15.99:0/4292669499 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7c030860 con 0x7f0e9406acd0 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.466+0000 7f0e927fc640 1 -- 172.21.15.99:0/4292669499 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c030c30 con 0x7f0e9406acd0 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 -- 172.21.15.99:0/4292669499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 msgr2=0x7f0e94161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 --2- 172.21.15.99:0/4292669499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e94161760 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f0e9406c830 tx=0x7f0e7c02b9a0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 -- 172.21.15.99:0/4292669499 shutdown_connections 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 --2- 172.21.15.99:0/4292669499 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e9415e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 --2- 172.21.15.99:0/4292669499 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e94161760 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 --2- 172.21.15.99:0/4292669499 >> v2:172.21.15.162:3300/0 conn(0x7f0e94158410 0x7f0e940b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.470 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 -- 172.21.15.99:0/4292669499 >> 172.21.15.99:0/4292669499 conn(0x7f0e940cfef0 msgr2=0x7f0e940d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:53.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 -- 172.21.15.99:0/4292669499 shutdown_connections 2024-03-04T10:09:53.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.467+0000 7f0e997a9640 1 -- 172.21.15.99:0/4292669499 wait complete. 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.468+0000 7f0e997a9640 1 Processor -- start 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.468+0000 7f0e997a9640 1 -- start start 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e997a9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e941f4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e997a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e94158410 0x7f0e9415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e997a9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e941f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e997a9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9406c5a0 con 0x7f0e9406acd0 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e997a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9406c080 con 0x7f0e94158410 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e997a9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9406b160 con 0x7f0e94161ca0 2024-03-04T10:09:53.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e941f4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e941f4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50498/0 (socket says 172.21.15.99:50498) 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 -- 172.21.15.99:0/3295543510 learned_addr learned my addr 172.21.15.99:0/3295543510 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e937fe640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e941f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e94158410 0x7f0e9415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 -- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 msgr2=0x7f0e941f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e941f4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 -- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e94158410 msgr2=0x7f0e9415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e94158410 0x7f0e9415d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e92ffd640 1 -- 172.21.15.99:0/3295543510 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e941f4670 con 0x7f0e9406acd0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.469+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e94158410 0x7f0e9415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.470+0000 7f0e92ffd640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e941f4870 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400b4d0 tx=0x7f0e8400b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.470+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84014070 con 0x7f0e9406acd0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.470+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e84010870 con 0x7f0e9406acd0 2024-03-04T10:09:53.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.470+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e940cd8e0 con 0x7f0e9406acd0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.470+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84010c60 con 0x7f0e9406acd0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.470+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e940cdea0 con 0x7f0e9406acd0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.471+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e940b3240 con 0x7f0e9406acd0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.471+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0e84017070 con 0x7f0e9406acd0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.471+0000 7f0e8bfff640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e7406b4d0 0x7f0e7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.472+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0e8408a0b0 con 0x7f0e9406acd0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.472+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e7406b4d0 0x7f0e7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.472+0000 7f0e8a7fc640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e7406b4d0 0x7f0e7406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:53.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.476+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0e940b3240 con 0x7f0e9406acd0 2024-03-04T10:09:53.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.683+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e94074f60 con 0x7f0e9406acd0 2024-03-04T10:09:53.685 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.684+0000 7f0e8bfff640 1 -- 172.21.15.99:0/3295543510 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4948 (secure 0 0 0) 0x7f0e941f4670 con 0x7f0e9406acd0 2024-03-04T10:09:53.685 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:53.685 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:48.638042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":12,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:53.686 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 71 2024-03-04T10:09:53.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e7406b4d0 msgr2=0x7f0e7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e7406b4d0 0x7f0e7406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c03b040 comp rx=0 tx=0).stop 2024-03-04T10:09:53.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 msgr2=0x7f0e941f4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e941f4870 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400b4d0 tx=0x7f0e8400b990 comp rx=0 tx=0).stop 2024-03-04T10:09:53.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 shutdown_connections 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0e7406b4d0 0x7f0e7406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e94161ca0 0x7f0e941f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e94158410 0x7f0e9415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.685+0000 7f0e997a9640 1 --2- 172.21.15.99:0/3295543510 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e9406acd0 0x7f0e941f4870 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.686+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 >> 172.21.15.99:0/3295543510 conn(0x7f0e940cfef0 msgr2=0x7f0e940b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.686+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 shutdown_connections 2024-03-04T10:09:53.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.686+0000 7f0e997a9640 1 -- 172.21.15.99:0/3295543510 wait complete. 2024-03-04T10:09:53.698 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:53.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.796+0000 7fd294232640 1 Processor -- start 2024-03-04T10:09:53.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.796+0000 7fd294232640 1 -- start start 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd294232640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd28c06c090 0x7fd28c06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd294232640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c0ad6c0 con 0x7fd28c066de0 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd294232640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c0ad960 con 0x7fd28c06c470 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd294232640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28c06ebc0 con 0x7fd28c06c090 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd291fce640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd28c06c090 0x7fd28c06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd2927cf640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd28c066de0 0x7fd28c15f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40470/0 (socket says 172.21.15.99:40470) 2024-03-04T10:09:53.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd2927cf640 1 -- 172.21.15.99:0/1072496858 learned_addr learned my addr 172.21.15.99:0/1072496858 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:53.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd291fce640 1 -- 172.21.15.99:0/1072496858 >> v1:172.21.15.162:6789/0 conn(0x7fd28c06c470 legacy=0x7fd28c0af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd291fce640 1 -- 172.21.15.99:0/1072496858 >> v1:172.21.15.99:6789/0 conn(0x7fd28c066de0 legacy=0x7fd28c15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd290fcc640 1 -- 172.21.15.99:0/1072496858 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850159054 0 0) 0x7fd28c0ad6c0 con 0x7fd28c066de0 2024-03-04T10:09:53.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.797+0000 7fd291fce640 1 -- 172.21.15.99:0/1072496858 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c06eea0 con 0x7fd28c06c090 2024-03-04T10:09:53.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.798+0000 7fd291fce640 1 --2- 172.21.15.99:0/1072496858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c06f680 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd288009d90 tx=0x7fd28802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b20b865b5d2f3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:53.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.798+0000 7fd290fcc640 1 -- 172.21.15.99:0/1072496858 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288039070 con 0x7fd28c06c090 2024-03-04T10:09:53.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.798+0000 7fd290fcc640 1 -- 172.21.15.99:0/1072496858 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28802cc60 con 0x7fd28c06c090 2024-03-04T10:09:53.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.798+0000 7fd290fcc640 1 -- 172.21.15.99:0/1072496858 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd288034660 con 0x7fd28c06c090 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 -- 172.21.15.99:0/1072496858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 msgr2=0x7fd28c06f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 --2- 172.21.15.99:0/1072496858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c06f680 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd288009d90 tx=0x7fd28802beb0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 -- 172.21.15.99:0/1072496858 shutdown_connections 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 --2- 172.21.15.99:0/1072496858 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c06f680 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 -- 172.21.15.99:0/1072496858 >> 172.21.15.99:0/1072496858 conn(0x7fd28c0cfef0 msgr2=0x7fd28c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 -- 172.21.15.99:0/1072496858 shutdown_connections 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.799+0000 7fd294232640 1 -- 172.21.15.99:0/1072496858 wait complete. 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.800+0000 7fd294232640 1 Processor -- start 2024-03-04T10:09:53.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.800+0000 7fd294232640 1 -- start start 2024-03-04T10:09:53.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd294232640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd28c066de0 0x7fd28c1f4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd294232640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c09a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd294232640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd28c0ceb20 0x7fd28c1f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd294232640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0addc0 con 0x7fd28c06c090 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd294232640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0adc40 con 0x7fd28c066de0 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd294232640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c15fc70 con 0x7fd28c0ceb20 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c09a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c09a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50522/0 (socket says 172.21.15.99:50522) 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 -- 172.21.15.99:0/829352983 learned_addr learned my addr 172.21.15.99:0/829352983 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd291fce640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd28c066de0 0x7fd28c1f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2927cf640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd28c0ceb20 0x7fd28c1f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 -- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd28c0ceb20 msgr2=0x7fd28c1f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd28c0ceb20 0x7fd28c1f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 -- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd28c066de0 msgr2=0x7fd28c1f4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd28c066de0 0x7fd28c1f4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 -- 172.21.15.99:0/829352983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c1f5280 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.801+0000 7fd2917cd640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c09a870 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c3c0 tx=0x7fd27c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.802+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c017070 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.802+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c0052d0 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.802+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c1f5560 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.802+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c003830 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.802+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c0cba10 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.802+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c074f60 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.803+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd27c01a070 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.803+0000 7fd2827fc640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27006b4d0 0x7fd27006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.803+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd27c0896a0 con 0x7fd28c06c090 2024-03-04T10:09:53.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.803+0000 7fd291fce640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27006b4d0 0x7fd27006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:53.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.804+0000 7fd291fce640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27006b4d0 0x7fd27006d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd288036000 tx=0x7fd288030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:53.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.808+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd27c058340 con 0x7fd28c06c090 2024-03-04T10:09:53.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.977+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd28c0ad3a0 con 0x7fd28c06c090 2024-03-04T10:09:53.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.978+0000 7fd2827fc640 1 -- 172.21.15.99:0/829352983 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+17038 (secure 0 0 0) 0x7fd27c057c40 con 0x7fd28c06c090 2024-03-04T10:09:53.980 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:53.980 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":82,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:46.616184+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3857383686":"2024-03-05T10:09:46.616133+0000","172.21.15.99:6833/3857383686":"2024-03-05T10:09:46.616133+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27006b4d0 msgr2=0x7fd27006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27006b4d0 0x7fd27006d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fd288036000 tx=0x7fd288030000 comp rx=0 tx=0).stop 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 msgr2=0x7fd28c09a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c09a870 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00c3c0 tx=0x7fd27c00c880 comp rx=0 tx=0).stop 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 shutdown_connections 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd27006b4d0 0x7fd27006d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd28c0ceb20 0x7fd28c1f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd28c06c090 0x7fd28c09a870 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 --2- 172.21.15.99:0/829352983 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd28c066de0 0x7fd28c1f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:53.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 >> 172.21.15.99:0/829352983 conn(0x7fd28c0cfef0 msgr2=0x7fd28c0af990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:53.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 shutdown_connections 2024-03-04T10:09:53.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:53.980+0000 7fd294232640 1 -- 172.21.15.99:0/829352983 wait complete. 2024-03-04T10:09:53.993 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:09:13.676364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:09:14.431767+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:09:53.993 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:09:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.095+0000 7fa960c9a640 1 Processor -- start 2024-03-04T10:09:54.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.095+0000 7fa960c9a640 1 -- start start 2024-03-04T10:09:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa960c9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c15b700 0x7fa95c0ae020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa960c9a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa95c06bce0 0x7fa95c06c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa960c9a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa95c15e100 0x7fa95c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa960c9a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c06c680 con 0x7fa95c15e100 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa960c9a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0adb90 con 0x7fa95c06bce0 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa960c9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c15bae0 con 0x7fa95c15b700 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa95c15e100 0x7fa95c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa95c15e100 0x7fa95c15e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50534/0 (socket says 172.21.15.99:50534) 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 -- 172.21.15.99:0/4243858668 learned_addr learned my addr 172.21.15.99:0/4243858668 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa953fff640 1 --2- 172.21.15.99:0/4243858668 >> v2:172.21.15.162:3300/0 conn(0x7fa95c06bce0 0x7fa95c06c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95a59c640 1 --2- 172.21.15.99:0/4243858668 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c15b700 0x7fa95c0ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 -- 172.21.15.99:0/4243858668 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c15b700 msgr2=0x7fa95c0ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 --2- 172.21.15.99:0/4243858668 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c15b700 0x7fa95c0ae020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 -- 172.21.15.99:0/4243858668 >> v2:172.21.15.162:3300/0 conn(0x7fa95c06bce0 msgr2=0x7fa95c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 --2- 172.21.15.99:0/4243858668 >> v2:172.21.15.162:3300/0 conn(0x7fa95c06bce0 0x7fa95c06c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.096+0000 7fa95ad9d640 1 -- 172.21.15.99:0/4243858668 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c15ee60 con 0x7fa95c15e100 2024-03-04T10:09:54.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.097+0000 7fa95ad9d640 1 --2- 172.21.15.99:0/4243858668 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c15e590 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa95c071440 tx=0x7fa95402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be707af49b972b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.097+0000 7fa959d9b640 1 -- 172.21.15.99:0/4243858668 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95402ca70 con 0x7fa95c15e100 2024-03-04T10:09:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.097+0000 7fa959d9b640 1 -- 172.21.15.99:0/4243858668 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95402cc10 con 0x7fa95c15e100 2024-03-04T10:09:54.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.097+0000 7fa959d9b640 1 -- 172.21.15.99:0/4243858668 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540343f0 con 0x7fa95c15e100 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 -- 172.21.15.99:0/4243858668 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 msgr2=0x7fa95c15e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 --2- 172.21.15.99:0/4243858668 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c15e590 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fa95c071440 tx=0x7fa95402bed0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 -- 172.21.15.99:0/4243858668 shutdown_connections 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 --2- 172.21.15.99:0/4243858668 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c15e590 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 --2- 172.21.15.99:0/4243858668 >> v2:172.21.15.162:3300/0 conn(0x7fa95c06bce0 0x7fa95c06c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 --2- 172.21.15.99:0/4243858668 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c15b700 0x7fa95c0ae020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 -- 172.21.15.99:0/4243858668 >> 172.21.15.99:0/4243858668 conn(0x7fa95c071bc0 msgr2=0x7fa95c071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 -- 172.21.15.99:0/4243858668 shutdown_connections 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.098+0000 7fa960c9a640 1 -- 172.21.15.99:0/4243858668 wait complete. 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.099+0000 7fa960c9a640 1 Processor -- start 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.099+0000 7fa960c9a640 1 -- start start 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c06bce0 0x7fa95c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa95c15b700 0x7fa95c06eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c15ead0 con 0x7fa95c15e100 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0ad820 con 0x7fa95c15b700 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0ae560 con 0x7fa95c06bce0 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c06f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50550/0 (socket says 172.21.15.99:50550) 2024-03-04T10:09:54.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 -- 172.21.15.99:0/1970334681 learned_addr learned my addr 172.21.15.99:0/1970334681 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 -- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c06bce0 msgr2=0x7fa95c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95a59c640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c06bce0 0x7fa95c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa953fff640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa95c15b700 0x7fa95c06eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c06bce0 0x7fa95c0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 -- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa95c15b700 msgr2=0x7fa95c06eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa95c15b700 0x7fa95c06eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 -- 172.21.15.99:0/1970334681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0ce8e0 con 0x7fa95c15e100 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95a59c640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c06bce0 0x7fa95c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa953fff640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa95c15b700 0x7fa95c06eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa95ad9d640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c06f0e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa954036040 tx=0x7fa954004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954044020 con 0x7fa95c15e100 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa954040690 con 0x7fa95c15e100 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c0cebc0 con 0x7fa95c15e100 2024-03-04T10:09:54.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954009510 con 0x7fa95c15e100 2024-03-04T10:09:54.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.100+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c0cf180 con 0x7fa95c15e100 2024-03-04T10:09:54.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.101+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c074f60 con 0x7fa95c15e100 2024-03-04T10:09:54.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.102+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa954047020 con 0x7fa95c15e100 2024-03-04T10:09:54.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.102+0000 7fa952ffd640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa93c06b4d0 0x7fa93c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.102+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa9540acf60 con 0x7fa95c15e100 2024-03-04T10:09:54.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.102+0000 7fa95a59c640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa93c06b4d0 0x7fa93c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.103+0000 7fa95a59c640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa93c06b4d0 0x7fa93c06d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa94c006fd0 tx=0x7fa94c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.106+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa95404e030 con 0x7fa95c15e100 2024-03-04T10:09:54.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.276+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa95c0b3920 con 0x7fa95c15e100 2024-03-04T10:09:54.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.276+0000 7fa952ffd640 1 -- 172.21.15.99:0/1970334681 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa95407bc00 con 0x7fa95c15e100 2024-03-04T10:09:54.278 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa93c06b4d0 msgr2=0x7fa93c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa93c06b4d0 0x7fa93c06d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa94c006fd0 tx=0x7fa94c008040 comp rx=0 tx=0).stop 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 msgr2=0x7fa95c06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c06f0e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fa954036040 tx=0x7fa954004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 shutdown_connections 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa93c06b4d0 0x7fa93c06d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa95c15e100 0x7fa95c06f0e0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa95c15b700 0x7fa95c06eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 --2- 172.21.15.99:0/1970334681 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa95c06bce0 0x7fa95c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 >> 172.21.15.99:0/1970334681 conn(0x7fa95c071bc0 msgr2=0x7fa95c0b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 shutdown_connections 2024-03-04T10:09:54.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.278+0000 7fa960c9a640 1 -- 172.21.15.99:0/1970334681 wait complete. 2024-03-04T10:09:54.291 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:09:54.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 Processor -- start 2024-03-04T10:09:54.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 -- start start 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f99940cec40 0x7f99940b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99940ca970 con 0x7f99940cf020 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9994161d90 con 0x7f99940cec40 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.393+0000 7f9999827640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9994162070 con 0x7f999415de40 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f99940cec40 0x7f99940b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f99940cec40 0x7f99940b6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50554/0 (socket says 172.21.15.99:50554) 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 -- 172.21.15.99:0/1451613639 learned_addr learned my addr 172.21.15.99:0/1451613639 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f99937fe640 1 --2- 172.21.15.99:0/1451613639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 -- 172.21.15.99:0/1451613639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 msgr2=0x7f99940d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 --2- 172.21.15.99:0/1451613639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 -- 172.21.15.99:0/1451613639 >> v1:172.21.15.162:6789/0 conn(0x7f99940cf020 legacy=0x7f999415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.394+0000 7f9992ffd640 1 -- 172.21.15.99:0/1451613639 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9994162350 con 0x7f99940cec40 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9992ffd640 1 --2- 172.21.15.99:0/1451613639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f99940b6ea0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f99940674a0 tx=0x7f998402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2141de6851da2779 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9991ffb640 1 -- 172.21.15.99:0/1451613639 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984035020 con 0x7f99940cec40 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9991ffb640 1 -- 172.21.15.99:0/1451613639 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9984031830 con 0x7f99940cec40 2024-03-04T10:09:54.397 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9991ffb640 1 -- 172.21.15.99:0/1451613639 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984031c00 con 0x7f99940cec40 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 -- 172.21.15.99:0/1451613639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 msgr2=0x7f99940b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 --2- 172.21.15.99:0/1451613639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f99940b6ea0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f99940674a0 tx=0x7f998402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 -- 172.21.15.99:0/1451613639 shutdown_connections 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 --2- 172.21.15.99:0/1451613639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 --2- 172.21.15.99:0/1451613639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f99940b6ea0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 -- 172.21.15.99:0/1451613639 >> 172.21.15.99:0/1451613639 conn(0x7f99940d0580 msgr2=0x7f99940d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 -- 172.21.15.99:0/1451613639 shutdown_connections 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.395+0000 7f9999827640 1 -- 172.21.15.99:0/1451613639 wait complete. 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.396+0000 7f9999827640 1 Processor -- start 2024-03-04T10:09:54.400 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 -- start start 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f9994151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99940cf020 0x7f99940c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940d3300 con 0x7f99940cec40 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940d1bf0 con 0x7f99940cf020 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9999827640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99940d3480 con 0x7f999415de40 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9992ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f9994151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9992ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f9994151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50562/0 (socket says 172.21.15.99:50562) 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f9992ffd640 1 -- 172.21.15.99:0/236357196 learned_addr learned my addr 172.21.15.99:0/236357196 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.397+0000 7f99937fe640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f99927fc640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99940cf020 0x7f99940c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9992ffd640 1 -- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 msgr2=0x7f99940d3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9992ffd640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9992ffd640 1 -- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99940cf020 msgr2=0x7f99940c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9992ffd640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99940cf020 0x7f99940c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9992ffd640 1 -- 172.21.15.99:0/236357196 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99940c7b10 con 0x7f99940cec40 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9992ffd640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f9994151dc0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9984000c00 tx=0x7f99840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984040070 con 0x7f99940cec40 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9984039940 con 0x7f99940cec40 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99940c7df0 con 0x7f99940cec40 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984035040 con 0x7f99940cec40 2024-03-04T10:09:54.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.398+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9994152300 con 0x7f99940cec40 2024-03-04T10:09:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.399+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994074f60 con 0x7f99940cec40 2024-03-04T10:09:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.399+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f998404b060 con 0x7f99940cec40 2024-03-04T10:09:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.400+0000 7f997bfff640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f998006b500 0x7f998006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.400+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f99840abb80 con 0x7f99940cec40 2024-03-04T10:09:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.400+0000 7f99927fc640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f998006b500 0x7f998006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.401+0000 7f99927fc640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f998006b500 0x7f998006d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f99941f7260 tx=0x7f9988009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.404+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f998407a820 con 0x7f99940cec40 2024-03-04T10:09:54.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.575+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f99940d1930 con 0x7f99940cec40 2024-03-04T10:09:54.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.575+0000 7f997bfff640 1 -- 172.21.15.99:0/236357196 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7f998407a120 con 0x7f99940cec40 2024-03-04T10:09:54.577 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f998006b500 msgr2=0x7f998006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f998006b500 0x7f998006d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f99941f7260 tx=0x7f9988009040 comp rx=0 tx=0).stop 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 msgr2=0x7f9994151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f9994151dc0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f9984000c00 tx=0x7f99840046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 shutdown_connections 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f998006b500 0x7f998006d9c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f999415de40 0x7f99940d3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f99940cf020 0x7f99940c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 --2- 172.21.15.99:0/236357196 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f99940cec40 0x7f9994151dc0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 >> 172.21.15.99:0/236357196 conn(0x7f99940d0580 msgr2=0x7f9994155f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.577+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 shutdown_connections 2024-03-04T10:09:54.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.578+0000 7f9999827640 1 -- 172.21.15.99:0/236357196 wait complete. 2024-03-04T10:09:54.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:09:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.692+0000 7ffb3a852640 1 Processor -- start 2024-03-04T10:09:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.692+0000 7ffb3a852640 1 -- start start 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb3a852640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cdd70 0x7ffb340cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb3a852640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ffb340cc900 0x7ffb340cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb3a852640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb340ccfe0 0x7ffb3415d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb3a852640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340ccde0 con 0x7ffb340ccfe0 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb3a852640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb340d0480 con 0x7ffb340cc900 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb3a852640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3406ae00 con 0x7ffb340cdd70 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb340ccfe0 0x7ffb3415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ffb340ccfe0 0x7ffb3415d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50572/0 (socket says 172.21.15.99:50572) 2024-03-04T10:09:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 -- 172.21.15.99:0/4206356298 learned_addr learned my addr 172.21.15.99:0/4206356298 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 -- 172.21.15.99:0/4206356298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cdd70 msgr2=0x7ffb340cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 --2- 172.21.15.99:0/4206356298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cdd70 0x7ffb340cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 -- 172.21.15.99:0/4206356298 >> v2:172.21.15.162:3300/0 conn(0x7ffb340cc900 msgr2=0x7ffb340cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 --2- 172.21.15.99:0/4206356298 >> v2:172.21.15.162:3300/0 conn(0x7ffb340cc900 0x7ffb340cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.693+0000 7ffb38def640 1 -- 172.21.15.99:0/4206356298 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3406b0e0 con 0x7ffb340ccfe0 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.694+0000 7ffb38def640 1 --2- 172.21.15.99:0/4206356298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340ccfe0 0x7ffb3415d410 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ffb3406def0 tx=0x7ffb2c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84625760974efe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.694+0000 7ffb32ffd640 1 -- 172.21.15.99:0/4206356298 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c02c9e0 con 0x7ffb340ccfe0 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.694+0000 7ffb32ffd640 1 -- 172.21.15.99:0/4206356298 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c02cb80 con 0x7ffb340ccfe0 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.694+0000 7ffb32ffd640 1 -- 172.21.15.99:0/4206356298 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c0343f0 con 0x7ffb340ccfe0 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 -- 172.21.15.99:0/4206356298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340ccfe0 msgr2=0x7ffb3415d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 --2- 172.21.15.99:0/4206356298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340ccfe0 0x7ffb3415d410 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ffb3406def0 tx=0x7ffb2c02be40 comp rx=0 tx=0).stop 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 -- 172.21.15.99:0/4206356298 shutdown_connections 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 --2- 172.21.15.99:0/4206356298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340ccfe0 0x7ffb3415d410 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 --2- 172.21.15.99:0/4206356298 >> v2:172.21.15.162:3300/0 conn(0x7ffb340cc900 0x7ffb340cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 --2- 172.21.15.99:0/4206356298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cdd70 0x7ffb340cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 -- 172.21.15.99:0/4206356298 >> 172.21.15.99:0/4206356298 conn(0x7ffb340b22e0 msgr2=0x7ffb340b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 -- 172.21.15.99:0/4206356298 shutdown_connections 2024-03-04T10:09:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.695+0000 7ffb3a852640 1 -- 172.21.15.99:0/4206356298 wait complete. 2024-03-04T10:09:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.696+0000 7ffb3a852640 1 Processor -- start 2024-03-04T10:09:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.696+0000 7ffb3a852640 1 -- start start 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb3a852640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cc900 0x7ffb340c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb3a852640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340ccfe0 0x7ffb34160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb3a852640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 0x7ffb341607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb3a852640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340b2e90 con 0x7ffb340cdd70 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb3a852640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3406b6a0 con 0x7ffb340ccfe0 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb3a852640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb340b3010 con 0x7ffb340cc900 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 0x7ffb341607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 0x7ffb341607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50588/0 (socket says 172.21.15.99:50588) 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 -- 172.21.15.99:0/1134214846 learned_addr learned my addr 172.21.15.99:0/1134214846 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb337fe640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340ccfe0 0x7ffb34160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb33fff640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cc900 0x7ffb340c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 -- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cc900 msgr2=0x7ffb340c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cc900 0x7ffb340c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 -- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340ccfe0 msgr2=0x7ffb34160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340ccfe0 0x7ffb34160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb38def640 1 -- 172.21.15.99:0/1134214846 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb341f7940 con 0x7ffb340cdd70 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb33fff640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cc900 0x7ffb340c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.697+0000 7ffb337fe640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340ccfe0 0x7ffb34160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.698+0000 7ffb38def640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 0x7ffb341607c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c036040 tx=0x7ffb2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.698+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c043020 con 0x7ffb340cdd70 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.698+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c03e450 con 0x7ffb340cdd70 2024-03-04T10:09:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.698+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c03e820 con 0x7ffb340cdd70 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.698+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb341f7c20 con 0x7ffb340cdd70 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.698+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb340c79f0 con 0x7ffb340cdd70 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.699+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb34074f60 con 0x7ffb340cdd70 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.699+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ffb2c046020 con 0x7ffb340cdd70 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.699+0000 7ffb30ff9640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.700+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffb2c0acd30 con 0x7ffb340cdd70 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.700+0000 7ffb33fff640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.700+0000 7ffb33fff640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009930 tx=0x7ffb24008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.704+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ffb2c04d030 con 0x7ffb340cdd70 2024-03-04T10:09:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.874+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffb340b6710 con 0x7ffb340cdd70 2024-03-04T10:09:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.874+0000 7ffb30ff9640 1 -- 172.21.15.99:0/1134214846 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffb2c07ba60 con 0x7ffb340cdd70 2024-03-04T10:09:54.876 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 msgr2=0x7ffb2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009930 tx=0x7ffb24008040 comp rx=0 tx=0).stop 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 msgr2=0x7ffb341607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 0x7ffb341607c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c036040 tx=0x7ffb2c004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 shutdown_connections 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ffb340cdd70 0x7ffb341607c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffb340ccfe0 0x7ffb34160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 --2- 172.21.15.99:0/1134214846 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ffb340cc900 0x7ffb340c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 >> 172.21.15.99:0/1134214846 conn(0x7ffb340b22e0 msgr2=0x7ffb3406f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 shutdown_connections 2024-03-04T10:09:54.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.876+0000 7ffb3a852640 1 -- 172.21.15.99:0/1134214846 wait complete. 2024-03-04T10:09:54.888 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:54.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.988+0000 7f8a12c4b640 1 Processor -- start 2024-03-04T10:09:54.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.988+0000 7f8a12c4b640 1 -- start start 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a12c4b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8a0c06d3b0 0x7f8a0c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a12c4b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c15b410 0x7f8a0c15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a12c4b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c15be00 con 0x7f8a0c15b030 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a12c4b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0c0b2720 con 0x7f8a0c06d3b0 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a12c4b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c06d8b0 con 0x7f8a0c15b410 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a109e7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8a0c15b030 0x7f8a0c15dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40486/0 (socket says 172.21.15.99:40486) 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a109e7640 1 -- 172.21.15.99:0/4253857478 learned_addr learned my addr 172.21.15.99:0/4253857478 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:54.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a111e8640 1 --2- 172.21.15.99:0/4253857478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c15b410 0x7f8a0c15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a0bfff640 1 --2- 172.21.15.99:0/4253857478 >> v2:172.21.15.162:3300/0 conn(0x7f8a0c06d3b0 0x7f8a0c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a0b7fe640 1 -- 172.21.15.99:0/4253857478 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989741435 0 0) 0x7f8a0c15be00 con 0x7f8a0c15b030 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a0b7fe640 1 -- 172.21.15.99:0/4253857478 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc003520 con 0x7f8a0c15b030 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a111e8640 1 -- 172.21.15.99:0/4253857478 >> v2:172.21.15.162:3300/0 conn(0x7f8a0c06d3b0 msgr2=0x7f8a0c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a111e8640 1 --2- 172.21.15.99:0/4253857478 >> v2:172.21.15.162:3300/0 conn(0x7f8a0c06d3b0 0x7f8a0c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a111e8640 1 -- 172.21.15.99:0/4253857478 >> v1:172.21.15.99:6789/0 conn(0x7f8a0c15b030 legacy=0x7f8a0c15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a0b7fe640 1 -- 172.21.15.99:0/4253857478 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2729455656 0 0) 0x7f89fc003520 con 0x7f8a0c15b030 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a111e8640 1 -- 172.21.15.99:0/4253857478 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c06db90 con 0x7f8a0c15b410 2024-03-04T10:09:54.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.989+0000 7f8a0bfff640 1 --2- 172.21.15.99:0/4253857478 >> v2:172.21.15.162:3300/0 conn(0x7f8a0c06d3b0 0x7f8a0c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.990+0000 7f8a111e8640 1 --2- 172.21.15.99:0/4253857478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c15b410 0x7f8a0c15b8c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0b28a0 tx=0x7f8a0402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5a578f2cebc68d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.990+0000 7f8a0b7fe640 1 -- 172.21.15.99:0/4253857478 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04039070 con 0x7f8a0c15b410 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.990+0000 7f8a0b7fe640 1 -- 172.21.15.99:0/4253857478 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0402c810 con 0x7f8a0c15b410 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.990+0000 7f8a0b7fe640 1 -- 172.21.15.99:0/4253857478 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a00001670 con 0x7f8a0c15b410 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 -- 172.21.15.99:0/4253857478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c15b410 msgr2=0x7f8a0c15b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/4253857478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c15b410 0x7f8a0c15b8c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0b28a0 tx=0x7f8a0402bce0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 -- 172.21.15.99:0/4253857478 shutdown_connections 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/4253857478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c15b410 0x7f8a0c15b8c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/4253857478 >> v2:172.21.15.162:3300/0 conn(0x7f8a0c06d3b0 0x7f8a0c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 -- 172.21.15.99:0/4253857478 >> 172.21.15.99:0/4253857478 conn(0x7f8a0c06acd0 msgr2=0x7f8a0c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 -- 172.21.15.99:0/4253857478 shutdown_connections 2024-03-04T10:09:54.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.991+0000 7f8a12c4b640 1 -- 172.21.15.99:0/4253857478 wait complete. 2024-03-04T10:09:54.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.992+0000 7f8a12c4b640 1 Processor -- start 2024-03-04T10:09:54.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.992+0000 7f8a12c4b640 1 -- start start 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a12c4b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c06d3b0 0x7f8a0c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a12c4b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 0x7f8a0c0b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a12c4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a0c15b410 0x7f8a0c0b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a12c4b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c06e690 con 0x7f8a0c15b030 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a12c4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c06e150 con 0x7f8a0c15b410 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a12c4b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c06e810 con 0x7f8a0c06d3b0 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 0x7f8a0c0b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 0x7f8a0c0b0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50600/0 (socket says 172.21.15.99:50600) 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 -- 172.21.15.99:0/2606014743 learned_addr learned my addr 172.21.15.99:0/2606014743 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a109e7640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c06d3b0 0x7f8a0c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a111e8640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a0c15b410 0x7f8a0c0b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 -- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c06d3b0 msgr2=0x7f8a0c157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c06d3b0 0x7f8a0c157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 -- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a0c15b410 msgr2=0x7f8a0c0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a0c15b410 0x7f8a0c0b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 -- 172.21.15.99:0/2606014743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c0ad8b0 con 0x7f8a0c15b030 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a111e8640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a0c15b410 0x7f8a0c0b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a109e7640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c06d3b0 0x7f8a0c157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.993+0000 7f8a0bfff640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 0x7f8a0c0b0640 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f89f800e9b0 tx=0x7f89f800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.994+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f8017070 con 0x7f8a0c15b030 2024-03-04T10:09:54.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.994+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f800ca00 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.994+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f800cdf0 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.994+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c0adb90 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.994+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c0ae150 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.995+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c0ae350 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.995+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f89f801a070 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.996+0000 7f8a097fa640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89f006b4d0 0x7f89f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.996+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f89f808a380 con 0x7f8a0c15b030 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.996+0000 7f8a109e7640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89f006b4d0 0x7f89f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:54.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:54.996+0000 7f8a109e7640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89f006b4d0 0x7f89f006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8a00007ae0 tx=0x7f8a00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.000+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f89f8059020 con 0x7f8a0c15b030 2024-03-04T10:09:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.208+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a0c074f60 con 0x7f8a0c15b030 2024-03-04T10:09:55.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.208+0000 7f8a097fa640 1 -- 172.21.15.99:0/2606014743 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4948 (secure 0 0 0) 0x7f89f8058920 con 0x7f8a0c15b030 2024-03-04T10:09:55.210 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:55.210 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:48.638042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":12,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:55.210 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 71 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89f006b4d0 msgr2=0x7f89f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89f006b4d0 0x7f89f006d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8a00007ae0 tx=0x7f8a00009040 comp rx=0 tx=0).stop 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 msgr2=0x7f8a0c0b0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 0x7f8a0c0b0640 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f89f800e9b0 tx=0x7f89f800ee70 comp rx=0 tx=0).stop 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 shutdown_connections 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89f006b4d0 0x7f89f006d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a0c15b410 0x7f8a0c0b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a0c15b030 0x7f8a0c0b0640 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 --2- 172.21.15.99:0/2606014743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a0c06d3b0 0x7f8a0c157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 >> 172.21.15.99:0/2606014743 conn(0x7f8a0c06acd0 msgr2=0x7f8a0c0af950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 shutdown_connections 2024-03-04T10:09:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.210+0000 7f8a12c4b640 1 -- 172.21.15.99:0/2606014743 wait complete. 2024-03-04T10:09:55.222 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:55.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.320+0000 7f47a6681640 1 Processor -- start 2024-03-04T10:09:55.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.320+0000 7f47a6681640 1 -- start start 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a6681640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47a006c090 0x7f47a006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a6681640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a006c470 0x7f47a00af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a6681640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a00ad6c0 con 0x7f47a0066de0 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a6681640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a00ad960 con 0x7f47a006c090 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a6681640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a006ebc0 con 0x7f47a006c470 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f479ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f47a006c090 0x7f47a006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a4c1e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f47a0066de0 0x7f47a015f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40498/0 (socket says 172.21.15.99:40498) 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.321+0000 7f47a4c1e640 1 -- 172.21.15.99:0/3120286487 learned_addr learned my addr 172.21.15.99:0/3120286487 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479ffff640 1 -- 172.21.15.99:0/3120286487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a006c470 msgr2=0x7f47a00af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479ffff640 1 --2- 172.21.15.99:0/3120286487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a006c470 0x7f47a00af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479ffff640 1 -- 172.21.15.99:0/3120286487 >> v1:172.21.15.99:6789/0 conn(0x7f47a0066de0 legacy=0x7f47a015f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479ffff640 1 -- 172.21.15.99:0/3120286487 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a006eea0 con 0x7f47a006c090 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479effd640 1 -- 172.21.15.99:0/3120286487 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447919622 0 0) 0x7f47a00ad6c0 con 0x7f47a0066de0 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479ffff640 1 --2- 172.21.15.99:0/3120286487 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a006f680 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f47900097a0 tx=0x7f479002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59381ceb813de67 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479effd640 1 -- 172.21.15.99:0/3120286487 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47900315a0 con 0x7f47a006c090 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.322+0000 7f479effd640 1 -- 172.21.15.99:0/3120286487 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790037070 con 0x7f47a006c090 2024-03-04T10:09:55.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f479effd640 1 -- 172.21.15.99:0/3120286487 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790031960 con 0x7f47a006c090 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 -- 172.21.15.99:0/3120286487 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 msgr2=0x7f47a006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 --2- 172.21.15.99:0/3120286487 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a006f680 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f47900097a0 tx=0x7f479002b840 comp rx=0 tx=0).stop 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 -- 172.21.15.99:0/3120286487 shutdown_connections 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 --2- 172.21.15.99:0/3120286487 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a006c470 0x7f47a00af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 --2- 172.21.15.99:0/3120286487 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a006f680 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 -- 172.21.15.99:0/3120286487 >> 172.21.15.99:0/3120286487 conn(0x7f47a00cfef0 msgr2=0x7f47a00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 -- 172.21.15.99:0/3120286487 shutdown_connections 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.323+0000 7f47a6681640 1 -- 172.21.15.99:0/3120286487 wait complete. 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.324+0000 7f47a6681640 1 Processor -- start 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.324+0000 7f47a6681640 1 -- start start 2024-03-04T10:09:55.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a6681640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47a0066de0 0x7f47a01f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a6681640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a01586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a6681640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a01eeb40 0x7f47a0158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a6681640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00adc40 con 0x7f47a006c090 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a6681640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00af760 con 0x7f47a0066de0 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a6681640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a00addc0 con 0x7f47a01eeb40 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a01586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a01586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50622/0 (socket says 172.21.15.99:50622) 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f479f7fe640 1 -- 172.21.15.99:0/2851905678 learned_addr learned my addr 172.21.15.99:0/2851905678 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f47a4c1e640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a01eeb40 0x7f47a0158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.325+0000 7f479ffff640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47a0066de0 0x7f47a01f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479f7fe640 1 -- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a01eeb40 msgr2=0x7f47a0158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479f7fe640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a01eeb40 0x7f47a0158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479f7fe640 1 -- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47a0066de0 msgr2=0x7f47a01f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479f7fe640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47a0066de0 0x7f47a01f1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479f7fe640 1 -- 172.21.15.99:0/2851905678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a01f2330 con 0x7f47a006c090 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f47a4c1e640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a01eeb40 0x7f47a0158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479ffff640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47a0066de0 0x7f47a01f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479f7fe640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a01586f0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f479400b4a0 tx=0x7f479400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794014070 con 0x7f47a006c090 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4794010610 con 0x7f47a006c090 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a0156f50 con 0x7f47a006c090 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794010a00 con 0x7f47a006c090 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.326+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a0157510 con 0x7f47a006c090 2024-03-04T10:09:55.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.327+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a0074f60 con 0x7f47a006c090 2024-03-04T10:09:55.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.327+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4794017070 con 0x7f47a006c090 2024-03-04T10:09:55.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.328+0000 7f479cff9640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f478c06b4d0 0x7f478c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.328+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4794089e40 con 0x7f47a006c090 2024-03-04T10:09:55.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.328+0000 7f479ffff640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f478c06b4d0 0x7f478c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.328+0000 7f479ffff640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f478c06b4d0 0x7f478c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4790036000 tx=0x7f479003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.332+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f47a01f2330 con 0x7f47a006c090 2024-03-04T10:09:55.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.502+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47a0157ef0 con 0x7f47a006c090 2024-03-04T10:09:55.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.503+0000 7f479cff9640 1 -- 172.21.15.99:0/2851905678 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+17038 (secure 0 0 0) 0x7f4794058ae0 con 0x7f47a006c090 2024-03-04T10:09:55.504 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:55.505 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":82,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:46.616184+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3857383686":"2024-03-05T10:09:46.616133+0000","172.21.15.99:6833/3857383686":"2024-03-05T10:09:46.616133+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:55.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f478c06b4d0 msgr2=0x7f478c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f478c06b4d0 0x7f478c06d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4790036000 tx=0x7f479003d040 comp rx=0 tx=0).stop 2024-03-04T10:09:55.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 msgr2=0x7f47a01586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a01586f0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f479400b4a0 tx=0x7f479400b960 comp rx=0 tx=0).stop 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 shutdown_connections 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f478c06b4d0 0x7f478c06d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f47a01eeb40 0x7f47a0158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f47a006c090 0x7f47a01586f0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 --2- 172.21.15.99:0/2851905678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47a0066de0 0x7f47a01f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 >> 172.21.15.99:0/2851905678 conn(0x7f47a00cfef0 msgr2=0x7f47a015fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 shutdown_connections 2024-03-04T10:09:55.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.505+0000 7f47a6681640 1 -- 172.21.15.99:0/2851905678 wait complete. 2024-03-04T10:09:55.517 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:09:55.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:55.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 Processor -- start 2024-03-04T10:09:55.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 -- start start 2024-03-04T10:09:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b030 0x7f95c415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f95c415b410 0x7f95c415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c415be00 con 0x7f95c406d3b0 2024-03-04T10:09:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c40b2720 con 0x7f95c415b410 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.616+0000 7f95c9085640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c406d8b0 con 0x7f95c415b030 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c259c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f95c406d3b0 0x7f95c415dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40500/0 (socket says 172.21.15.99:40500) 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c359e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f95c415b410 0x7f95c415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c259c640 1 -- 172.21.15.99:0/1061877425 learned_addr learned my addr 172.21.15.99:0/1061877425 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c359e640 1 -- 172.21.15.99:0/1061877425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b030 msgr2=0x7f95c415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c359e640 1 --2- 172.21.15.99:0/1061877425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b030 0x7f95c415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c359e640 1 -- 172.21.15.99:0/1061877425 >> v1:172.21.15.99:6789/0 conn(0x7f95c406d3b0 legacy=0x7f95c415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c359e640 1 -- 172.21.15.99:0/1061877425 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c406db90 con 0x7f95c415b410 2024-03-04T10:09:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c1d9b640 1 -- 172.21.15.99:0/1061877425 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682192761 0 0) 0x7f95c415be00 con 0x7f95c406d3b0 2024-03-04T10:09:55.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.617+0000 7f95c359e640 1 --2- 172.21.15.99:0/1061877425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c415b410 0x7f95c415b8c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f95b40097a0 tx=0x7f95b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b133229e638be09 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c1d9b640 1 -- 172.21.15.99:0/1061877425 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b40315a0 con 0x7f95c415b410 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c1d9b640 1 -- 172.21.15.99:0/1061877425 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b4037070 con 0x7f95c415b410 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c1d9b640 1 -- 172.21.15.99:0/1061877425 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4031960 con 0x7f95c415b410 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 -- 172.21.15.99:0/1061877425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c415b410 msgr2=0x7f95c415b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 --2- 172.21.15.99:0/1061877425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c415b410 0x7f95c415b8c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f95b40097a0 tx=0x7f95b402b840 comp rx=0 tx=0).stop 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 -- 172.21.15.99:0/1061877425 shutdown_connections 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 --2- 172.21.15.99:0/1061877425 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c415b410 0x7f95c415b8c0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 --2- 172.21.15.99:0/1061877425 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b030 0x7f95c415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 -- 172.21.15.99:0/1061877425 >> 172.21.15.99:0/1061877425 conn(0x7f95c406acd0 msgr2=0x7f95c406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 -- 172.21.15.99:0/1061877425 shutdown_connections 2024-03-04T10:09:55.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.618+0000 7f95c9085640 1 -- 172.21.15.99:0/1061877425 wait complete. 2024-03-04T10:09:55.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 Processor -- start 2024-03-04T10:09:55.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 -- start start 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 0x7f95c41556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c415b030 0x7f95c4156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b410 0x7f95c4153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c406e690 con 0x7f95c406d3b0 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c406e150 con 0x7f95c415b030 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c9085640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c406e810 con 0x7f95c415b410 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c2d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 0x7f95c41556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c2d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 0x7f95c41556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50626/0 (socket says 172.21.15.99:50626) 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.620+0000 7f95c2d9d640 1 -- 172.21.15.99:0/580427733 learned_addr learned my addr 172.21.15.99:0/580427733 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c259c640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c415b030 0x7f95c4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c359e640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b410 0x7f95c4153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c2d9d640 1 -- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b410 msgr2=0x7f95c4153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c2d9d640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b410 0x7f95c4153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c2d9d640 1 -- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c415b030 msgr2=0x7f95c4156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c2d9d640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c415b030 0x7f95c4156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c2d9d640 1 -- 172.21.15.99:0/580427733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4158940 con 0x7f95c406d3b0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c2d9d640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 0x7f95c41556f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00ba80 tx=0x7f95bc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc019070 con 0x7f95c406d3b0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc00cc20 con 0x7f95c406d3b0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c41578a0 con 0x7f95c406d3b0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc014440 con 0x7f95c406d3b0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.621+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c4157e60 con 0x7f95c406d3b0 2024-03-04T10:09:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.622+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c4074f60 con 0x7f95c406d3b0 2024-03-04T10:09:55.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.623+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f95bc01c070 con 0x7f95c406d3b0 2024-03-04T10:09:55.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.623+0000 7f95bb7fe640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f95a406b500 0x7f95a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.623+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f95bc08a070 con 0x7f95c406d3b0 2024-03-04T10:09:55.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.623+0000 7f95c259c640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f95a406b500 0x7f95a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.624+0000 7f95c259c640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f95a406b500 0x7f95a406d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f95c40b0fa0 tx=0x7f95ac007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.627+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f95bc058640 con 0x7f95c406d3b0 2024-03-04T10:09:55.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.833+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95c406b360 con 0x7f95c406d3b0 2024-03-04T10:09:55.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.834+0000 7f95bb7fe640 1 -- 172.21.15.99:0/580427733 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4948 (secure 0 0 0) 0x7f95bc05c6e0 con 0x7f95c406d3b0 2024-03-04T10:09:55.835 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:55.835 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:48.638042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":12,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:55.836 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 71 2024-03-04T10:09:55.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f95a406b500 msgr2=0x7f95a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f95a406b500 0x7f95a406d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f95c40b0fa0 tx=0x7f95ac007c00 comp rx=0 tx=0).stop 2024-03-04T10:09:55.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 msgr2=0x7f95c41556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 0x7f95c41556f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00ba80 tx=0x7f95bc00bf40 comp rx=0 tx=0).stop 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 shutdown_connections 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f95a406b500 0x7f95a406d9c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f95c415b410 0x7f95c4153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f95c415b030 0x7f95c4156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 --2- 172.21.15.99:0/580427733 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f95c406d3b0 0x7f95c41556f0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 >> 172.21.15.99:0/580427733 conn(0x7f95c406acd0 msgr2=0x7f95c415d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 shutdown_connections 2024-03-04T10:09:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.836+0000 7f95c9085640 1 -- 172.21.15.99:0/580427733 wait complete. 2024-03-04T10:09:55.848 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:09:55.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.947+0000 7f5c40612640 1 Processor -- start 2024-03-04T10:09:55.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.947+0000 7f5c40612640 1 -- start start 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c40612640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5c380aeb20 0x7f5c38071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c40612640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3815da40 con 0x7f5c3815d660 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c40612640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380af090 con 0x7f5c38071890 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c40612640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c38071ce0 con 0x7f5c380aeb20 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c3dbad640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5c380aeb20 0x7f5c38071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c3e3ae640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5c3815d660 0x7f5c3815e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40506/0 (socket says 172.21.15.99:40506) 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c3e3ae640 1 -- 172.21.15.99:0/2754182409 learned_addr learned my addr 172.21.15.99:0/2754182409 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2754182409 >> v1:172.21.15.162:6789/0 conn(0x7f5c38071890 legacy=0x7f5c38150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2754182409 >> v1:172.21.15.99:6789/0 conn(0x7f5c3815d660 legacy=0x7f5c3815e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.948+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2754182409 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3815a9f0 con 0x7f5c380aeb20 2024-03-04T10:09:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.949+0000 7f5c3dbad640 1 --2- 172.21.15.99:0/2754182409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38071350 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c2802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4303b3ea444d12c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.949+0000 7f5c3d3ac640 1 -- 172.21.15.99:0/2754182409 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28038070 con 0x7f5c380aeb20 2024-03-04T10:09:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.949+0000 7f5c3d3ac640 1 -- 172.21.15.99:0/2754182409 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c28031700 con 0x7f5c380aeb20 2024-03-04T10:09:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.949+0000 7f5c3d3ac640 1 -- 172.21.15.99:0/2754182409 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28031ad0 con 0x7f5c380aeb20 2024-03-04T10:09:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.949+0000 7f5c40612640 1 -- 172.21.15.99:0/2754182409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 msgr2=0x7f5c38071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.949+0000 7f5c40612640 1 --2- 172.21.15.99:0/2754182409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38071350 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f5c280097a0 tx=0x7f5c2802b7a0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.950+0000 7f5c40612640 1 -- 172.21.15.99:0/2754182409 shutdown_connections 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.950+0000 7f5c40612640 1 --2- 172.21.15.99:0/2754182409 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38071350 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.950+0000 7f5c40612640 1 -- 172.21.15.99:0/2754182409 >> 172.21.15.99:0/2754182409 conn(0x7f5c3806acd0 msgr2=0x7f5c3806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.950+0000 7f5c40612640 1 -- 172.21.15.99:0/2754182409 shutdown_connections 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.950+0000 7f5c40612640 1 -- 172.21.15.99:0/2754182409 wait complete. 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 Processor -- start 2024-03-04T10:09:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 -- start start 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c38071890 0x7f5c3814bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3815d660 0x7f5c3814a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3815afb0 con 0x7f5c380aeb20 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3815cbc0 con 0x7f5c3815d660 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.951+0000 7f5c40612640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3815a660 con 0x7f5c38071890 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50632/0 (socket says 172.21.15.99:50632) 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2678127996 learned_addr learned my addr 172.21.15.99:0/2678127996 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3ebaf640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3815d660 0x7f5c3814a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3e3ae640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c38071890 0x7f5c3814bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c38071890 msgr2=0x7f5c3814bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c38071890 0x7f5c3814bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3815d660 msgr2=0x7f5c3814a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3815d660 0x7f5c3814a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 -- 172.21.15.99:0/2678127996 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3814ed20 con 0x7f5c380aeb20 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c3dbad640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38151850 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5c280098c0 tx=0x7f5c280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28043070 con 0x7f5c380aeb20 2024-03-04T10:09:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c28033a30 con 0x7f5c380aeb20 2024-03-04T10:09:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c28038040 con 0x7f5c380aeb20 2024-03-04T10:09:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c38152080 con 0x7f5c380aeb20 2024-03-04T10:09:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.952+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c38152920 con 0x7f5c380aeb20 2024-03-04T10:09:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.953+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c38074f60 con 0x7f5c380aeb20 2024-03-04T10:09:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.953+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5c28046070 con 0x7f5c380aeb20 2024-03-04T10:09:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.954+0000 7f5c2effd640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c1c06b500 0x7f5c1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.954+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5c280ac3c0 con 0x7f5c380aeb20 2024-03-04T10:09:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.954+0000 7f5c3e3ae640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c1c06b500 0x7f5c1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.955+0000 7f5c3e3ae640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c1c06b500 0x7f5c1c06d9c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5c34000f60 tx=0x7f5c34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:55.958+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5c28030070 con 0x7f5c380aeb20 2024-03-04T10:09:56.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.165+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c38072460 con 0x7f5c380aeb20 2024-03-04T10:09:56.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.165+0000 7f5c2effd640 1 -- 172.21.15.99:0/2678127996 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4948 (secure 0 0 0) 0x7f5c2807b210 con 0x7f5c380aeb20 2024-03-04T10:09:56.167 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:56.167 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:09:15.461841+0000","modified":"2024-03-04T10:09:48.638042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7134},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7134":{"gid":7134,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":12,"addr":"172.21.15.99:6834/3121450352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":3121450352},{"type":"v1","addr":"172.21.15.99:6834","nonce":3121450352}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-04T10:09:56.167 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 71 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c1c06b500 msgr2=0x7f5c1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c1c06b500 0x7f5c1c06d9c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5c34000f60 tx=0x7f5c34009040 comp rx=0 tx=0).stop 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 msgr2=0x7f5c38151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38151850 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5c280098c0 tx=0x7f5c280046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 shutdown_connections 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5c1c06b500 0x7f5c1c06d9c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3815d660 0x7f5c3814a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5c380aeb20 0x7f5c38151850 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 --2- 172.21.15.99:0/2678127996 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5c38071890 0x7f5c3814bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 >> 172.21.15.99:0/2678127996 conn(0x7f5c3806acd0 msgr2=0x7f5c3806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 shutdown_connections 2024-03-04T10:09:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.167+0000 7f5c40612640 1 -- 172.21.15.99:0/2678127996 wait complete. 2024-03-04T10:09:56.179 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:09:56.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.278+0000 7f5e782fc640 1 Processor -- start 2024-03-04T10:09:56.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.278+0000 7f5e782fc640 1 -- start start 2024-03-04T10:09:56.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e782fc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5e7006c090 0x7f5e7006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e782fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5e70066de0 0x7f5e7015f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e782fc640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700ad6c0 con 0x7f5e7006c470 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e782fc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e700ad960 con 0x7f5e70066de0 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e782fc640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7006ebc0 con 0x7f5e7006c090 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5e70066de0 0x7f5e7015f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f5e70066de0 0x7f5e7015f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50644/0 (socket says 172.21.15.99:50644) 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 -- 172.21.15.99:0/1826390435 learned_addr learned my addr 172.21.15.99:0/1826390435 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 -- 172.21.15.99:0/1826390435 >> v2:172.21.15.162:3300/0 conn(0x7f5e7006c090 msgr2=0x7f5e7006f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76098640 1 --2- 172.21.15.99:0/1826390435 >> v2:172.21.15.162:3300/0 conn(0x7f5e7006c090 0x7f5e7006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 --2- 172.21.15.99:0/1826390435 >> v2:172.21.15.162:3300/0 conn(0x7f5e7006c090 0x7f5e7006f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 -- 172.21.15.99:0/1826390435 >> v1:172.21.15.162:6789/0 conn(0x7f5e7006c470 legacy=0x7f5e700af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76899640 1 -- 172.21.15.99:0/1826390435 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7006eea0 con 0x7f5e70066de0 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.279+0000 7f5e76098640 1 --2- 172.21.15.99:0/1826390435 >> v2:172.21.15.162:3300/0 conn(0x7f5e7006c090 0x7f5e7006f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.280+0000 7f5e76899640 1 --2- 172.21.15.99:0/1826390435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e70066de0 0x7f5e7015f560 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5e700ae7d0 tx=0x7f5e6802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd78b491094e3b0c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.280+0000 7f5e75096640 1 -- 172.21.15.99:0/1826390435 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6802c9e0 con 0x7f5e70066de0 2024-03-04T10:09:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.280+0000 7f5e75096640 1 -- 172.21.15.99:0/1826390435 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6802cb80 con 0x7f5e70066de0 2024-03-04T10:09:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.280+0000 7f5e75096640 1 -- 172.21.15.99:0/1826390435 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680343f0 con 0x7f5e70066de0 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 -- 172.21.15.99:0/1826390435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e70066de0 msgr2=0x7f5e7015f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 --2- 172.21.15.99:0/1826390435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e70066de0 0x7f5e7015f560 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f5e700ae7d0 tx=0x7f5e6802be40 comp rx=0 tx=0).stop 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 -- 172.21.15.99:0/1826390435 shutdown_connections 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 --2- 172.21.15.99:0/1826390435 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e70066de0 0x7f5e7015f560 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 --2- 172.21.15.99:0/1826390435 >> v2:172.21.15.162:3300/0 conn(0x7f5e7006c090 0x7f5e7006f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 -- 172.21.15.99:0/1826390435 >> 172.21.15.99:0/1826390435 conn(0x7f5e700cfef0 msgr2=0x7f5e700d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 -- 172.21.15.99:0/1826390435 shutdown_connections 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.281+0000 7f5e782fc640 1 -- 172.21.15.99:0/1826390435 wait complete. 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.282+0000 7f5e782fc640 1 Processor -- start 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.282+0000 7f5e782fc640 1 -- start start 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.282+0000 7f5e782fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e70066de0 0x7f5e70149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.282+0000 7f5e782fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e7006c090 0x7f5e7009b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e782fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 0x7f5e70156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e782fc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700addc0 con 0x7f5e701f4ff0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e782fc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700adc40 con 0x7f5e70066de0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e782fc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7015faa0 con 0x7f5e7006c090 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 0x7f5e70156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 0x7f5e70156fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50654/0 (socket says 172.21.15.99:50654) 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 -- 172.21.15.99:0/3823472978 learned_addr learned my addr 172.21.15.99:0/3823472978 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 -- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e7006c090 msgr2=0x7f5e7009b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e7006c090 0x7f5e7009b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 -- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e70066de0 msgr2=0x7f5e70149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76098640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e70066de0 0x7f5e70149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e70066de0 0x7f5e70149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 -- 172.21.15.99:0/3823472978 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e701f5480 con 0x7f5e701f4ff0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e76899640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 0x7f5e70156fb0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f5e68036040 tx=0x7f5e68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68043020 con 0x7f5e701f4ff0 2024-03-04T10:09:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6803e450 con 0x7f5e701f4ff0 2024-03-04T10:09:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e701f5760 con 0x7f5e701f4ff0 2024-03-04T10:09:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6803e860 con 0x7f5e701f4ff0 2024-03-04T10:09:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.283+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e701f5d20 con 0x7f5e701f4ff0 2024-03-04T10:09:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.284+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70074f60 con 0x7f5e701f4ff0 2024-03-04T10:09:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.285+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5e68046020 con 0x7f5e701f4ff0 2024-03-04T10:09:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.285+0000 7f5e66ffd640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e5406b4d0 0x7f5e5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.285+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5e680accc0 con 0x7f5e701f4ff0 2024-03-04T10:09:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.285+0000 7f5e76098640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e5406b4d0 0x7f5e5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.286+0000 7f5e76098640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e5406b4d0 0x7f5e5406d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c008850 tx=0x7f5e6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.289+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5e6804d030 con 0x7f5e701f4ff0 2024-03-04T10:09:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.458+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e7015cd90 con 0x7f5e701f4ff0 2024-03-04T10:09:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.459+0000 7f5e66ffd640 1 -- 172.21.15.99:0/3823472978 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+17038 (secure 0 0 0) 0x7f5e6807b9f0 con 0x7f5e701f4ff0 2024-03-04T10:09:56.461 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:09:56.461 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":82,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:46.616184+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:09:13.676364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-04T10:09:14.431767+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3857383686":"2024-03-05T10:09:46.616133+0000","172.21.15.99:6833/3857383686":"2024-03-05T10:09:46.616133+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e5406b4d0 msgr2=0x7f5e5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e5406b4d0 0x7f5e5406d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c008850 tx=0x7f5e6c008040 comp rx=0 tx=0).stop 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 msgr2=0x7f5e70156fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 0x7f5e70156fb0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f5e68036040 tx=0x7f5e68004a40 comp rx=0 tx=0).stop 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 shutdown_connections 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5e5406b4d0 0x7f5e5406d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5e701f4ff0 0x7f5e70156fb0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5e7006c090 0x7f5e7009b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 --2- 172.21.15.99:0/3823472978 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e70066de0 0x7f5e70149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 >> 172.21.15.99:0/3823472978 conn(0x7f5e700cfef0 msgr2=0x7f5e70073830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 shutdown_connections 2024-03-04T10:09:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.461+0000 7f5e782fc640 1 -- 172.21.15.99:0/3823472978 wait complete. 2024-03-04T10:09:56.474 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:09:56.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.573+0000 7fdf5c89a640 1 Processor -- start 2024-03-04T10:09:56.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.573+0000 7fdf5c89a640 1 -- start start 2024-03-04T10:09:56.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf5c89a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 0x7fdf5815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf5c89a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5806d8b0 con 0x7fdf5815b410 2024-03-04T10:09:56.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf5c89a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5806db90 con 0x7fdf5806d3b0 2024-03-04T10:09:56.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf5c89a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf580b2720 con 0x7fdf5815b030 2024-03-04T10:09:56.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf56d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdf5815b410 0x7fdf5815b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40516/0 (socket says 172.21.15.99:40516) 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf56d9d640 1 -- 172.21.15.99:0/2661448736 learned_addr learned my addr 172.21.15.99:0/2661448736 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989816699 0 0) 0x7fdf5806d8b0 con 0x7fdf5815b410 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf5659c640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 0x7fdf5815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.574+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf44003520 con 0x7fdf5815b410 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3681576851 0 0) 0x7fdf44003520 con 0x7fdf5815b410 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 msgr2=0x7fdf5815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 0x7fdf5815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 >> v1:172.21.15.162:6789/0 conn(0x7fdf5806d3b0 legacy=0x7fdf5815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5806de70 con 0x7fdf5815b410 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fdf500019f0 con 0x7fdf5815b410 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 >> v1:172.21.15.99:6789/0 conn(0x7fdf5815b410 legacy=0x7fdf5815b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf440039e0 0x7fdf44003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf44004800 0x7fdf44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 0x7fdf44005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf580b2720 con 0x7fdf440039e0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4400dd00 con 0x7fdf44004800 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4400df00 con 0x7fdf5815b030 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf5659c640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf440039e0 0x7fdf44003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf4ffff640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf44004800 0x7fdf44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf56d9d640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 0x7fdf44005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf5659c640 1 -- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 msgr2=0x7fdf44005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf5659c640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf5815b030 0x7fdf44005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf5659c640 1 -- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf44004800 msgr2=0x7fdf44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf5659c640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf44004800 0x7fdf44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.575+0000 7fdf5659c640 1 -- 172.21.15.99:0/2661448736 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44003520 con 0x7fdf440039e0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.576+0000 7fdf5659c640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf440039e0 0x7fdf44003e00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009ac0 tx=0x7fdf44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84fd68141d84a5bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.576+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48039070 con 0x7fdf440039e0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.576+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4802c6b0 con 0x7fdf440039e0 2024-03-04T10:09:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.576+0000 7fdf55d9b640 1 -- 172.21.15.99:0/2661448736 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48009690 con 0x7fdf440039e0 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf5c89a640 1 -- 172.21.15.99:0/2661448736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf440039e0 msgr2=0x7fdf44003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/2661448736 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf440039e0 0x7fdf44003e00 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009ac0 tx=0x7fdf44005830 comp rx=0 tx=0).stop 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf56d9d640 1 -- 172.21.15.99:0/2661448736 reap_dead start 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf5c89a640 1 -- 172.21.15.99:0/2661448736 shutdown_connections 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf5c89a640 1 -- 172.21.15.99:0/2661448736 >> 172.21.15.99:0/2661448736 conn(0x7fdf5806acd0 msgr2=0x7fdf5806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf5c89a640 1 -- 172.21.15.99:0/2661448736 shutdown_connections 2024-03-04T10:09:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.577+0000 7fdf5c89a640 1 -- 172.21.15.99:0/2661448736 wait complete. 2024-03-04T10:09:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.578+0000 7fdf5c89a640 1 Processor -- start 2024-03-04T10:09:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.578+0000 7fdf5c89a640 1 -- start start 2024-03-04T10:09:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf58144af0 0x7fdf58148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf58144ed0 0x7fdf580ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 0x7fdf580a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5806e810 con 0x7fdf580ced80 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5806e690 con 0x7fdf58144ed0 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5815c120 con 0x7fdf58144af0 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 0x7fdf580a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 0x7fdf580a8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50664/0 (socket says 172.21.15.99:50664) 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 -- 172.21.15.99:0/294660529 learned_addr learned my addr 172.21.15.99:0/294660529 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf4ffff640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf58144ed0 0x7fdf580ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5659c640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf58144af0 0x7fdf58148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 -- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf58144af0 msgr2=0x7fdf58148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf58144af0 0x7fdf58148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 -- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf58144ed0 msgr2=0x7fdf580ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf58144ed0 0x7fdf580ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 -- 172.21.15.99:0/294660529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58145460 con 0x7fdf580ced80 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf4ffff640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf58144ed0 0x7fdf580ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf56d9d640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 0x7fdf580a8460 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fdf44004780 tx=0x7fdf5000e220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf5000ec30 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5000ee10 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf58145700 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50013420 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.579+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf58145cc0 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.580+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf58074f60 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.581+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdf5001a020 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.581+0000 7fdf4effd640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b500 0x7fdf3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.581+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdf5008a0b0 con 0x7fdf580ced80 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.581+0000 7fdf5659c640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b500 0x7fdf3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.582+0000 7fdf5659c640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b500 0x7fdf3806d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdf48036000 tx=0x7fdf48030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.585+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdf50058de0 con 0x7fdf580ced80 2024-03-04T10:09:56.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.796+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdf58159f60 con 0x7fdf580ced80 2024-03-04T10:09:56.797 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:09:56.796+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:09:56.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.799+0000 7fdf4effd640 1 -- 172.21.15.99:0/294660529 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdf500586e0 con 0x7fdf580ced80 2024-03-04T10:09:56.801 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:09:56.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b500 msgr2=0x7fdf3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b500 0x7fdf3806d9c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdf48036000 tx=0x7fdf48030000 comp rx=0 tx=0).stop 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 msgr2=0x7fdf580a8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 0x7fdf580a8460 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fdf44004780 tx=0x7fdf5000e220 comp rx=0 tx=0).stop 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 shutdown_connections 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf3806b500 0x7fdf3806d9c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf580ced80 0x7fdf580a8460 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf58144ed0 0x7fdf580ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 --2- 172.21.15.99:0/294660529 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf58144af0 0x7fdf58148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 >> 172.21.15.99:0/294660529 conn(0x7fdf5806acd0 msgr2=0x7fdf581591d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 shutdown_connections 2024-03-04T10:09:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.802+0000 7fdf5c89a640 1 -- 172.21.15.99:0/294660529 wait complete. 2024-03-04T10:09:56.808 INFO:tasks.ceph.mds.c.smithi099.stderr: -259> 2024-03-04T10:09:52.100+0000 7f07d4dfd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:09:56.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T10:09:56.831 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:09:56.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 Processor -- start 2024-03-04T10:09:56.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 -- start start 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1f3415a810 0x7f1f340b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3406d8a0 con 0x7f1f340cbfb0 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3406db80 con 0x7f1f3415a810 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.917+0000 7f1f39569640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3406de60 con 0x7f1f3406d3c0 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1f3415a810 0x7f1f340b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1f3415a810 0x7f1f340b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50686/0 (socket says 172.21.15.99:50686) 2024-03-04T10:09:56.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 -- 172.21.15.99:0/3104098297 learned_addr learned my addr 172.21.15.99:0/3104098297 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f327fc640 1 --2- 172.21.15.99:0/3104098297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 -- 172.21.15.99:0/3104098297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 msgr2=0x7f1f340b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 --2- 172.21.15.99:0/3104098297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 -- 172.21.15.99:0/3104098297 >> v1:172.21.15.162:6789/0 conn(0x7f1f340cbfb0 legacy=0x7f1f3415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.918+0000 7f1f32ffd640 1 -- 172.21.15.99:0/3104098297 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3406e140 con 0x7f1f3415a810 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f32ffd640 1 --2- 172.21.15.99:0/3104098297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f3415a810 0x7f1f340b4190 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400d150 tx=0x7f1f2402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da9b42aea417c1a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f31ffb640 1 -- 172.21.15.99:0/3104098297 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2402ea80 con 0x7f1f3415a810 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f31ffb640 1 -- 172.21.15.99:0/3104098297 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2402ec20 con 0x7f1f3415a810 2024-03-04T10:09:56.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f31ffb640 1 -- 172.21.15.99:0/3104098297 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24009600 con 0x7f1f3415a810 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 -- 172.21.15.99:0/3104098297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f3415a810 msgr2=0x7f1f340b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 --2- 172.21.15.99:0/3104098297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f3415a810 0x7f1f340b4190 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400d150 tx=0x7f1f2402e0e0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 -- 172.21.15.99:0/3104098297 shutdown_connections 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 --2- 172.21.15.99:0/3104098297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 --2- 172.21.15.99:0/3104098297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f3415a810 0x7f1f340b4190 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 -- 172.21.15.99:0/3104098297 >> 172.21.15.99:0/3104098297 conn(0x7f1f3406acd0 msgr2=0x7f1f3406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:56.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 -- 172.21.15.99:0/3104098297 shutdown_connections 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.919+0000 7f1f39569640 1 -- 172.21.15.99:0/3104098297 wait complete. 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 Processor -- start 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 -- start start 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 0x7f1f3415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f3415a810 0x7f1f341417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3415e3c0 con 0x7f1f340cbfb0 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340b3c10 con 0x7f1f3415a810 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f39569640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3415e540 con 0x7f1f3406d3c0 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f327fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 0x7f1f3415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f327fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 0x7f1f3415ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50700/0 (socket says 172.21.15.99:50700) 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.921+0000 7f1f327fc640 1 -- 172.21.15.99:0/1160718544 learned_addr learned my addr 172.21.15.99:0/1160718544 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:56.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f337fe640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f3415a810 0x7f1f341417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f32ffd640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f327fc640 1 -- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 msgr2=0x7f1f340ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f327fc640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f327fc640 1 -- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f3415a810 msgr2=0x7f1f341417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f327fc640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f3415a810 0x7f1f341417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f327fc640 1 -- 172.21.15.99:0/1160718544 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f341f3d20 con 0x7f1f340cbfb0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f327fc640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 0x7f1f3415ae60 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800d800 tx=0x7f1f2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2800b660 con 0x7f1f340cbfb0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f28019070 con 0x7f1f340cbfb0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f340ae120 con 0x7f1f340cbfb0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2800ba20 con 0x7f1f340cbfb0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.922+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f340ad080 con 0x7f1f340cbfb0 2024-03-04T10:09:56.926 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.923+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f34074f60 con 0x7f1f340cbfb0 2024-03-04T10:09:56.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.923+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1f2800bbc0 con 0x7f1f340cbfb0 2024-03-04T10:09:56.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.924+0000 7f1f1b7fe640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1f2006b500 0x7f1f2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:56.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.924+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f1f2808a8a0 con 0x7f1f340cbfb0 2024-03-04T10:09:56.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.924+0000 7f1f32ffd640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1f2006b500 0x7f1f2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:56.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.924+0000 7f1f32ffd640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1f2006b500 0x7f1f2006d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1f24037000 tx=0x7f1f24032000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:56.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:56.928+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1f2801c360 con 0x7f1f340cbfb0 2024-03-04T10:09:57.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.133+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1f3406ec30 con 0x7f1f340cbfb0 2024-03-04T10:09:57.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.808+0000 7f1f1b7fe640 1 -- 172.21.15.99:0/1160718544 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v73) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1f280594e0 con 0x7f1f340cbfb0 2024-03-04T10:09:57.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1f2006b500 msgr2=0x7f1f2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:57.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1f2006b500 0x7f1f2006d9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1f24037000 tx=0x7f1f24032000 comp rx=0 tx=0).stop 2024-03-04T10:09:57.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 msgr2=0x7f1f3415ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:57.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 0x7f1f3415ae60 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800d800 tx=0x7f1f2800dcc0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 shutdown_connections 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1f2006b500 0x7f1f2006d9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f3415a810 0x7f1f341417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1f340cbfb0 0x7f1f3415ae60 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 --2- 172.21.15.99:0/1160718544 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1f3406d3c0 0x7f1f340ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.811+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 >> 172.21.15.99:0/1160718544 conn(0x7f1f3406acd0 msgr2=0x7f1f34071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.812+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 shutdown_connections 2024-03-04T10:09:57.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.812+0000 7f1f39569640 1 -- 172.21.15.99:0/1160718544 wait complete. 2024-03-04T10:09:57.826 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T10:09:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.972+0000 7f33d3e97640 1 Processor -- start 2024-03-04T10:09:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.972+0000 7f33d3e97640 1 -- start start 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d3e97640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f33cc15f1f0 0x7f33cc0b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d3e97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc0b1190 0x7f33cc0b1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d3e97640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b4270 con 0x7f33cc099a70 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d3e97640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc0b2a10 con 0x7f33cc15f1f0 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d3e97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0d07b0 con 0x7f33cc0b1190 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d1c33640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f33cc15f1f0 0x7f33cc0b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d1432640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc0b1190 0x7f33cc0b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d2434640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f33cc099a70 0x7f33cc099ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:37194/0 (socket says 172.21.15.99:37194) 2024-03-04T10:09:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.973+0000 7f33d2434640 1 -- 172.21.15.99:0/1025350865 learned_addr learned my addr 172.21.15.99:0/1025350865 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d0c31640 1 -- 172.21.15.99:0/1025350865 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168256979 0 0) 0x7f33c4001490 con 0x7f33cc099a70 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d0c31640 1 -- 172.21.15.99:0/1025350865 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b8003520 con 0x7f33cc099a70 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d1432640 1 -- 172.21.15.99:0/1025350865 >> v2:172.21.15.162:3300/0 conn(0x7f33cc15f1f0 msgr2=0x7f33cc0b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d1432640 1 --2- 172.21.15.99:0/1025350865 >> v2:172.21.15.162:3300/0 conn(0x7f33cc15f1f0 0x7f33cc0b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d1432640 1 -- 172.21.15.99:0/1025350865 >> v1:172.21.15.162:6789/0 conn(0x7f33cc099a70 legacy=0x7f33cc099ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d1432640 1 -- 172.21.15.99:0/1025350865 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0b1f10 con 0x7f33cc0b1190 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d1c33640 1 --2- 172.21.15.99:0/1025350865 >> v2:172.21.15.162:3300/0 conn(0x7f33cc15f1f0 0x7f33cc0b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.974+0000 7f33d1432640 1 --2- 172.21.15.99:0/1025350865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc0b1190 0x7f33cc0b1610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0097a0 tx=0x7f33bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4668be4973a7ec91 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d0c31640 1 -- 172.21.15.99:0/1025350865 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc035020 con 0x7f33cc0b1190 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d0c31640 1 -- 172.21.15.99:0/1025350865 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc031830 con 0x7f33cc0b1190 2024-03-04T10:09:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d0c31640 1 -- 172.21.15.99:0/1025350865 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc031c00 con 0x7f33cc0b1190 2024-03-04T10:09:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 -- 172.21.15.99:0/1025350865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc0b1190 msgr2=0x7f33cc0b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 --2- 172.21.15.99:0/1025350865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc0b1190 0x7f33cc0b1610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0097a0 tx=0x7f33bc02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 -- 172.21.15.99:0/1025350865 shutdown_connections 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 --2- 172.21.15.99:0/1025350865 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc0b1190 0x7f33cc0b1610 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 --2- 172.21.15.99:0/1025350865 >> v2:172.21.15.162:3300/0 conn(0x7f33cc15f1f0 0x7f33cc0b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 -- 172.21.15.99:0/1025350865 >> 172.21.15.99:0/1025350865 conn(0x7f33cc0b3320 msgr2=0x7f33cc0b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 -- 172.21.15.99:0/1025350865 shutdown_connections 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.975+0000 7f33d3e97640 1 -- 172.21.15.99:0/1025350865 wait complete. 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.976+0000 7f33d3e97640 1 Processor -- start 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.976+0000 7f33d3e97640 1 -- start start 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d3e97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33cc099a70 0x7f33cc155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d3e97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 0x7f33cc15b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d3e97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc15f1f0 0x7f33cc154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d3e97640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0d0290 con 0x7f33cc0b1190 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d3e97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0cffb0 con 0x7f33cc099a70 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d3e97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0b1b50 con 0x7f33cc15f1f0 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d1432640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 0x7f33cc15b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d1432640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 0x7f33cc15b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50712/0 (socket says 172.21.15.99:50712) 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d1432640 1 -- 172.21.15.99:0/3940690794 learned_addr learned my addr 172.21.15.99:0/3940690794 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d1c33640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33cc099a70 0x7f33cc155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.977+0000 7f33d2434640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc15f1f0 0x7f33cc154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d1432640 1 -- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc15f1f0 msgr2=0x7f33cc154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d1432640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc15f1f0 0x7f33cc154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d1432640 1 -- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33cc099a70 msgr2=0x7f33cc155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d1432640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33cc099a70 0x7f33cc155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d1432640 1 -- 172.21.15.99:0/3940690794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc15bf00 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d2434640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc15f1f0 0x7f33cc154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d1432640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 0x7f33cc15b9c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f33bc02bee0 tx=0x7f33bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc035040 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc03b920 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc158f20 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc03bd10 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.978+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc1594e0 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.979+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc074f60 con 0x7f33cc0b1190 2024-03-04T10:09:57.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.979+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f33bc038020 con 0x7f33cc0b1190 2024-03-04T10:09:57.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.980+0000 7f33c27fc640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33b006b4d0 0x7f33b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:57.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.980+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f33bc0ac4c0 con 0x7f33cc0b1190 2024-03-04T10:09:57.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.980+0000 7f33d1c33640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33b006b4d0 0x7f33b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:57.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.981+0000 7f33d1c33640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33b006b4d0 0x7f33b006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0cdbd0 tx=0x7f33c8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:57.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:57.984+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33bc07b190 con 0x7f33cc0b1190 2024-03-04T10:09:58.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.154+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f33cc0d0f00 con 0x7f33cc0b1190 2024-03-04T10:09:58.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.811+0000 7f33c27fc640 1 -- 172.21.15.99:0/3940690794 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v84) v1 ==== 181+0+0 (secure 0 0 0) 0x7f33bc07aa90 con 0x7f33cc0b1190 2024-03-04T10:09:58.813 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' removed 2024-03-04T10:09:58.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.813+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33b006b4d0 msgr2=0x7f33b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:58.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33b006b4d0 0x7f33b006d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0cdbd0 tx=0x7f33c8008040 comp rx=0 tx=0).stop 2024-03-04T10:09:58.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 msgr2=0x7f33cc15b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:58.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 0x7f33cc15b9c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f33bc02bee0 tx=0x7f33bc0046c0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 shutdown_connections 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f33b006b4d0 0x7f33b006d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33cc15f1f0 0x7f33cc154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33cc0b1190 0x7f33cc15b9c0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 --2- 172.21.15.99:0/3940690794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33cc099a70 0x7f33cc155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 >> 172.21.15.99:0/3940690794 conn(0x7f33cc0b3320 msgr2=0x7f33cc0cee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 shutdown_connections 2024-03-04T10:09:58.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.814+0000 7f33d3e97640 1 -- 172.21.15.99:0/3940690794 wait complete. 2024-03-04T10:09:58.828 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T10:09:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 Processor -- start 2024-03-04T10:09:58.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 -- start start 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f86040cfe20 0x7f860409a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 0x7f86040cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040cd360 con 0x7f86040d0200 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860409aea0 con 0x7f86040cfe20 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.931+0000 7f860aba9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860415ca80 con 0x7f860415bf10 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8609146640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f86040d0200 0x7f860415f720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40526/0 (socket says 172.21.15.99:40526) 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8609146640 1 -- 172.21.15.99:0/2910647551 learned_addr learned my addr 172.21.15.99:0/2910647551 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8608945640 1 --2- 172.21.15.99:0/2910647551 >> v2:172.21.15.162:3300/0 conn(0x7f86040cfe20 0x7f860409a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449911771 0 0) 0x7f86040cd360 con 0x7f86040d0200 2024-03-04T10:09:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8603fff640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 0x7f86040cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f4003520 con 0x7f86040d0200 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772894587 0 0) 0x7f85f4003520 con 0x7f86040d0200 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 msgr2=0x7f86040cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 0x7f86040cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 >> v2:172.21.15.162:3300/0 conn(0x7f86040cfe20 msgr2=0x7f860409a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 --2- 172.21.15.99:0/2910647551 >> v2:172.21.15.162:3300/0 conn(0x7f86040cfe20 0x7f860409a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040cfaa0 con 0x7f86040d0200 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f85fc0019f0 con 0x7f86040d0200 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 >> v1:172.21.15.99:6789/0 conn(0x7f86040d0200 legacy=0x7f860415f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f40039b0 0x7f85f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f4004800 0x7f85f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 0x7f85f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860415ca80 con 0x7f85f40039b0 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f400b9c0 con 0x7f85f4004800 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f400bbc0 con 0x7f860415bf10 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8608945640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f40039b0 0x7f85f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8603fff640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f4004800 0x7f85f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8609146640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 0x7f85f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8608945640 1 -- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 msgr2=0x7f85f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8608945640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f860415bf10 0x7f85f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8608945640 1 -- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f4004800 msgr2=0x7f85f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.932+0000 7f8608945640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f4004800 0x7f85f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.933+0000 7f8608945640 1 -- 172.21.15.99:0/2910647551 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4003520 con 0x7f85f40039b0 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.933+0000 7f8603fff640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f4004800 0x7f85f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.933+0000 7f8608945640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f40039b0 0x7f85f4003dd0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009ac0 tx=0x7f85f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3f5e04d56f1455 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.933+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8039070 con 0x7f85f40039b0 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.933+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f802c710 con 0x7f85f40039b0 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.933+0000 7f86037fe640 1 -- 172.21.15.99:0/2910647551 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8009690 con 0x7f85f40039b0 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f860aba9640 1 -- 172.21.15.99:0/2910647551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f40039b0 msgr2=0x7f85f4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f860aba9640 1 --2- 172.21.15.99:0/2910647551 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85f40039b0 0x7f85f4003dd0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009ac0 tx=0x7f85f4005830 comp rx=0 tx=0).stop 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f8609146640 1 -- 172.21.15.99:0/2910647551 reap_dead start 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f860aba9640 1 -- 172.21.15.99:0/2910647551 shutdown_connections 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f860aba9640 1 -- 172.21.15.99:0/2910647551 >> 172.21.15.99:0/2910647551 conn(0x7f86040d1840 msgr2=0x7f86040d1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f860aba9640 1 -- 172.21.15.99:0/2910647551 shutdown_connections 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.934+0000 7f860aba9640 1 -- 172.21.15.99:0/2910647551 wait complete. 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.935+0000 7f860aba9640 1 Processor -- start 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.935+0000 7f860aba9640 1 -- start start 2024-03-04T10:09:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f860aba9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f40039b0 0x7f8604154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f860aba9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86041555c0 0x7f8604155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f860aba9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 0x7f86040c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f860aba9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040d06b0 con 0x7f86040c9960 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f860aba9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860415c710 con 0x7f85f40039b0 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f860aba9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860415cf10 con 0x7f86041555c0 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 0x7f86040c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 0x7f86040c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50724/0 (socket says 172.21.15.99:50724) 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 -- 172.21.15.99:0/650093640 learned_addr learned my addr 172.21.15.99:0/650093640 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8608945640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f40039b0 0x7f8604154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8603fff640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86041555c0 0x7f8604155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 -- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86041555c0 msgr2=0x7f8604155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86041555c0 0x7f8604155a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 -- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f40039b0 msgr2=0x7f8604154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f40039b0 0x7f8604154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 -- 172.21.15.99:0/650093640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040ca570 con 0x7f86040c9960 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.936+0000 7f8609146640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 0x7f86040c9de0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f85fc001fe0 tx=0x7f85fc00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.937+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00eb90 con 0x7f86040c9960 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.937+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85fc018070 con 0x7f86040c9960 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.937+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86040ca850 con 0x7f86040c9960 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.937+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc013420 con 0x7f86040c9960 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.937+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86040cae10 con 0x7f86040c9960 2024-03-04T10:09:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.938+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8604157850 con 0x7f86040c9960 2024-03-04T10:09:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.938+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f85fc0135c0 con 0x7f86040c9960 2024-03-04T10:09:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.938+0000 7f86017fa640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85e806b500 0x7f85e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.938+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f85fc08a570 con 0x7f86040c9960 2024-03-04T10:09:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.939+0000 7f8608945640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85e806b500 0x7f85e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.939+0000 7f8608945640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85e806b500 0x7f85e806d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009ac0 tx=0x7f85f8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:58.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:58.943+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f85fc059370 con 0x7f86040c9960 2024-03-04T10:09:59.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.111+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8604074f60 con 0x7f86040c9960 2024-03-04T10:09:59.817 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.816+0000 7f86017fa640 1 -- 172.21.15.99:0/650093640 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f85fc058c70 con 0x7f86040c9960 2024-03-04T10:09:59.818 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' removed 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.819+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85e806b500 msgr2=0x7f85e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.819+0000 7f860aba9640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85e806b500 0x7f85e806d9c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009ac0 tx=0x7f85f8030000 comp rx=0 tx=0).stop 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.819+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 msgr2=0x7f86040c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.819+0000 7f860aba9640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 0x7f86040c9de0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f85fc001fe0 tx=0x7f85fc00e530 comp rx=0 tx=0).stop 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 shutdown_connections 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f85e806b500 0x7f85e806d9c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f86040c9960 0x7f86040c9de0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f86041555c0 0x7f8604155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 --2- 172.21.15.99:0/650093640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85f40039b0 0x7f8604154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 >> 172.21.15.99:0/650093640 conn(0x7f86040d1840 msgr2=0x7f8604099ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:59.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.820+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 shutdown_connections 2024-03-04T10:09:59.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.821+0000 7f860aba9640 1 -- 172.21.15.99:0/650093640 wait complete. 2024-03-04T10:09:59.837 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T10:09:59.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.945+0000 7f6e684f1640 1 Processor -- start 2024-03-04T10:09:59.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.946+0000 7f6e684f1640 1 -- start start 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e684f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 0x7f6e6006d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e684f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f6e600b3340 0x7f6e60071b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e684f1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e600720c0 con 0x7f6e6006efa0 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e684f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e600b2c70 con 0x7f6e600b3340 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e684f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60147890 con 0x7f6e600717a0 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e66a8e640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6e6006efa0 0x7f6e600ace30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40542/0 (socket says 172.21.15.99:40542) 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e66a8e640 1 -- 172.21.15.99:0/1036111764 learned_addr learned my addr 172.21.15.99:0/1036111764 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124219086 0 0) 0x7f6e600720c0 con 0x7f6e6006efa0 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c003520 con 0x7f6e6006efa0 2024-03-04T10:09:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e65a8c640 1 --2- 172.21.15.99:0/1036111764 >> v2:172.21.15.162:3300/0 conn(0x7f6e600b3340 0x7f6e60071b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6628d640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 0x7f6e6006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 333705330 0 0) 0x7f6e4c003520 con 0x7f6e6006efa0 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 msgr2=0x7f6e6006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 0x7f6e6006d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 >> v2:172.21.15.162:3300/0 conn(0x7f6e600b3340 msgr2=0x7f6e60071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 --2- 172.21.15.99:0/1036111764 >> v2:172.21.15.162:3300/0 conn(0x7f6e600b3340 0x7f6e60071b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6006df00 con 0x7f6e6006efa0 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f6e580019f0 con 0x7f6e6006efa0 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 >> v1:172.21.15.99:6789/0 conn(0x7f6e6006efa0 legacy=0x7f6e600ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0039b0 0x7f6e4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 0x7f6e4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c004d40 0x7f6e4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e600b2c70 con 0x7f6e4c0039b0 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60147890 con 0x7f6e4c004d40 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c00b9c0 con 0x7f6e600717a0 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.947+0000 7f6e6628d640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0039b0 0x7f6e4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e65a8c640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 0x7f6e4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e66a8e640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c004d40 0x7f6e4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e6628d640 1 -- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 msgr2=0x7f6e4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e6628d640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e600717a0 0x7f6e4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e6628d640 1 -- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c004d40 msgr2=0x7f6e4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e6628d640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e4c004d40 0x7f6e4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e6628d640 1 -- 172.21.15.99:0/1036111764 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c003520 con 0x7f6e4c0039b0 2024-03-04T10:09:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.948+0000 7f6e6628d640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0039b0 0x7f6e4c003dd0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009ac0 tx=0x7f6e4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9ce705e5940d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c039070 con 0x7f6e4c0039b0 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c02c6b0 con 0x7f6e4c0039b0 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e6528b640 1 -- 172.21.15.99:0/1036111764 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c009690 con 0x7f6e4c0039b0 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e684f1640 1 -- 172.21.15.99:0/1036111764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0039b0 msgr2=0x7f6e4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e684f1640 1 --2- 172.21.15.99:0/1036111764 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e4c0039b0 0x7f6e4c003dd0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009ac0 tx=0x7f6e4c005830 comp rx=0 tx=0).stop 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e66a8e640 1 -- 172.21.15.99:0/1036111764 reap_dead start 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e684f1640 1 -- 172.21.15.99:0/1036111764 shutdown_connections 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e684f1640 1 -- 172.21.15.99:0/1036111764 >> 172.21.15.99:0/1036111764 conn(0x7f6e600cfad0 msgr2=0x7f6e600cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e684f1640 1 -- 172.21.15.99:0/1036111764 shutdown_connections 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.949+0000 7f6e684f1640 1 -- 172.21.15.99:0/1036111764 wait complete. 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.950+0000 7f6e684f1640 1 Processor -- start 2024-03-04T10:09:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.950+0000 7f6e684f1640 1 -- start start 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e684f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0039b0 0x7f6e60138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e684f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e6015faf0 0x7f6e6015ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e684f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 0x7f6e6013b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e684f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6006e7a0 con 0x7f6e6013b100 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e684f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6006e4c0 con 0x7f6e6015faf0 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e684f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6006ea80 con 0x7f6e4c0039b0 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e66a8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 0x7f6e6013b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e66a8e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 0x7f6e6013b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50738/0 (socket says 172.21.15.99:50738) 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e66a8e640 1 -- 172.21.15.99:0/3101857682 learned_addr learned my addr 172.21.15.99:0/3101857682 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e6628d640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0039b0 0x7f6e60138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e65a8c640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e6015faf0 0x7f6e6015ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e66a8e640 1 -- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0039b0 msgr2=0x7f6e60138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e66a8e640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0039b0 0x7f6e60138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.951+0000 7f6e66a8e640 1 -- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e6015faf0 msgr2=0x7f6e6015ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e66a8e640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e6015faf0 0x7f6e6015ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e66a8e640 1 -- 172.21.15.99:0/3101857682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6013bdf0 con 0x7f6e6013b100 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e6628d640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0039b0 0x7f6e60138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e66a8e640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 0x7f6e6013b580 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f6e60071750 tx=0x7f6e58007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e58017020 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5800ab80 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e580133f0 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e60151cf0 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.952+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6013c0d0 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.953+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6015dad0 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.953+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6e5801a020 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.954+0000 7f6e56ffd640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e4406b4d0 0x7f6e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.954+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f6e5808a870 con 0x7f6e6013b100 2024-03-04T10:09:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.954+0000 7f6e6628d640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e4406b4d0 0x7f6e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:09:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.954+0000 7f6e6628d640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e4406b4d0 0x7f6e4406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c036000 tx=0x7f6e5c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:09:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:09:59.958+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6e58021020 con 0x7f6e6013b100 2024-03-04T10:10:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.128+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6e60074f60 con 0x7f6e6013b100 2024-03-04T10:10:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.131+0000 7f6e56ffd640 1 -- 172.21.15.99:0/3101857682 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6e58059250 con 0x7f6e6013b100 2024-03-04T10:10:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.133+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e4406b4d0 msgr2=0x7f6e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.133+0000 7f6e684f1640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e4406b4d0 0x7f6e4406d990 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c036000 tx=0x7f6e5c030000 comp rx=0 tx=0).stop 2024-03-04T10:10:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.133+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 msgr2=0x7f6e6013b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.133+0000 7f6e684f1640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 0x7f6e6013b580 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f6e60071750 tx=0x7f6e58007b10 comp rx=0 tx=0).stop 2024-03-04T10:10:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 shutdown_connections 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6e4406b4d0 0x7f6e4406d990 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6e6013b100 0x7f6e6013b580 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e6015faf0 0x7f6e6015ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 --2- 172.21.15.99:0/3101857682 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6e4c0039b0 0x7f6e60138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 >> 172.21.15.99:0/3101857682 conn(0x7f6e600cfad0 msgr2=0x7f6e600b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 shutdown_connections 2024-03-04T10:10:00.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.134+0000 7f6e684f1640 1 -- 172.21.15.99:0/3101857682 wait complete. 2024-03-04T10:10:00.146 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_inject_rename_corrupt_dentry_first 2024-03-04T10:10:00.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.253+0000 7ff859812640 1 Processor -- start 2024-03-04T10:10:00.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.253+0000 7ff859812640 1 -- start start 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff859812640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff8540b5700 0x7ff8540d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff859812640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b9410 0x7ff85406c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff859812640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8540d2400 con 0x7ff8540d34a0 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff859812640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8540d26e0 con 0x7ff8540b5700 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff859812640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540d29c0 con 0x7ff8540b9410 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff8540b5700 0x7ff8540d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff8540b5700 0x7ff8540d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50752/0 (socket says 172.21.15.99:50752) 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 -- 172.21.15.99:0/85657457 learned_addr learned my addr 172.21.15.99:0/85657457 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff8537fe640 1 --2- 172.21.15.99:0/85657457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b9410 0x7ff85406c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 -- 172.21.15.99:0/85657457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b9410 msgr2=0x7ff85406c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 --2- 172.21.15.99:0/85657457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b9410 0x7ff85406c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 -- 172.21.15.99:0/85657457 >> v1:172.21.15.162:6789/0 conn(0x7ff8540d34a0 legacy=0x7ff85406c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:00.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.254+0000 7ff852ffd640 1 -- 172.21.15.99:0/85657457 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540d2ca0 con 0x7ff8540b5700 2024-03-04T10:10:00.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.255+0000 7ff852ffd640 1 --2- 172.21.15.99:0/85657457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b5700 0x7ff8540d1b00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7ff844009870 tx=0x7ff84402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdec921195249146 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.255+0000 7ff8527fc640 1 -- 172.21.15.99:0/85657457 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8440317c0 con 0x7ff8540b5700 2024-03-04T10:10:00.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.255+0000 7ff8527fc640 1 -- 172.21.15.99:0/85657457 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844037070 con 0x7ff8540b5700 2024-03-04T10:10:00.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.255+0000 7ff8527fc640 1 -- 172.21.15.99:0/85657457 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844031b80 con 0x7ff8540b5700 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 -- 172.21.15.99:0/85657457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b5700 msgr2=0x7ff8540d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 --2- 172.21.15.99:0/85657457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b5700 0x7ff8540d1b00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7ff844009870 tx=0x7ff84402b860 comp rx=0 tx=0).stop 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 -- 172.21.15.99:0/85657457 shutdown_connections 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 --2- 172.21.15.99:0/85657457 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b9410 0x7ff85406c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 --2- 172.21.15.99:0/85657457 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b5700 0x7ff8540d1b00 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 -- 172.21.15.99:0/85657457 >> 172.21.15.99:0/85657457 conn(0x7ff8540d0490 msgr2=0x7ff8540d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 -- 172.21.15.99:0/85657457 shutdown_connections 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.256+0000 7ff859812640 1 -- 172.21.15.99:0/85657457 wait complete. 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.257+0000 7ff859812640 1 Processor -- start 2024-03-04T10:10:00.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.257+0000 7ff859812640 1 -- start start 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff859812640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b5700 0x7ff854158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff859812640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 0x7ff8541f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff859812640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8540d34a0 0x7ff85414dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff859812640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85406ceb0 con 0x7ff8540b9410 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff859812640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85406cd30 con 0x7ff8540d34a0 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff859812640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540b5ae0 con 0x7ff8540b5700 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff84bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 0x7ff8541f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff84bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 0x7ff8541f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50762/0 (socket says 172.21.15.99:50762) 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff84bfff640 1 -- 172.21.15.99:0/3520341787 learned_addr learned my addr 172.21.15.99:0/3520341787 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff8537fe640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8540d34a0 0x7ff85414dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.258+0000 7ff852ffd640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b5700 0x7ff854158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff84bfff640 1 -- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b5700 msgr2=0x7ff854158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff84bfff640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b5700 0x7ff854158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff84bfff640 1 -- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8540d34a0 msgr2=0x7ff85414dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff84bfff640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8540d34a0 0x7ff85414dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff84bfff640 1 -- 172.21.15.99:0/3520341787 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8540cbe60 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff84bfff640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 0x7ff8541f3c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00b3c0 tx=0x7ff83c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c011830 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c016070 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8540cc140 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c011c10 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.259+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85415a730 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.260+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff854074f60 con 0x7ff8540b9410 2024-03-04T10:10:00.263 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.261+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff83c01b460 con 0x7ff8540b9410 2024-03-04T10:10:00.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.261+0000 7ff850ff9640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.261+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7ff83c089ab0 con 0x7ff8540b9410 2024-03-04T10:10:00.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.261+0000 7ff852ffd640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.264 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.262+0000 7ff852ffd640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff844036000 tx=0x7ff84403d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.266+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff83c020020 con 0x7ff8540b9410 2024-03-04T10:10:00.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.446+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"} v 0) v1 -- 0x7ff8540cf650 con 0x7ff8540b9410 2024-03-04T10:10:00.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.447+0000 7ff850ff9640 1 -- 172.21.15.99:0/3520341787 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_inject_rename_corrupt_dentry_first"}]=0 v25) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff83c058370 con 0x7ff8540b9410 2024-03-04T10:10:00.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff83406b4d0 msgr2=0x7ff83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff844036000 tx=0x7ff84403d040 comp rx=0 tx=0).stop 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 msgr2=0x7ff8541f3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 0x7ff8541f3c10 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00b3c0 tx=0x7ff83c00b880 comp rx=0 tx=0).stop 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 shutdown_connections 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff8540d34a0 0x7ff85414dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff8540b9410 0x7ff8541f3c10 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 --2- 172.21.15.99:0/3520341787 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff8540b5700 0x7ff854158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 >> 172.21.15.99:0/3520341787 conn(0x7ff8540d0490 msgr2=0x7ff8540b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 shutdown_connections 2024-03-04T10:10:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.449+0000 7ff859812640 1 -- 172.21.15.99:0/3520341787 wait complete. 2024-03-04T10:10:00.461 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush' 2024-03-04T10:10:00.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.561+0000 7fa849612640 1 Processor -- start 2024-03-04T10:10:00.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.561+0000 7fa849612640 1 -- start start 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa849612640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa849612640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa8440b5470 0x7fa8440b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa849612640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84415e1e0 con 0x7fa84415de00 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa849612640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8440b98c0 con 0x7fa8440b5470 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa849612640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440b32c0 con 0x7fa8440b2e60 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa8427fc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa8437fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa8440b5470 0x7fa8440b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa842ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa84415de00 0x7fa8440b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:37198/0 (socket says 172.21.15.99:37198) 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.562+0000 7fa842ffd640 1 -- 172.21.15.99:0/2089491514 learned_addr learned my addr 172.21.15.99:0/2089491514 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.563+0000 7fa8437fe640 1 -- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 msgr2=0x7fa8440b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.563+0000 7fa8437fe640 1 --2- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.563+0000 7fa8437fe640 1 -- 172.21.15.99:0/2089491514 >> v1:172.21.15.162:6789/0 conn(0x7fa84415de00 legacy=0x7fa8440b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.563+0000 7fa8437fe640 1 -- 172.21.15.99:0/2089491514 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8440d1c10 con 0x7fa8440b5470 2024-03-04T10:10:00.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.563+0000 7fa8427fc640 1 --2- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa8437fe640 1 --2- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa8440b5900 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097a0 tx=0x7fa83c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26d94df86db75e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa841ffb640 1 -- 172.21.15.99:0/2089491514 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c035020 con 0x7fa8440b5470 2024-03-04T10:10:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa841ffb640 1 -- 172.21.15.99:0/2089491514 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c031830 con 0x7fa8440b5470 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa841ffb640 1 -- 172.21.15.99:0/2089491514 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c031c00 con 0x7fa8440b5470 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 -- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 msgr2=0x7fa8440b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 --2- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa8440b5900 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa83c0097a0 tx=0x7fa83c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 -- 172.21.15.99:0/2089491514 shutdown_connections 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 --2- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 --2- 172.21.15.99:0/2089491514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa8440b5900 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 -- 172.21.15.99:0/2089491514 >> 172.21.15.99:0/2089491514 conn(0x7fa8440d0470 msgr2=0x7fa8440d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 -- 172.21.15.99:0/2089491514 shutdown_connections 2024-03-04T10:10:00.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.564+0000 7fa849612640 1 -- 172.21.15.99:0/2089491514 wait complete. 2024-03-04T10:10:00.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 Processor -- start 2024-03-04T10:10:00.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 -- start start 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa84415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 0x7fa8440b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440d21d0 con 0x7fa84415de00 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440d2820 con 0x7fa8440b5470 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa849612640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440d24b0 con 0x7fa8440b2e60 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa8437fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 0x7fa8440b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa8437fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 0x7fa8440b8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50778/0 (socket says 172.21.15.99:50778) 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.566+0000 7fa8437fe640 1 -- 172.21.15.99:0/2466486200 learned_addr learned my addr 172.21.15.99:0/2466486200 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8427fc640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa84415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa842ffd640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8437fe640 1 -- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 msgr2=0x7fa8440c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8437fe640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8437fe640 1 -- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 msgr2=0x7fa84415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8437fe640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa84415faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8437fe640 1 -- 172.21.15.99:0/2466486200 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8441604e0 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa8437fe640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 0x7fa8440b8350 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa83c02b910 tx=0x7fa83c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c035040 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c03b840 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c03bc30 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8441607c0 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.567+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa844160d80 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.568+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa844074f60 con 0x7fa84415de00 2024-03-04T10:10:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.568+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa83c038020 con 0x7fa84415de00 2024-03-04T10:10:00.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.569+0000 7fa82b7fe640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa83006b4d0 0x7fa83006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.569+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fa83c0ac230 con 0x7fa84415de00 2024-03-04T10:10:00.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.569+0000 7fa842ffd640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa83006b4d0 0x7fa83006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.570+0000 7fa842ffd640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa83006b4d0 0x7fa83006d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa8440b27a0 tx=0x7fa834009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.573+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa83c031800 con 0x7fa84415de00 2024-03-04T10:10:00.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.809+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]} v 0) v1 -- 0x7fa8440d09f0 con 0x7fa84415de00 2024-03-04T10:10:00.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.817+0000 7fa82b7fe640 1 -- 172.21.15.99:0/2466486200 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_dentry_first_preflush"]}]=0 v316) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa83c07b1e0 con 0x7fa84415de00 2024-03-04T10:10:00.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa83006b4d0 msgr2=0x7fa83006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa83006b4d0 0x7fa83006d990 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa8440b27a0 tx=0x7fa834009040 comp rx=0 tx=0).stop 2024-03-04T10:10:00.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 msgr2=0x7fa8440b8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 0x7fa8440b8350 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fa83c02b910 tx=0x7fa83c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 shutdown_connections 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa83006b4d0 0x7fa83006d990 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa84415de00 0x7fa8440b8350 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa8440b5470 0x7fa84415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.819+0000 7fa849612640 1 --2- 172.21.15.99:0/2466486200 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa8440b2e60 0x7fa8440c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.820+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 >> 172.21.15.99:0/2466486200 conn(0x7fa8440d0470 msgr2=0x7fa8440c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.820+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 shutdown_connections 2024-03-04T10:10:00.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.820+0000 7fa849612640 1 -- 172.21.15.99:0/2466486200 wait complete. 2024-03-04T10:10:00.832 INFO:tasks.cephfs_test_runner:That the MDS won't write a dentry with new damage to CDentry::first ... ok 2024-03-04T10:10:00.832 INFO:tasks.cephfs_test_runner:Starting test: test_object_deletion (tasks.cephfs.test_damage.TestDamage) That the MDS has a clean 'damaged' response to loss of any single metadata object 2024-03-04T10:10:00.832 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:10:00.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.931+0000 7f6573668640 1 Processor -- start 2024-03-04T10:10:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.931+0000 7f6573668640 1 -- start start 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6573668640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f656c158410 0x7f656c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6573668640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c06acd0 0x7f656c161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6573668640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656c06b500 con 0x7f656c161ca0 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6573668640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656c0b3f00 con 0x7f656c158410 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6573668640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c06b7e0 con 0x7f656c06acd0 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6571c05640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f656c161ca0 0x7f656c15e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40554/0 (socket says 172.21.15.99:40554) 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6571c05640 1 -- 172.21.15.99:0/3043763578 learned_addr learned my addr 172.21.15.99:0/3043763578 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6570c03640 1 --2- 172.21.15.99:0/3043763578 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c06acd0 0x7f656c161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6571404640 1 --2- 172.21.15.99:0/3043763578 >> v2:172.21.15.162:3300/0 conn(0x7f656c158410 0x7f656c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6563fff640 1 -- 172.21.15.99:0/3043763578 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322458047 0 0) 0x7f656c06b500 con 0x7f656c161ca0 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.932+0000 7f6563fff640 1 -- 172.21.15.99:0/3043763578 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6558003520 con 0x7f656c161ca0 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6570c03640 1 -- 172.21.15.99:0/3043763578 >> v2:172.21.15.162:3300/0 conn(0x7f656c158410 msgr2=0x7f656c0b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6570c03640 1 --2- 172.21.15.99:0/3043763578 >> v2:172.21.15.162:3300/0 conn(0x7f656c158410 0x7f656c0b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6563fff640 1 -- 172.21.15.99:0/3043763578 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2718221325 0 0) 0x7f6558003520 con 0x7f656c161ca0 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6570c03640 1 -- 172.21.15.99:0/3043763578 >> v1:172.21.15.99:6789/0 conn(0x7f656c161ca0 legacy=0x7f656c15e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6570c03640 1 -- 172.21.15.99:0/3043763578 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c06bac0 con 0x7f656c06acd0 2024-03-04T10:10:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6571404640 1 --2- 172.21.15.99:0/3043763578 >> v2:172.21.15.162:3300/0 conn(0x7f656c158410 0x7f656c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.933+0000 7f6570c03640 1 --2- 172.21.15.99:0/3043763578 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c06acd0 0x7f656c161760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f655c0097a0 tx=0x7f655c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d1ee15c184194e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6563fff640 1 -- 172.21.15.99:0/3043763578 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c038070 con 0x7f656c06acd0 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6563fff640 1 -- 172.21.15.99:0/3043763578 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c031500 con 0x7f656c06acd0 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6563fff640 1 -- 172.21.15.99:0/3043763578 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65640019f0 con 0x7f656c06acd0 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 -- 172.21.15.99:0/3043763578 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c06acd0 msgr2=0x7f656c161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 --2- 172.21.15.99:0/3043763578 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c06acd0 0x7f656c161760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f655c0097a0 tx=0x7f655c02b820 comp rx=0 tx=0).stop 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 -- 172.21.15.99:0/3043763578 shutdown_connections 2024-03-04T10:10:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 --2- 172.21.15.99:0/3043763578 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c06acd0 0x7f656c161760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 --2- 172.21.15.99:0/3043763578 >> v2:172.21.15.162:3300/0 conn(0x7f656c158410 0x7f656c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 -- 172.21.15.99:0/3043763578 >> 172.21.15.99:0/3043763578 conn(0x7f656c0cfef0 msgr2=0x7f656c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 -- 172.21.15.99:0/3043763578 shutdown_connections 2024-03-04T10:10:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.934+0000 7f6573668640 1 -- 172.21.15.99:0/3043763578 wait complete. 2024-03-04T10:10:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.935+0000 7f6573668640 1 Processor -- start 2024-03-04T10:10:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.935+0000 7f6573668640 1 -- start start 2024-03-04T10:10:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6573668640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 0x7f656c14a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6573668640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f656c158410 0x7f656c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6573668640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c161ca0 0x7f656c15cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6573668640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c06c5a0 con 0x7f656c06acd0 2024-03-04T10:10:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6573668640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c06c080 con 0x7f656c158410 2024-03-04T10:10:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6573668640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656c06ee60 con 0x7f656c161ca0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 0x7f656c14a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 0x7f656c14a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50792/0 (socket says 172.21.15.99:50792) 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 -- 172.21.15.99:0/1516375396 learned_addr learned my addr 172.21.15.99:0/1516375396 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 -- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c161ca0 msgr2=0x7f656c15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c161ca0 0x7f656c15cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 -- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f656c158410 msgr2=0x7f656c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f656c158410 0x7f656c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.936+0000 7f6571404640 1 -- 172.21.15.99:0/1516375396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c14aef0 con 0x7f656c06acd0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.937+0000 7f6571404640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 0x7f656c14a9b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f656800ebe0 tx=0x7f656800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.937+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568017070 con 0x7f656c06acd0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.937+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656800cc30 con 0x7f656c06acd0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.937+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c15bc00 con 0x7f656c06acd0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.937+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568013470 con 0x7f656c06acd0 2024-03-04T10:10:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.937+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c15c1c0 con 0x7f656c06acd0 2024-03-04T10:10:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.938+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c074f60 con 0x7f656c06acd0 2024-03-04T10:10:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.938+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f656801a070 con 0x7f656c06acd0 2024-03-04T10:10:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.939+0000 7f6561ffb640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f655006b4d0 0x7f655006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.939+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f656808a040 con 0x7f656c06acd0 2024-03-04T10:10:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.939+0000 7f6570c03640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f655006b4d0 0x7f655006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.939+0000 7f6570c03640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f655006b4d0 0x7f655006d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f655c035000 tx=0x7f655c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:00.943+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f656c14aef0 con 0x7f656c06acd0 2024-03-04T10:10:01.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.114+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f656c0b05d0 con 0x7f656c06acd0 2024-03-04T10:10:01.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.115+0000 7f6561ffb640 1 -- 172.21.15.99:0/1516375396 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+13388 (secure 0 0 0) 0x7f6568058fe0 con 0x7f656c06acd0 2024-03-04T10:10:01.116 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:01.116 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":85,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:09:59.812715+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6834/3121450352":"2024-03-05T10:09:56.797094+0000","172.21.15.99:6832/3121450352":"2024-03-05T10:09:56.797094+0000","172.21.15.99:6835/3857383686":"2024-03-05T10:09:46.616133+0000","172.21.15.99:6833/3857383686":"2024-03-05T10:09:46.616133+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f655006b4d0 msgr2=0x7f655006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f655006b4d0 0x7f655006d990 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f655c035000 tx=0x7f655c030000 comp rx=0 tx=0).stop 2024-03-04T10:10:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 msgr2=0x7f656c14a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 0x7f656c14a9b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f656800ebe0 tx=0x7f656800c520 comp rx=0 tx=0).stop 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 shutdown_connections 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f655006b4d0 0x7f655006d990 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f656c161ca0 0x7f656c15cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f656c158410 0x7f656c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 --2- 172.21.15.99:0/1516375396 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f656c06acd0 0x7f656c14a9b0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 >> 172.21.15.99:0/1516375396 conn(0x7f656c0cfef0 msgr2=0x7f656c0d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 shutdown_connections 2024-03-04T10:10:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.117+0000 7f6573668640 1 -- 172.21.15.99:0/1516375396 wait complete. 2024-03-04T10:10:01.129 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:10:01.129 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:10:01.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.230+0000 7f3fe0fc4640 1 Processor -- start 2024-03-04T10:10:01.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.230+0000 7f3fe0fc4640 1 -- start start 2024-03-04T10:10:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.231+0000 7f3fe0fc4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3fdc0ccfe0 0x7f3fdc15db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.231+0000 7f3fe0fc4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fdc06ae00 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.231+0000 7f3fe0fc4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fdc06b0e0 con 0x7f3fdc0cc900 2024-03-04T10:10:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.231+0000 7f3fe0fc4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fdc0ccde0 con 0x7f3fdc0ccfe0 2024-03-04T10:10:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.231+0000 7f3fda59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3fdc0cdd70 0x7f3fdc0cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40560/0 (socket says 172.21.15.99:40560) 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fda59c640 1 -- 172.21.15.99:0/179210681 learned_addr learned my addr 172.21.15.99:0/179210681 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fdad9d640 1 --2- 172.21.15.99:0/179210681 >> v2:172.21.15.99:3300/0 conn(0x7f3fdc0ccfe0 0x7f3fdc15db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fd959a640 1 -- 172.21.15.99:0/179210681 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334652478 0 0) 0x7f3fdc06ae00 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fd959a640 1 -- 172.21.15.99:0/179210681 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc4003520 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fdad9d640 1 -- 172.21.15.99:0/179210681 >> v1:172.21.15.162:6789/0 conn(0x7f3fdc0cc900 legacy=0x7f3fdc15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fdad9d640 1 -- 172.21.15.99:0/179210681 >> v1:172.21.15.99:6789/0 conn(0x7f3fdc0cdd70 legacy=0x7f3fdc0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.232+0000 7f3fdad9d640 1 -- 172.21.15.99:0/179210681 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc06b3c0 con 0x7f3fdc0ccfe0 2024-03-04T10:10:01.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fdad9d640 1 --2- 172.21.15.99:0/179210681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15db20 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40097a0 tx=0x7f3fd402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e242488557081 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fd959a640 1 -- 172.21.15.99:0/179210681 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40315a0 con 0x7f3fdc0ccfe0 2024-03-04T10:10:01.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fd959a640 1 -- 172.21.15.99:0/179210681 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd4037070 con 0x7f3fdc0ccfe0 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fd959a640 1 -- 172.21.15.99:0/179210681 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4031960 con 0x7f3fdc0ccfe0 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/179210681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0ccfe0 msgr2=0x7f3fdc15db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/179210681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15db20 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40097a0 tx=0x7f3fd402b840 comp rx=0 tx=0).stop 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/179210681 shutdown_connections 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/179210681 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15db20 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/179210681 >> 172.21.15.99:0/179210681 conn(0x7f3fdc0b22e0 msgr2=0x7f3fdc0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/179210681 shutdown_connections 2024-03-04T10:10:01.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.233+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/179210681 wait complete. 2024-03-04T10:10:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 Processor -- start 2024-03-04T10:10:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 -- start start 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fdc0cc900 0x7f3fdc14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 0x7f3fdc160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc15e1e0 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc15e060 con 0x7f3fdc0cc900 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fe0fc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fdc0b2e90 con 0x7f3fdc0ccfe0 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fdad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 0x7f3fdc160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fdad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 0x7f3fdc160470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50808/0 (socket says 172.21.15.99:50808) 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.235+0000 7f3fdad9d640 1 -- 172.21.15.99:0/3392529627 learned_addr learned my addr 172.21.15.99:0/3392529627 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fd9d9b640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fda59c640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fdc0cc900 0x7f3fdc14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fdad9d640 1 -- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fdc0ccfe0 msgr2=0x7f3fdc15ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fdad9d640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fdad9d640 1 -- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fdc0cc900 msgr2=0x7f3fdc14fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fdad9d640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fdc0cc900 0x7f3fdc14fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fdad9d640 1 -- 172.21.15.99:0/3392529627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc1e8380 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fda59c640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fdc0cc900 0x7f3fdc14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fdad9d640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 0x7f3fdc160470 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40097a0 tx=0x7f3fd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4032630 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd40327f0 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd4032be0 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fdc1e8660 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.236+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fdc1e8c20 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.237+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fdc074f60 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.238+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3fd403b3f0 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.238+0000 7f3fcaffd640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fbc06b4d0 0x7f3fbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.238+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3fd402f030 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.238+0000 7f3fda59c640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fbc06b4d0 0x7f3fbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.239+0000 7f3fda59c640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fbc06b4d0 0x7f3fbc06d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc005c00 tx=0x7f3fcc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.242+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3fd40b1020 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.414+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3fdc0b1840 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.414+0000 7f3fcaffd640 1 -- 172.21.15.99:0/3392529627 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3fd4040070 con 0x7f3fdc0cdd70 2024-03-04T10:10:01.416 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fbc06b4d0 msgr2=0x7f3fbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fbc06b4d0 0x7f3fbc06d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc005c00 tx=0x7f3fcc009040 comp rx=0 tx=0).stop 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 msgr2=0x7f3fdc160470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 0x7f3fdc160470 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f3fd40097a0 tx=0x7f3fd40046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 shutdown_connections 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fbc06b4d0 0x7f3fbc06d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fdc0cdd70 0x7f3fdc160470 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fdc0ccfe0 0x7f3fdc15ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 --2- 172.21.15.99:0/3392529627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fdc0cc900 0x7f3fdc14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 >> 172.21.15.99:0/3392529627 conn(0x7f3fdc0b22e0 msgr2=0x7f3fdc0ad550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 shutdown_connections 2024-03-04T10:10:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.416+0000 7f3fe0fc4640 1 -- 172.21.15.99:0/3392529627 wait complete. 2024-03-04T10:10:01.428 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-03-04T10:10:01.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.526+0000 7fc89a53b640 1 Processor -- start 2024-03-04T10:10:01.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.526+0000 7fc89a53b640 1 -- start start 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc89a53b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc89415de00 0x7fc8940b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc89a53b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b5470 0x7fc8940b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc89a53b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89415e1e0 con 0x7fc8940b9380 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc89a53b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8940d2820 con 0x7fc89415de00 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc89a53b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8940d1c10 con 0x7fc8940b5470 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc893fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc89415de00 0x7fc8940b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc8937fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc8940b9380 0x7fc8940b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40562/0 (socket says 172.21.15.99:40562) 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.527+0000 7fc8937fe640 1 -- 172.21.15.99:0/1478188315 learned_addr learned my addr 172.21.15.99:0/1478188315 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc893fff640 1 -- 172.21.15.99:0/1478188315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b5470 msgr2=0x7fc8940b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc898ad8640 1 --2- 172.21.15.99:0/1478188315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b5470 0x7fc8940b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc892ffd640 1 -- 172.21.15.99:0/1478188315 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071574519 0 0) 0x7fc89415e1e0 con 0x7fc8940b9380 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc893fff640 1 --2- 172.21.15.99:0/1478188315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b5470 0x7fc8940b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc893fff640 1 -- 172.21.15.99:0/1478188315 >> v1:172.21.15.99:6789/0 conn(0x7fc8940b9380 legacy=0x7fc8940b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc893fff640 1 -- 172.21.15.99:0/1478188315 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8940d1ef0 con 0x7fc89415de00 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.528+0000 7fc893fff640 1 --2- 172.21.15.99:0/1478188315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b2e60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc8840097a0 tx=0x7fc88402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb24ecbffdef218 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc892ffd640 1 -- 172.21.15.99:0/1478188315 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884035020 con 0x7fc89415de00 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc892ffd640 1 -- 172.21.15.99:0/1478188315 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc884031830 con 0x7fc89415de00 2024-03-04T10:10:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc892ffd640 1 -- 172.21.15.99:0/1478188315 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884031c00 con 0x7fc89415de00 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 -- 172.21.15.99:0/1478188315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 msgr2=0x7fc8940b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 --2- 172.21.15.99:0/1478188315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b2e60 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fc8840097a0 tx=0x7fc88402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 -- 172.21.15.99:0/1478188315 shutdown_connections 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 --2- 172.21.15.99:0/1478188315 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b5470 0x7fc8940b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 --2- 172.21.15.99:0/1478188315 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b2e60 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 -- 172.21.15.99:0/1478188315 >> 172.21.15.99:0/1478188315 conn(0x7fc8940d0470 msgr2=0x7fc8940d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 -- 172.21.15.99:0/1478188315 shutdown_connections 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.529+0000 7fc89a53b640 1 -- 172.21.15.99:0/1478188315 wait complete. 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.530+0000 7fc89a53b640 1 Processor -- start 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 -- start start 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8940b5470 0x7fc89414ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b9380 0x7fc89414b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8940b97e0 con 0x7fc89415de00 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8940d24b0 con 0x7fc8940b5470 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc89a53b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8940b72a0 con 0x7fc8940b9380 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc898ad8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc898ad8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50820/0 (socket says 172.21.15.99:50820) 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.531+0000 7fc898ad8640 1 -- 172.21.15.99:0/2620059703 learned_addr learned my addr 172.21.15.99:0/2620059703 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc8937fe640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b9380 0x7fc89414b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc893fff640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8940b5470 0x7fc89414ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc898ad8640 1 -- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b9380 msgr2=0x7fc89414b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc898ad8640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b9380 0x7fc89414b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc898ad8640 1 -- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8940b5470 msgr2=0x7fc89414ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc898ad8640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8940b5470 0x7fc89414ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc898ad8640 1 -- 172.21.15.99:0/2620059703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89414b340 con 0x7fc89415de00 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc898ad8640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b45d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00cda0 tx=0x7fc88c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c003b90 con 0x7fc89415de00 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c003d30 con 0x7fc89415de00 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89415f800 con 0x7fc89415de00 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c004240 con 0x7fc89415de00 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.532+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89415fdc0 con 0x7fc89415de00 2024-03-04T10:10:01.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.533+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc894074f60 con 0x7fc89415de00 2024-03-04T10:10:01.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.533+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc88c0043e0 con 0x7fc89415de00 2024-03-04T10:10:01.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.534+0000 7fc890ff9640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88006b4d0 0x7fc88006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.534+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc88c08ac20 con 0x7fc89415de00 2024-03-04T10:10:01.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.534+0000 7fc893fff640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88006b4d0 0x7fc88006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.534+0000 7fc893fff640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88006b4d0 0x7fc88006d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc884036000 tx=0x7fc88403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.538+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc89414b340 con 0x7fc89415de00 2024-03-04T10:10:01.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.771+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) v1 -- 0x7fc89406b1b0 con 0x7fc89415de00 2024-03-04T10:10:01.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.822+0000 7fc890ff9640 1 -- 172.21.15.99:0/2620059703 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v317) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc88c059bc0 con 0x7fc89415de00 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88006b4d0 msgr2=0x7fc88006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88006b4d0 0x7fc88006d990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc884036000 tx=0x7fc88403c040 comp rx=0 tx=0).stop 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 msgr2=0x7fc8940b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b45d0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fc88c00cda0 tx=0x7fc88c00a660 comp rx=0 tx=0).stop 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 shutdown_connections 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc88006b4d0 0x7fc88006d990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc89415de00 0x7fc8940b45d0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc8940b9380 0x7fc89414b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 --2- 172.21.15.99:0/2620059703 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc8940b5470 0x7fc89414ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 >> 172.21.15.99:0/2620059703 conn(0x7fc8940d0470 msgr2=0x7fc89406cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 shutdown_connections 2024-03-04T10:10:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.824+0000 7fc89a53b640 1 -- 172.21.15.99:0/2620059703 wait complete. 2024-03-04T10:10:01.836 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-04T10:10:01.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.939+0000 7f853e75b640 1 Processor -- start 2024-03-04T10:10:01.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.939+0000 7f853e75b640 1 -- start start 2024-03-04T10:10:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.939+0000 7f853e75b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f85380cfc70 0x7f85380cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.939+0000 7f853e75b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380cebd0 con 0x7f85381450e0 2024-03-04T10:10:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.939+0000 7f853e75b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85380ceeb0 con 0x7f85380cc340 2024-03-04T10:10:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.939+0000 7f853e75b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85380cc7b0 con 0x7f85380cfc70 2024-03-04T10:10:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f85377fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f85380cfc70 0x7f85380cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f8537fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f85381450e0 0x7f853815c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40566/0 (socket says 172.21.15.99:40566) 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f8537fff640 1 -- 172.21.15.99:0/735868300 learned_addr learned my addr 172.21.15.99:0/735868300 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f85377fe640 1 -- 172.21.15.99:0/735868300 >> v1:172.21.15.162:6789/0 conn(0x7f85380cc340 legacy=0x7f85380b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f85377fe640 1 -- 172.21.15.99:0/735868300 >> v1:172.21.15.99:6789/0 conn(0x7f85381450e0 legacy=0x7f853815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f8536ffd640 1 -- 172.21.15.99:0/735868300 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854260862 0 0) 0x7f85380cebd0 con 0x7f85381450e0 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.940+0000 7f85377fe640 1 -- 172.21.15.99:0/735868300 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85380cf190 con 0x7f85380cfc70 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.941+0000 7f85377fe640 1 --2- 172.21.15.99:0/735868300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f85380cbe00 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f852c0097a0 tx=0x7f852c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8770468c1a905ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.941+0000 7f8536ffd640 1 -- 172.21.15.99:0/735868300 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c035020 con 0x7f85380cfc70 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.941+0000 7f8536ffd640 1 -- 172.21.15.99:0/735868300 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c031830 con 0x7f85380cfc70 2024-03-04T10:10:01.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.941+0000 7f8536ffd640 1 -- 172.21.15.99:0/735868300 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c031c00 con 0x7f85380cfc70 2024-03-04T10:10:01.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.941+0000 7f853e75b640 1 -- 172.21.15.99:0/735868300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 msgr2=0x7f85380cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.941+0000 7f853e75b640 1 --2- 172.21.15.99:0/735868300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f85380cbe00 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f852c0097a0 tx=0x7f852c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.942+0000 7f853e75b640 1 -- 172.21.15.99:0/735868300 shutdown_connections 2024-03-04T10:10:01.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.942+0000 7f853e75b640 1 --2- 172.21.15.99:0/735868300 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f85380cbe00 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.942+0000 7f853e75b640 1 -- 172.21.15.99:0/735868300 >> 172.21.15.99:0/735868300 conn(0x7f853806acd0 msgr2=0x7f853806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.942+0000 7f853e75b640 1 -- 172.21.15.99:0/735868300 shutdown_connections 2024-03-04T10:10:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.942+0000 7f853e75b640 1 -- 172.21.15.99:0/735868300 wait complete. 2024-03-04T10:10:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.943+0000 7f853e75b640 1 Processor -- start 2024-03-04T10:10:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.943+0000 7f853e75b640 1 -- start start 2024-03-04T10:10:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85380cc340 0x7f853813afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f853813c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85381450e0 0x7f853813c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85381454c0 con 0x7f85380cfc70 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380cf7e0 con 0x7f85381450e0 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85380ccd10 con 0x7f85380cc340 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f853813c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f853813c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50828/0 (socket says 172.21.15.99:50828) 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 -- 172.21.15.99:0/245164189 learned_addr learned my addr 172.21.15.99:0/245164189 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 -- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85380cc340 msgr2=0x7f853813afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85380cc340 0x7f853813afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 -- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85381450e0 msgr2=0x7f853813c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853ccf8640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85381450e0 0x7f853813c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85381450e0 0x7f853813c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 -- 172.21.15.99:0/245164189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853813e7a0 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853ccf8640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85381450e0 0x7f853813c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f85377fe640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f853813c420 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098c0 tx=0x7f852c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c03ba80 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c03bc20 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c0453f0 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853813cf90 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.944+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853813d550 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.946+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85380ce760 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.946+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f852c04c060 con 0x7f85380cfc70 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.946+0000 7f8534ff9640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f852406b4d0 0x7f852406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.946+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f852c0abf10 con 0x7f85380cfc70 2024-03-04T10:10:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.946+0000 7f8537fff640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f852406b4d0 0x7f852406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.947+0000 7f8537fff640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f852406b4d0 0x7f852406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8528007fd0 tx=0x7f8528007960 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:01.950+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f85380ce760 con 0x7f85380cfc70 2024-03-04T10:10:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.120+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8538074f60 con 0x7f85380cfc70 2024-03-04T10:10:02.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.123+0000 7f8534ff9640 1 -- 172.21.15.99:0/245164189 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f852c040070 con 0x7f85380cfc70 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f852406b4d0 msgr2=0x7f852406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f852406b4d0 0x7f852406d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8528007fd0 tx=0x7f8528007960 comp rx=0 tx=0).stop 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 msgr2=0x7f853813c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f853813c420 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f852c0098c0 tx=0x7f852c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 shutdown_connections 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f852406b4d0 0x7f852406d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85381450e0 0x7f853813c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f85380cfc70 0x7f853813c420 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 --2- 172.21.15.99:0/245164189 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f85380cc340 0x7f853813afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 >> 172.21.15.99:0/245164189 conn(0x7f853806acd0 msgr2=0x7f853815c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 shutdown_connections 2024-03-04T10:10:02.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.125+0000 7f853e75b640 1 -- 172.21.15.99:0/245164189 wait complete. 2024-03-04T10:10:02.137 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:10:02.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:10:02.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:10:02.223 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:10:02.224 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:02.224 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:02.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:10:02.241 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:10:02.241 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:02.264 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:10:02.264 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:02.264 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:02.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:10:02.300 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:10:02.300 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:02.337 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:10:02.337 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-04T10:10:02.337 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.446+0000 7f8b646e8640 1 Processor -- start 2024-03-04T10:10:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.446+0000 7f8b646e8640 1 -- start start 2024-03-04T10:10:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b646e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b646e8640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8b5c06d3b0 0x7f8b5c15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b646e8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5c15e2d0 con 0x7f8b5c15b410 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b646e8640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5c0b2720 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b646e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c06d8b0 con 0x7f8b5c15b030 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b61c83640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8b5c06d3b0 0x7f8b5c15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b62c85640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8b5c15b410 0x7f8b5c15b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40578/0 (socket says 172.21.15.99:40578) 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.447+0000 7f8b62c85640 1 -- 172.21.15.99:0/1026921745 learned_addr learned my addr 172.21.15.99:0/1026921745 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b62484640 1 --2- 172.21.15.99:0/1026921745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b61c83640 1 -- 172.21.15.99:0/1026921745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 msgr2=0x7f8b5c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b61c83640 1 --2- 172.21.15.99:0/1026921745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b61c83640 1 -- 172.21.15.99:0/1026921745 >> v1:172.21.15.99:6789/0 conn(0x7f8b5c15b410 legacy=0x7f8b5c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b61c83640 1 -- 172.21.15.99:0/1026921745 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c06db90 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b61482640 1 -- 172.21.15.99:0/1026921745 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356773388 0 0) 0x7f8b5c15e2d0 con 0x7f8b5c15b410 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.448+0000 7f8b61c83640 1 --2- 172.21.15.99:0/1026921745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c15dd90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8b580097a0 tx=0x7f8b5802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3816249f8b602e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b61482640 1 -- 172.21.15.99:0/1026921745 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b580315a0 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b61482640 1 -- 172.21.15.99:0/1026921745 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b58037070 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b61482640 1 -- 172.21.15.99:0/1026921745 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b58031960 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 -- 172.21.15.99:0/1026921745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 msgr2=0x7f8b5c15dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 --2- 172.21.15.99:0/1026921745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c15dd90 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8b580097a0 tx=0x7f8b5802b840 comp rx=0 tx=0).stop 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 -- 172.21.15.99:0/1026921745 shutdown_connections 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 --2- 172.21.15.99:0/1026921745 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c15dd90 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 --2- 172.21.15.99:0/1026921745 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 -- 172.21.15.99:0/1026921745 >> 172.21.15.99:0/1026921745 conn(0x7f8b5c06acd0 msgr2=0x7f8b5c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 -- 172.21.15.99:0/1026921745 shutdown_connections 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.449+0000 7f8b646e8640 1 -- 172.21.15.99:0/1026921745 wait complete. 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.450+0000 7f8b646e8640 1 Processor -- start 2024-03-04T10:10:02.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.450+0000 7f8b646e8640 1 -- start start 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b646e8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c0aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b646e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c0acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b646e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b5c15b410 0x7f8b5c0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b646e8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c06e690 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b646e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c06e150 con 0x7f8b5c15b410 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b646e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5c06e810 con 0x7f8b5c15b030 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c0aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c0aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50846/0 (socket says 172.21.15.99:50846) 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 -- 172.21.15.99:0/3052296350 learned_addr learned my addr 172.21.15.99:0/3052296350 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b61c83640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c0acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62c85640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b5c15b410 0x7f8b5c0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 -- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 msgr2=0x7f8b5c0acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c0acb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 -- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b5c15b410 msgr2=0x7f8b5c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b5c15b410 0x7f8b5c0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b62484640 1 -- 172.21.15.99:0/3052296350 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c0a92c0 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.451+0000 7f8b61c83640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c0acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.452+0000 7f8b62484640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c0aa1e0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000d800 tx=0x7f8b5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.452+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000b6c0 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.452+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50019070 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.452+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5c0a95a0 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.452+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000ba80 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.452+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c0a9b60 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.453+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5c074f60 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.453+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8b5000bc20 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.453+0000 7f8b4effd640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b4006b4d0 0x7f8b4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.453+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f8b5008a530 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.454+0000 7f8b61c83640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b4006b4d0 0x7f8b4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.454+0000 7f8b61c83640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b4006b4d0 0x7f8b4006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8b58036000 tx=0x7f8b5803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:02.463 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.458+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8b50059560 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.667+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b5c0b2cc0 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.668+0000 7f8b4effd640 1 -- 172.21.15.99:0/3052296350 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f8b50058e60 con 0x7f8b5c06d3b0 2024-03-04T10:10:02.669 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:02.669 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":73,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5789,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6836/3106637055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":3106637055},{"type":"v1","addr":"172.21.15.162:6836","nonce":3106637055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":5792,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/2493683410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6835","nonce":2493683410},{"type":"v1","addr":"172.21.15.162:6837","nonce":2493683410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":7389,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/2281805004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6833","nonce":2281805004},{"type":"v1","addr":"172.21.15.99:6835","nonce":2281805004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7554,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6834/792302876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":792302876},{"type":"v1","addr":"172.21.15.99:6834","nonce":792302876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":73}],"filesystems":[]} 2024-03-04T10:10:02.669 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 73 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b4006b4d0 msgr2=0x7f8b4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b4006b4d0 0x7f8b4006d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8b58036000 tx=0x7f8b5803d040 comp rx=0 tx=0).stop 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 msgr2=0x7f8b5c0aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c0aa1e0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000d800 tx=0x7f8b5000dcc0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 shutdown_connections 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b4006b4d0 0x7f8b4006d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8b5c15b410 0x7f8b5c0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8b5c15b030 0x7f8b5c0acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 --2- 172.21.15.99:0/3052296350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8b5c06d3b0 0x7f8b5c0aa1e0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 >> 172.21.15.99:0/3052296350 conn(0x7f8b5c06acd0 msgr2=0x7f8b5c15b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 shutdown_connections 2024-03-04T10:10:02.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.670+0000 7f8b646e8640 1 -- 172.21.15.99:0/3052296350 wait complete. 2024-03-04T10:10:02.683 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-04T10:10:02.683 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-04T10:10:02.683 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:10:02.683 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:10:02.683 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-04T10:10:02.683 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-04T10:10:02.684 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:10:02.684 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:10:02.684 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-04T10:10:02.684 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-04T10:10:02.684 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T10:10:02.684 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:10:02.684 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-04T10:10:02.684 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-04T10:10:02.685 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T10:10:02.685 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:10:02.698 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:10:02.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-04T10:10:02.701 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:10:02.701 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-04T10:10:02.702 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T10:10:02.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-04T10:10:02.707 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T10:10:02.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-04T10:10:02.743 INFO:tasks.ceph.mds.a:Started 2024-03-04T10:10:02.744 DEBUG:teuthology.parallel:result is None 2024-03-04T10:10:02.746 INFO:tasks.ceph.mds.c:Started 2024-03-04T10:10:02.746 DEBUG:teuthology.parallel:result is None 2024-03-04T10:10:02.757 INFO:tasks.ceph.mds.b:Started 2024-03-04T10:10:02.758 DEBUG:teuthology.parallel:result is None 2024-03-04T10:10:02.758 INFO:tasks.ceph.mds.d:Started 2024-03-04T10:10:02.759 DEBUG:teuthology.parallel:result is None 2024-03-04T10:10:02.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-04T10:10:02.819 INFO:tasks.ceph.mds.a.smithi099.stdout:starting mds.a at 2024-03-04T10:10:02.820 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:10:02.830 INFO:tasks.ceph.mds.b.smithi162.stdout:starting mds.b at 2024-03-04T10:10:02.835 INFO:tasks.ceph.mds.d.smithi162.stdout:starting mds.d at 2024-03-04T10:10:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.864+0000 7f204df2b640 1 Processor -- start 2024-03-04T10:10:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.864+0000 7f204df2b640 1 -- start start 2024-03-04T10:10:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f204df2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815bab0 0x7f20480b7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f204df2b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204815f8c0 con 0x7f204815c2f0 2024-03-04T10:10:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f204df2b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204815c7f0 con 0x7f204815be90 2024-03-04T10:10:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f204df2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204815cad0 con 0x7f204815bab0 2024-03-04T10:10:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f2047fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f204815c2f0 0x7f20480cc1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:40602/0 (socket says 172.21.15.99:40602) 2024-03-04T10:10:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f2047fff640 1 -- 172.21.15.99:0/2922886779 learned_addr learned my addr 172.21.15.99:0/2922886779 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f20477fe640 1 --2- 172.21.15.99:0/2922886779 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815bab0 0x7f20480b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f2046ffd640 1 -- 172.21.15.99:0/2922886779 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027408660 0 0) 0x7f204815f8c0 con 0x7f204815c2f0 2024-03-04T10:10:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.865+0000 7f2046ffd640 1 -- 172.21.15.99:0/2922886779 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2028003520 con 0x7f204815c2f0 2024-03-04T10:10:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.866+0000 7f20477fe640 1 -- 172.21.15.99:0/2922886779 >> v1:172.21.15.162:6789/0 conn(0x7f204815be90 legacy=0x7f20480cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.866+0000 7f2046ffd640 1 -- 172.21.15.99:0/2922886779 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270968077 0 0) 0x7f204815c7f0 con 0x7f204815be90 2024-03-04T10:10:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.866+0000 7f20477fe640 1 -- 172.21.15.99:0/2922886779 >> v1:172.21.15.99:6789/0 conn(0x7f204815c2f0 legacy=0x7f20480cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.866+0000 7f20477fe640 1 -- 172.21.15.99:0/2922886779 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204815cdb0 con 0x7f204815bab0 2024-03-04T10:10:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.866+0000 7f20477fe640 1 --2- 172.21.15.99:0/2922886779 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815bab0 0x7f20480b7260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f20380097d0 tx=0x7f203802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b903d99af37a02c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.866+0000 7f2046ffd640 1 -- 172.21.15.99:0/2922886779 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038035020 con 0x7f204815bab0 2024-03-04T10:10:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f2046ffd640 1 -- 172.21.15.99:0/2922886779 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2038031890 con 0x7f204815bab0 2024-03-04T10:10:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f2046ffd640 1 -- 172.21.15.99:0/2922886779 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038031c60 con 0x7f204815bab0 2024-03-04T10:10:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 -- 172.21.15.99:0/2922886779 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815bab0 msgr2=0x7f20480b7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 --2- 172.21.15.99:0/2922886779 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815bab0 0x7f20480b7260 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f20380097d0 tx=0x7f203802b930 comp rx=0 tx=0).stop 2024-03-04T10:10:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 -- 172.21.15.99:0/2922886779 shutdown_connections 2024-03-04T10:10:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 --2- 172.21.15.99:0/2922886779 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815bab0 0x7f20480b7260 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 -- 172.21.15.99:0/2922886779 >> 172.21.15.99:0/2922886779 conn(0x7f204806f6e0 msgr2=0x7f204806faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 -- 172.21.15.99:0/2922886779 shutdown_connections 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.867+0000 7f204df2b640 1 -- 172.21.15.99:0/2922886779 wait complete. 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.868+0000 7f204df2b640 1 Processor -- start 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.868+0000 7f204df2b640 1 -- start start 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f204df2b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f204815bab0 0x7f204814c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f204df2b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815be90 0x7f204814f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f204df2b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 0x7f204814ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f204df2b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20480b77a0 con 0x7f204815c2f0 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f204df2b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204815d090 con 0x7f204815bab0 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f204df2b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20480b7920 con 0x7f204815be90 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f2047fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 0x7f204814ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f2047fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 0x7f204814ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:50872/0 (socket says 172.21.15.99:50872) 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f2047fff640 1 -- 172.21.15.99:0/159427760 learned_addr learned my addr 172.21.15.99:0/159427760 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f20477fe640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f204815bab0 0x7f204814c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.869+0000 7f203effd640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815be90 0x7f204814f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2047fff640 1 -- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815be90 msgr2=0x7f204814f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2047fff640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815be90 0x7f204814f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2047fff640 1 -- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f204815bab0 msgr2=0x7f204814c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2047fff640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f204815bab0 0x7f204814c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2047fff640 1 -- 172.21.15.99:0/159427760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204814fde0 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2047fff640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 0x7f204814ce50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f204000ca70 tx=0x7f204000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040017070 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204000abd0 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20481f4060 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040012400 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.870+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20481f4620 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.871+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2048067d20 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.871+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f204001a070 con 0x7f204815c2f0 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.872+0000 7f2044ff9640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f202c06b4d0 0x7f202c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.872+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2040089c30 con 0x7f204815c2f0 2024-03-04T10:10:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.872+0000 7f20477fe640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f202c06b4d0 0x7f202c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.872+0000 7f20477fe640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f202c06b4d0 0x7f202c06d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2038036000 tx=0x7f2038008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:02.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:02.876+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f2048067d20 con 0x7f204815c2f0 2024-03-04T10:10:03.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.057+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2048074f60 con 0x7f204815c2f0 2024-03-04T10:10:03.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.830+0000 7f2044ff9640 1 -- 172.21.15.99:0/159427760 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v87) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2040058610 con 0x7f204815c2f0 2024-03-04T10:10:03.833 INFO:teuthology.orchestra.run.smithi099.stderr: removed all blocklist entries 2024-03-04T10:10:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f202c06b4d0 msgr2=0x7f202c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f202c06b4d0 0x7f202c06d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2038036000 tx=0x7f2038008030 comp rx=0 tx=0).stop 2024-03-04T10:10:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 msgr2=0x7f204814ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 0x7f204814ce50 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f204000ca70 tx=0x7f204000cf30 comp rx=0 tx=0).stop 2024-03-04T10:10:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 shutdown_connections 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f202c06b4d0 0x7f202c06d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f204815c2f0 0x7f204814ce50 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f204815be90 0x7f204814f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 --2- 172.21.15.99:0/159427760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f204815bab0 0x7f204814c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.833+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 >> 172.21.15.99:0/159427760 conn(0x7f204806f6e0 msgr2=0x7f204815ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.834+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 shutdown_connections 2024-03-04T10:10:03.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.834+0000 7f204df2b640 1 -- 172.21.15.99:0/159427760 wait complete. 2024-03-04T10:10:03.849 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-04T10:10:03.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.950+0000 7f4c146e8640 1 Processor -- start 2024-03-04T10:10:03.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.950+0000 7f4c146e8640 1 -- start start 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c146e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4c0c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c146e8640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c06d8b0 con 0x7f4c0c15b410 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c146e8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c06db90 con 0x7f4c0c06d3b0 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c146e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0b2720 con 0x7f4c0c15b030 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c12c85640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f4c0c15b410 0x7f4c0c15b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59528/0 (socket says 172.21.15.99:59528) 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c12c85640 1 -- 172.21.15.99:0/3550606459 learned_addr learned my addr 172.21.15.99:0/3550606459 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c12484640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4c0c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362342187 0 0) 0x7f4c0c06d8b0 con 0x7f4c0c15b410 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.951+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf8003520 con 0x7f4c0c15b410 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231892494 0 0) 0x7f4c0c06db90 con 0x7f4c0c06d3b0 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c06d8b0 con 0x7f4c0c06d3b0 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537316751 0 0) 0x7f4bf8003520 con 0x7f4c0c15b410 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 msgr2=0x7f4c0c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4c0c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 >> v1:172.21.15.162:6789/0 conn(0x7f4c0c06d3b0 legacy=0x7f4c0c15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c06de70 con 0x7f4c0c15b410 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c12484640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4c0c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f4c04001670 con 0x7f4c0c15b410 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 >> v1:172.21.15.99:6789/0 conn(0x7f4c0c15b410 legacy=0x7f4c0c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4bf8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4bf8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bf8004d40 0x7f4bf80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c0b2720 con 0x7f4bf8004100 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf800bb80 con 0x7f4bf8004d40 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf800bd80 con 0x7f4c0c15b030 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11c83640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4bf8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c12484640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4bf8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c12c85640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bf8004d40 0x7f4bf80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11c83640 1 -- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 msgr2=0x7f4bf8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11c83640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c15b030 0x7f4bf8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11c83640 1 -- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bf8004d40 msgr2=0x7f4bf80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11c83640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bf8004d40 0x7f4bf80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c11c83640 1 -- 172.21.15.99:0/3550606459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bf8003520 con 0x7f4bf8004100 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.952+0000 7f4c12c85640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4bf8004d40 0x7f4bf80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:03.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.953+0000 7f4c11c83640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4bf8004800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0018d0 tx=0x7f4bf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f55358289059dca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.953+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc037070 con 0x7f4bf8004100 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.953+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc0052d0 con 0x7f4bf8004100 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.953+0000 7f4c11482640 1 -- 172.21.15.99:0/3550606459 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0014f0 con 0x7f4bf8004100 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c146e8640 1 -- 172.21.15.99:0/3550606459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 msgr2=0x7f4bf8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c146e8640 1 --2- 172.21.15.99:0/3550606459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4bf8004800 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0018d0 tx=0x7f4bf8005830 comp rx=0 tx=0).stop 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c12c85640 1 -- 172.21.15.99:0/3550606459 reap_dead start 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c146e8640 1 -- 172.21.15.99:0/3550606459 shutdown_connections 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c146e8640 1 -- 172.21.15.99:0/3550606459 >> 172.21.15.99:0/3550606459 conn(0x7f4c0c06acd0 msgr2=0x7f4c0c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c146e8640 1 -- 172.21.15.99:0/3550606459 shutdown_connections 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.954+0000 7f4c146e8640 1 -- 172.21.15.99:0/3550606459 wait complete. 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.955+0000 7f4c146e8640 1 Processor -- start 2024-03-04T10:10:03.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.955+0000 7f4c146e8640 1 -- start start 2024-03-04T10:10:03.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.955+0000 7f4c146e8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4c0c1411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c0c1f1aa0 0x7f4c0c1f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c071350 0x7f4c0c0717d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c06e810 con 0x7f4bf8004100 2024-03-04T10:10:03.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c06e690 con 0x7f4c0c1f1aa0 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c15c120 con 0x7f4c0c071350 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4c0c1411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4c0c1411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45332/0 (socket says 172.21.15.99:45332) 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 -- 172.21.15.99:0/324750606 learned_addr learned my addr 172.21.15.99:0/324750606 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c11c83640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c0c1f1aa0 0x7f4c0c1f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12c85640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c071350 0x7f4c0c0717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 -- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c071350 msgr2=0x7f4c0c0717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c071350 0x7f4c0c0717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 -- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c0c1f1aa0 msgr2=0x7f4c0c1f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c0c1f1aa0 0x7f4c0c1f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 -- 172.21.15.99:0/324750606 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c1f25b0 con 0x7f4bf8004100 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c11c83640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c0c1f1aa0 0x7f4c0c1f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:03.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c12484640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4c0c1411e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f4c08011e30 tx=0x7f4c0800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08015630 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c080157d0 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0c1f2890 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08015bc0 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.956+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c151630 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.957+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c071d10 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.958+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4c0801d400 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.958+0000 7f4c02ffd640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4bf006b500 0x7f4bf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.958+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4c0808aff0 con 0x7f4bf8004100 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.958+0000 7f4c11c83640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4bf006b500 0x7f4bf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:03.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.959+0000 7f4c11c83640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4bf006b500 0x7f4bf006d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc034000 tx=0x7f4bfc031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:03.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:03.962+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4c08019030 con 0x7f4bf8004100 2024-03-04T10:10:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.199+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4c0c151140 con 0x7f4bf8004100 2024-03-04T10:10:04.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.199+0000 7f4c02ffd640 1 -- 172.21.15.99:0/324750606 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4c08010040 con 0x7f4bf8004100 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "auth_dump": [ 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.a", 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlxGqUFxAAlHochSpUgc2FjY/g8y+dLg==", 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:10:04.202 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.b", 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlcd32IxAAGd9qDPQIYVX4AsqAnvYCvw==", 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.c", 2024-03-04T10:10:04.203 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVleJzzHhAA/HL5aHVoHqOveMeRDIdGFg==", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mds.d", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlJ7NdKxAAJxymFqv0BLhFl8D2ZhoYYw==", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:10:04.204 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.0", 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBFnOVlU2vDEBAAVv4SLy6mwI/Vo5GF7ELOVg==", 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.205 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.1", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBInOVlA9vKDxAAHUbopSKA8nrKJPK0gsd1zQ==", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.2", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBLnOVl5CjyDRAAmxk6V/1a8AYU9VZxvMD+Vw==", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.206 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.3", 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBOnOVlOEepDBAAjpu7Khf4udD5FnXN0XxKeA==", 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.4", 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBUnOVl21CkExAAR++XGwqDjMxkbel7NG8/Sw==", 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.207 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.5", 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBXnOVlKwHeERAAEKlon8wCE3RsoNW1MNJtpg==", 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.208 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.6", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBanOVl7Nx6DxAAyBWMocaCUaVVnXwsHscykg==", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "osd.7", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBdnOVlw45yDRAATtEJceHpN8hyzay/QMvlDA==", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow profile osd", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile osd", 2024-03-04T10:10:04.209 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.0", 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ==", 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow r", 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.1", 2024-03-04T10:10:04.210 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlL/jiOBAAVRA0AVAOEX/35/TUM7PFWA==", 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.2", 2024-03-04T10:10:04.211 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlyql5NRAA32AFQwjud15iyjepkl45Vg==", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.3", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBCnOVlusQsAxAAJNgNar5c2zN4SOylY9a5qQ==", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow r", 2024-03-04T10:10:04.212 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow rw", 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow rwx" 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "client.admin", 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBAnOVlV1MhNhAAJTJsIc9qGmeQBnHKjrkPvw==", 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "mgr": "allow *", 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow *", 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.213 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.x", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVlMF4uFBAA16IsvezOb7xJSICREFCJdw==", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: }, 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "entity": "mgr.y", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "key": "AQBBnOVl3XTmDxAAxQUIEH4iLI40XgH5HB3FPQ==", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "caps": { 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "mds": "allow *", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "mon": "allow profile mgr", 2024-03-04T10:10:04.214 INFO:teuthology.orchestra.run.smithi099.stdout: "osd": "allow *" 2024-03-04T10:10:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:04.215 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-03-04T10:10:04.215 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:10:04.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4bf006b500 msgr2=0x7f4bf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4bf006b500 0x7f4bf006d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc034000 tx=0x7f4bfc031000 comp rx=0 tx=0).stop 2024-03-04T10:10:04.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 msgr2=0x7f4c0c1411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4c0c1411e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f4c08011e30 tx=0x7f4c0800f580 comp rx=0 tx=0).stop 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 shutdown_connections 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4bf006b500 0x7f4bf006d9c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4c0c071350 0x7f4c0c0717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4c0c1f1aa0 0x7f4c0c1f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 --2- 172.21.15.99:0/324750606 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4bf8004100 0x7f4c0c1411e0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 >> 172.21.15.99:0/324750606 conn(0x7f4c0c06acd0 msgr2=0x7f4c0c0cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 shutdown_connections 2024-03-04T10:10:04.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.201+0000 7f4c146e8640 1 -- 172.21.15.99:0/324750606 wait complete. 2024-03-04T10:10:04.218 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:04.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.314+0000 7f891ed05640 1 Processor -- start 2024-03-04T10:10:04.317 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.314+0000 7f891ed05640 1 -- start start 2024-03-04T10:10:04.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f891ed05640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f891815dcd0 0x7f89180ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f891ed05640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f891ed05640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89180cf200 con 0x7f89180cfeb0 2024-03-04T10:10:04.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f891ed05640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89180cc0c0 con 0x7f891815dcd0 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f891ed05640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89180ce680 con 0x7f8918161fb0 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f891caa1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f891815dcd0 0x7f89180ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f8917fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f89180cfeb0 0x7f89181618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59544/0 (socket says 172.21.15.99:59544) 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.315+0000 7f8917fff640 1 -- 172.21.15.99:0/1858464946 learned_addr learned my addr 172.21.15.99:0/1858464946 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.316+0000 7f891d2a2640 1 --2- 172.21.15.99:0/1858464946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.316+0000 7f891caa1640 1 -- 172.21.15.99:0/1858464946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 msgr2=0x7f89180cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.316+0000 7f891caa1640 1 --2- 172.21.15.99:0/1858464946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.316+0000 7f891caa1640 1 -- 172.21.15.99:0/1858464946 >> v1:172.21.15.99:6789/0 conn(0x7f89180cfeb0 legacy=0x7f89181618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.316+0000 7f891caa1640 1 -- 172.21.15.99:0/1858464946 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89180ce960 con 0x7f891815dcd0 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.316+0000 7f891caa1640 1 --2- 172.21.15.99:0/1858464946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180ce0a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f89080097a0 tx=0x7f890802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b55118ece955927 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f89177fe640 1 -- 172.21.15.99:0/1858464946 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908035020 con 0x7f891815dcd0 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f89177fe640 1 -- 172.21.15.99:0/1858464946 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8908031830 con 0x7f891815dcd0 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f89177fe640 1 -- 172.21.15.99:0/1858464946 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908031c00 con 0x7f891815dcd0 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 -- 172.21.15.99:0/1858464946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 msgr2=0x7f89180ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 --2- 172.21.15.99:0/1858464946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180ce0a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f89080097a0 tx=0x7f890802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 -- 172.21.15.99:0/1858464946 shutdown_connections 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 --2- 172.21.15.99:0/1858464946 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 --2- 172.21.15.99:0/1858464946 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180ce0a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 -- 172.21.15.99:0/1858464946 >> 172.21.15.99:0/1858464946 conn(0x7f89180b3480 msgr2=0x7f89180b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 -- 172.21.15.99:0/1858464946 shutdown_connections 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.317+0000 7f891ed05640 1 -- 172.21.15.99:0/1858464946 wait complete. 2024-03-04T10:10:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.318+0000 7f891ed05640 1 Processor -- start 2024-03-04T10:10:04.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.318+0000 7f891ed05640 1 -- start start 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891ed05640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89180cfeb0 0x7f891814eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891ed05640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891ed05640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891ed05640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f891806bff0 con 0x7f891815dcd0 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891ed05640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89180cef20 con 0x7f89180cfeb0 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891ed05640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891815e0b0 con 0x7f8918161fb0 2024-03-04T10:10:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f8917fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f8917fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180c9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45346/0 (socket says 172.21.15.99:45346) 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f8917fff640 1 -- 172.21.15.99:0/1668322273 learned_addr learned my addr 172.21.15.99:0/1668322273 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891caa1640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89180cfeb0 0x7f891814eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.319+0000 7f891d2a2640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f8917fff640 1 -- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 msgr2=0x7f89180c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f8917fff640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f8917fff640 1 -- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89180cfeb0 msgr2=0x7f891814eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f8917fff640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89180cfeb0 0x7f891814eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f8917fff640 1 -- 172.21.15.99:0/1668322273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89180caca0 con 0x7f891815dcd0 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f891caa1640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89180cfeb0 0x7f891814eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f891d2a2640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f8917fff640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180c9a50 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f890c00c3c0 tx=0x7f890c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c017070 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c005ce0 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89180c7f50 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c007310 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.320+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8918159100 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.321+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8918074f60 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.321+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f890c01a070 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.322+0000 7f89157fa640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f890406b4d0 0x7f890406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.322+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f890c0893e0 con 0x7f891815dcd0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.322+0000 7f891caa1640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f890406b4d0 0x7f890406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.322+0000 7f891caa1640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f890406b4d0 0x7f890406d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8908036000 tx=0x7f890803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.326+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f890c058420 con 0x7f891815dcd0 2024-03-04T10:10:04.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.497+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89180b78b0 con 0x7f891815dcd0 2024-03-04T10:10:04.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.498+0000 7f89157fa640 1 -- 172.21.15.99:0/1668322273 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+13129 (secure 0 0 0) 0x7f890c057d20 con 0x7f891815dcd0 2024-03-04T10:10:04.500 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:04.500 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":89,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:10:03.837997+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f890406b4d0 msgr2=0x7f890406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f890406b4d0 0x7f890406d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8908036000 tx=0x7f890803c040 comp rx=0 tx=0).stop 2024-03-04T10:10:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 msgr2=0x7f89180c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180c9a50 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f890c00c3c0 tx=0x7f890c00c880 comp rx=0 tx=0).stop 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 shutdown_connections 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f890406b4d0 0x7f890406d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8918161fb0 0x7f89180c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f891815dcd0 0x7f89180c9a50 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 --2- 172.21.15.99:0/1668322273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89180cfeb0 0x7f891814eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 >> 172.21.15.99:0/1668322273 conn(0x7f89180b3480 msgr2=0x7f89180d1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 shutdown_connections 2024-03-04T10:10:04.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.500+0000 7f891ed05640 1 -- 172.21.15.99:0/1668322273 wait complete. 2024-03-04T10:10:04.513 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:10:04.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:10:04.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.614+0000 7f500bd7e640 1 Processor -- start 2024-03-04T10:10:04.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.614+0000 7f500bd7e640 1 -- start start 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f500bd7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50040cfc70 0x7f50040cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f500bd7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f50040cc340 0x7f50040b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f500bd7e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50040cebd0 con 0x7f50041450e0 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f500bd7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50040cf7e0 con 0x7f50040cc340 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f500bd7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cc780 con 0x7f50040cfc70 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f500a31b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f50040cc340 0x7f50040b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f5009b1a640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f50041450e0 0x7f500415c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59558/0 (socket says 172.21.15.99:59558) 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.615+0000 7f5009b1a640 1 -- 172.21.15.99:0/288989291 learned_addr learned my addr 172.21.15.99:0/288989291 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f5009319640 1 --2- 172.21.15.99:0/288989291 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50040cfc70 0x7f50040cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f500a31b640 1 -- 172.21.15.99:0/288989291 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50040cfc70 msgr2=0x7f50040cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f500a31b640 1 --2- 172.21.15.99:0/288989291 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50040cfc70 0x7f50040cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f500a31b640 1 -- 172.21.15.99:0/288989291 >> v1:172.21.15.99:6789/0 conn(0x7f50041450e0 legacy=0x7f500415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f5008b18640 1 -- 172.21.15.99:0/288989291 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828617137 0 0) 0x7f50040cebd0 con 0x7f50041450e0 2024-03-04T10:10:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f500a31b640 1 -- 172.21.15.99:0/288989291 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50040ceeb0 con 0x7f50040cc340 2024-03-04T10:10:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f500a31b640 1 --2- 172.21.15.99:0/288989291 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040b3250 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0097a0 tx=0x7f4ffc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed58ad92fc782b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f5008b18640 1 -- 172.21.15.99:0/288989291 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc035020 con 0x7f50040cc340 2024-03-04T10:10:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f5008b18640 1 -- 172.21.15.99:0/288989291 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ffc031830 con 0x7f50040cc340 2024-03-04T10:10:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.616+0000 7f5008b18640 1 -- 172.21.15.99:0/288989291 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc031c00 con 0x7f50040cc340 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 -- 172.21.15.99:0/288989291 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 msgr2=0x7f50040b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 --2- 172.21.15.99:0/288989291 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040b3250 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0097a0 tx=0x7f4ffc02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 -- 172.21.15.99:0/288989291 shutdown_connections 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 --2- 172.21.15.99:0/288989291 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040b3250 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 --2- 172.21.15.99:0/288989291 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50040cfc70 0x7f50040cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 -- 172.21.15.99:0/288989291 >> 172.21.15.99:0/288989291 conn(0x7f500406acd0 msgr2=0x7f500406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 -- 172.21.15.99:0/288989291 shutdown_connections 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.617+0000 7f500bd7e640 1 -- 172.21.15.99:0/288989291 wait complete. 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.618+0000 7f500bd7e640 1 Processor -- start 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.618+0000 7f500bd7e640 1 -- start start 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500bd7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500bd7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50040cfc70 0x7f500413c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500bd7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50041450e0 0x7f500413ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500bd7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50041454c0 con 0x7f50040cc340 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500bd7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50040cf470 con 0x7f50040cfc70 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500bd7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50040ccd10 con 0x7f50041450e0 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45364/0 (socket says 172.21.15.99:45364) 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 -- 172.21.15.99:0/886623678 learned_addr learned my addr 172.21.15.99:0/886623678 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f500a31b640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50041450e0 0x7f500413ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009319640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50040cfc70 0x7f500413c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 -- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50041450e0 msgr2=0x7f500413ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50041450e0 0x7f500413ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 -- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50040cfc70 msgr2=0x7f500413c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50040cfc70 0x7f500413c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.619+0000 7f5009b1a640 1 -- 172.21.15.99:0/886623678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500406f3d0 con 0x7f50040cc340 2024-03-04T10:10:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.620+0000 7f5009b1a640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040adb90 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f500000c7c0 tx=0x7f500000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.620+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5000018070 con 0x7f50040cc340 2024-03-04T10:10:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.620+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500000a640 con 0x7f50040cc340 2024-03-04T10:10:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.620+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500406ee10 con 0x7f50040cc340 2024-03-04T10:10:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.620+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500000aa30 con 0x7f50040cc340 2024-03-04T10:10:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.620+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50040ad390 con 0x7f50040cc340 2024-03-04T10:10:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.621+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5004074f60 con 0x7f50040cc340 2024-03-04T10:10:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.621+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f500000abd0 con 0x7f50040cc340 2024-03-04T10:10:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.621+0000 7f4ffa7fc640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4fe806b500 0x7f4fe806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.621+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5000089590 con 0x7f50040cc340 2024-03-04T10:10:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.622+0000 7f5009319640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4fe806b500 0x7f4fe806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.622+0000 7f5009319640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4fe806b500 0x7f4fe806d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f500415f0d0 tx=0x7f4ff4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.626+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f50000585e0 con 0x7f50040cc340 2024-03-04T10:10:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.794+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50040b4350 con 0x7f50040cc340 2024-03-04T10:10:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.795+0000 7f4ffa7fc640 1 -- 172.21.15.99:0/886623678 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5000057ee0 con 0x7f50040cc340 2024-03-04T10:10:04.797 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:10:04.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4fe806b500 msgr2=0x7f4fe806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4fe806b500 0x7f4fe806d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f500415f0d0 tx=0x7f4ff4008040 comp rx=0 tx=0).stop 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 msgr2=0x7f50040adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040adb90 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f500000c7c0 tx=0x7f500000cc80 comp rx=0 tx=0).stop 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 shutdown_connections 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4fe806b500 0x7f4fe806d9c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f50041450e0 0x7f500413ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f50040cfc70 0x7f500413c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 --2- 172.21.15.99:0/886623678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50040cc340 0x7f50040adb90 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 >> 172.21.15.99:0/886623678 conn(0x7f500406acd0 msgr2=0x7f500406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 shutdown_connections 2024-03-04T10:10:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.797+0000 7f500bd7e640 1 -- 172.21.15.99:0/886623678 wait complete. 2024-03-04T10:10:04.810 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T10:10:04.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.908+0000 7fe931261640 1 Processor -- start 2024-03-04T10:10:04.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.909+0000 7fe931261640 1 -- start start 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.909+0000 7fe931261640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe92c0cefb0 0x7fe92c06e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.909+0000 7fe931261640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0afb30 0x7fe92c06ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.909+0000 7fe931261640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0d1550 con 0x7fe92c15abe0 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.909+0000 7fe931261640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0cbfa0 con 0x7fe92c0cefb0 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.909+0000 7fe931261640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0cc280 con 0x7fe92c0afb30 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe92c0cefb0 0x7fe92c06e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe92c0cefb0 0x7fe92c06e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45366/0 (socket says 172.21.15.99:45366) 2024-03-04T10:10:04.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 -- 172.21.15.99:0/22789077 learned_addr learned my addr 172.21.15.99:0/22789077 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 -- 172.21.15.99:0/22789077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0afb30 msgr2=0x7fe92c06ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92b7fe640 1 --2- 172.21.15.99:0/22789077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0afb30 0x7fe92c06ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 --2- 172.21.15.99:0/22789077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0afb30 0x7fe92c06ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 -- 172.21.15.99:0/22789077 >> v1:172.21.15.162:6789/0 conn(0x7fe92c15abe0 legacy=0x7fe92c0ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.910+0000 7fe92a7fc640 1 -- 172.21.15.99:0/22789077 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c0cc560 con 0x7fe92c0cefb0 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe92a7fc640 1 --2- 172.21.15.99:0/22789077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c0cefb0 0x7fe92c06e740 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fe920009e50 tx=0x7fe92002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6691489b7e2840e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe929ffb640 1 -- 172.21.15.99:0/22789077 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92002ca70 con 0x7fe92c0cefb0 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe929ffb640 1 -- 172.21.15.99:0/22789077 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92002cc10 con 0x7fe92c0cefb0 2024-03-04T10:10:04.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe929ffb640 1 -- 172.21.15.99:0/22789077 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200343f0 con 0x7fe92c0cefb0 2024-03-04T10:10:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 -- 172.21.15.99:0/22789077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c0cefb0 msgr2=0x7fe92c06e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 --2- 172.21.15.99:0/22789077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c0cefb0 0x7fe92c06e740 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fe920009e50 tx=0x7fe92002bed0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 -- 172.21.15.99:0/22789077 shutdown_connections 2024-03-04T10:10:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 --2- 172.21.15.99:0/22789077 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0afb30 0x7fe92c06ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 --2- 172.21.15.99:0/22789077 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c0cefb0 0x7fe92c06e740 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 -- 172.21.15.99:0/22789077 >> 172.21.15.99:0/22789077 conn(0x7fe92c0d0710 msgr2=0x7fe92c0d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 -- 172.21.15.99:0/22789077 shutdown_connections 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.911+0000 7fe931261640 1 -- 172.21.15.99:0/22789077 wait complete. 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 Processor -- start 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 -- start start 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c0afb30 0x7fe92c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0cefb0 0x7fe92c09c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 0x7fe92c09cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0ce2e0 con 0x7fe92c15abe0 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0d1200 con 0x7fe92c0afb30 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe931261640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0ce460 con 0x7fe92c0cefb0 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe92b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 0x7fe92c09cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe92b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 0x7fe92c09cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45368/0 (socket says 172.21.15.99:45368) 2024-03-04T10:10:04.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.913+0000 7fe92b7fe640 1 -- 172.21.15.99:0/243754640 learned_addr learned my addr 172.21.15.99:0/243754640 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92b7fe640 1 -- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0cefb0 msgr2=0x7fe92c09c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92b7fe640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0cefb0 0x7fe92c09c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92affd640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c0afb30 0x7fe92c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92b7fe640 1 -- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c0afb30 msgr2=0x7fe92c151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92b7fe640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c0afb30 0x7fe92c151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92b7fe640 1 -- 172.21.15.99:0/243754640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c0b64a0 con 0x7fe92c15abe0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92affd640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c0afb30 0x7fe92c151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe92b7fe640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 0x7fe92c09cad0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00c7f0 tx=0x7fe91c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c015070 con 0x7fe92c15abe0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c004220 con 0x7fe92c15abe0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c0b41c0 con 0x7fe92c15abe0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c007620 con 0x7fe92c15abe0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.914+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c157f70 con 0x7fe92c15abe0 2024-03-04T10:10:04.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.915+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c074f60 con 0x7fe92c15abe0 2024-03-04T10:10:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.916+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe91c0077c0 con 0x7fe92c15abe0 2024-03-04T10:10:04.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.916+0000 7fe9137fe640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe91806b4d0 0x7fe91806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.916+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe91c089dc0 con 0x7fe92c15abe0 2024-03-04T10:10:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.916+0000 7fe92affd640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe91806b4d0 0x7fe91806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:04.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.917+0000 7fe92affd640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe91806b4d0 0x7fe91806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe92400a830 tx=0x7fe924008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:04.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:04.920+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe91c058e90 con 0x7fe92c15abe0 2024-03-04T10:10:05.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.091+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe92c1596f0 con 0x7fe92c15abe0 2024-03-04T10:10:05.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.092+0000 7fe9137fe640 1 -- 172.21.15.99:0/243754640 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe91c058790 con 0x7fe92c15abe0 2024-03-04T10:10:05.094 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:05.095 INFO:teuthology.orchestra.run.smithi099.stdout:[ 2024-03-04T10:10:05.095 INFO:teuthology.orchestra.run.smithi099.stdout: { 2024-03-04T10:10:05.095 INFO:teuthology.orchestra.run.smithi099.stdout: "poolnum": 1, 2024-03-04T10:10:05.095 INFO:teuthology.orchestra.run.smithi099.stdout: "poolname": ".mgr" 2024-03-04T10:10:05.095 INFO:teuthology.orchestra.run.smithi099.stdout: } 2024-03-04T10:10:05.095 INFO:teuthology.orchestra.run.smithi099.stdout:] 2024-03-04T10:10:05.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe91806b4d0 msgr2=0x7fe91806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe91806b4d0 0x7fe91806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe92400a830 tx=0x7fe924008040 comp rx=0 tx=0).stop 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 msgr2=0x7fe92c09cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 0x7fe92c09cad0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00c7f0 tx=0x7fe91c00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 shutdown_connections 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe91806b4d0 0x7fe91806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe92c15abe0 0x7fe92c09cad0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe92c0cefb0 0x7fe92c09c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 --2- 172.21.15.99:0/243754640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe92c0afb30 0x7fe92c151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 >> 172.21.15.99:0/243754640 conn(0x7fe92c0d0710 msgr2=0x7fe92c0b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 shutdown_connections 2024-03-04T10:10:05.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.094+0000 7fe931261640 1 -- 172.21.15.99:0/243754640 wait complete. 2024-03-04T10:10:05.107 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T10:10:05.107 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T10:10:05.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.208+0000 7f3db5881640 1 Processor -- start 2024-03-04T10:10:05.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.208+0000 7f3db5881640 1 -- start start 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3db5881640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3db015dff0 0x7f3db00cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3db5881640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 0x7f3db00cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3db5881640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db00d0000 con 0x7f3db00afb30 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3db5881640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3db00cb310 con 0x7f3db015dff0 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3db5881640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00d02e0 con 0x7f3db006b3e0 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3dae7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3db00afb30 0x7f3db006acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59570/0 (socket says 172.21.15.99:59570) 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3dae7fc640 1 -- 172.21.15.99:0/2552479971 learned_addr learned my addr 172.21.15.99:0/2552479971 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3daf7fe640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 0x7f3db00cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3daeffd640 1 --2- 172.21.15.99:0/2552479971 >> v2:172.21.15.162:3300/0 conn(0x7f3db015dff0 0x7f3db00cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305998240 0 0) 0x7f3db00d0000 con 0x7f3db00afb30 2024-03-04T10:10:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.209+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d98003520 con 0x7f3db00afb30 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1545452758 0 0) 0x7f3d98003520 con 0x7f3db00afb30 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 msgr2=0x7f3db00cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 0x7f3db00cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 >> v2:172.21.15.162:3300/0 conn(0x7f3db015dff0 msgr2=0x7f3db00cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 --2- 172.21.15.99:0/2552479971 >> v2:172.21.15.162:3300/0 conn(0x7f3db015dff0 0x7f3db00cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00cafa0 con 0x7f3db00afb30 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3da4001670 con 0x7f3db00afb30 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 >> v1:172.21.15.99:6789/0 conn(0x7f3db00afb30 legacy=0x7f3db006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d980039b0 0x7f3d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d98004800 0x7f3d98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 0x7f3d980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00cb310 con 0x7f3d980039b0 2024-03-04T10:10:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00d02e0 con 0x7f3d98004800 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9800b9c0 con 0x7f3db006b3e0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daeffd640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d980039b0 0x7f3d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dae7fc640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d98004800 0x7f3d98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daf7fe640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 0x7f3d980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daeffd640 1 -- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 msgr2=0x7f3d980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daeffd640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3db006b3e0 0x7f3d980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daeffd640 1 -- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d98004800 msgr2=0x7f3d98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daeffd640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d98004800 0x7f3d98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3daeffd640 1 -- 172.21.15.99:0/2552479971 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d98003520 con 0x7f3d980039b0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.210+0000 7f3dae7fc640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3d98004800 0x7f3d98004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.211+0000 7f3daeffd640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d980039b0 0x7f3d98003dd0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009fc0 tx=0x7f3d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c63350638ec5c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.211+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0039070 con 0x7f3d980039b0 2024-03-04T10:10:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.211+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da002cc60 con 0x7f3d980039b0 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.211+0000 7f3dadffb640 1 -- 172.21.15.99:0/2552479971 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0009c10 con 0x7f3d980039b0 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3db5881640 1 -- 172.21.15.99:0/2552479971 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d980039b0 msgr2=0x7f3d98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3db5881640 1 --2- 172.21.15.99:0/2552479971 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3d980039b0 0x7f3d98003dd0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009fc0 tx=0x7f3d98005830 comp rx=0 tx=0).stop 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3daf7fe640 1 -- 172.21.15.99:0/2552479971 reap_dead start 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3db5881640 1 -- 172.21.15.99:0/2552479971 shutdown_connections 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3db5881640 1 -- 172.21.15.99:0/2552479971 >> 172.21.15.99:0/2552479971 conn(0x7f3db00b22e0 msgr2=0x7f3db00b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3db5881640 1 -- 172.21.15.99:0/2552479971 shutdown_connections 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.212+0000 7f3db5881640 1 -- 172.21.15.99:0/2552479971 wait complete. 2024-03-04T10:10:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.213+0000 7f3db5881640 1 Processor -- start 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.213+0000 7f3db5881640 1 -- start start 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3db5881640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d980039b0 0x7f3db0158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3db5881640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 0x7f3db01e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3db5881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3db01e9250 0x7f3db00c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3db5881640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00cb7a0 con 0x7f3db01e8890 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3db5881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00cfcb0 con 0x7f3db01e9250 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3db5881640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db00cb920 con 0x7f3d980039b0 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 0x7f3db01e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 0x7f3db01e8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45386/0 (socket says 172.21.15.99:45386) 2024-03-04T10:10:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 -- 172.21.15.99:0/800181236 learned_addr learned my addr 172.21.15.99:0/800181236 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3daeffd640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d980039b0 0x7f3db0158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3daf7fe640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3db01e9250 0x7f3db00c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 -- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d980039b0 msgr2=0x7f3db0158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d980039b0 0x7f3db0158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 -- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3db01e9250 msgr2=0x7f3db00c5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3db01e9250 0x7f3db00c5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 -- 172.21.15.99:0/800181236 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db01e96d0 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.214+0000 7f3dae7fc640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 0x7f3db01e8d10 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3da400e730 tx=0x7f3da400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.215+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da400ee40 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.215+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3da4018070 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.215+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db00c4ca0 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.215+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da400cb10 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.215+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db00c5260 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.216+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db0074f60 con 0x7f3db01e8890 2024-03-04T10:10:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.216+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3da400ccb0 con 0x7f3db01e8890 2024-03-04T10:10:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.216+0000 7f3d97fff640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d9c06b4d0 0x7f3d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.216+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3da4089dc0 con 0x7f3db01e8890 2024-03-04T10:10:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.216+0000 7f3daeffd640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d9c06b4d0 0x7f3d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.217+0000 7f3daeffd640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d9c06b4d0 0x7f3d9c06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3da0036000 tx=0x7f3da002e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.221+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3da4058e90 con 0x7f3db01e8890 2024-03-04T10:10:05.394 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.391+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3db00b6750 con 0x7f3db01e8890 2024-03-04T10:10:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.835+0000 7f3d97fff640 1 -- 172.21.15.99:0/800181236 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3da4058790 con 0x7f3db01e8890 2024-03-04T10:10:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' created 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d9c06b4d0 msgr2=0x7f3d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d9c06b4d0 0x7f3d9c06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3da0036000 tx=0x7f3da002e000 comp rx=0 tx=0).stop 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 msgr2=0x7f3db01e8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 0x7f3db01e8d10 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3da400e730 tx=0x7f3da400ebf0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 shutdown_connections 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3d9c06b4d0 0x7f3d9c06d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3db01e9250 0x7f3db00c5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3db01e8890 0x7f3db01e8d10 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 --2- 172.21.15.99:0/800181236 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3d980039b0 0x7f3db0158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 >> 172.21.15.99:0/800181236 conn(0x7f3db00b22e0 msgr2=0x7f3db006ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 shutdown_connections 2024-03-04T10:10:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.837+0000 7f3db5881640 1 -- 172.21.15.99:0/800181236 wait complete. 2024-03-04T10:10:05.856 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T10:10:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.964+0000 7fdfd2df4640 1 Processor -- start 2024-03-04T10:10:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.964+0000 7fdfd2df4640 1 -- start start 2024-03-04T10:10:05.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.965+0000 7fdfd2df4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fdfcc0b5700 0x7fdfcc0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.965+0000 7fdfd2df4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 0x7fdfcc06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.965+0000 7fdfd2df4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc0d2400 con 0x7fdfcc0b9410 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.965+0000 7fdfd2df4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfcc0d26e0 con 0x7fdfcc0b5700 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.965+0000 7fdfd2df4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0d29c0 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfd0b90640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fdfcc0b5700 0x7fdfcc0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcbfff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 0x7fdfcc06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfd1391640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fdfcc0b9410 0x7fdfcc06c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:35864/0 (socket says 172.21.15.99:35864) 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcbfff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 0x7fdfcc06c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:34584/0 (socket says 172.21.15.99:34584) 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfd1391640 1 -- 172.21.15.99:0/2066230511 learned_addr learned my addr 172.21.15.99:0/2066230511 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcb7fe640 1 -- 172.21.15.99:0/2066230511 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106173684 0 0) 0x7fdfcc0d2400 con 0x7fdfcc0b9410 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcb7fe640 1 -- 172.21.15.99:0/2066230511 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb4003520 con 0x7fdfcc0b9410 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcbfff640 1 -- 172.21.15.99:0/2066230511 >> v2:172.21.15.162:3300/0 conn(0x7fdfcc0b5700 msgr2=0x7fdfcc0d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcbfff640 1 --2- 172.21.15.99:0/2066230511 >> v2:172.21.15.162:3300/0 conn(0x7fdfcc0b5700 0x7fdfcc0d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcbfff640 1 -- 172.21.15.99:0/2066230511 >> v1:172.21.15.162:6789/0 conn(0x7fdfcc0b9410 legacy=0x7fdfcc06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfcbfff640 1 -- 172.21.15.99:0/2066230511 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0d2ca0 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.966+0000 7fdfd0b90640 1 --2- 172.21.15.99:0/2066230511 >> v2:172.21.15.162:3300/0 conn(0x7fdfcc0b5700 0x7fdfcc0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.967+0000 7fdfcbfff640 1 --2- 172.21.15.99:0/2066230511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 0x7fdfcc06c0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00097a0 tx=0x7fdfc002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=676cfc11aa7f83e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.967+0000 7fdfcb7fe640 1 -- 172.21.15.99:0/2066230511 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc00315a0 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.967+0000 7fdfcb7fe640 1 -- 172.21.15.99:0/2066230511 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc0037070 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.967+0000 7fdfcb7fe640 1 -- 172.21.15.99:0/2066230511 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0031940 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 -- 172.21.15.99:0/2066230511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 msgr2=0x7fdfcc06c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/2066230511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 0x7fdfcc06c0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00097a0 tx=0x7fdfc002b840 comp rx=0 tx=0).stop 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 -- 172.21.15.99:0/2066230511 shutdown_connections 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/2066230511 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0d34a0 0x7fdfcc06c0e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/2066230511 >> v2:172.21.15.162:3300/0 conn(0x7fdfcc0b5700 0x7fdfcc0d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 -- 172.21.15.99:0/2066230511 >> 172.21.15.99:0/2066230511 conn(0x7fdfcc0d0490 msgr2=0x7fdfcc0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 -- 172.21.15.99:0/2066230511 shutdown_connections 2024-03-04T10:10:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.968+0000 7fdfd2df4640 1 -- 172.21.15.99:0/2066230511 wait complete. 2024-03-04T10:10:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.969+0000 7fdfd2df4640 1 Processor -- start 2024-03-04T10:10:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.969+0000 7fdfd2df4640 1 -- start start 2024-03-04T10:10:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfcc0b5700 0x7fdfcc1f3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0b9410 0x7fdfcc156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 0x7fdfcc1586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc06ceb0 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc06cd30 con 0x7fdfcc0b5700 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc0b5ae0 con 0x7fdfcc0b9410 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 0x7fdfcc1586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 0x7fdfcc1586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45394/0 (socket says 172.21.15.99:45394) 2024-03-04T10:10:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 -- 172.21.15.99:0/3086599566 learned_addr learned my addr 172.21.15.99:0/3086599566 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd0b90640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfcc0b5700 0x7fdfcc1f3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfcbfff640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0b9410 0x7fdfcc156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 -- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0b9410 msgr2=0x7fdfcc156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0b9410 0x7fdfcc156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 -- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfcc0b5700 msgr2=0x7fdfcc1f3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfcc0b5700 0x7fdfcc1f3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 -- 172.21.15.99:0/3086599566 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc0cc440 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd1391640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 0x7fdfcc1586c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400cb70 tx=0x7fdfc400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4017070 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc400ace0 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc0cc720 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4012620 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.970+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc1590d0 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.971+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc0ccb10 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.972+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdfc401a070 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.972+0000 7fdfc97fa640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfb806b4d0 0x7fdfb806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.972+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fdfc4089f80 con 0x7fdfcc0d34a0 2024-03-04T10:10:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.973+0000 7fdfd0b90640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfb806b4d0 0x7fdfb806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.973+0000 7fdfd0b90640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfb806b4d0 0x7fdfb806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009560 tx=0x7fdfbc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:05.977+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdfc4058800 con 0x7fdfcc0d34a0 2024-03-04T10:10:06.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.146+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdfcc074f60 con 0x7fdfcc0d34a0 2024-03-04T10:10:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.841+0000 7fdfc97fa640 1 -- 172.21.15.99:0/3086599566 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7fdfc405c8a0 con 0x7fdfcc0d34a0 2024-03-04T10:10:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' created 2024-03-04T10:10:06.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.845+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfb806b4d0 msgr2=0x7fdfb806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:06.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.845+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfb806b4d0 0x7fdfb806d990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009560 tx=0x7fdfbc009190 comp rx=0 tx=0).stop 2024-03-04T10:10:06.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.845+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 msgr2=0x7fdfcc1586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:06.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.845+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 0x7fdfcc1586c0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400cb70 tx=0x7fdfc400a430 comp rx=0 tx=0).stop 2024-03-04T10:10:06.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.849+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 shutdown_connections 2024-03-04T10:10:06.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.849+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfb806b4d0 0x7fdfb806d990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.849+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdfcc0d34a0 0x7fdfcc1586c0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.849+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdfcc0b9410 0x7fdfcc156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.849+0000 7fdfd2df4640 1 --2- 172.21.15.99:0/3086599566 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdfcc0b5700 0x7fdfcc1f3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.849+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 >> 172.21.15.99:0/3086599566 conn(0x7fdfcc0d0490 msgr2=0x7fdfcc0b6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:06.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.850+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 shutdown_connections 2024-03-04T10:10:06.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.850+0000 7fdfd2df4640 1 -- 172.21.15.99:0/3086599566 wait complete. 2024-03-04T10:10:06.873 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T10:10:06.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.978+0000 7f44215e6640 1 Processor -- start 2024-03-04T10:10:06.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.978+0000 7f44215e6640 1 -- start start 2024-03-04T10:10:06.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f44215e6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f441c0b5700 0x7f441c0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:06.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f44215e6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0d34a0 0x7f441c06c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:06.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f44215e6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0d2400 con 0x7f441c0b9410 2024-03-04T10:10:06.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f44215e6640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c0d26e0 con 0x7f441c0b5700 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f44215e6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0d29c0 con 0x7f441c0d34a0 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441affd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f441c0b5700 0x7f441c0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441b7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f441c0b9410 0x7f441c06c620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59586/0 (socket says 172.21.15.99:59586) 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441b7fe640 1 -- 172.21.15.99:0/4280173398 learned_addr learned my addr 172.21.15.99:0/4280173398 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441a7fc640 1 --2- 172.21.15.99:0/4280173398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0d34a0 0x7f441c06c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441affd640 1 -- 172.21.15.99:0/4280173398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0d34a0 msgr2=0x7f441c06c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441affd640 1 --2- 172.21.15.99:0/4280173398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0d34a0 0x7f441c06c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441affd640 1 -- 172.21.15.99:0/4280173398 >> v1:172.21.15.99:6789/0 conn(0x7f441c0b9410 legacy=0x7f441c06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f441affd640 1 -- 172.21.15.99:0/4280173398 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c0d2ca0 con 0x7f441c0b5700 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.979+0000 7f4419ffb640 1 -- 172.21.15.99:0/4280173398 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958570009 0 0) 0x7f441c0d2400 con 0x7f441c0b9410 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.980+0000 7f441affd640 1 --2- 172.21.15.99:0/4280173398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0b5700 0x7f441c0d1b00 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3214063a38fa7314 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.980+0000 7f4419ffb640 1 -- 172.21.15.99:0/4280173398 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c0315a0 con 0x7f441c0b5700 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.980+0000 7f4419ffb640 1 -- 172.21.15.99:0/4280173398 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440c037070 con 0x7f441c0b5700 2024-03-04T10:10:06.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.980+0000 7f4419ffb640 1 -- 172.21.15.99:0/4280173398 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440c031960 con 0x7f441c0b5700 2024-03-04T10:10:06.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 -- 172.21.15.99:0/4280173398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0b5700 msgr2=0x7f441c0d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:06.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 --2- 172.21.15.99:0/4280173398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0b5700 0x7f441c0d1b00 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b840 comp rx=0 tx=0).stop 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 -- 172.21.15.99:0/4280173398 shutdown_connections 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 --2- 172.21.15.99:0/4280173398 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0d34a0 0x7f441c06c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 --2- 172.21.15.99:0/4280173398 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0b5700 0x7f441c0d1b00 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 -- 172.21.15.99:0/4280173398 >> 172.21.15.99:0/4280173398 conn(0x7f441c0d0490 msgr2=0x7f441c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 -- 172.21.15.99:0/4280173398 shutdown_connections 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.981+0000 7f44215e6640 1 -- 172.21.15.99:0/4280173398 wait complete. 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.982+0000 7f44215e6640 1 Processor -- start 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.982+0000 7f44215e6640 1 -- start start 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0b5700 0x7f441c15e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0b9410 0x7f441c0aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 0x7f441c0abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c06ceb0 con 0x7f441c0d34a0 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c06cd30 con 0x7f441c0b5700 2024-03-04T10:10:06.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0b5ae0 con 0x7f441c0b9410 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 0x7f441c0abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 0x7f441c0abe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45410/0 (socket says 172.21.15.99:45410) 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 -- 172.21.15.99:0/385654182 learned_addr learned my addr 172.21.15.99:0/385654182 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441affd640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0b5700 0x7f441c15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441a7fc640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0b9410 0x7f441c0aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 -- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0b9410 msgr2=0x7f441c0aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0b9410 0x7f441c0aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 -- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0b5700 msgr2=0x7f441c15e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0b5700 0x7f441c15e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 -- 172.21.15.99:0/385654182 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c1f3fb0 con 0x7f441c0d34a0 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441affd640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0b5700 0x7f441c15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441a7fc640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0b9410 0x7f441c0aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:06.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f441b7fe640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 0x7f441c0abe30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f441400c7a0 tx=0x7f441400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:06.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414018070 con 0x7f441c0d34a0 2024-03-04T10:10:06.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441400a640 con 0x7f441c0d34a0 2024-03-04T10:10:06.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441400aab0 con 0x7f441c0d34a0 2024-03-04T10:10:06.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c1f4290 con 0x7f441c0d34a0 2024-03-04T10:10:06.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.983+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c1f4850 con 0x7f441c0d34a0 2024-03-04T10:10:06.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.984+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f441c074f60 con 0x7f441c0d34a0 2024-03-04T10:10:06.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.985+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f441401b070 con 0x7f441c0d34a0 2024-03-04T10:10:06.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.985+0000 7f44037fe640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f440806b4d0 0x7f440806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:06.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.986+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f44140898c0 con 0x7f441c0d34a0 2024-03-04T10:10:06.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.986+0000 7f441affd640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f440806b4d0 0x7f440806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:06.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.986+0000 7f441affd640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f440806b4d0 0x7f440806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f440c036000 tx=0x7f440c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:06.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:06.990+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f44140586a0 con 0x7f441c0d34a0 2024-03-04T10:10:07.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.193+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f441c0b4190 con 0x7f441c0d34a0 2024-03-04T10:10:07.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.842+0000 7f44037fe640 1 -- 172.21.15.99:0/385654182 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v76) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4414057fa0 con 0x7f441c0d34a0 2024-03-04T10:10:07.845 INFO:teuthology.orchestra.run.smithi099.stderr:new fs with metadata pool 13 and data pool 14 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f440806b4d0 msgr2=0x7f440806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f440806b4d0 0x7f440806d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f440c036000 tx=0x7f440c03d040 comp rx=0 tx=0).stop 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 msgr2=0x7f441c0abe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 0x7f441c0abe30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f441400c7a0 tx=0x7f441400cc60 comp rx=0 tx=0).stop 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 shutdown_connections 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f440806b4d0 0x7f440806d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f441c0d34a0 0x7f441c0abe30 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f441c0b9410 0x7f441c0aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 --2- 172.21.15.99:0/385654182 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f441c0b5700 0x7f441c15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 >> 172.21.15.99:0/385654182 conn(0x7f441c0d0490 msgr2=0x7f441c0b6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:07.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 shutdown_connections 2024-03-04T10:10:07.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.845+0000 7f44215e6640 1 -- 172.21.15.99:0/385654182 wait complete. 2024-03-04T10:10:07.874 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:07.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.976+0000 7ff19b8d1640 1 Processor -- start 2024-03-04T10:10:07.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.976+0000 7ff19b8d1640 1 -- start start 2024-03-04T10:10:07.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19b8d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff19415dea0 0x7ff1940cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19b8d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1940cc8d0 0x7ff19406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19b8d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19406fd20 con 0x7ff1940d1200 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19b8d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19406fad0 con 0x7ff19415dea0 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19b8d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff194070000 con 0x7ff1940cc8d0 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff19415dea0 0x7ff1940cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff19415dea0 0x7ff1940cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45424/0 (socket says 172.21.15.99:45424) 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 -- 172.21.15.99:0/2990392912 learned_addr learned my addr 172.21.15.99:0/2990392912 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 -- 172.21.15.99:0/2990392912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1940cc8d0 msgr2=0x7ff19406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 --2- 172.21.15.99:0/2990392912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1940cc8d0 0x7ff19406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 -- 172.21.15.99:0/2990392912 >> v1:172.21.15.162:6789/0 conn(0x7ff1940d1200 legacy=0x7ff1940cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.977+0000 7ff19966d640 1 -- 172.21.15.99:0/2990392912 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1940702e0 con 0x7ff19415dea0 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.978+0000 7ff19966d640 1 --2- 172.21.15.99:0/2990392912 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff19415dea0 0x7ff1940cad60 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff190009870 tx=0x7ff19002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe2a545f21a70d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.978+0000 7ff18bfff640 1 -- 172.21.15.99:0/2990392912 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190035020 con 0x7ff19415dea0 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.978+0000 7ff18bfff640 1 -- 172.21.15.99:0/2990392912 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff190031640 con 0x7ff19415dea0 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.978+0000 7ff18bfff640 1 -- 172.21.15.99:0/2990392912 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19002d040 con 0x7ff19415dea0 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 -- 172.21.15.99:0/2990392912 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff19415dea0 msgr2=0x7ff1940cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/2990392912 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff19415dea0 0x7ff1940cad60 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff190009870 tx=0x7ff19002b8f0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 -- 172.21.15.99:0/2990392912 shutdown_connections 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/2990392912 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff1940cc8d0 0x7ff19406acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/2990392912 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff19415dea0 0x7ff1940cad60 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 -- 172.21.15.99:0/2990392912 >> 172.21.15.99:0/2990392912 conn(0x7ff1940d0710 msgr2=0x7ff1940d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 -- 172.21.15.99:0/2990392912 shutdown_connections 2024-03-04T10:10:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.979+0000 7ff19b8d1640 1 -- 172.21.15.99:0/2990392912 wait complete. 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.980+0000 7ff19b8d1640 1 Processor -- start 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.980+0000 7ff19b8d1640 1 -- start start 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 0x7ff194154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1940d1200 0x7ff19415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19415dea0 0x7ff19415c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19406b390 con 0x7ff1940cc8d0 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19406b210 con 0x7ff1940d1200 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19406b510 con 0x7ff19415dea0 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 0x7ff194154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 0x7ff194154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45430/0 (socket says 172.21.15.99:45430) 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 -- 172.21.15.99:0/1442719229 learned_addr learned my addr 172.21.15.99:0/1442719229 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff199e6e640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19415dea0 0x7ff19415c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff198e6c640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1940d1200 0x7ff19415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 -- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19415dea0 msgr2=0x7ff19415c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19415dea0 0x7ff19415c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 -- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1940d1200 msgr2=0x7ff19415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1940d1200 0x7ff19415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 -- 172.21.15.99:0/1442719229 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff194154840 con 0x7ff1940cc8d0 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff199e6e640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19415dea0 0x7ff19415c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff198e6c640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1940d1200 0x7ff19415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19966d640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 0x7ff194154300 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ff19002b930 tx=0x7ff1900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19003ac60 con 0x7ff1940cc8d0 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19003ae20 con 0x7ff1940cc8d0 2024-03-04T10:10:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1940c4ac0 con 0x7ff1940cc8d0 2024-03-04T10:10:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.981+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190044630 con 0x7ff1940cc8d0 2024-03-04T10:10:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.982+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1940c5080 con 0x7ff1940cc8d0 2024-03-04T10:10:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.983+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff194074f60 con 0x7ff1940cc8d0 2024-03-04T10:10:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.983+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff19004b060 con 0x7ff1940cc8d0 2024-03-04T10:10:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.983+0000 7ff189ffb640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff17806b4d0 0x7ff17806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.983+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff1900ac160 con 0x7ff1940cc8d0 2024-03-04T10:10:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.984+0000 7ff198e6c640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff17806b4d0 0x7ff17806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.984+0000 7ff198e6c640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff17806b4d0 0x7ff17806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff194155680 tx=0x7ff184008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:07.988+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff19003f070 con 0x7ff1940cc8d0 2024-03-04T10:10:08.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.158+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1940cbad0 con 0x7ff1940cc8d0 2024-03-04T10:10:08.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.159+0000 7ff189ffb640 1 -- 172.21.15.99:0/1442719229 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16862 (secure 0 0 0) 0x7ff19007ae50 con 0x7ff1940cc8d0 2024-03-04T10:10:08.162 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:08.162 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":92,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:10:07.195609+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:08.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff17806b4d0 msgr2=0x7ff17806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff17806b4d0 0x7ff17806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff194155680 tx=0x7ff184008040 comp rx=0 tx=0).stop 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 msgr2=0x7ff194154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 0x7ff194154300 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ff19002b930 tx=0x7ff1900046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 shutdown_connections 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff17806b4d0 0x7ff17806d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff19415dea0 0x7ff19415c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff1940d1200 0x7ff19415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 --2- 172.21.15.99:0/1442719229 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff1940cc8d0 0x7ff194154300 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 >> 172.21.15.99:0/1442719229 conn(0x7ff1940d0710 msgr2=0x7ff1940b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 shutdown_connections 2024-03-04T10:10:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.161+0000 7ff19b8d1640 1 -- 172.21.15.99:0/1442719229 wait complete. 2024-03-04T10:10:08.174 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:08.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.272+0000 7f3336bcd640 1 Processor -- start 2024-03-04T10:10:08.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.272+0000 7f3336bcd640 1 -- start start 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3336bcd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f333015abe0 0x7f33300ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3336bcd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33300afb30 0x7f33300cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3336bcd640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33300d1550 con 0x7f33300ce2e0 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3336bcd640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33300cbfa0 con 0x7f333015abe0 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3336bcd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33300cc280 con 0x7f33300afb30 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3334969640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f333015abe0 0x7f33300ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f333516a640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f33300ce2e0 0x7f33300cd000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59594/0 (socket says 172.21.15.99:59594) 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3334969640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f333015abe0 0x7f33300ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45442/0 (socket says 172.21.15.99:45442) 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f333516a640 1 -- 172.21.15.99:0/1454525399 learned_addr learned my addr 172.21.15.99:0/1454525399 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f332ffff640 1 --2- 172.21.15.99:0/1454525399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33300afb30 0x7f33300cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3334969640 1 -- 172.21.15.99:0/1454525399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33300afb30 msgr2=0x7f33300cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3334969640 1 --2- 172.21.15.99:0/1454525399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33300afb30 0x7f33300cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3334969640 1 -- 172.21.15.99:0/1454525399 >> v1:172.21.15.99:6789/0 conn(0x7f33300ce2e0 legacy=0x7f33300cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f3334969640 1 -- 172.21.15.99:0/1454525399 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33300cc560 con 0x7f333015abe0 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.273+0000 7f332f7fe640 1 -- 172.21.15.99:0/1454525399 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607330543 0 0) 0x7f33300d1550 con 0x7f33300ce2e0 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3334969640 1 --2- 172.21.15.99:0/1454525399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f333015abe0 0x7f33300ce8a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f33240097a0 tx=0x7f332402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4c5640143771fe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f332f7fe640 1 -- 172.21.15.99:0/1454525399 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324035020 con 0x7f333015abe0 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f332f7fe640 1 -- 172.21.15.99:0/1454525399 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324031830 con 0x7f333015abe0 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f332f7fe640 1 -- 172.21.15.99:0/1454525399 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324031c00 con 0x7f333015abe0 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 -- 172.21.15.99:0/1454525399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f333015abe0 msgr2=0x7f33300ce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 --2- 172.21.15.99:0/1454525399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f333015abe0 0x7f33300ce8a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f33240097a0 tx=0x7f332402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 -- 172.21.15.99:0/1454525399 shutdown_connections 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 --2- 172.21.15.99:0/1454525399 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f33300afb30 0x7f33300cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 --2- 172.21.15.99:0/1454525399 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f333015abe0 0x7f33300ce8a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 -- 172.21.15.99:0/1454525399 >> 172.21.15.99:0/1454525399 conn(0x7f33300d0710 msgr2=0x7f33300d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:08.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.274+0000 7f3336bcd640 1 -- 172.21.15.99:0/1454525399 shutdown_connections 2024-03-04T10:10:08.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.275+0000 7f3336bcd640 1 -- 172.21.15.99:0/1454525399 wait complete. 2024-03-04T10:10:08.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 Processor -- start 2024-03-04T10:10:08.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 -- start start 2024-03-04T10:10:08.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 0x7f33300b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33300ce2e0 0x7f33300b5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f333015abe0 0x7f333014e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333006f750 con 0x7f33300afb30 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33300d1200 con 0x7f33300ce2e0 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.276+0000 7f3336bcd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333006f8d0 con 0x7f333015abe0 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 0x7f33300b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 0x7f33300b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45454/0 (socket says 172.21.15.99:45454) 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 -- 172.21.15.99:0/3045506394 learned_addr learned my addr 172.21.15.99:0/3045506394 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f332ffff640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33300ce2e0 0x7f33300b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f333516a640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f333015abe0 0x7f333014e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 -- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f333015abe0 msgr2=0x7f333014e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f333015abe0 0x7f333014e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 -- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33300ce2e0 msgr2=0x7f33300b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33300ce2e0 0x7f33300b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 -- 172.21.15.99:0/3045506394 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333014f780 con 0x7f33300afb30 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3334969640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 0x7f33300b4b50 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f3324000c00 tx=0x7f33240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324040070 con 0x7f33300afb30 2024-03-04T10:10:08.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324039a00 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333014fa60 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3324035040 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.277+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3330150020 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.278+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3330074f60 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.279+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f332404b060 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.279+0000 7f332d7fa640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f331406b4d0 0x7f331406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.279+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f33240abc00 con 0x7f33300afb30 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.279+0000 7f332ffff640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f331406b4d0 0x7f331406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.280+0000 7f332ffff640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f331406b4d0 0x7f331406d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f33300c6610 tx=0x7f331c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.283+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f33240b2080 con 0x7f33300afb30 2024-03-04T10:10:08.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.454+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33300cefa0 con 0x7f33300afb30 2024-03-04T10:10:08.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.455+0000 7f332d7fa640 1 -- 172.21.15.99:0/3045506394 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16862 (secure 0 0 0) 0x7f332407a980 con 0x7f33300afb30 2024-03-04T10:10:08.457 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:08.458 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":92,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:10:07.195609+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:08.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f331406b4d0 msgr2=0x7f331406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.459 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f331406b4d0 0x7f331406d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f33300c6610 tx=0x7f331c008040 comp rx=0 tx=0).stop 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 msgr2=0x7f33300b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 0x7f33300b4b50 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f3324000c00 tx=0x7f33240046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 shutdown_connections 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f331406b4d0 0x7f331406d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f333015abe0 0x7f333014e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f33300ce2e0 0x7f33300b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 --2- 172.21.15.99:0/3045506394 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f33300afb30 0x7f33300b4b50 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 >> 172.21.15.99:0/3045506394 conn(0x7f33300d0710 msgr2=0x7f333006acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 shutdown_connections 2024-03-04T10:10:08.460 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.457+0000 7f3336bcd640 1 -- 172.21.15.99:0/3045506394 wait complete. 2024-03-04T10:10:08.471 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T10:10:08.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.572+0000 7f114da32640 1 Processor -- start 2024-03-04T10:10:08.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.572+0000 7f114da32640 1 -- start start 2024-03-04T10:10:08.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f114da32640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b5700 0x7f11480d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f114da32640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f11480b9410 0x7f114806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f114da32640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11480d2400 con 0x7f11480d34a0 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f114da32640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11480d26e0 con 0x7f11480b9410 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f114da32640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11480d29c0 con 0x7f11480b5700 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f11480b9410 0x7f114806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f11480b9410 0x7f114806c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45468/0 (socket says 172.21.15.99:45468) 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 -- 172.21.15.99:0/1315023256 learned_addr learned my addr 172.21.15.99:0/1315023256 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f1146ffd640 1 --2- 172.21.15.99:0/1315023256 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b5700 0x7f11480d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 -- 172.21.15.99:0/1315023256 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b5700 msgr2=0x7f11480d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 --2- 172.21.15.99:0/1315023256 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b5700 0x7f11480d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 -- 172.21.15.99:0/1315023256 >> v1:172.21.15.162:6789/0 conn(0x7f11480d34a0 legacy=0x7f114806c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.573+0000 7f11477fe640 1 -- 172.21.15.99:0/1315023256 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11480d2ca0 con 0x7f11480b9410 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.574+0000 7f11477fe640 1 --2- 172.21.15.99:0/1315023256 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b9410 0x7f114806c7f0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f1138009e50 tx=0x7f113802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21a82198c40f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.574+0000 7f1145ffb640 1 -- 172.21.15.99:0/1315023256 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113802c9e0 con 0x7f11480b9410 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.574+0000 7f1145ffb640 1 -- 172.21.15.99:0/1315023256 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113802cb80 con 0x7f11480b9410 2024-03-04T10:10:08.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.574+0000 7f1145ffb640 1 -- 172.21.15.99:0/1315023256 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11380343f0 con 0x7f11480b9410 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 -- 172.21.15.99:0/1315023256 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b9410 msgr2=0x7f114806c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 --2- 172.21.15.99:0/1315023256 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b9410 0x7f114806c7f0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f1138009e50 tx=0x7f113802be40 comp rx=0 tx=0).stop 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 -- 172.21.15.99:0/1315023256 shutdown_connections 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 --2- 172.21.15.99:0/1315023256 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b9410 0x7f114806c7f0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 --2- 172.21.15.99:0/1315023256 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b5700 0x7f11480d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 -- 172.21.15.99:0/1315023256 >> 172.21.15.99:0/1315023256 conn(0x7f11480d0490 msgr2=0x7f11480d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 -- 172.21.15.99:0/1315023256 shutdown_connections 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.575+0000 7f114da32640 1 -- 172.21.15.99:0/1315023256 wait complete. 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.576+0000 7f114da32640 1 Processor -- start 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.576+0000 7f114da32640 1 -- start start 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 0x7f11480b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b9410 0x7f1148156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11480d34a0 0x7f11480cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114806ceb0 con 0x7f11480b5700 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114806cd30 con 0x7f11480d34a0 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11480b5ae0 con 0x7f11480b9410 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 0x7f11480b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 0x7f11480b2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45474/0 (socket says 172.21.15.99:45474) 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 -- 172.21.15.99:0/2670167785 learned_addr learned my addr 172.21.15.99:0/2670167785 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 -- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b9410 msgr2=0x7f1148156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b9410 0x7f1148156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 -- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11480d34a0 msgr2=0x7f11480cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f11477fe640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11480d34a0 0x7f11480cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11480d34a0 0x7f11480cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 -- 172.21.15.99:0/2670167785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11480c8db0 con 0x7f11480b5700 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f1146ffd640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 0x7f11480b2820 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f114000c7f0 tx=0x7f114000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140015070 con 0x7f11480b5700 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140004240 con 0x7f11480b5700 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11480c9050 con 0x7f11480b5700 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140007620 con 0x7f11480b5700 2024-03-04T10:10:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.577+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11480c9610 con 0x7f11480b5700 2024-03-04T10:10:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.578+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1148074f60 con 0x7f11480b5700 2024-03-04T10:10:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.579+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f11400079d0 con 0x7f11480b5700 2024-03-04T10:10:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.579+0000 7f112ffff640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f113406b4d0 0x7f113406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.579+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f114008a360 con 0x7f11480b5700 2024-03-04T10:10:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.580+0000 7f11467fc640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f113406b4d0 0x7f113406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.580+0000 7f11467fc640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f113406b4d0 0x7f113406d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f11480cc110 tx=0x7f113c006ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.584+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f11400590e0 con 0x7f11480b5700 2024-03-04T10:10:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.789+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f11480c7110 con 0x7f11480b5700 2024-03-04T10:10:08.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.853+0000 7f112ffff640 1 -- 172.21.15.99:0/2670167785 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f11400589e0 con 0x7f11480b5700 2024-03-04T10:10:08.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f113406b4d0 msgr2=0x7f113406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f113406b4d0 0x7f113406d990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f11480cc110 tx=0x7f113c006ba0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 msgr2=0x7f11480b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 0x7f11480b2820 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f114000c7f0 tx=0x7f114000ccb0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 shutdown_connections 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f113406b4d0 0x7f113406d990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11480d34a0 0x7f11480cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f11480b9410 0x7f1148156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 --2- 172.21.15.99:0/2670167785 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f11480b5700 0x7f11480b2820 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 >> 172.21.15.99:0/2670167785 conn(0x7f11480d0490 msgr2=0x7f11480d39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 shutdown_connections 2024-03-04T10:10:08.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.855+0000 7f114da32640 1 -- 172.21.15.99:0/2670167785 wait complete. 2024-03-04T10:10:08.869 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-04T10:10:08.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-04T10:10:08.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.969+0000 7fdecb7d4640 1 Processor -- start 2024-03-04T10:10:08.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.970+0000 7fdecb7d4640 1 -- start start 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.970+0000 7fdecb7d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec406bce0 0x7fdec406c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.970+0000 7fdecb7d4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdec415e100 0x7fdec415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.970+0000 7fdecb7d4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec415bae0 con 0x7fdec415b700 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.970+0000 7fdecb7d4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec406c680 con 0x7fdec415e100 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.970+0000 7fdecb7d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40adb90 con 0x7fdec406bce0 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdec415e100 0x7fdec415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdec415e100 0x7fdec415e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45478/0 (socket says 172.21.15.99:45478) 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 -- 172.21.15.99:0/1885446663 learned_addr learned my addr 172.21.15.99:0/1885446663 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec8d6f640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec406bce0 0x7fdec406c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 -- 172.21.15.99:0/1885446663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec406bce0 msgr2=0x7fdec406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec406bce0 0x7fdec406c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 -- 172.21.15.99:0/1885446663 >> v1:172.21.15.162:6789/0 conn(0x7fdec415b700 legacy=0x7fdec40ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec9d71640 1 -- 172.21.15.99:0/1885446663 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec415ee60 con 0x7fdec415e100 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.971+0000 7fdec8d6f640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec406bce0 0x7fdec406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdec9d71640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415e590 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009e50 tx=0x7fdebc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214ccb8c6c5c1fec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdebbfff640 1 -- 172.21.15.99:0/1885446663 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc02ca70 con 0x7fdec415e100 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdebbfff640 1 -- 172.21.15.99:0/1885446663 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc02cc10 con 0x7fdec415e100 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdebbfff640 1 -- 172.21.15.99:0/1885446663 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0343f0 con 0x7fdec415e100 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 -- 172.21.15.99:0/1885446663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 msgr2=0x7fdec415e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415e590 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009e50 tx=0x7fdebc02bed0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 -- 172.21.15.99:0/1885446663 shutdown_connections 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415e590 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/1885446663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec406bce0 0x7fdec406c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 -- 172.21.15.99:0/1885446663 >> 172.21.15.99:0/1885446663 conn(0x7fdec4071bc0 msgr2=0x7fdec4071fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:08.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 -- 172.21.15.99:0/1885446663 shutdown_connections 2024-03-04T10:10:08.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.972+0000 7fdecb7d4640 1 -- 172.21.15.99:0/1885446663 wait complete. 2024-03-04T10:10:08.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.973+0000 7fdecb7d4640 1 Processor -- start 2024-03-04T10:10:08.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 -- start start 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdec406bce0 0x7fdec41f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec415b700 0x7fdec415d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415ead0 con 0x7fdec415e100 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40ad820 con 0x7fdec406bce0 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdecb7d4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415f2c0 con 0x7fdec415b700 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdec9d71640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdec9d71640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45494/0 (socket says 172.21.15.99:45494) 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdec9d71640 1 -- 172.21.15.99:0/3732111387 learned_addr learned my addr 172.21.15.99:0/3732111387 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdec8d6f640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec415b700 0x7fdec415d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.974+0000 7fdec9570640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdec406bce0 0x7fdec41f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9d71640 1 -- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec415b700 msgr2=0x7fdec415d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9d71640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec415b700 0x7fdec415d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9d71640 1 -- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdec406bce0 msgr2=0x7fdec41f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9d71640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdec406bce0 0x7fdec41f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9d71640 1 -- 172.21.15.99:0/3732111387 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec41f7900 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9570640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdec406bce0 0x7fdec41f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdec9d71640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415d7e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009e50 tx=0x7fdebc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc044020 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc040670 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec41f3280 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc009510 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.975+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec41f6e20 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.976+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec414b240 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.977+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdebc047020 con 0x7fdec415e100 2024-03-04T10:10:08.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.977+0000 7fdeb9ffb640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdea806b4d0 0x7fdea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:08.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.977+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdebc0acec0 con 0x7fdec415e100 2024-03-04T10:10:08.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.977+0000 7fdec9570640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdea806b4d0 0x7fdea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:08.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.978+0000 7fdec9570640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdea806b4d0 0x7fdea806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdec0003eb0 tx=0x7fdec0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:08.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:08.981+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdec414b240 con 0x7fdec415e100 2024-03-04T10:10:09.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.186+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdec4074f60 con 0x7fdec415e100 2024-03-04T10:10:09.837 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:09.835+0000 7f30973fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:10:09.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.866+0000 7fdeb9ffb640 1 -- 172.21.15.99:0/3732111387 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdebc04e030 con 0x7fdec415e100 2024-03-04T10:10:09.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdea806b4d0 msgr2=0x7fdea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdea806b4d0 0x7fdea806d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdec0003eb0 tx=0x7fdec0009040 comp rx=0 tx=0).stop 2024-03-04T10:10:09.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 msgr2=0x7fdec415d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415d7e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009e50 tx=0x7fdebc004a40 comp rx=0 tx=0).stop 2024-03-04T10:10:09.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 shutdown_connections 2024-03-04T10:10:09.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdea806b4d0 0x7fdea806d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdec415e100 0x7fdec415d7e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdec415b700 0x7fdec415d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 --2- 172.21.15.99:0/3732111387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdec406bce0 0x7fdec41f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.867+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 >> 172.21.15.99:0/3732111387 conn(0x7fdec4071bc0 msgr2=0x7fdec415f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.868+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 shutdown_connections 2024-03-04T10:10:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.868+0000 7fdecb7d4640 1 -- 172.21.15.99:0/3732111387 wait complete. 2024-03-04T10:10:09.882 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-04T10:10:09.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.982+0000 7fccb2b97640 1 Processor -- start 2024-03-04T10:10:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.982+0000 7fccb2b97640 1 -- start start 2024-03-04T10:10:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.983+0000 7fccb2b97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0af7b0 0x7fccac0d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.983+0000 7fccb2b97640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0cc760 con 0x7fccac15ac40 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.983+0000 7fccb2b97640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0ce720 con 0x7fccac0cf080 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.983+0000 7fccb2b97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac15b020 con 0x7fccac0af7b0 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.983+0000 7fccb0933640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fccac15ac40 0x7fccac0ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59602/0 (socket says 172.21.15.99:59602) 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.983+0000 7fccb0933640 1 -- 172.21.15.99:0/3936117882 learned_addr learned my addr 172.21.15.99:0/3936117882 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccabfff640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0af7b0 0x7fccac0d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350986701 0 0) 0x7fccac0cc760 con 0x7fccac15ac40 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc98003520 con 0x7fccac15ac40 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058781998 0 0) 0x7fcc98003520 con 0x7fccac15ac40 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0af7b0 msgr2=0x7fccac0d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0af7b0 0x7fccac0d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 >> v1:172.21.15.162:6789/0 conn(0x7fccac0cf080 legacy=0x7fccac0ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0cbe30 con 0x7fccac15ac40 2024-03-04T10:10:09.989 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fcca40019f0 con 0x7fccac15ac40 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 >> v1:172.21.15.99:6789/0 conn(0x7fccac15ac40 legacy=0x7fccac0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc980039e0 0x7fcc98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc98004300 0x7fcc98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc98004f90 0x7fcc9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac15b020 con 0x7fcc98004f90 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9800bf00 con 0x7fcc98004300 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9800c080 con 0x7fcc980039e0 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb1134640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc98004f90 0x7fcc9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccabfff640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc98004300 0x7fcc98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb0933640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc980039e0 0x7fcc98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb1134640 1 -- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc980039e0 msgr2=0x7fcc98003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb1134640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc980039e0 0x7fcc98003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb1134640 1 -- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc98004300 msgr2=0x7fcc98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb1134640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc98004300 0x7fcc98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.990 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccb1134640 1 -- 172.21.15.99:0/3936117882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc98003520 con 0x7fcc98004f90 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.984+0000 7fccabfff640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc98004300 0x7fcc98004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.985+0000 7fccb1134640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc98004f90 0x7fcc9800b9c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fccac06e2a0 tx=0x7fcc98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d93f121c12a8702 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.985+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94001430 con 0x7fcc98004f90 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.985+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94032620 con 0x7fcc98004f90 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.985+0000 7fccab7fe640 1 -- 172.21.15.99:0/3936117882 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94032a10 con 0x7fcc98004f90 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb2b97640 1 -- 172.21.15.99:0/3936117882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc98004f90 msgr2=0x7fcc9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb2b97640 1 --2- 172.21.15.99:0/3936117882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc98004f90 0x7fcc9800b9c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fccac06e2a0 tx=0x7fcc98005830 comp rx=0 tx=0).stop 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb1134640 1 -- 172.21.15.99:0/3936117882 reap_dead start 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb2b97640 1 -- 172.21.15.99:0/3936117882 shutdown_connections 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb2b97640 1 -- 172.21.15.99:0/3936117882 >> 172.21.15.99:0/3936117882 conn(0x7fccac0d0770 msgr2=0x7fccac0d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb2b97640 1 -- 172.21.15.99:0/3936117882 shutdown_connections 2024-03-04T10:10:09.991 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.986+0000 7fccb2b97640 1 -- 172.21.15.99:0/3936117882 wait complete. 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.987+0000 7fccb2b97640 1 Processor -- start 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.987+0000 7fccb2b97640 1 -- start start 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb2b97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 0x7fccac15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb2b97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0c7c20 0x7fccac1f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb2b97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac15c8c0 0x7fccac15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb2b97640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0ce340 con 0x7fccac0c7840 2024-03-04T10:10:09.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb2b97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0cc3f0 con 0x7fccac15c8c0 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb2b97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccac0ce4c0 con 0x7fccac0c7c20 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 0x7fccac15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 0x7fccac15b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45504/0 (socket says 172.21.15.99:45504) 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 -- 172.21.15.99:0/1124158219 learned_addr learned my addr 172.21.15.99:0/1124158219 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccabfff640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0c7c20 0x7fccac1f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb1134640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac15c8c0 0x7fccac15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 -- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0c7c20 msgr2=0x7fccac1f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0c7c20 0x7fccac1f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 -- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac15c8c0 msgr2=0x7fccac15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac15c8c0 0x7fccac15cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 -- 172.21.15.99:0/1124158219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac15b6e0 con 0x7fccac0c7840 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb1134640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac15c8c0 0x7fccac15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:09.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fccb0933640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 0x7fccac15b1a0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fcca400a160 tx=0x7fcca400a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.988+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4017020 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.989+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca4013600 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.989+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac06e630 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.989+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca40139d0 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.989+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac06ebf0 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.990+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac06cd70 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.990+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcca401a020 con 0x7fccac0c7840 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.990+0000 7fcca97fa640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc8c06b4d0 0x7fcc8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:09.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.991+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcca408b270 con 0x7fccac0c7840 2024-03-04T10:10:09.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.991+0000 7fccabfff640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc8c06b4d0 0x7fcc8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:09.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.991+0000 7fccabfff640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc8c06b4d0 0x7fcc8c06d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fccac0b63f0 tx=0x7fcc9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:09.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:09.995+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcca4021020 con 0x7fccac0c7840 2024-03-04T10:10:10.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.201+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccac0af450 con 0x7fccac0c7840 2024-03-04T10:10:10.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.202+0000 7fcca97fa640 1 -- 172.21.15.99:0/1124158219 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7fcca4059ff0 con 0x7fccac0c7840 2024-03-04T10:10:10.204 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:10.204 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":78,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:09.188560+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7644},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7644":{"gid":7644,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3671020073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3671020073},{"type":"v1","addr":"172.21.15.99:6835","nonce":3671020073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:10.205 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 78 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc8c06b4d0 msgr2=0x7fcc8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc8c06b4d0 0x7fcc8c06d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fccac0b63f0 tx=0x7fcc9c008040 comp rx=0 tx=0).stop 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 msgr2=0x7fccac15b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 0x7fccac15b1a0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fcca400a160 tx=0x7fcca400a810 comp rx=0 tx=0).stop 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 shutdown_connections 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc8c06b4d0 0x7fcc8c06d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fccac15c8c0 0x7fccac15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fccac0c7c20 0x7fccac1f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 --2- 172.21.15.99:0/1124158219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fccac0c7840 0x7fccac15b1a0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.203+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 >> 172.21.15.99:0/1124158219 conn(0x7fccac0d0770 msgr2=0x7fccac0ad6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.204+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 shutdown_connections 2024-03-04T10:10:10.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.204+0000 7fccb2b97640 1 -- 172.21.15.99:0/1124158219 wait complete. 2024-03-04T10:10:10.217 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.315+0000 7f1ec3b0a640 1 Processor -- start 2024-03-04T10:10:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec3b0a640 1 -- start start 2024-03-04T10:10:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec3b0a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1ebc0ce0a0 0x7f1ebc1618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec3b0a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1ebc161de0 0x7f1ebc0cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec3b0a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc162200 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec3b0a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc0cc0c0 con 0x7f1ebc161de0 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec3b0a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc0cf200 con 0x7f1ebc0ce0a0 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec20a7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1ebc161de0 0x7f1ebc0cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec20a7640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f1ebc161de0 0x7f1ebc0cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45508/0 (socket says 172.21.15.99:45508) 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.316+0000 7f1ec20a7640 1 -- 172.21.15.99:0/3632059940 learned_addr learned my addr 172.21.15.99:0/3632059940 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.317+0000 7f1ec10a5640 1 --2- 172.21.15.99:0/3632059940 >> v2:172.21.15.162:3300/0 conn(0x7f1ebc0ce0a0 0x7f1ebc1618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.317+0000 7f1ec20a7640 1 -- 172.21.15.99:0/3632059940 >> v2:172.21.15.162:3300/0 conn(0x7f1ebc0ce0a0 msgr2=0x7f1ebc1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.317+0000 7f1ec20a7640 1 --2- 172.21.15.99:0/3632059940 >> v2:172.21.15.162:3300/0 conn(0x7f1ebc0ce0a0 0x7f1ebc1618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.317+0000 7f1ec20a7640 1 -- 172.21.15.99:0/3632059940 >> v1:172.21.15.162:6789/0 conn(0x7f1ebc15dcd0 legacy=0x7f1ebc0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.317+0000 7f1ec20a7640 1 -- 172.21.15.99:0/3632059940 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0ce680 con 0x7f1ebc161de0 2024-03-04T10:10:10.321 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.317+0000 7f1ec20a7640 1 --2- 172.21.15.99:0/3632059940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0cc510 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc070bf0 tx=0x7f1eb402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8aecc7b552d18d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec08a4640 1 -- 172.21.15.99:0/3632059940 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb402ca80 con 0x7f1ebc161de0 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec08a4640 1 -- 172.21.15.99:0/3632059940 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb402cc20 con 0x7f1ebc161de0 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec08a4640 1 -- 172.21.15.99:0/3632059940 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40343f0 con 0x7f1ebc161de0 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/3632059940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc161de0 msgr2=0x7f1ebc0cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/3632059940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0cc510 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc070bf0 tx=0x7f1eb402bee0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/3632059940 shutdown_connections 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/3632059940 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0cc510 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/3632059940 >> v2:172.21.15.162:3300/0 conn(0x7f1ebc0ce0a0 0x7f1ebc1618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/3632059940 >> 172.21.15.99:0/3632059940 conn(0x7f1ebc0b3480 msgr2=0x7f1ebc0b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/3632059940 shutdown_connections 2024-03-04T10:10:10.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.318+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/3632059940 wait complete. 2024-03-04T10:10:10.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.320+0000 7f1ec3b0a640 1 Processor -- start 2024-03-04T10:10:10.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.320+0000 7f1ec3b0a640 1 -- start start 2024-03-04T10:10:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.321+0000 7f1ec3b0a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ebc0ce0a0 0x7f1ebc151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.321+0000 7f1ec3b0a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 0x7f1ebc151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.321+0000 7f1ec3b0a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.321+0000 7f1ec3b0a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0cef20 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.321+0000 7f1ec3b0a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0cec40 con 0x7f1ebc161de0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.321+0000 7f1ec3b0a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc06bff0 con 0x7f1ebc0ce0a0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 0x7f1ebc151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 0x7f1ebc151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45524/0 (socket says 172.21.15.99:45524) 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 -- 172.21.15.99:0/1849613105 learned_addr learned my addr 172.21.15.99:0/1849613105 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec20a7640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec18a6640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ebc0ce0a0 0x7f1ebc151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 -- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ebc0ce0a0 msgr2=0x7f1ebc151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ebc0ce0a0 0x7f1ebc151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 -- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ebc161de0 msgr2=0x7f1ebc0c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 -- 172.21.15.99:0/1849613105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0b4860 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec20a7640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec10a5640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 0x7f1ebc151e70 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800d5f0 tx=0x7f1eb800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80125f0 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb8019070 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0b4b40 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb80129d0 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.322+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc0b5100 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.323+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc0d1310 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.324+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1eb8012b70 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.324+0000 7f1eae7fc640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ea006b4d0 0x7f1ea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.324+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1eb808a6f0 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.324+0000 7f1ec18a6640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ea006b4d0 0x7f1ea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.325+0000 7f1ec18a6640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ea006b4d0 0x7f1ea006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0006fd0 tx=0x7f1eb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:10.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.328+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1ebc0d1310 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.426 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:10.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.508+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ebc074f60 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.509+0000 7f1eae7fc640 1 -- 172.21.15.99:0/1849613105 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16849 (secure 0 0 0) 0x7f1eb8059470 con 0x7f1ebc15dcd0 2024-03-04T10:10:10.511 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:10.512 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":93,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:10:08.853420+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ea006b4d0 msgr2=0x7f1ea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ea006b4d0 0x7f1ea006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0006fd0 tx=0x7f1eb0008040 comp rx=0 tx=0).stop 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 msgr2=0x7f1ebc151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 0x7f1ebc151e70 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800d5f0 tx=0x7f1eb800dab0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 shutdown_connections 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1ea006b4d0 0x7f1ea006d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ebc161de0 0x7f1ebc0c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1ebc15dcd0 0x7f1ebc151e70 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 --2- 172.21.15.99:0/1849613105 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1ebc0ce0a0 0x7f1ebc151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 >> 172.21.15.99:0/1849613105 conn(0x7f1ebc0b3480 msgr2=0x7f1ebc0b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 shutdown_connections 2024-03-04T10:10:10.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.511+0000 7f1ec3b0a640 1 -- 172.21.15.99:0/1849613105 wait complete. 2024-03-04T10:10:10.525 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T10:10:10.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:10:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbca5ea4640 1 Processor -- start 2024-03-04T10:10:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbca5ea4640 1 -- start start 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbca5ea4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbca00d23b0 0x7fbca00cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbca5ea4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca00cfb40 con 0x7fbca015bbb0 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbca5ea4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca00cfe20 con 0x7fbca00cbf80 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbca5ea4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca00d2880 con 0x7fbca00d23b0 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbc9ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbca00d23b0 0x7fbca00cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbc9f7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fbca015bbb0 0x7fbca00d0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59606/0 (socket says 172.21.15.99:59606) 2024-03-04T10:10:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.667+0000 7fbc9f7fe640 1 -- 172.21.15.99:0/3820047093 learned_addr learned my addr 172.21.15.99:0/3820047093 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.668+0000 7fbc9ffff640 1 -- 172.21.15.99:0/3820047093 >> v1:172.21.15.162:6789/0 conn(0x7fbca00cbf80 legacy=0x7fbca00d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.668+0000 7fbc9ffff640 1 -- 172.21.15.99:0/3820047093 >> v1:172.21.15.99:6789/0 conn(0x7fbca015bbb0 legacy=0x7fbca00d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.668+0000 7fbc9e7fc640 1 -- 172.21.15.99:0/3820047093 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157891349 0 0) 0x7fbca00cfb40 con 0x7fbca015bbb0 2024-03-04T10:10:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.668+0000 7fbc9ffff640 1 -- 172.21.15.99:0/3820047093 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca00d0100 con 0x7fbca00d23b0 2024-03-04T10:10:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbc9ffff640 1 --2- 172.21.15.99:0/3820047093 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca00cc3c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009d90 tx=0x7fbc9802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b131bbc7ae08e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbc9e7fc640 1 -- 172.21.15.99:0/3820047093 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98039070 con 0x7fbca00d23b0 2024-03-04T10:10:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbc9e7fc640 1 -- 172.21.15.99:0/3820047093 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9802cd00 con 0x7fbca00d23b0 2024-03-04T10:10:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbc9e7fc640 1 -- 172.21.15.99:0/3820047093 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98034660 con 0x7fbca00d23b0 2024-03-04T10:10:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 -- 172.21.15.99:0/3820047093 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 msgr2=0x7fbca00cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/3820047093 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca00cc3c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009d90 tx=0x7fbc9802bf50 comp rx=0 tx=0).stop 2024-03-04T10:10:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 -- 172.21.15.99:0/3820047093 shutdown_connections 2024-03-04T10:10:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/3820047093 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca00cc3c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 -- 172.21.15.99:0/3820047093 >> 172.21.15.99:0/3820047093 conn(0x7fbca00b3480 msgr2=0x7fbca00b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 -- 172.21.15.99:0/3820047093 shutdown_connections 2024-03-04T10:10:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.669+0000 7fbca5ea4640 1 -- 172.21.15.99:0/3820047093 wait complete. 2024-03-04T10:10:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.670+0000 7fbca5ea4640 1 Processor -- start 2024-03-04T10:10:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 -- start start 2024-03-04T10:10:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbca00cbf80 0x7fbca00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca014c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca015bbb0 0x7fbca00ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00d1010 con 0x7fbca00d23b0 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca00cc900 con 0x7fbca015bbb0 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbca5ea4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca015bf90 con 0x7fbca00cbf80 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbc9effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca014c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbc9effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca014c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45550/0 (socket says 172.21.15.99:45550) 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.671+0000 7fbc9effd640 1 -- 172.21.15.99:0/1039476090 learned_addr learned my addr 172.21.15.99:0/1039476090 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9f7fe640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbca00cbf80 0x7fbca00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9effd640 1 -- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbca00cbf80 msgr2=0x7fbca00c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9effd640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbca00cbf80 0x7fbca00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9effd640 1 -- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca015bbb0 msgr2=0x7fbca00ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9ffff640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca015bbb0 0x7fbca00ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9effd640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca015bbb0 0x7fbca00ac9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9effd640 1 -- 172.21.15.99:0/1039476090 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca014ce30 con 0x7fbca00d23b0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc9effd640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca014c650 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000c400 tx=0x7fbc9000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90017070 con 0x7fbca00d23b0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900052d0 con 0x7fbca00d23b0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca014d110 con 0x7fbca00d23b0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90003830 con 0x7fbca00d23b0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.672+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca014d6d0 con 0x7fbca00d23b0 2024-03-04T10:10:10.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.673+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca0074f60 con 0x7fbca00d23b0 2024-03-04T10:10:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.674+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbc9001a070 con 0x7fbca00d23b0 2024-03-04T10:10:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.674+0000 7fbc87fff640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc8c06b500 0x7fbc8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.674+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbc90089670 con 0x7fbca00d23b0 2024-03-04T10:10:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.674+0000 7fbc9f7fe640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc8c06b500 0x7fbc8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.675+0000 7fbc9f7fe640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc8c06b500 0x7fbc8c06d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbc940011e0 tx=0x7fbc94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:10.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.678+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbc90058360 con 0x7fbca00d23b0 2024-03-04T10:10:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.848+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbca00d2030 con 0x7fbc8c06b500 2024-03-04T10:10:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.854+0000 7fbc87fff640 1 -- 172.21.15.99:0/1039476090 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246767 (secure 0 0 0) 0x7fbca00d2030 con 0x7fbc8c06b500 2024-03-04T10:10:10.860 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:10:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc8c06b500 msgr2=0x7fbc8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc8c06b500 0x7fbc8c06d9c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbc940011e0 tx=0x7fbc94009040 comp rx=0 tx=0).stop 2024-03-04T10:10:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 msgr2=0x7fbca014c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca014c650 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000c400 tx=0x7fbc9000c8c0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 shutdown_connections 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbc8c06b500 0x7fbc8c06d9c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbca015bbb0 0x7fbca00ac9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbca00d23b0 0x7fbca014c650 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 --2- 172.21.15.99:0/1039476090 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbca00cbf80 0x7fbca00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 >> 172.21.15.99:0/1039476090 conn(0x7fbca00b3480 msgr2=0x7fbca00ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 shutdown_connections 2024-03-04T10:10:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:10.857+0000 7fbca5ea4640 1 -- 172.21.15.99:0/1039476090 wait complete. 2024-03-04T10:10:10.871 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-04T10:10:09.532313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242772,"kb_used_data":9928,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754731948,"statfs":{"total":773094113280,"available":772845514752,"internally_reserved":0,"allocated":10166272,"data_stored":5347640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001198"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852633+0000","last_change":"2024-03-04T10:10:06.852633+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852633+0000","last_undegraded":"2024-03-04T10:10:06.852633+0000","last_fullsized":"2024-03-04T10:10:06.852633+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854680+0000","last_change":"2024-03-04T10:10:06.854680+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854680+0000","last_undegraded":"2024-03-04T10:10:06.854680+0000","last_fullsized":"2024-03-04T10:10:06.854680+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852344+0000","last_change":"2024-03-04T10:10:06.852344+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852344+0000","last_undegraded":"2024-03-04T10:10:06.852344+0000","last_fullsized":"2024-03-04T10:10:06.852344+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850373+0000","last_change":"2024-03-04T10:10:06.850373+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850373+0000","last_undegraded":"2024-03-04T10:10:06.850373+0000","last_fullsized":"2024-03-04T10:10:06.850373+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850436+0000","last_change":"2024-03-04T10:10:06.850436+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850436+0000","last_undegraded":"2024-03-04T10:10:06.850436+0000","last_fullsized":"2024-03-04T10:10:06.850436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854406+0000","last_change":"2024-03-04T10:10:06.854406+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854406+0000","last_undegraded":"2024-03-04T10:10:06.854406+0000","last_fullsized":"2024-03-04T10:10:06.854406+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853075+0000","last_change":"2024-03-04T10:10:06.853075+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853075+0000","last_undegraded":"2024-03-04T10:10:06.853075+0000","last_fullsized":"2024-03-04T10:10:06.853075+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851456+0000","last_change":"2024-03-04T10:10:06.851456+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851456+0000","last_undegraded":"2024-03-04T10:10:06.851456+0000","last_fullsized":"2024-03-04T10:10:06.851456+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850204+0000","last_change":"2024-03-04T10:10:06.850204+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850204+0000","last_undegraded":"2024-03-04T10:10:06.850204+0000","last_fullsized":"2024-03-04T10:10:06.850204+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848505+0000","last_change":"2024-03-04T10:10:06.848505+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848505+0000","last_undegraded":"2024-03-04T10:10:06.848505+0000","last_fullsized":"2024-03-04T10:10:06.848505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853200+0000","last_change":"2024-03-04T10:10:06.853200+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853200+0000","last_undegraded":"2024-03-04T10:10:06.853200+0000","last_fullsized":"2024-03-04T10:10:06.853200+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849214+0000","last_change":"2024-03-04T10:10:06.849214+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849214+0000","last_undegraded":"2024-03-04T10:10:06.849214+0000","last_fullsized":"2024-03-04T10:10:06.849214+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849674+0000","last_change":"2024-03-04T10:10:06.849674+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849674+0000","last_undegraded":"2024-03-04T10:10:06.849674+0000","last_fullsized":"2024-03-04T10:10:06.849674+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853490+0000","last_change":"2024-03-04T10:10:06.853490+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853490+0000","last_undegraded":"2024-03-04T10:10:06.853490+0000","last_fullsized":"2024-03-04T10:10:06.853490+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848629+0000","last_change":"2024-03-04T10:10:06.848629+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848629+0000","last_undegraded":"2024-03-04T10:10:06.848629+0000","last_fullsized":"2024-03-04T10:10:06.848629+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854342+0000","last_change":"2024-03-04T10:10:06.854342+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854342+0000","last_undegraded":"2024-03-04T10:10:06.854342+0000","last_fullsized":"2024-03-04T10:10:06.854342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852040+0000","last_change":"2024-03-04T10:10:06.852040+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852040+0000","last_undegraded":"2024-03-04T10:10:06.852040+0000","last_fullsized":"2024-03-04T10:10:06.852040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852330+0000","last_change":"2024-03-04T10:10:06.852330+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852330+0000","last_undegraded":"2024-03-04T10:10:06.852330+0000","last_fullsized":"2024-03-04T10:10:06.852330+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851041+0000","last_change":"2024-03-04T10:10:06.851041+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851041+0000","last_undegraded":"2024-03-04T10:10:06.851041+0000","last_fullsized":"2024-03-04T10:10:06.851041+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849980+0000","last_change":"2024-03-04T10:10:06.849980+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849980+0000","last_undegraded":"2024-03-04T10:10:06.849980+0000","last_fullsized":"2024-03-04T10:10:06.849980+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851155+0000","last_change":"2024-03-04T10:10:06.851155+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851155+0000","last_undegraded":"2024-03-04T10:10:06.851155+0000","last_fullsized":"2024-03-04T10:10:06.851155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853010+0000","last_change":"2024-03-04T10:10:06.853010+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853010+0000","last_undegraded":"2024-03-04T10:10:06.853010+0000","last_fullsized":"2024-03-04T10:10:06.853010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855177+0000","last_change":"2024-03-04T10:10:06.855177+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855177+0000","last_undegraded":"2024-03-04T10:10:06.855177+0000","last_fullsized":"2024-03-04T10:10:06.855177+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851940+0000","last_change":"2024-03-04T10:10:06.851940+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851940+0000","last_undegraded":"2024-03-04T10:10:06.851940+0000","last_fullsized":"2024-03-04T10:10:06.851940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854658+0000","last_change":"2024-03-04T10:10:06.854658+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854658+0000","last_undegraded":"2024-03-04T10:10:06.854658+0000","last_fullsized":"2024-03-04T10:10:06.854658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855850+0000","last_change":"2024-03-04T10:10:06.855850+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855850+0000","last_undegraded":"2024-03-04T10:10:06.855850+0000","last_fullsized":"2024-03-04T10:10:06.855850+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848047+0000","last_change":"2024-03-04T10:10:06.848047+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848047+0000","last_undegraded":"2024-03-04T10:10:06.848047+0000","last_fullsized":"2024-03-04T10:10:06.848047+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849542+0000","last_change":"2024-03-04T10:10:06.849542+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849542+0000","last_undegraded":"2024-03-04T10:10:06.849542+0000","last_fullsized":"2024-03-04T10:10:06.849542+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851655+0000","last_change":"2024-03-04T10:10:06.851655+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851655+0000","last_undegraded":"2024-03-04T10:10:06.851655+0000","last_fullsized":"2024-03-04T10:10:06.851655+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850703+0000","last_change":"2024-03-04T10:10:06.850703+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850703+0000","last_undegraded":"2024-03-04T10:10:06.850703+0000","last_fullsized":"2024-03-04T10:10:06.850703+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852821+0000","last_change":"2024-03-04T10:10:06.852821+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852821+0000","last_undegraded":"2024-03-04T10:10:06.852821+0000","last_fullsized":"2024-03-04T10:10:06.852821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851229+0000","last_change":"2024-03-04T10:10:06.851229+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851229+0000","last_undegraded":"2024-03-04T10:10:06.851229+0000","last_fullsized":"2024-03-04T10:10:06.851229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849419+0000","last_change":"2024-03-04T10:10:06.849419+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849419+0000","last_undegraded":"2024-03-04T10:10:06.849419+0000","last_fullsized":"2024-03-04T10:10:06.849419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.864395+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.864082+0000","last_became_peered":"2024-03-04T10:10:06.864082+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:45:18.950003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849862+0000","last_change":"2024-03-04T10:10:06.849862+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849862+0000","last_undegraded":"2024-03-04T10:10:06.849862+0000","last_fullsized":"2024-03-04T10:10:06.849862+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861011+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.860571+0000","last_became_peered":"2024-03-04T10:10:06.860571+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:35.744190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851356+0000","last_change":"2024-03-04T10:10:06.851356+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851356+0000","last_undegraded":"2024-03-04T10:10:06.851356+0000","last_fullsized":"2024-03-04T10:10:06.851356+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862539+0000","last_change":"2024-03-04T10:10:06.860271+0000","last_active":"2024-03-04T10:10:06.862539+0000","last_peered":"2024-03-04T10:10:06.862539+0000","last_clean":"2024-03-04T10:10:06.862539+0000","last_became_active":"2024-03-04T10:10:06.859984+0000","last_became_peered":"2024-03-04T10:10:06.859984+0000","last_unstale":"2024-03-04T10:10:06.862539+0000","last_undegraded":"2024-03-04T10:10:06.862539+0000","last_fullsized":"2024-03-04T10:10:06.862539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848478+0000","last_change":"2024-03-04T10:10:06.848478+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848478+0000","last_undegraded":"2024-03-04T10:10:06.848478+0000","last_fullsized":"2024-03-04T10:10:06.848478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870416+0000","last_change":"2024-03-04T10:10:06.865348+0000","last_active":"2024-03-04T10:10:06.870416+0000","last_peered":"2024-03-04T10:10:06.870416+0000","last_clean":"2024-03-04T10:10:06.870416+0000","last_became_active":"2024-03-04T10:10:06.865084+0000","last_became_peered":"2024-03-04T10:10:06.865084+0000","last_unstale":"2024-03-04T10:10:06.870416+0000","last_undegraded":"2024-03-04T10:10:06.870416+0000","last_fullsized":"2024-03-04T10:10:06.870416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:01:06.838971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851037+0000","last_change":"2024-03-04T10:10:06.851037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851037+0000","last_undegraded":"2024-03-04T10:10:06.851037+0000","last_fullsized":"2024-03-04T10:10:06.851037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867495+0000","last_change":"2024-03-04T10:10:06.863293+0000","last_active":"2024-03-04T10:10:06.867495+0000","last_peered":"2024-03-04T10:10:06.867495+0000","last_clean":"2024-03-04T10:10:06.867495+0000","last_became_active":"2024-03-04T10:10:06.863021+0000","last_became_peered":"2024-03-04T10:10:06.863021+0000","last_unstale":"2024-03-04T10:10:06.867495+0000","last_undegraded":"2024-03-04T10:10:06.867495+0000","last_fullsized":"2024-03-04T10:10:06.867495+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:02.877820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854753+0000","last_change":"2024-03-04T10:10:06.854753+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854753+0000","last_undegraded":"2024-03-04T10:10:06.854753+0000","last_fullsized":"2024-03-04T10:10:06.854753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.860442+0000","last_change":"2024-03-04T10:10:06.857045+0000","last_active":"2024-03-04T10:10:06.860442+0000","last_peered":"2024-03-04T10:10:06.860442+0000","last_clean":"2024-03-04T10:10:06.860442+0000","last_became_active":"2024-03-04T10:10:06.856753+0000","last_became_peered":"2024-03-04T10:10:06.856753+0000","last_unstale":"2024-03-04T10:10:06.860442+0000","last_undegraded":"2024-03-04T10:10:06.860442+0000","last_fullsized":"2024-03-04T10:10:06.860442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:51:19.769656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848193+0000","last_change":"2024-03-04T10:10:06.848193+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848193+0000","last_undegraded":"2024-03-04T10:10:06.848193+0000","last_fullsized":"2024-03-04T10:10:06.848193+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866070+0000","last_change":"2024-03-04T10:10:06.862082+0000","last_active":"2024-03-04T10:10:06.866070+0000","last_peered":"2024-03-04T10:10:06.866070+0000","last_clean":"2024-03-04T10:10:06.866070+0000","last_became_active":"2024-03-04T10:10:06.861780+0000","last_became_peered":"2024-03-04T10:10:06.861780+0000","last_unstale":"2024-03-04T10:10:06.866070+0000","last_undegraded":"2024-03-04T10:10:06.866070+0000","last_fullsized":"2024-03-04T10:10:06.866070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849240+0000","last_change":"2024-03-04T10:10:06.849240+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849240+0000","last_undegraded":"2024-03-04T10:10:06.849240+0000","last_fullsized":"2024-03-04T10:10:06.849240+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870454+0000","last_change":"2024-03-04T10:10:06.865722+0000","last_active":"2024-03-04T10:10:06.870454+0000","last_peered":"2024-03-04T10:10:06.870454+0000","last_clean":"2024-03-04T10:10:06.870454+0000","last_became_active":"2024-03-04T10:10:06.865459+0000","last_became_peered":"2024-03-04T10:10:06.865459+0000","last_unstale":"2024-03-04T10:10:06.870454+0000","last_undegraded":"2024-03-04T10:10:06.870454+0000","last_fullsized":"2024-03-04T10:10:06.870454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:50.057767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849672+0000","last_change":"2024-03-04T10:10:06.849672+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849672+0000","last_undegraded":"2024-03-04T10:10:06.849672+0000","last_fullsized":"2024-03-04T10:10:06.849672+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867470+0000","last_change":"2024-03-04T10:10:06.860947+0000","last_active":"2024-03-04T10:10:06.867470+0000","last_peered":"2024-03-04T10:10:06.867470+0000","last_clean":"2024-03-04T10:10:06.867470+0000","last_became_active":"2024-03-04T10:10:06.860654+0000","last_became_peered":"2024-03-04T10:10:06.860654+0000","last_unstale":"2024-03-04T10:10:06.867470+0000","last_undegraded":"2024-03-04T10:10:06.867470+0000","last_fullsized":"2024-03-04T10:10:06.867470+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:27.491076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853492+0000","last_change":"2024-03-04T10:10:06.853492+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853492+0000","last_undegraded":"2024-03-04T10:10:06.853492+0000","last_fullsized":"2024-03-04T10:10:06.853492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863318+0000","last_change":"2024-03-04T10:10:06.861468+0000","last_active":"2024-03-04T10:10:06.863318+0000","last_peered":"2024-03-04T10:10:06.863318+0000","last_clean":"2024-03-04T10:10:06.863318+0000","last_became_active":"2024-03-04T10:10:06.861177+0000","last_became_peered":"2024-03-04T10:10:06.861177+0000","last_unstale":"2024-03-04T10:10:06.863318+0000","last_undegraded":"2024-03-04T10:10:06.863318+0000","last_fullsized":"2024-03-04T10:10:06.863318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851887+0000","last_change":"2024-03-04T10:10:06.851887+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851887+0000","last_undegraded":"2024-03-04T10:10:06.851887+0000","last_fullsized":"2024-03-04T10:10:06.851887+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870472+0000","last_change":"2024-03-04T10:10:06.866005+0000","last_active":"2024-03-04T10:10:06.870472+0000","last_peered":"2024-03-04T10:10:06.870472+0000","last_clean":"2024-03-04T10:10:06.870472+0000","last_became_active":"2024-03-04T10:10:06.865735+0000","last_became_peered":"2024-03-04T10:10:06.865735+0000","last_unstale":"2024-03-04T10:10:06.870472+0000","last_undegraded":"2024-03-04T10:10:06.870472+0000","last_fullsized":"2024-03-04T10:10:06.870472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:28.683327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849865+0000","last_change":"2024-03-04T10:10:06.849865+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849865+0000","last_undegraded":"2024-03-04T10:10:06.849865+0000","last_fullsized":"2024-03-04T10:10:06.849865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861324+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.861027+0000","last_became_peered":"2024-03-04T10:10:06.861027+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:43.919308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850421+0000","last_change":"2024-03-04T10:10:06.850421+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850421+0000","last_undegraded":"2024-03-04T10:10:06.850421+0000","last_fullsized":"2024-03-04T10:10:06.850421+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867748+0000","last_change":"2024-03-04T10:10:06.864064+0000","last_active":"2024-03-04T10:10:06.867748+0000","last_peered":"2024-03-04T10:10:06.867748+0000","last_clean":"2024-03-04T10:10:06.867748+0000","last_became_active":"2024-03-04T10:10:06.863773+0000","last_became_peered":"2024-03-04T10:10:06.863773+0000","last_unstale":"2024-03-04T10:10:06.867748+0000","last_undegraded":"2024-03-04T10:10:06.867748+0000","last_fullsized":"2024-03-04T10:10:06.867748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:20.480886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849880+0000","last_change":"2024-03-04T10:10:06.849880+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849880+0000","last_undegraded":"2024-03-04T10:10:06.849880+0000","last_fullsized":"2024-03-04T10:10:06.849880+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863554+0000","last_change":"2024-03-04T10:10:06.858647+0000","last_active":"2024-03-04T10:10:06.863554+0000","last_peered":"2024-03-04T10:10:06.863554+0000","last_clean":"2024-03-04T10:10:06.863554+0000","last_became_active":"2024-03-04T10:10:06.858363+0000","last_became_peered":"2024-03-04T10:10:06.858363+0000","last_unstale":"2024-03-04T10:10:06.863554+0000","last_undegraded":"2024-03-04T10:10:06.863554+0000","last_fullsized":"2024-03-04T10:10:06.863554+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:59.513633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848011+0000","last_change":"2024-03-04T10:10:06.848011+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848011+0000","last_undegraded":"2024-03-04T10:10:06.848011+0000","last_fullsized":"2024-03-04T10:10:06.848011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864363+0000","last_change":"2024-03-04T10:10:06.860311+0000","last_active":"2024-03-04T10:10:06.864363+0000","last_peered":"2024-03-04T10:10:06.864363+0000","last_clean":"2024-03-04T10:10:06.864363+0000","last_became_active":"2024-03-04T10:10:06.860013+0000","last_became_peered":"2024-03-04T10:10:06.860013+0000","last_unstale":"2024-03-04T10:10:06.864363+0000","last_undegraded":"2024-03-04T10:10:06.864363+0000","last_fullsized":"2024-03-04T10:10:06.864363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:25.671449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848785+0000","last_change":"2024-03-04T10:10:06.848785+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848785+0000","last_undegraded":"2024-03-04T10:10:06.848785+0000","last_fullsized":"2024-03-04T10:10:06.848785+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864248+0000","last_change":"2024-03-04T10:10:06.859945+0000","last_active":"2024-03-04T10:10:06.864248+0000","last_peered":"2024-03-04T10:10:06.864248+0000","last_clean":"2024-03-04T10:10:06.864248+0000","last_became_active":"2024-03-04T10:10:06.859616+0000","last_became_peered":"2024-03-04T10:10:06.859616+0000","last_unstale":"2024-03-04T10:10:06.864248+0000","last_undegraded":"2024-03-04T10:10:06.864248+0000","last_fullsized":"2024-03-04T10:10:06.864248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:22.139606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849037+0000","last_change":"2024-03-04T10:10:06.849037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849037+0000","last_undegraded":"2024-03-04T10:10:06.849037+0000","last_fullsized":"2024-03-04T10:10:06.849037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865077+0000","last_change":"2024-03-04T10:10:06.862337+0000","last_active":"2024-03-04T10:10:06.865077+0000","last_peered":"2024-03-04T10:10:06.865077+0000","last_clean":"2024-03-04T10:10:06.865077+0000","last_became_active":"2024-03-04T10:10:06.861981+0000","last_became_peered":"2024-03-04T10:10:06.861981+0000","last_unstale":"2024-03-04T10:10:06.865077+0000","last_undegraded":"2024-03-04T10:10:06.865077+0000","last_fullsized":"2024-03-04T10:10:06.865077+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.881925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853333+0000","last_change":"2024-03-04T10:10:06.853333+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853333+0000","last_undegraded":"2024-03-04T10:10:06.853333+0000","last_fullsized":"2024-03-04T10:10:06.853333+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.859862+0000","last_change":"2024-03-04T10:10:06.858629+0000","last_active":"2024-03-04T10:10:06.859862+0000","last_peered":"2024-03-04T10:10:06.859862+0000","last_clean":"2024-03-04T10:10:06.859862+0000","last_became_active":"2024-03-04T10:10:06.858362+0000","last_became_peered":"2024-03-04T10:10:06.858362+0000","last_unstale":"2024-03-04T10:10:06.859862+0000","last_undegraded":"2024-03-04T10:10:06.859862+0000","last_fullsized":"2024-03-04T10:10:06.859862+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:28.667220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851428+0000","last_change":"2024-03-04T10:10:06.851428+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851428+0000","last_undegraded":"2024-03-04T10:10:06.851428+0000","last_fullsized":"2024-03-04T10:10:06.851428+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870494+0000","last_change":"2024-03-04T10:10:06.866361+0000","last_active":"2024-03-04T10:10:06.870494+0000","last_peered":"2024-03-04T10:10:06.870494+0000","last_clean":"2024-03-04T10:10:06.870494+0000","last_became_active":"2024-03-04T10:10:06.866083+0000","last_became_peered":"2024-03-04T10:10:06.866083+0000","last_unstale":"2024-03-04T10:10:06.870494+0000","last_undegraded":"2024-03-04T10:10:06.870494+0000","last_fullsized":"2024-03-04T10:10:06.870494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:57.646340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847765+0000","last_change":"2024-03-04T10:10:06.847765+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847765+0000","last_undegraded":"2024-03-04T10:10:06.847765+0000","last_fullsized":"2024-03-04T10:10:06.847765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.861629+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.861360+0000","last_became_peered":"2024-03-04T10:10:06.861360+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:34:32.909646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852388+0000","last_change":"2024-03-04T10:10:06.852388+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852388+0000","last_undegraded":"2024-03-04T10:10:06.852388+0000","last_fullsized":"2024-03-04T10:10:06.852388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.861908+0000","last_change":"2024-03-04T10:10:06.857599+0000","last_active":"2024-03-04T10:10:06.861908+0000","last_peered":"2024-03-04T10:10:06.861908+0000","last_clean":"2024-03-04T10:10:06.861908+0000","last_became_active":"2024-03-04T10:10:06.857319+0000","last_became_peered":"2024-03-04T10:10:06.857319+0000","last_unstale":"2024-03-04T10:10:06.861908+0000","last_undegraded":"2024-03-04T10:10:06.861908+0000","last_fullsized":"2024-03-04T10:10:06.861908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:49.470487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851702+0000","last_change":"2024-03-04T10:10:06.851702+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851702+0000","last_undegraded":"2024-03-04T10:10:06.851702+0000","last_fullsized":"2024-03-04T10:10:06.851702+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870325+0000","last_change":"2024-03-04T10:10:06.865016+0000","last_active":"2024-03-04T10:10:06.870325+0000","last_peered":"2024-03-04T10:10:06.870325+0000","last_clean":"2024-03-04T10:10:06.870325+0000","last_became_active":"2024-03-04T10:10:06.864692+0000","last_became_peered":"2024-03-04T10:10:06.864692+0000","last_unstale":"2024-03-04T10:10:06.870325+0000","last_undegraded":"2024-03-04T10:10:06.870325+0000","last_fullsized":"2024-03-04T10:10:06.870325+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:28:10.331623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849996+0000","last_change":"2024-03-04T10:10:06.849996+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849996+0000","last_undegraded":"2024-03-04T10:10:06.849996+0000","last_fullsized":"2024-03-04T10:10:06.849996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862041+0000","last_change":"2024-03-04T10:10:06.859809+0000","last_active":"2024-03-04T10:10:06.862041+0000","last_peered":"2024-03-04T10:10:06.862041+0000","last_clean":"2024-03-04T10:10:06.862041+0000","last_became_active":"2024-03-04T10:10:06.859499+0000","last_became_peered":"2024-03-04T10:10:06.859499+0000","last_unstale":"2024-03-04T10:10:06.862041+0000","last_undegraded":"2024-03-04T10:10:06.862041+0000","last_fullsized":"2024-03-04T10:10:06.862041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:17:31.377873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.856931+0000","last_change":"2024-03-04T10:10:06.856931+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.856931+0000","last_undegraded":"2024-03-04T10:10:06.856931+0000","last_fullsized":"2024-03-04T10:10:06.856931+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862008+0000","last_change":"2024-03-04T10:10:06.859472+0000","last_active":"2024-03-04T10:10:06.862008+0000","last_peered":"2024-03-04T10:10:06.862008+0000","last_clean":"2024-03-04T10:10:06.862008+0000","last_became_active":"2024-03-04T10:10:06.859206+0000","last_became_peered":"2024-03-04T10:10:06.859206+0000","last_unstale":"2024-03-04T10:10:06.862008+0000","last_undegraded":"2024-03-04T10:10:06.862008+0000","last_fullsized":"2024-03-04T10:10:06.862008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848039+0000","last_change":"2024-03-04T10:10:06.848039+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848039+0000","last_undegraded":"2024-03-04T10:10:06.848039+0000","last_fullsized":"2024-03-04T10:10:06.848039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866014+0000","last_change":"2024-03-04T10:10:06.860264+0000","last_active":"2024-03-04T10:10:06.866014+0000","last_peered":"2024-03-04T10:10:06.866014+0000","last_clean":"2024-03-04T10:10:06.866014+0000","last_became_active":"2024-03-04T10:10:06.859974+0000","last_became_peered":"2024-03-04T10:10:06.859974+0000","last_unstale":"2024-03-04T10:10:06.866014+0000","last_undegraded":"2024-03-04T10:10:06.866014+0000","last_fullsized":"2024-03-04T10:10:06.866014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:28:05.990062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848251+0000","last_change":"2024-03-04T10:10:06.848251+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848251+0000","last_undegraded":"2024-03-04T10:10:06.848251+0000","last_fullsized":"2024-03-04T10:10:06.848251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867523+0000","last_change":"2024-03-04T10:10:06.864593+0000","last_active":"2024-03-04T10:10:06.867523+0000","last_peered":"2024-03-04T10:10:06.867523+0000","last_clean":"2024-03-04T10:10:06.867523+0000","last_became_active":"2024-03-04T10:10:06.864298+0000","last_became_peered":"2024-03-04T10:10:06.864298+0000","last_unstale":"2024-03-04T10:10:06.867523+0000","last_undegraded":"2024-03-04T10:10:06.867523+0000","last_fullsized":"2024-03-04T10:10:06.867523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:44.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854228+0000","last_change":"2024-03-04T10:10:06.854228+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854228+0000","last_undegraded":"2024-03-04T10:10:06.854228+0000","last_fullsized":"2024-03-04T10:10:06.854228+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865410+0000","last_change":"2024-03-04T10:10:06.862581+0000","last_active":"2024-03-04T10:10:06.865410+0000","last_peered":"2024-03-04T10:10:06.865410+0000","last_clean":"2024-03-04T10:10:06.865410+0000","last_became_active":"2024-03-04T10:10:06.862283+0000","last_became_peered":"2024-03-04T10:10:06.862283+0000","last_unstale":"2024-03-04T10:10:06.865410+0000","last_undegraded":"2024-03-04T10:10:06.865410+0000","last_fullsized":"2024-03-04T10:10:06.865410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:44.951154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847286+0000","last_change":"2024-03-04T10:10:06.847286+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847286+0000","last_undegraded":"2024-03-04T10:10:06.847286+0000","last_fullsized":"2024-03-04T10:10:06.847286+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863815+0000","last_change":"2024-03-04T10:10:06.859084+0000","last_active":"2024-03-04T10:10:06.863815+0000","last_peered":"2024-03-04T10:10:06.863815+0000","last_clean":"2024-03-04T10:10:06.863815+0000","last_became_active":"2024-03-04T10:10:06.858820+0000","last_became_peered":"2024-03-04T10:10:06.858820+0000","last_unstale":"2024-03-04T10:10:06.863815+0000","last_undegraded":"2024-03-04T10:10:06.863815+0000","last_fullsized":"2024-03-04T10:10:06.863815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:29.284872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851394+0000","last_change":"2024-03-04T10:10:06.851394+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851394+0000","last_undegraded":"2024-03-04T10:10:06.851394+0000","last_fullsized":"2024-03-04T10:10:06.851394+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867553+0000","last_change":"2024-03-04T10:10:06.862727+0000","last_active":"2024-03-04T10:10:06.867553+0000","last_peered":"2024-03-04T10:10:06.867553+0000","last_clean":"2024-03-04T10:10:06.867553+0000","last_became_active":"2024-03-04T10:10:06.861472+0000","last_became_peered":"2024-03-04T10:10:06.861472+0000","last_unstale":"2024-03-04T10:10:06.867553+0000","last_undegraded":"2024-03-04T10:10:06.867553+0000","last_fullsized":"2024-03-04T10:10:06.867553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:33.683923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854465+0000","last_change":"2024-03-04T10:10:06.854465+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854465+0000","last_undegraded":"2024-03-04T10:10:06.854465+0000","last_fullsized":"2024-03-04T10:10:06.854465+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867555+0000","last_change":"2024-03-04T10:10:06.864463+0000","last_active":"2024-03-04T10:10:06.867555+0000","last_peered":"2024-03-04T10:10:06.867555+0000","last_clean":"2024-03-04T10:10:06.867555+0000","last_became_active":"2024-03-04T10:10:06.861931+0000","last_became_peered":"2024-03-04T10:10:06.861931+0000","last_unstale":"2024-03-04T10:10:06.867555+0000","last_undegraded":"2024-03-04T10:10:06.867555+0000","last_fullsized":"2024-03-04T10:10:06.867555+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:15.494976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849584+0000","last_change":"2024-03-04T10:10:06.849584+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849584+0000","last_undegraded":"2024-03-04T10:10:06.849584+0000","last_fullsized":"2024-03-04T10:10:06.849584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870433+0000","last_change":"2024-03-04T10:10:06.865079+0000","last_active":"2024-03-04T10:10:06.870433+0000","last_peered":"2024-03-04T10:10:06.870433+0000","last_clean":"2024-03-04T10:10:06.870433+0000","last_became_active":"2024-03-04T10:10:06.864808+0000","last_became_peered":"2024-03-04T10:10:06.864808+0000","last_unstale":"2024-03-04T10:10:06.870433+0000","last_undegraded":"2024-03-04T10:10:06.870433+0000","last_fullsized":"2024-03-04T10:10:06.870433+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:32.258399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851418+0000","last_change":"2024-03-04T10:10:06.851418+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851418+0000","last_undegraded":"2024-03-04T10:10:06.851418+0000","last_fullsized":"2024-03-04T10:10:06.851418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866583+0000","last_change":"2024-03-04T10:10:06.861954+0000","last_active":"2024-03-04T10:10:06.866583+0000","last_peered":"2024-03-04T10:10:06.866583+0000","last_clean":"2024-03-04T10:10:06.866583+0000","last_became_active":"2024-03-04T10:10:06.859757+0000","last_became_peered":"2024-03-04T10:10:06.859757+0000","last_unstale":"2024-03-04T10:10:06.866583+0000","last_undegraded":"2024-03-04T10:10:06.866583+0000","last_fullsized":"2024-03-04T10:10:06.866583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:58:45.314856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":315,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.854556+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:10:06.854556+0000","last_peered":"2024-03-04T10:10:06.854556+0000","last_clean":"2024-03-04T10:10:06.854556+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:10:06.854556+0000","last_undegraded":"2024-03-04T10:10:06.854556+0000","last_fullsized":"2024-03-04T10:10:06.854556+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343140,"statfs":{"total":96636764160,"available":96607375360,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:10:10.875 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:10:11.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.010+0000 7f648222b640 1 Processor -- start 2024-03-04T10:10:11.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.010+0000 7f648222b640 1 -- start start 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f648222b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f647c15de00 0x7f647c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f648222b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f647c0b9380 0x7f647c0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f648222b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c15e1e0 con 0x7f647c0b70d0 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f648222b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c0d2820 con 0x7f647c0b9380 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f648222b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c0d1c10 con 0x7f647c15de00 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f6472ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f647c0b9380 0x7f647c0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f647bfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f647c0b70d0 0x7f647c0b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59614/0 (socket says 172.21.15.99:59614) 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f647bfff640 1 -- 172.21.15.99:0/3691316725 learned_addr learned my addr 172.21.15.99:0/3691316725 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f6472ffd640 1 -- 172.21.15.99:0/3691316725 >> v2:172.21.15.162:3300/0 conn(0x7f647c15de00 msgr2=0x7f647c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f6472ffd640 1 --2- 172.21.15.99:0/3691316725 >> v2:172.21.15.162:3300/0 conn(0x7f647c15de00 0x7f647c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f6472ffd640 1 -- 172.21.15.99:0/3691316725 >> v1:172.21.15.99:6789/0 conn(0x7f647c0b70d0 legacy=0x7f647c0b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f6472ffd640 1 -- 172.21.15.99:0/3691316725 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c0d1ef0 con 0x7f647c0b9380 2024-03-04T10:10:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.011+0000 7f647affd640 1 -- 172.21.15.99:0/3691316725 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623188579 0 0) 0x7f647c15e1e0 con 0x7f647c0b70d0 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.012+0000 7f6472ffd640 1 --2- 172.21.15.99:0/3691316725 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b9380 0x7f647c0b6b90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f647c0ce240 tx=0x7f646c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a0c36d6e7d44db server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.012+0000 7f647affd640 1 -- 172.21.15.99:0/3691316725 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c035020 con 0x7f647c0b9380 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.012+0000 7f647affd640 1 -- 172.21.15.99:0/3691316725 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c031830 con 0x7f647c0b9380 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.012+0000 7f647affd640 1 -- 172.21.15.99:0/3691316725 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c031c00 con 0x7f647c0b9380 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 -- 172.21.15.99:0/3691316725 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b9380 msgr2=0x7f647c0b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 --2- 172.21.15.99:0/3691316725 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b9380 0x7f647c0b6b90 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f647c0ce240 tx=0x7f646c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 -- 172.21.15.99:0/3691316725 shutdown_connections 2024-03-04T10:10:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 --2- 172.21.15.99:0/3691316725 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b9380 0x7f647c0b6b90 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 --2- 172.21.15.99:0/3691316725 >> v2:172.21.15.162:3300/0 conn(0x7f647c15de00 0x7f647c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 -- 172.21.15.99:0/3691316725 >> 172.21.15.99:0/3691316725 conn(0x7f647c0d0470 msgr2=0x7f647c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 -- 172.21.15.99:0/3691316725 shutdown_connections 2024-03-04T10:10:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.013+0000 7f648222b640 1 -- 172.21.15.99:0/3691316725 wait complete. 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.014+0000 7f648222b640 1 Processor -- start 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.014+0000 7f648222b640 1 -- start start 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f648222b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 0x7f647c0aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f648222b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f647c0b9380 0x7f647c0ab170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f648222b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f647c15de00 0x7f647c159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f648222b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b97e0 con 0x7f647c0b70d0 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f648222b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0d24b0 con 0x7f647c15de00 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f648222b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b5b80 con 0x7f647c0b9380 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 0x7f647c0aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 0x7f647c0aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45566/0 (socket says 172.21.15.99:45566) 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 -- 172.21.15.99:0/108928740 learned_addr learned my addr 172.21.15.99:0/108928740 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647bfff640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f647c15de00 0x7f647c159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 -- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f647c0b9380 msgr2=0x7f647c0ab170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f647c0b9380 0x7f647c0ab170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 -- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f647c15de00 msgr2=0x7f647c159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f647c15de00 0x7f647c159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.015+0000 7f647b7fe640 1 -- 172.21.15.99:0/108928740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c15a550 con 0x7f647c0b70d0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.016+0000 7f647b7fe640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 0x7f647c0aa3d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f646400b440 tx=0x7f646400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.016+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464015020 con 0x7f647c0b70d0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.016+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6464011890 con 0x7f647c0b70d0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.016+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c0cdff0 con 0x7f647c0b70d0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.016+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464011c80 con 0x7f647c0b70d0 2024-03-04T10:10:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.016+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c0cae40 con 0x7f647c0b70d0 2024-03-04T10:10:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.017+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c074f60 con 0x7f647c0b70d0 2024-03-04T10:10:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.017+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6464018020 con 0x7f647c0b70d0 2024-03-04T10:10:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.018+0000 7f6478ff9640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f646006b4d0 0x7f646006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.018+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f646408a410 con 0x7f647c0b70d0 2024-03-04T10:10:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.018+0000 7f6472ffd640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f646006b4d0 0x7f646006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.019+0000 7f6472ffd640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f646006b4d0 0x7f646006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f646c036000 tx=0x7f646c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.022+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f646401f020 con 0x7f647c0b70d0 2024-03-04T10:10:11.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.195+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f647c0b2a00 con 0x7f646006b4d0 2024-03-04T10:10:11.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.201+0000 7f6478ff9640 1 -- 172.21.15.99:0/108928740 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246767 (secure 0 0 0) 0x7f647c0b2a00 con 0x7f646006b4d0 2024-03-04T10:10:11.206 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f646006b4d0 msgr2=0x7f646006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f646006b4d0 0x7f646006d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f646c036000 tx=0x7f646c030000 comp rx=0 tx=0).stop 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 msgr2=0x7f647c0aa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 0x7f647c0aa3d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f646400b440 tx=0x7f646400b900 comp rx=0 tx=0).stop 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 shutdown_connections 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f646006b4d0 0x7f646006d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f647c15de00 0x7f647c159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f647c0b9380 0x7f647c0ab170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 --2- 172.21.15.99:0/108928740 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f647c0b70d0 0x7f647c0aa3d0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 >> 172.21.15.99:0/108928740 conn(0x7f647c0d0470 msgr2=0x7f647c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 shutdown_connections 2024-03-04T10:10:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.204+0000 7f648222b640 1 -- 172.21.15.99:0/108928740 wait complete. 2024-03-04T10:10:11.221 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-03-04T10:10:09.532313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242772,"kb_used_data":9928,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754731948,"statfs":{"total":773094113280,"available":772845514752,"internally_reserved":0,"allocated":10166272,"data_stored":5347640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001198"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852633+0000","last_change":"2024-03-04T10:10:06.852633+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852633+0000","last_undegraded":"2024-03-04T10:10:06.852633+0000","last_fullsized":"2024-03-04T10:10:06.852633+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854680+0000","last_change":"2024-03-04T10:10:06.854680+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854680+0000","last_undegraded":"2024-03-04T10:10:06.854680+0000","last_fullsized":"2024-03-04T10:10:06.854680+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852344+0000","last_change":"2024-03-04T10:10:06.852344+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852344+0000","last_undegraded":"2024-03-04T10:10:06.852344+0000","last_fullsized":"2024-03-04T10:10:06.852344+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850373+0000","last_change":"2024-03-04T10:10:06.850373+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850373+0000","last_undegraded":"2024-03-04T10:10:06.850373+0000","last_fullsized":"2024-03-04T10:10:06.850373+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850436+0000","last_change":"2024-03-04T10:10:06.850436+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850436+0000","last_undegraded":"2024-03-04T10:10:06.850436+0000","last_fullsized":"2024-03-04T10:10:06.850436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854406+0000","last_change":"2024-03-04T10:10:06.854406+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854406+0000","last_undegraded":"2024-03-04T10:10:06.854406+0000","last_fullsized":"2024-03-04T10:10:06.854406+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853075+0000","last_change":"2024-03-04T10:10:06.853075+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853075+0000","last_undegraded":"2024-03-04T10:10:06.853075+0000","last_fullsized":"2024-03-04T10:10:06.853075+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851456+0000","last_change":"2024-03-04T10:10:06.851456+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851456+0000","last_undegraded":"2024-03-04T10:10:06.851456+0000","last_fullsized":"2024-03-04T10:10:06.851456+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850204+0000","last_change":"2024-03-04T10:10:06.850204+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850204+0000","last_undegraded":"2024-03-04T10:10:06.850204+0000","last_fullsized":"2024-03-04T10:10:06.850204+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848505+0000","last_change":"2024-03-04T10:10:06.848505+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848505+0000","last_undegraded":"2024-03-04T10:10:06.848505+0000","last_fullsized":"2024-03-04T10:10:06.848505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853200+0000","last_change":"2024-03-04T10:10:06.853200+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853200+0000","last_undegraded":"2024-03-04T10:10:06.853200+0000","last_fullsized":"2024-03-04T10:10:06.853200+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849214+0000","last_change":"2024-03-04T10:10:06.849214+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849214+0000","last_undegraded":"2024-03-04T10:10:06.849214+0000","last_fullsized":"2024-03-04T10:10:06.849214+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849674+0000","last_change":"2024-03-04T10:10:06.849674+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849674+0000","last_undegraded":"2024-03-04T10:10:06.849674+0000","last_fullsized":"2024-03-04T10:10:06.849674+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853490+0000","last_change":"2024-03-04T10:10:06.853490+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853490+0000","last_undegraded":"2024-03-04T10:10:06.853490+0000","last_fullsized":"2024-03-04T10:10:06.853490+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848629+0000","last_change":"2024-03-04T10:10:06.848629+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848629+0000","last_undegraded":"2024-03-04T10:10:06.848629+0000","last_fullsized":"2024-03-04T10:10:06.848629+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854342+0000","last_change":"2024-03-04T10:10:06.854342+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854342+0000","last_undegraded":"2024-03-04T10:10:06.854342+0000","last_fullsized":"2024-03-04T10:10:06.854342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852040+0000","last_change":"2024-03-04T10:10:06.852040+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852040+0000","last_undegraded":"2024-03-04T10:10:06.852040+0000","last_fullsized":"2024-03-04T10:10:06.852040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852330+0000","last_change":"2024-03-04T10:10:06.852330+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852330+0000","last_undegraded":"2024-03-04T10:10:06.852330+0000","last_fullsized":"2024-03-04T10:10:06.852330+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851041+0000","last_change":"2024-03-04T10:10:06.851041+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851041+0000","last_undegraded":"2024-03-04T10:10:06.851041+0000","last_fullsized":"2024-03-04T10:10:06.851041+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849980+0000","last_change":"2024-03-04T10:10:06.849980+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849980+0000","last_undegraded":"2024-03-04T10:10:06.849980+0000","last_fullsized":"2024-03-04T10:10:06.849980+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851155+0000","last_change":"2024-03-04T10:10:06.851155+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851155+0000","last_undegraded":"2024-03-04T10:10:06.851155+0000","last_fullsized":"2024-03-04T10:10:06.851155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853010+0000","last_change":"2024-03-04T10:10:06.853010+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853010+0000","last_undegraded":"2024-03-04T10:10:06.853010+0000","last_fullsized":"2024-03-04T10:10:06.853010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855177+0000","last_change":"2024-03-04T10:10:06.855177+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855177+0000","last_undegraded":"2024-03-04T10:10:06.855177+0000","last_fullsized":"2024-03-04T10:10:06.855177+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851940+0000","last_change":"2024-03-04T10:10:06.851940+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851940+0000","last_undegraded":"2024-03-04T10:10:06.851940+0000","last_fullsized":"2024-03-04T10:10:06.851940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854658+0000","last_change":"2024-03-04T10:10:06.854658+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854658+0000","last_undegraded":"2024-03-04T10:10:06.854658+0000","last_fullsized":"2024-03-04T10:10:06.854658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855850+0000","last_change":"2024-03-04T10:10:06.855850+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855850+0000","last_undegraded":"2024-03-04T10:10:06.855850+0000","last_fullsized":"2024-03-04T10:10:06.855850+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848047+0000","last_change":"2024-03-04T10:10:06.848047+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848047+0000","last_undegraded":"2024-03-04T10:10:06.848047+0000","last_fullsized":"2024-03-04T10:10:06.848047+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849542+0000","last_change":"2024-03-04T10:10:06.849542+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849542+0000","last_undegraded":"2024-03-04T10:10:06.849542+0000","last_fullsized":"2024-03-04T10:10:06.849542+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851655+0000","last_change":"2024-03-04T10:10:06.851655+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851655+0000","last_undegraded":"2024-03-04T10:10:06.851655+0000","last_fullsized":"2024-03-04T10:10:06.851655+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850703+0000","last_change":"2024-03-04T10:10:06.850703+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850703+0000","last_undegraded":"2024-03-04T10:10:06.850703+0000","last_fullsized":"2024-03-04T10:10:06.850703+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852821+0000","last_change":"2024-03-04T10:10:06.852821+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852821+0000","last_undegraded":"2024-03-04T10:10:06.852821+0000","last_fullsized":"2024-03-04T10:10:06.852821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851229+0000","last_change":"2024-03-04T10:10:06.851229+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851229+0000","last_undegraded":"2024-03-04T10:10:06.851229+0000","last_fullsized":"2024-03-04T10:10:06.851229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849419+0000","last_change":"2024-03-04T10:10:06.849419+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849419+0000","last_undegraded":"2024-03-04T10:10:06.849419+0000","last_fullsized":"2024-03-04T10:10:06.849419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.864395+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.864082+0000","last_became_peered":"2024-03-04T10:10:06.864082+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:45:18.950003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849862+0000","last_change":"2024-03-04T10:10:06.849862+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849862+0000","last_undegraded":"2024-03-04T10:10:06.849862+0000","last_fullsized":"2024-03-04T10:10:06.849862+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861011+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.860571+0000","last_became_peered":"2024-03-04T10:10:06.860571+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:35.744190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851356+0000","last_change":"2024-03-04T10:10:06.851356+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851356+0000","last_undegraded":"2024-03-04T10:10:06.851356+0000","last_fullsized":"2024-03-04T10:10:06.851356+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862539+0000","last_change":"2024-03-04T10:10:06.860271+0000","last_active":"2024-03-04T10:10:06.862539+0000","last_peered":"2024-03-04T10:10:06.862539+0000","last_clean":"2024-03-04T10:10:06.862539+0000","last_became_active":"2024-03-04T10:10:06.859984+0000","last_became_peered":"2024-03-04T10:10:06.859984+0000","last_unstale":"2024-03-04T10:10:06.862539+0000","last_undegraded":"2024-03-04T10:10:06.862539+0000","last_fullsized":"2024-03-04T10:10:06.862539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848478+0000","last_change":"2024-03-04T10:10:06.848478+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848478+0000","last_undegraded":"2024-03-04T10:10:06.848478+0000","last_fullsized":"2024-03-04T10:10:06.848478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870416+0000","last_change":"2024-03-04T10:10:06.865348+0000","last_active":"2024-03-04T10:10:06.870416+0000","last_peered":"2024-03-04T10:10:06.870416+0000","last_clean":"2024-03-04T10:10:06.870416+0000","last_became_active":"2024-03-04T10:10:06.865084+0000","last_became_peered":"2024-03-04T10:10:06.865084+0000","last_unstale":"2024-03-04T10:10:06.870416+0000","last_undegraded":"2024-03-04T10:10:06.870416+0000","last_fullsized":"2024-03-04T10:10:06.870416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:01:06.838971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851037+0000","last_change":"2024-03-04T10:10:06.851037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851037+0000","last_undegraded":"2024-03-04T10:10:06.851037+0000","last_fullsized":"2024-03-04T10:10:06.851037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867495+0000","last_change":"2024-03-04T10:10:06.863293+0000","last_active":"2024-03-04T10:10:06.867495+0000","last_peered":"2024-03-04T10:10:06.867495+0000","last_clean":"2024-03-04T10:10:06.867495+0000","last_became_active":"2024-03-04T10:10:06.863021+0000","last_became_peered":"2024-03-04T10:10:06.863021+0000","last_unstale":"2024-03-04T10:10:06.867495+0000","last_undegraded":"2024-03-04T10:10:06.867495+0000","last_fullsized":"2024-03-04T10:10:06.867495+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:02.877820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854753+0000","last_change":"2024-03-04T10:10:06.854753+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854753+0000","last_undegraded":"2024-03-04T10:10:06.854753+0000","last_fullsized":"2024-03-04T10:10:06.854753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.860442+0000","last_change":"2024-03-04T10:10:06.857045+0000","last_active":"2024-03-04T10:10:06.860442+0000","last_peered":"2024-03-04T10:10:06.860442+0000","last_clean":"2024-03-04T10:10:06.860442+0000","last_became_active":"2024-03-04T10:10:06.856753+0000","last_became_peered":"2024-03-04T10:10:06.856753+0000","last_unstale":"2024-03-04T10:10:06.860442+0000","last_undegraded":"2024-03-04T10:10:06.860442+0000","last_fullsized":"2024-03-04T10:10:06.860442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:51:19.769656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848193+0000","last_change":"2024-03-04T10:10:06.848193+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848193+0000","last_undegraded":"2024-03-04T10:10:06.848193+0000","last_fullsized":"2024-03-04T10:10:06.848193+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866070+0000","last_change":"2024-03-04T10:10:06.862082+0000","last_active":"2024-03-04T10:10:06.866070+0000","last_peered":"2024-03-04T10:10:06.866070+0000","last_clean":"2024-03-04T10:10:06.866070+0000","last_became_active":"2024-03-04T10:10:06.861780+0000","last_became_peered":"2024-03-04T10:10:06.861780+0000","last_unstale":"2024-03-04T10:10:06.866070+0000","last_undegraded":"2024-03-04T10:10:06.866070+0000","last_fullsized":"2024-03-04T10:10:06.866070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849240+0000","last_change":"2024-03-04T10:10:06.849240+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849240+0000","last_undegraded":"2024-03-04T10:10:06.849240+0000","last_fullsized":"2024-03-04T10:10:06.849240+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870454+0000","last_change":"2024-03-04T10:10:06.865722+0000","last_active":"2024-03-04T10:10:06.870454+0000","last_peered":"2024-03-04T10:10:06.870454+0000","last_clean":"2024-03-04T10:10:06.870454+0000","last_became_active":"2024-03-04T10:10:06.865459+0000","last_became_peered":"2024-03-04T10:10:06.865459+0000","last_unstale":"2024-03-04T10:10:06.870454+0000","last_undegraded":"2024-03-04T10:10:06.870454+0000","last_fullsized":"2024-03-04T10:10:06.870454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:50.057767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849672+0000","last_change":"2024-03-04T10:10:06.849672+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849672+0000","last_undegraded":"2024-03-04T10:10:06.849672+0000","last_fullsized":"2024-03-04T10:10:06.849672+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867470+0000","last_change":"2024-03-04T10:10:06.860947+0000","last_active":"2024-03-04T10:10:06.867470+0000","last_peered":"2024-03-04T10:10:06.867470+0000","last_clean":"2024-03-04T10:10:06.867470+0000","last_became_active":"2024-03-04T10:10:06.860654+0000","last_became_peered":"2024-03-04T10:10:06.860654+0000","last_unstale":"2024-03-04T10:10:06.867470+0000","last_undegraded":"2024-03-04T10:10:06.867470+0000","last_fullsized":"2024-03-04T10:10:06.867470+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:27.491076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853492+0000","last_change":"2024-03-04T10:10:06.853492+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853492+0000","last_undegraded":"2024-03-04T10:10:06.853492+0000","last_fullsized":"2024-03-04T10:10:06.853492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863318+0000","last_change":"2024-03-04T10:10:06.861468+0000","last_active":"2024-03-04T10:10:06.863318+0000","last_peered":"2024-03-04T10:10:06.863318+0000","last_clean":"2024-03-04T10:10:06.863318+0000","last_became_active":"2024-03-04T10:10:06.861177+0000","last_became_peered":"2024-03-04T10:10:06.861177+0000","last_unstale":"2024-03-04T10:10:06.863318+0000","last_undegraded":"2024-03-04T10:10:06.863318+0000","last_fullsized":"2024-03-04T10:10:06.863318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851887+0000","last_change":"2024-03-04T10:10:06.851887+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851887+0000","last_undegraded":"2024-03-04T10:10:06.851887+0000","last_fullsized":"2024-03-04T10:10:06.851887+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870472+0000","last_change":"2024-03-04T10:10:06.866005+0000","last_active":"2024-03-04T10:10:06.870472+0000","last_peered":"2024-03-04T10:10:06.870472+0000","last_clean":"2024-03-04T10:10:06.870472+0000","last_became_active":"2024-03-04T10:10:06.865735+0000","last_became_peered":"2024-03-04T10:10:06.865735+0000","last_unstale":"2024-03-04T10:10:06.870472+0000","last_undegraded":"2024-03-04T10:10:06.870472+0000","last_fullsized":"2024-03-04T10:10:06.870472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:28.683327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849865+0000","last_change":"2024-03-04T10:10:06.849865+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849865+0000","last_undegraded":"2024-03-04T10:10:06.849865+0000","last_fullsized":"2024-03-04T10:10:06.849865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861324+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.861027+0000","last_became_peered":"2024-03-04T10:10:06.861027+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:43.919308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850421+0000","last_change":"2024-03-04T10:10:06.850421+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850421+0000","last_undegraded":"2024-03-04T10:10:06.850421+0000","last_fullsized":"2024-03-04T10:10:06.850421+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867748+0000","last_change":"2024-03-04T10:10:06.864064+0000","last_active":"2024-03-04T10:10:06.867748+0000","last_peered":"2024-03-04T10:10:06.867748+0000","last_clean":"2024-03-04T10:10:06.867748+0000","last_became_active":"2024-03-04T10:10:06.863773+0000","last_became_peered":"2024-03-04T10:10:06.863773+0000","last_unstale":"2024-03-04T10:10:06.867748+0000","last_undegraded":"2024-03-04T10:10:06.867748+0000","last_fullsized":"2024-03-04T10:10:06.867748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:20.480886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849880+0000","last_change":"2024-03-04T10:10:06.849880+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849880+0000","last_undegraded":"2024-03-04T10:10:06.849880+0000","last_fullsized":"2024-03-04T10:10:06.849880+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863554+0000","last_change":"2024-03-04T10:10:06.858647+0000","last_active":"2024-03-04T10:10:06.863554+0000","last_peered":"2024-03-04T10:10:06.863554+0000","last_clean":"2024-03-04T10:10:06.863554+0000","last_became_active":"2024-03-04T10:10:06.858363+0000","last_became_peered":"2024-03-04T10:10:06.858363+0000","last_unstale":"2024-03-04T10:10:06.863554+0000","last_undegraded":"2024-03-04T10:10:06.863554+0000","last_fullsized":"2024-03-04T10:10:06.863554+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:59.513633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848011+0000","last_change":"2024-03-04T10:10:06.848011+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848011+0000","last_undegraded":"2024-03-04T10:10:06.848011+0000","last_fullsized":"2024-03-04T10:10:06.848011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864363+0000","last_change":"2024-03-04T10:10:06.860311+0000","last_active":"2024-03-04T10:10:06.864363+0000","last_peered":"2024-03-04T10:10:06.864363+0000","last_clean":"2024-03-04T10:10:06.864363+0000","last_became_active":"2024-03-04T10:10:06.860013+0000","last_became_peered":"2024-03-04T10:10:06.860013+0000","last_unstale":"2024-03-04T10:10:06.864363+0000","last_undegraded":"2024-03-04T10:10:06.864363+0000","last_fullsized":"2024-03-04T10:10:06.864363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:25.671449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848785+0000","last_change":"2024-03-04T10:10:06.848785+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848785+0000","last_undegraded":"2024-03-04T10:10:06.848785+0000","last_fullsized":"2024-03-04T10:10:06.848785+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864248+0000","last_change":"2024-03-04T10:10:06.859945+0000","last_active":"2024-03-04T10:10:06.864248+0000","last_peered":"2024-03-04T10:10:06.864248+0000","last_clean":"2024-03-04T10:10:06.864248+0000","last_became_active":"2024-03-04T10:10:06.859616+0000","last_became_peered":"2024-03-04T10:10:06.859616+0000","last_unstale":"2024-03-04T10:10:06.864248+0000","last_undegraded":"2024-03-04T10:10:06.864248+0000","last_fullsized":"2024-03-04T10:10:06.864248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:22.139606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849037+0000","last_change":"2024-03-04T10:10:06.849037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849037+0000","last_undegraded":"2024-03-04T10:10:06.849037+0000","last_fullsized":"2024-03-04T10:10:06.849037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865077+0000","last_change":"2024-03-04T10:10:06.862337+0000","last_active":"2024-03-04T10:10:06.865077+0000","last_peered":"2024-03-04T10:10:06.865077+0000","last_clean":"2024-03-04T10:10:06.865077+0000","last_became_active":"2024-03-04T10:10:06.861981+0000","last_became_peered":"2024-03-04T10:10:06.861981+0000","last_unstale":"2024-03-04T10:10:06.865077+0000","last_undegraded":"2024-03-04T10:10:06.865077+0000","last_fullsized":"2024-03-04T10:10:06.865077+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.881925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853333+0000","last_change":"2024-03-04T10:10:06.853333+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853333+0000","last_undegraded":"2024-03-04T10:10:06.853333+0000","last_fullsized":"2024-03-04T10:10:06.853333+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.859862+0000","last_change":"2024-03-04T10:10:06.858629+0000","last_active":"2024-03-04T10:10:06.859862+0000","last_peered":"2024-03-04T10:10:06.859862+0000","last_clean":"2024-03-04T10:10:06.859862+0000","last_became_active":"2024-03-04T10:10:06.858362+0000","last_became_peered":"2024-03-04T10:10:06.858362+0000","last_unstale":"2024-03-04T10:10:06.859862+0000","last_undegraded":"2024-03-04T10:10:06.859862+0000","last_fullsized":"2024-03-04T10:10:06.859862+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:28.667220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851428+0000","last_change":"2024-03-04T10:10:06.851428+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851428+0000","last_undegraded":"2024-03-04T10:10:06.851428+0000","last_fullsized":"2024-03-04T10:10:06.851428+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870494+0000","last_change":"2024-03-04T10:10:06.866361+0000","last_active":"2024-03-04T10:10:06.870494+0000","last_peered":"2024-03-04T10:10:06.870494+0000","last_clean":"2024-03-04T10:10:06.870494+0000","last_became_active":"2024-03-04T10:10:06.866083+0000","last_became_peered":"2024-03-04T10:10:06.866083+0000","last_unstale":"2024-03-04T10:10:06.870494+0000","last_undegraded":"2024-03-04T10:10:06.870494+0000","last_fullsized":"2024-03-04T10:10:06.870494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:57.646340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847765+0000","last_change":"2024-03-04T10:10:06.847765+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847765+0000","last_undegraded":"2024-03-04T10:10:06.847765+0000","last_fullsized":"2024-03-04T10:10:06.847765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.861629+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.861360+0000","last_became_peered":"2024-03-04T10:10:06.861360+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:34:32.909646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852388+0000","last_change":"2024-03-04T10:10:06.852388+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852388+0000","last_undegraded":"2024-03-04T10:10:06.852388+0000","last_fullsized":"2024-03-04T10:10:06.852388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.861908+0000","last_change":"2024-03-04T10:10:06.857599+0000","last_active":"2024-03-04T10:10:06.861908+0000","last_peered":"2024-03-04T10:10:06.861908+0000","last_clean":"2024-03-04T10:10:06.861908+0000","last_became_active":"2024-03-04T10:10:06.857319+0000","last_became_peered":"2024-03-04T10:10:06.857319+0000","last_unstale":"2024-03-04T10:10:06.861908+0000","last_undegraded":"2024-03-04T10:10:06.861908+0000","last_fullsized":"2024-03-04T10:10:06.861908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:49.470487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851702+0000","last_change":"2024-03-04T10:10:06.851702+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851702+0000","last_undegraded":"2024-03-04T10:10:06.851702+0000","last_fullsized":"2024-03-04T10:10:06.851702+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870325+0000","last_change":"2024-03-04T10:10:06.865016+0000","last_active":"2024-03-04T10:10:06.870325+0000","last_peered":"2024-03-04T10:10:06.870325+0000","last_clean":"2024-03-04T10:10:06.870325+0000","last_became_active":"2024-03-04T10:10:06.864692+0000","last_became_peered":"2024-03-04T10:10:06.864692+0000","last_unstale":"2024-03-04T10:10:06.870325+0000","last_undegraded":"2024-03-04T10:10:06.870325+0000","last_fullsized":"2024-03-04T10:10:06.870325+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:28:10.331623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849996+0000","last_change":"2024-03-04T10:10:06.849996+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849996+0000","last_undegraded":"2024-03-04T10:10:06.849996+0000","last_fullsized":"2024-03-04T10:10:06.849996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862041+0000","last_change":"2024-03-04T10:10:06.859809+0000","last_active":"2024-03-04T10:10:06.862041+0000","last_peered":"2024-03-04T10:10:06.862041+0000","last_clean":"2024-03-04T10:10:06.862041+0000","last_became_active":"2024-03-04T10:10:06.859499+0000","last_became_peered":"2024-03-04T10:10:06.859499+0000","last_unstale":"2024-03-04T10:10:06.862041+0000","last_undegraded":"2024-03-04T10:10:06.862041+0000","last_fullsized":"2024-03-04T10:10:06.862041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:17:31.377873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.856931+0000","last_change":"2024-03-04T10:10:06.856931+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.856931+0000","last_undegraded":"2024-03-04T10:10:06.856931+0000","last_fullsized":"2024-03-04T10:10:06.856931+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862008+0000","last_change":"2024-03-04T10:10:06.859472+0000","last_active":"2024-03-04T10:10:06.862008+0000","last_peered":"2024-03-04T10:10:06.862008+0000","last_clean":"2024-03-04T10:10:06.862008+0000","last_became_active":"2024-03-04T10:10:06.859206+0000","last_became_peered":"2024-03-04T10:10:06.859206+0000","last_unstale":"2024-03-04T10:10:06.862008+0000","last_undegraded":"2024-03-04T10:10:06.862008+0000","last_fullsized":"2024-03-04T10:10:06.862008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848039+0000","last_change":"2024-03-04T10:10:06.848039+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848039+0000","last_undegraded":"2024-03-04T10:10:06.848039+0000","last_fullsized":"2024-03-04T10:10:06.848039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866014+0000","last_change":"2024-03-04T10:10:06.860264+0000","last_active":"2024-03-04T10:10:06.866014+0000","last_peered":"2024-03-04T10:10:06.866014+0000","last_clean":"2024-03-04T10:10:06.866014+0000","last_became_active":"2024-03-04T10:10:06.859974+0000","last_became_peered":"2024-03-04T10:10:06.859974+0000","last_unstale":"2024-03-04T10:10:06.866014+0000","last_undegraded":"2024-03-04T10:10:06.866014+0000","last_fullsized":"2024-03-04T10:10:06.866014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:28:05.990062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848251+0000","last_change":"2024-03-04T10:10:06.848251+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848251+0000","last_undegraded":"2024-03-04T10:10:06.848251+0000","last_fullsized":"2024-03-04T10:10:06.848251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867523+0000","last_change":"2024-03-04T10:10:06.864593+0000","last_active":"2024-03-04T10:10:06.867523+0000","last_peered":"2024-03-04T10:10:06.867523+0000","last_clean":"2024-03-04T10:10:06.867523+0000","last_became_active":"2024-03-04T10:10:06.864298+0000","last_became_peered":"2024-03-04T10:10:06.864298+0000","last_unstale":"2024-03-04T10:10:06.867523+0000","last_undegraded":"2024-03-04T10:10:06.867523+0000","last_fullsized":"2024-03-04T10:10:06.867523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:44.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854228+0000","last_change":"2024-03-04T10:10:06.854228+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854228+0000","last_undegraded":"2024-03-04T10:10:06.854228+0000","last_fullsized":"2024-03-04T10:10:06.854228+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865410+0000","last_change":"2024-03-04T10:10:06.862581+0000","last_active":"2024-03-04T10:10:06.865410+0000","last_peered":"2024-03-04T10:10:06.865410+0000","last_clean":"2024-03-04T10:10:06.865410+0000","last_became_active":"2024-03-04T10:10:06.862283+0000","last_became_peered":"2024-03-04T10:10:06.862283+0000","last_unstale":"2024-03-04T10:10:06.865410+0000","last_undegraded":"2024-03-04T10:10:06.865410+0000","last_fullsized":"2024-03-04T10:10:06.865410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:44.951154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847286+0000","last_change":"2024-03-04T10:10:06.847286+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847286+0000","last_undegraded":"2024-03-04T10:10:06.847286+0000","last_fullsized":"2024-03-04T10:10:06.847286+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863815+0000","last_change":"2024-03-04T10:10:06.859084+0000","last_active":"2024-03-04T10:10:06.863815+0000","last_peered":"2024-03-04T10:10:06.863815+0000","last_clean":"2024-03-04T10:10:06.863815+0000","last_became_active":"2024-03-04T10:10:06.858820+0000","last_became_peered":"2024-03-04T10:10:06.858820+0000","last_unstale":"2024-03-04T10:10:06.863815+0000","last_undegraded":"2024-03-04T10:10:06.863815+0000","last_fullsized":"2024-03-04T10:10:06.863815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:29.284872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851394+0000","last_change":"2024-03-04T10:10:06.851394+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851394+0000","last_undegraded":"2024-03-04T10:10:06.851394+0000","last_fullsized":"2024-03-04T10:10:06.851394+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867553+0000","last_change":"2024-03-04T10:10:06.862727+0000","last_active":"2024-03-04T10:10:06.867553+0000","last_peered":"2024-03-04T10:10:06.867553+0000","last_clean":"2024-03-04T10:10:06.867553+0000","last_became_active":"2024-03-04T10:10:06.861472+0000","last_became_peered":"2024-03-04T10:10:06.861472+0000","last_unstale":"2024-03-04T10:10:06.867553+0000","last_undegraded":"2024-03-04T10:10:06.867553+0000","last_fullsized":"2024-03-04T10:10:06.867553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:33.683923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854465+0000","last_change":"2024-03-04T10:10:06.854465+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854465+0000","last_undegraded":"2024-03-04T10:10:06.854465+0000","last_fullsized":"2024-03-04T10:10:06.854465+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867555+0000","last_change":"2024-03-04T10:10:06.864463+0000","last_active":"2024-03-04T10:10:06.867555+0000","last_peered":"2024-03-04T10:10:06.867555+0000","last_clean":"2024-03-04T10:10:06.867555+0000","last_became_active":"2024-03-04T10:10:06.861931+0000","last_became_peered":"2024-03-04T10:10:06.861931+0000","last_unstale":"2024-03-04T10:10:06.867555+0000","last_undegraded":"2024-03-04T10:10:06.867555+0000","last_fullsized":"2024-03-04T10:10:06.867555+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:15.494976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849584+0000","last_change":"2024-03-04T10:10:06.849584+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849584+0000","last_undegraded":"2024-03-04T10:10:06.849584+0000","last_fullsized":"2024-03-04T10:10:06.849584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870433+0000","last_change":"2024-03-04T10:10:06.865079+0000","last_active":"2024-03-04T10:10:06.870433+0000","last_peered":"2024-03-04T10:10:06.870433+0000","last_clean":"2024-03-04T10:10:06.870433+0000","last_became_active":"2024-03-04T10:10:06.864808+0000","last_became_peered":"2024-03-04T10:10:06.864808+0000","last_unstale":"2024-03-04T10:10:06.870433+0000","last_undegraded":"2024-03-04T10:10:06.870433+0000","last_fullsized":"2024-03-04T10:10:06.870433+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:32.258399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851418+0000","last_change":"2024-03-04T10:10:06.851418+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851418+0000","last_undegraded":"2024-03-04T10:10:06.851418+0000","last_fullsized":"2024-03-04T10:10:06.851418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866583+0000","last_change":"2024-03-04T10:10:06.861954+0000","last_active":"2024-03-04T10:10:06.866583+0000","last_peered":"2024-03-04T10:10:06.866583+0000","last_clean":"2024-03-04T10:10:06.866583+0000","last_became_active":"2024-03-04T10:10:06.859757+0000","last_became_peered":"2024-03-04T10:10:06.859757+0000","last_unstale":"2024-03-04T10:10:06.866583+0000","last_undegraded":"2024-03-04T10:10:06.866583+0000","last_fullsized":"2024-03-04T10:10:06.866583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:58:45.314856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":315,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.854556+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:10:06.854556+0000","last_peered":"2024-03-04T10:10:06.854556+0000","last_clean":"2024-03-04T10:10:06.854556+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:10:06.854556+0000","last_undegraded":"2024-03-04T10:10:06.854556+0000","last_fullsized":"2024-03-04T10:10:06.854556+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343140,"statfs":{"total":96636764160,"available":96607375360,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:10:11.224 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:10:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.359+0000 7f0e18f22640 1 Processor -- start 2024-03-04T10:10:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.359+0000 7f0e18f22640 1 -- start start 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e18f22640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e140d1290 0x7f0e140cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e18f22640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1406ae50 con 0x7f0e140cfae0 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e18f22640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1406b130 con 0x7f0e140cc8e0 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e18f22640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1406b410 con 0x7f0e140d1290 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e12d9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0e140d1290 0x7f0e140cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e1259c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f0e140cfae0 0x7f0e140cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59626/0 (socket says 172.21.15.99:59626) 2024-03-04T10:10:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.360+0000 7f0e1259c640 1 -- 172.21.15.99:0/3625904190 learned_addr learned my addr 172.21.15.99:0/3625904190 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.361+0000 7f0e12d9d640 1 -- 172.21.15.99:0/3625904190 >> v1:172.21.15.162:6789/0 conn(0x7f0e140cc8e0 legacy=0x7f0e140cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.361+0000 7f0e12d9d640 1 -- 172.21.15.99:0/3625904190 >> v1:172.21.15.99:6789/0 conn(0x7f0e140cfae0 legacy=0x7f0e140cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.361+0000 7f0e12d9d640 1 -- 172.21.15.99:0/3625904190 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1406b6f0 con 0x7f0e140d1290 2024-03-04T10:10:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.361+0000 7f0e11d9b640 1 -- 172.21.15.99:0/3625904190 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140066020 0 0) 0x7f0e1406ae50 con 0x7f0e140cfae0 2024-03-04T10:10:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e12d9d640 1 --2- 172.21.15.99:0/3625904190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140d1290 0x7f0e140cd440 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009d90 tx=0x7f0e0c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b0542d340d0e9c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e11d9b640 1 -- 172.21.15.99:0/3625904190 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c039070 con 0x7f0e140d1290 2024-03-04T10:10:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e11d9b640 1 -- 172.21.15.99:0/3625904190 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0c02cc60 con 0x7f0e140d1290 2024-03-04T10:10:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e11d9b640 1 -- 172.21.15.99:0/3625904190 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c034660 con 0x7f0e140d1290 2024-03-04T10:10:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e18f22640 1 -- 172.21.15.99:0/3625904190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140d1290 msgr2=0x7f0e140cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e18f22640 1 --2- 172.21.15.99:0/3625904190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140d1290 0x7f0e140cd440 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009d90 tx=0x7f0e0c02beb0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e18f22640 1 -- 172.21.15.99:0/3625904190 shutdown_connections 2024-03-04T10:10:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e18f22640 1 --2- 172.21.15.99:0/3625904190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140d1290 0x7f0e140cd440 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.362+0000 7f0e18f22640 1 -- 172.21.15.99:0/3625904190 >> 172.21.15.99:0/3625904190 conn(0x7f0e140d0770 msgr2=0x7f0e140d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.363+0000 7f0e18f22640 1 -- 172.21.15.99:0/3625904190 shutdown_connections 2024-03-04T10:10:11.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.363+0000 7f0e18f22640 1 -- 172.21.15.99:0/3625904190 wait complete. 2024-03-04T10:10:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 Processor -- start 2024-03-04T10:10:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 -- start start 2024-03-04T10:10:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e140cc8e0 0x7f0e140c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 0x7f0e14151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e140d1290 0x7f0e140b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140cd980 con 0x7f0e140cfae0 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1406b9d0 con 0x7f0e140d1290 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.364+0000 7f0e18f22640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140cdb00 con 0x7f0e140cc8e0 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 0x7f0e14151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 0x7f0e14151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45580/0 (socket says 172.21.15.99:45580) 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 -- 172.21.15.99:0/380572238 learned_addr learned my addr 172.21.15.99:0/380572238 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e12d9d640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e140d1290 0x7f0e140b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e1259c640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e140cc8e0 0x7f0e140c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 -- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e140cc8e0 msgr2=0x7f0e140c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e140cc8e0 0x7f0e140c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 -- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e140d1290 msgr2=0x7f0e140b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e140d1290 0x7f0e140b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 -- 172.21.15.99:0/380572238 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e140c85e0 con 0x7f0e140cfae0 2024-03-04T10:10:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e1259c640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e140cc8e0 0x7f0e140c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e12d9d640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e140d1290 0x7f0e140b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0bfff640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 0x7f0e14151860 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c410 tx=0x7f0dfc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc017070 con 0x7f0e140cfae0 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc0035f0 con 0x7f0e140cfae0 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e140c88c0 con 0x7f0e140cfae0 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0039e0 con 0x7f0e140cfae0 2024-03-04T10:10:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.365+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e140c8e80 con 0x7f0e140cfae0 2024-03-04T10:10:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.367+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1406cd70 con 0x7f0e140cfae0 2024-03-04T10:10:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.367+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0dfc01a070 con 0x7f0e140cfae0 2024-03-04T10:10:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.367+0000 7f0e0affd640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0df406b4d0 0x7f0df406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.367+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0dfc089740 con 0x7f0e140cfae0 2024-03-04T10:10:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.367+0000 7f0e1259c640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0df406b4d0 0x7f0df406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.368+0000 7f0e1259c640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0df406b4d0 0x7f0df406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0e04000f60 tx=0x7f0e04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.371+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0dfc0584c0 con 0x7f0e140cfae0 2024-03-04T10:10:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.554+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e14074f60 con 0x7f0df406b4d0 2024-03-04T10:10:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.561+0000 7f0e0affd640 1 -- 172.21.15.99:0/380572238 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246767 (secure 0 0 0) 0x7f0e14074f60 con 0x7f0df406b4d0 2024-03-04T10:10:11.566 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:10:11.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.563+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0df406b4d0 msgr2=0x7f0df406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.563+0000 7f0e18f22640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0df406b4d0 0x7f0df406d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0e04000f60 tx=0x7f0e04009040 comp rx=0 tx=0).stop 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.563+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 msgr2=0x7f0e14151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.563+0000 7f0e18f22640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 0x7f0e14151860 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c410 tx=0x7f0dfc00c8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 shutdown_connections 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0df406b4d0 0x7f0df406d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0e140d1290 0x7f0e140b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0e140cfae0 0x7f0e14151860 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 --2- 172.21.15.99:0/380572238 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0e140cc8e0 0x7f0e140c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 >> 172.21.15.99:0/380572238 conn(0x7f0e140d0770 msgr2=0x7f0e140cada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 shutdown_connections 2024-03-04T10:10:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.564+0000 7f0e18f22640 1 -- 172.21.15.99:0/380572238 wait complete. 2024-03-04T10:10:11.576 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-04T10:10:11.532750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242772,"kb_used_data":9928,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754731948,"statfs":{"total":773094113280,"available":772845514752,"internally_reserved":0,"allocated":10166272,"data_stored":5347640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.695135"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852633+0000","last_change":"2024-03-04T10:10:06.852633+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852633+0000","last_undegraded":"2024-03-04T10:10:06.852633+0000","last_fullsized":"2024-03-04T10:10:06.852633+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854680+0000","last_change":"2024-03-04T10:10:06.854680+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854680+0000","last_undegraded":"2024-03-04T10:10:06.854680+0000","last_fullsized":"2024-03-04T10:10:06.854680+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852344+0000","last_change":"2024-03-04T10:10:06.852344+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852344+0000","last_undegraded":"2024-03-04T10:10:06.852344+0000","last_fullsized":"2024-03-04T10:10:06.852344+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850373+0000","last_change":"2024-03-04T10:10:06.850373+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850373+0000","last_undegraded":"2024-03-04T10:10:06.850373+0000","last_fullsized":"2024-03-04T10:10:06.850373+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850436+0000","last_change":"2024-03-04T10:10:06.850436+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850436+0000","last_undegraded":"2024-03-04T10:10:06.850436+0000","last_fullsized":"2024-03-04T10:10:06.850436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854406+0000","last_change":"2024-03-04T10:10:06.854406+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854406+0000","last_undegraded":"2024-03-04T10:10:06.854406+0000","last_fullsized":"2024-03-04T10:10:06.854406+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853075+0000","last_change":"2024-03-04T10:10:06.853075+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853075+0000","last_undegraded":"2024-03-04T10:10:06.853075+0000","last_fullsized":"2024-03-04T10:10:06.853075+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851456+0000","last_change":"2024-03-04T10:10:06.851456+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851456+0000","last_undegraded":"2024-03-04T10:10:06.851456+0000","last_fullsized":"2024-03-04T10:10:06.851456+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850204+0000","last_change":"2024-03-04T10:10:06.850204+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850204+0000","last_undegraded":"2024-03-04T10:10:06.850204+0000","last_fullsized":"2024-03-04T10:10:06.850204+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848505+0000","last_change":"2024-03-04T10:10:06.848505+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848505+0000","last_undegraded":"2024-03-04T10:10:06.848505+0000","last_fullsized":"2024-03-04T10:10:06.848505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853200+0000","last_change":"2024-03-04T10:10:06.853200+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853200+0000","last_undegraded":"2024-03-04T10:10:06.853200+0000","last_fullsized":"2024-03-04T10:10:06.853200+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849214+0000","last_change":"2024-03-04T10:10:06.849214+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849214+0000","last_undegraded":"2024-03-04T10:10:06.849214+0000","last_fullsized":"2024-03-04T10:10:06.849214+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849674+0000","last_change":"2024-03-04T10:10:06.849674+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849674+0000","last_undegraded":"2024-03-04T10:10:06.849674+0000","last_fullsized":"2024-03-04T10:10:06.849674+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853490+0000","last_change":"2024-03-04T10:10:06.853490+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853490+0000","last_undegraded":"2024-03-04T10:10:06.853490+0000","last_fullsized":"2024-03-04T10:10:06.853490+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848629+0000","last_change":"2024-03-04T10:10:06.848629+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848629+0000","last_undegraded":"2024-03-04T10:10:06.848629+0000","last_fullsized":"2024-03-04T10:10:06.848629+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854342+0000","last_change":"2024-03-04T10:10:06.854342+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854342+0000","last_undegraded":"2024-03-04T10:10:06.854342+0000","last_fullsized":"2024-03-04T10:10:06.854342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852040+0000","last_change":"2024-03-04T10:10:06.852040+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852040+0000","last_undegraded":"2024-03-04T10:10:06.852040+0000","last_fullsized":"2024-03-04T10:10:06.852040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852330+0000","last_change":"2024-03-04T10:10:06.852330+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852330+0000","last_undegraded":"2024-03-04T10:10:06.852330+0000","last_fullsized":"2024-03-04T10:10:06.852330+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851041+0000","last_change":"2024-03-04T10:10:06.851041+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851041+0000","last_undegraded":"2024-03-04T10:10:06.851041+0000","last_fullsized":"2024-03-04T10:10:06.851041+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849980+0000","last_change":"2024-03-04T10:10:06.849980+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849980+0000","last_undegraded":"2024-03-04T10:10:06.849980+0000","last_fullsized":"2024-03-04T10:10:06.849980+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851155+0000","last_change":"2024-03-04T10:10:06.851155+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851155+0000","last_undegraded":"2024-03-04T10:10:06.851155+0000","last_fullsized":"2024-03-04T10:10:06.851155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853010+0000","last_change":"2024-03-04T10:10:06.853010+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853010+0000","last_undegraded":"2024-03-04T10:10:06.853010+0000","last_fullsized":"2024-03-04T10:10:06.853010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855177+0000","last_change":"2024-03-04T10:10:06.855177+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855177+0000","last_undegraded":"2024-03-04T10:10:06.855177+0000","last_fullsized":"2024-03-04T10:10:06.855177+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851940+0000","last_change":"2024-03-04T10:10:06.851940+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851940+0000","last_undegraded":"2024-03-04T10:10:06.851940+0000","last_fullsized":"2024-03-04T10:10:06.851940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854658+0000","last_change":"2024-03-04T10:10:06.854658+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854658+0000","last_undegraded":"2024-03-04T10:10:06.854658+0000","last_fullsized":"2024-03-04T10:10:06.854658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855850+0000","last_change":"2024-03-04T10:10:06.855850+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855850+0000","last_undegraded":"2024-03-04T10:10:06.855850+0000","last_fullsized":"2024-03-04T10:10:06.855850+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848047+0000","last_change":"2024-03-04T10:10:06.848047+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848047+0000","last_undegraded":"2024-03-04T10:10:06.848047+0000","last_fullsized":"2024-03-04T10:10:06.848047+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849542+0000","last_change":"2024-03-04T10:10:06.849542+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849542+0000","last_undegraded":"2024-03-04T10:10:06.849542+0000","last_fullsized":"2024-03-04T10:10:06.849542+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851655+0000","last_change":"2024-03-04T10:10:06.851655+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851655+0000","last_undegraded":"2024-03-04T10:10:06.851655+0000","last_fullsized":"2024-03-04T10:10:06.851655+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850703+0000","last_change":"2024-03-04T10:10:06.850703+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850703+0000","last_undegraded":"2024-03-04T10:10:06.850703+0000","last_fullsized":"2024-03-04T10:10:06.850703+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852821+0000","last_change":"2024-03-04T10:10:06.852821+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852821+0000","last_undegraded":"2024-03-04T10:10:06.852821+0000","last_fullsized":"2024-03-04T10:10:06.852821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851229+0000","last_change":"2024-03-04T10:10:06.851229+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851229+0000","last_undegraded":"2024-03-04T10:10:06.851229+0000","last_fullsized":"2024-03-04T10:10:06.851229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849419+0000","last_change":"2024-03-04T10:10:06.849419+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849419+0000","last_undegraded":"2024-03-04T10:10:06.849419+0000","last_fullsized":"2024-03-04T10:10:06.849419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.864395+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.864082+0000","last_became_peered":"2024-03-04T10:10:06.864082+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:45:18.950003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849862+0000","last_change":"2024-03-04T10:10:06.849862+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849862+0000","last_undegraded":"2024-03-04T10:10:06.849862+0000","last_fullsized":"2024-03-04T10:10:06.849862+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861011+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.860571+0000","last_became_peered":"2024-03-04T10:10:06.860571+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:35.744190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851356+0000","last_change":"2024-03-04T10:10:06.851356+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851356+0000","last_undegraded":"2024-03-04T10:10:06.851356+0000","last_fullsized":"2024-03-04T10:10:06.851356+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862539+0000","last_change":"2024-03-04T10:10:06.860271+0000","last_active":"2024-03-04T10:10:06.862539+0000","last_peered":"2024-03-04T10:10:06.862539+0000","last_clean":"2024-03-04T10:10:06.862539+0000","last_became_active":"2024-03-04T10:10:06.859984+0000","last_became_peered":"2024-03-04T10:10:06.859984+0000","last_unstale":"2024-03-04T10:10:06.862539+0000","last_undegraded":"2024-03-04T10:10:06.862539+0000","last_fullsized":"2024-03-04T10:10:06.862539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848478+0000","last_change":"2024-03-04T10:10:06.848478+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848478+0000","last_undegraded":"2024-03-04T10:10:06.848478+0000","last_fullsized":"2024-03-04T10:10:06.848478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870416+0000","last_change":"2024-03-04T10:10:06.865348+0000","last_active":"2024-03-04T10:10:06.870416+0000","last_peered":"2024-03-04T10:10:06.870416+0000","last_clean":"2024-03-04T10:10:06.870416+0000","last_became_active":"2024-03-04T10:10:06.865084+0000","last_became_peered":"2024-03-04T10:10:06.865084+0000","last_unstale":"2024-03-04T10:10:06.870416+0000","last_undegraded":"2024-03-04T10:10:06.870416+0000","last_fullsized":"2024-03-04T10:10:06.870416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:01:06.838971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851037+0000","last_change":"2024-03-04T10:10:06.851037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851037+0000","last_undegraded":"2024-03-04T10:10:06.851037+0000","last_fullsized":"2024-03-04T10:10:06.851037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867495+0000","last_change":"2024-03-04T10:10:06.863293+0000","last_active":"2024-03-04T10:10:06.867495+0000","last_peered":"2024-03-04T10:10:06.867495+0000","last_clean":"2024-03-04T10:10:06.867495+0000","last_became_active":"2024-03-04T10:10:06.863021+0000","last_became_peered":"2024-03-04T10:10:06.863021+0000","last_unstale":"2024-03-04T10:10:06.867495+0000","last_undegraded":"2024-03-04T10:10:06.867495+0000","last_fullsized":"2024-03-04T10:10:06.867495+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:02.877820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854753+0000","last_change":"2024-03-04T10:10:06.854753+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854753+0000","last_undegraded":"2024-03-04T10:10:06.854753+0000","last_fullsized":"2024-03-04T10:10:06.854753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.860442+0000","last_change":"2024-03-04T10:10:06.857045+0000","last_active":"2024-03-04T10:10:06.860442+0000","last_peered":"2024-03-04T10:10:06.860442+0000","last_clean":"2024-03-04T10:10:06.860442+0000","last_became_active":"2024-03-04T10:10:06.856753+0000","last_became_peered":"2024-03-04T10:10:06.856753+0000","last_unstale":"2024-03-04T10:10:06.860442+0000","last_undegraded":"2024-03-04T10:10:06.860442+0000","last_fullsized":"2024-03-04T10:10:06.860442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:51:19.769656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848193+0000","last_change":"2024-03-04T10:10:06.848193+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848193+0000","last_undegraded":"2024-03-04T10:10:06.848193+0000","last_fullsized":"2024-03-04T10:10:06.848193+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866070+0000","last_change":"2024-03-04T10:10:06.862082+0000","last_active":"2024-03-04T10:10:06.866070+0000","last_peered":"2024-03-04T10:10:06.866070+0000","last_clean":"2024-03-04T10:10:06.866070+0000","last_became_active":"2024-03-04T10:10:06.861780+0000","last_became_peered":"2024-03-04T10:10:06.861780+0000","last_unstale":"2024-03-04T10:10:06.866070+0000","last_undegraded":"2024-03-04T10:10:06.866070+0000","last_fullsized":"2024-03-04T10:10:06.866070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849240+0000","last_change":"2024-03-04T10:10:06.849240+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849240+0000","last_undegraded":"2024-03-04T10:10:06.849240+0000","last_fullsized":"2024-03-04T10:10:06.849240+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870454+0000","last_change":"2024-03-04T10:10:06.865722+0000","last_active":"2024-03-04T10:10:06.870454+0000","last_peered":"2024-03-04T10:10:06.870454+0000","last_clean":"2024-03-04T10:10:06.870454+0000","last_became_active":"2024-03-04T10:10:06.865459+0000","last_became_peered":"2024-03-04T10:10:06.865459+0000","last_unstale":"2024-03-04T10:10:06.870454+0000","last_undegraded":"2024-03-04T10:10:06.870454+0000","last_fullsized":"2024-03-04T10:10:06.870454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:50.057767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849672+0000","last_change":"2024-03-04T10:10:06.849672+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849672+0000","last_undegraded":"2024-03-04T10:10:06.849672+0000","last_fullsized":"2024-03-04T10:10:06.849672+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867470+0000","last_change":"2024-03-04T10:10:06.860947+0000","last_active":"2024-03-04T10:10:06.867470+0000","last_peered":"2024-03-04T10:10:06.867470+0000","last_clean":"2024-03-04T10:10:06.867470+0000","last_became_active":"2024-03-04T10:10:06.860654+0000","last_became_peered":"2024-03-04T10:10:06.860654+0000","last_unstale":"2024-03-04T10:10:06.867470+0000","last_undegraded":"2024-03-04T10:10:06.867470+0000","last_fullsized":"2024-03-04T10:10:06.867470+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:27.491076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853492+0000","last_change":"2024-03-04T10:10:06.853492+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853492+0000","last_undegraded":"2024-03-04T10:10:06.853492+0000","last_fullsized":"2024-03-04T10:10:06.853492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863318+0000","last_change":"2024-03-04T10:10:06.861468+0000","last_active":"2024-03-04T10:10:06.863318+0000","last_peered":"2024-03-04T10:10:06.863318+0000","last_clean":"2024-03-04T10:10:06.863318+0000","last_became_active":"2024-03-04T10:10:06.861177+0000","last_became_peered":"2024-03-04T10:10:06.861177+0000","last_unstale":"2024-03-04T10:10:06.863318+0000","last_undegraded":"2024-03-04T10:10:06.863318+0000","last_fullsized":"2024-03-04T10:10:06.863318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851887+0000","last_change":"2024-03-04T10:10:06.851887+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851887+0000","last_undegraded":"2024-03-04T10:10:06.851887+0000","last_fullsized":"2024-03-04T10:10:06.851887+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870472+0000","last_change":"2024-03-04T10:10:06.866005+0000","last_active":"2024-03-04T10:10:06.870472+0000","last_peered":"2024-03-04T10:10:06.870472+0000","last_clean":"2024-03-04T10:10:06.870472+0000","last_became_active":"2024-03-04T10:10:06.865735+0000","last_became_peered":"2024-03-04T10:10:06.865735+0000","last_unstale":"2024-03-04T10:10:06.870472+0000","last_undegraded":"2024-03-04T10:10:06.870472+0000","last_fullsized":"2024-03-04T10:10:06.870472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:28.683327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849865+0000","last_change":"2024-03-04T10:10:06.849865+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849865+0000","last_undegraded":"2024-03-04T10:10:06.849865+0000","last_fullsized":"2024-03-04T10:10:06.849865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861324+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.861027+0000","last_became_peered":"2024-03-04T10:10:06.861027+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:43.919308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850421+0000","last_change":"2024-03-04T10:10:06.850421+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850421+0000","last_undegraded":"2024-03-04T10:10:06.850421+0000","last_fullsized":"2024-03-04T10:10:06.850421+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867748+0000","last_change":"2024-03-04T10:10:06.864064+0000","last_active":"2024-03-04T10:10:06.867748+0000","last_peered":"2024-03-04T10:10:06.867748+0000","last_clean":"2024-03-04T10:10:06.867748+0000","last_became_active":"2024-03-04T10:10:06.863773+0000","last_became_peered":"2024-03-04T10:10:06.863773+0000","last_unstale":"2024-03-04T10:10:06.867748+0000","last_undegraded":"2024-03-04T10:10:06.867748+0000","last_fullsized":"2024-03-04T10:10:06.867748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:20.480886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849880+0000","last_change":"2024-03-04T10:10:06.849880+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849880+0000","last_undegraded":"2024-03-04T10:10:06.849880+0000","last_fullsized":"2024-03-04T10:10:06.849880+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863554+0000","last_change":"2024-03-04T10:10:06.858647+0000","last_active":"2024-03-04T10:10:06.863554+0000","last_peered":"2024-03-04T10:10:06.863554+0000","last_clean":"2024-03-04T10:10:06.863554+0000","last_became_active":"2024-03-04T10:10:06.858363+0000","last_became_peered":"2024-03-04T10:10:06.858363+0000","last_unstale":"2024-03-04T10:10:06.863554+0000","last_undegraded":"2024-03-04T10:10:06.863554+0000","last_fullsized":"2024-03-04T10:10:06.863554+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:59.513633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848011+0000","last_change":"2024-03-04T10:10:06.848011+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848011+0000","last_undegraded":"2024-03-04T10:10:06.848011+0000","last_fullsized":"2024-03-04T10:10:06.848011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864363+0000","last_change":"2024-03-04T10:10:06.860311+0000","last_active":"2024-03-04T10:10:06.864363+0000","last_peered":"2024-03-04T10:10:06.864363+0000","last_clean":"2024-03-04T10:10:06.864363+0000","last_became_active":"2024-03-04T10:10:06.860013+0000","last_became_peered":"2024-03-04T10:10:06.860013+0000","last_unstale":"2024-03-04T10:10:06.864363+0000","last_undegraded":"2024-03-04T10:10:06.864363+0000","last_fullsized":"2024-03-04T10:10:06.864363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:25.671449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848785+0000","last_change":"2024-03-04T10:10:06.848785+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848785+0000","last_undegraded":"2024-03-04T10:10:06.848785+0000","last_fullsized":"2024-03-04T10:10:06.848785+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864248+0000","last_change":"2024-03-04T10:10:06.859945+0000","last_active":"2024-03-04T10:10:06.864248+0000","last_peered":"2024-03-04T10:10:06.864248+0000","last_clean":"2024-03-04T10:10:06.864248+0000","last_became_active":"2024-03-04T10:10:06.859616+0000","last_became_peered":"2024-03-04T10:10:06.859616+0000","last_unstale":"2024-03-04T10:10:06.864248+0000","last_undegraded":"2024-03-04T10:10:06.864248+0000","last_fullsized":"2024-03-04T10:10:06.864248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:22.139606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849037+0000","last_change":"2024-03-04T10:10:06.849037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849037+0000","last_undegraded":"2024-03-04T10:10:06.849037+0000","last_fullsized":"2024-03-04T10:10:06.849037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865077+0000","last_change":"2024-03-04T10:10:06.862337+0000","last_active":"2024-03-04T10:10:06.865077+0000","last_peered":"2024-03-04T10:10:06.865077+0000","last_clean":"2024-03-04T10:10:06.865077+0000","last_became_active":"2024-03-04T10:10:06.861981+0000","last_became_peered":"2024-03-04T10:10:06.861981+0000","last_unstale":"2024-03-04T10:10:06.865077+0000","last_undegraded":"2024-03-04T10:10:06.865077+0000","last_fullsized":"2024-03-04T10:10:06.865077+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.881925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853333+0000","last_change":"2024-03-04T10:10:06.853333+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853333+0000","last_undegraded":"2024-03-04T10:10:06.853333+0000","last_fullsized":"2024-03-04T10:10:06.853333+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.859862+0000","last_change":"2024-03-04T10:10:06.858629+0000","last_active":"2024-03-04T10:10:06.859862+0000","last_peered":"2024-03-04T10:10:06.859862+0000","last_clean":"2024-03-04T10:10:06.859862+0000","last_became_active":"2024-03-04T10:10:06.858362+0000","last_became_peered":"2024-03-04T10:10:06.858362+0000","last_unstale":"2024-03-04T10:10:06.859862+0000","last_undegraded":"2024-03-04T10:10:06.859862+0000","last_fullsized":"2024-03-04T10:10:06.859862+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:28.667220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851428+0000","last_change":"2024-03-04T10:10:06.851428+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851428+0000","last_undegraded":"2024-03-04T10:10:06.851428+0000","last_fullsized":"2024-03-04T10:10:06.851428+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870494+0000","last_change":"2024-03-04T10:10:06.866361+0000","last_active":"2024-03-04T10:10:06.870494+0000","last_peered":"2024-03-04T10:10:06.870494+0000","last_clean":"2024-03-04T10:10:06.870494+0000","last_became_active":"2024-03-04T10:10:06.866083+0000","last_became_peered":"2024-03-04T10:10:06.866083+0000","last_unstale":"2024-03-04T10:10:06.870494+0000","last_undegraded":"2024-03-04T10:10:06.870494+0000","last_fullsized":"2024-03-04T10:10:06.870494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:57.646340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847765+0000","last_change":"2024-03-04T10:10:06.847765+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847765+0000","last_undegraded":"2024-03-04T10:10:06.847765+0000","last_fullsized":"2024-03-04T10:10:06.847765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.861629+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.861360+0000","last_became_peered":"2024-03-04T10:10:06.861360+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:34:32.909646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852388+0000","last_change":"2024-03-04T10:10:06.852388+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852388+0000","last_undegraded":"2024-03-04T10:10:06.852388+0000","last_fullsized":"2024-03-04T10:10:06.852388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.861908+0000","last_change":"2024-03-04T10:10:06.857599+0000","last_active":"2024-03-04T10:10:06.861908+0000","last_peered":"2024-03-04T10:10:06.861908+0000","last_clean":"2024-03-04T10:10:06.861908+0000","last_became_active":"2024-03-04T10:10:06.857319+0000","last_became_peered":"2024-03-04T10:10:06.857319+0000","last_unstale":"2024-03-04T10:10:06.861908+0000","last_undegraded":"2024-03-04T10:10:06.861908+0000","last_fullsized":"2024-03-04T10:10:06.861908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:49.470487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851702+0000","last_change":"2024-03-04T10:10:06.851702+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851702+0000","last_undegraded":"2024-03-04T10:10:06.851702+0000","last_fullsized":"2024-03-04T10:10:06.851702+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870325+0000","last_change":"2024-03-04T10:10:06.865016+0000","last_active":"2024-03-04T10:10:06.870325+0000","last_peered":"2024-03-04T10:10:06.870325+0000","last_clean":"2024-03-04T10:10:06.870325+0000","last_became_active":"2024-03-04T10:10:06.864692+0000","last_became_peered":"2024-03-04T10:10:06.864692+0000","last_unstale":"2024-03-04T10:10:06.870325+0000","last_undegraded":"2024-03-04T10:10:06.870325+0000","last_fullsized":"2024-03-04T10:10:06.870325+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:28:10.331623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849996+0000","last_change":"2024-03-04T10:10:06.849996+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849996+0000","last_undegraded":"2024-03-04T10:10:06.849996+0000","last_fullsized":"2024-03-04T10:10:06.849996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862041+0000","last_change":"2024-03-04T10:10:06.859809+0000","last_active":"2024-03-04T10:10:06.862041+0000","last_peered":"2024-03-04T10:10:06.862041+0000","last_clean":"2024-03-04T10:10:06.862041+0000","last_became_active":"2024-03-04T10:10:06.859499+0000","last_became_peered":"2024-03-04T10:10:06.859499+0000","last_unstale":"2024-03-04T10:10:06.862041+0000","last_undegraded":"2024-03-04T10:10:06.862041+0000","last_fullsized":"2024-03-04T10:10:06.862041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:17:31.377873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.856931+0000","last_change":"2024-03-04T10:10:06.856931+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.856931+0000","last_undegraded":"2024-03-04T10:10:06.856931+0000","last_fullsized":"2024-03-04T10:10:06.856931+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862008+0000","last_change":"2024-03-04T10:10:06.859472+0000","last_active":"2024-03-04T10:10:06.862008+0000","last_peered":"2024-03-04T10:10:06.862008+0000","last_clean":"2024-03-04T10:10:06.862008+0000","last_became_active":"2024-03-04T10:10:06.859206+0000","last_became_peered":"2024-03-04T10:10:06.859206+0000","last_unstale":"2024-03-04T10:10:06.862008+0000","last_undegraded":"2024-03-04T10:10:06.862008+0000","last_fullsized":"2024-03-04T10:10:06.862008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848039+0000","last_change":"2024-03-04T10:10:06.848039+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848039+0000","last_undegraded":"2024-03-04T10:10:06.848039+0000","last_fullsized":"2024-03-04T10:10:06.848039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866014+0000","last_change":"2024-03-04T10:10:06.860264+0000","last_active":"2024-03-04T10:10:06.866014+0000","last_peered":"2024-03-04T10:10:06.866014+0000","last_clean":"2024-03-04T10:10:06.866014+0000","last_became_active":"2024-03-04T10:10:06.859974+0000","last_became_peered":"2024-03-04T10:10:06.859974+0000","last_unstale":"2024-03-04T10:10:06.866014+0000","last_undegraded":"2024-03-04T10:10:06.866014+0000","last_fullsized":"2024-03-04T10:10:06.866014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:28:05.990062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848251+0000","last_change":"2024-03-04T10:10:06.848251+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848251+0000","last_undegraded":"2024-03-04T10:10:06.848251+0000","last_fullsized":"2024-03-04T10:10:06.848251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867523+0000","last_change":"2024-03-04T10:10:06.864593+0000","last_active":"2024-03-04T10:10:06.867523+0000","last_peered":"2024-03-04T10:10:06.867523+0000","last_clean":"2024-03-04T10:10:06.867523+0000","last_became_active":"2024-03-04T10:10:06.864298+0000","last_became_peered":"2024-03-04T10:10:06.864298+0000","last_unstale":"2024-03-04T10:10:06.867523+0000","last_undegraded":"2024-03-04T10:10:06.867523+0000","last_fullsized":"2024-03-04T10:10:06.867523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:44.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854228+0000","last_change":"2024-03-04T10:10:06.854228+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854228+0000","last_undegraded":"2024-03-04T10:10:06.854228+0000","last_fullsized":"2024-03-04T10:10:06.854228+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865410+0000","last_change":"2024-03-04T10:10:06.862581+0000","last_active":"2024-03-04T10:10:06.865410+0000","last_peered":"2024-03-04T10:10:06.865410+0000","last_clean":"2024-03-04T10:10:06.865410+0000","last_became_active":"2024-03-04T10:10:06.862283+0000","last_became_peered":"2024-03-04T10:10:06.862283+0000","last_unstale":"2024-03-04T10:10:06.865410+0000","last_undegraded":"2024-03-04T10:10:06.865410+0000","last_fullsized":"2024-03-04T10:10:06.865410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:44.951154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847286+0000","last_change":"2024-03-04T10:10:06.847286+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847286+0000","last_undegraded":"2024-03-04T10:10:06.847286+0000","last_fullsized":"2024-03-04T10:10:06.847286+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863815+0000","last_change":"2024-03-04T10:10:06.859084+0000","last_active":"2024-03-04T10:10:06.863815+0000","last_peered":"2024-03-04T10:10:06.863815+0000","last_clean":"2024-03-04T10:10:06.863815+0000","last_became_active":"2024-03-04T10:10:06.858820+0000","last_became_peered":"2024-03-04T10:10:06.858820+0000","last_unstale":"2024-03-04T10:10:06.863815+0000","last_undegraded":"2024-03-04T10:10:06.863815+0000","last_fullsized":"2024-03-04T10:10:06.863815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:29.284872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851394+0000","last_change":"2024-03-04T10:10:06.851394+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851394+0000","last_undegraded":"2024-03-04T10:10:06.851394+0000","last_fullsized":"2024-03-04T10:10:06.851394+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867553+0000","last_change":"2024-03-04T10:10:06.862727+0000","last_active":"2024-03-04T10:10:06.867553+0000","last_peered":"2024-03-04T10:10:06.867553+0000","last_clean":"2024-03-04T10:10:06.867553+0000","last_became_active":"2024-03-04T10:10:06.861472+0000","last_became_peered":"2024-03-04T10:10:06.861472+0000","last_unstale":"2024-03-04T10:10:06.867553+0000","last_undegraded":"2024-03-04T10:10:06.867553+0000","last_fullsized":"2024-03-04T10:10:06.867553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:33.683923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854465+0000","last_change":"2024-03-04T10:10:06.854465+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854465+0000","last_undegraded":"2024-03-04T10:10:06.854465+0000","last_fullsized":"2024-03-04T10:10:06.854465+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867555+0000","last_change":"2024-03-04T10:10:06.864463+0000","last_active":"2024-03-04T10:10:06.867555+0000","last_peered":"2024-03-04T10:10:06.867555+0000","last_clean":"2024-03-04T10:10:06.867555+0000","last_became_active":"2024-03-04T10:10:06.861931+0000","last_became_peered":"2024-03-04T10:10:06.861931+0000","last_unstale":"2024-03-04T10:10:06.867555+0000","last_undegraded":"2024-03-04T10:10:06.867555+0000","last_fullsized":"2024-03-04T10:10:06.867555+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:15.494976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849584+0000","last_change":"2024-03-04T10:10:06.849584+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849584+0000","last_undegraded":"2024-03-04T10:10:06.849584+0000","last_fullsized":"2024-03-04T10:10:06.849584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870433+0000","last_change":"2024-03-04T10:10:06.865079+0000","last_active":"2024-03-04T10:10:06.870433+0000","last_peered":"2024-03-04T10:10:06.870433+0000","last_clean":"2024-03-04T10:10:06.870433+0000","last_became_active":"2024-03-04T10:10:06.864808+0000","last_became_peered":"2024-03-04T10:10:06.864808+0000","last_unstale":"2024-03-04T10:10:06.870433+0000","last_undegraded":"2024-03-04T10:10:06.870433+0000","last_fullsized":"2024-03-04T10:10:06.870433+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:32.258399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851418+0000","last_change":"2024-03-04T10:10:06.851418+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851418+0000","last_undegraded":"2024-03-04T10:10:06.851418+0000","last_fullsized":"2024-03-04T10:10:06.851418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866583+0000","last_change":"2024-03-04T10:10:06.861954+0000","last_active":"2024-03-04T10:10:06.866583+0000","last_peered":"2024-03-04T10:10:06.866583+0000","last_clean":"2024-03-04T10:10:06.866583+0000","last_became_active":"2024-03-04T10:10:06.859757+0000","last_became_peered":"2024-03-04T10:10:06.859757+0000","last_unstale":"2024-03-04T10:10:06.866583+0000","last_undegraded":"2024-03-04T10:10:06.866583+0000","last_fullsized":"2024-03-04T10:10:06.866583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:58:45.314856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":315,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.854556+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:10:06.854556+0000","last_peered":"2024-03-04T10:10:06.854556+0000","last_clean":"2024-03-04T10:10:06.854556+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:10:06.854556+0000","last_undegraded":"2024-03-04T10:10:06.854556+0000","last_fullsized":"2024-03-04T10:10:06.854556+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343140,"statfs":{"total":96636764160,"available":96607375360,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:10:11.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3f is not active+clean 2024-03-04T10:10:11.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.852633+0000', 'last_change': '2024-03-04T10:10:06.852633+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.852633+0000', 'last_undegraded': '2024-03-04T10:10:06.852633+0000', 'last_fullsized': '2024-03-04T10:10:06.852633+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:10:11.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854680+0000', 'last_change': '2024-03-04T10:10:06.854680+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854680+0000', 'last_undegraded': '2024-03-04T10:10:06.854680+0000', 'last_fullsized': '2024-03-04T10:10:06.854680+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3d is not active+clean 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.852344+0000', 'last_change': '2024-03-04T10:10:06.852344+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.852344+0000', 'last_undegraded': '2024-03-04T10:10:06.852344+0000', 'last_fullsized': '2024-03-04T10:10:06.852344+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3c is not active+clean 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.850373+0000', 'last_change': '2024-03-04T10:10:06.850373+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.850373+0000', 'last_undegraded': '2024-03-04T10:10:06.850373+0000', 'last_fullsized': '2024-03-04T10:10:06.850373+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.850436+0000', 'last_change': '2024-03-04T10:10:06.850436+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.850436+0000', 'last_undegraded': '2024-03-04T10:10:06.850436+0000', 'last_fullsized': '2024-03-04T10:10:06.850436+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3a is not active+clean 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854406+0000', 'last_change': '2024-03-04T10:10:06.854406+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854406+0000', 'last_undegraded': '2024-03-04T10:10:06.854406+0000', 'last_fullsized': '2024-03-04T10:10:06.854406+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-03-04T10:10:11.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.853075+0000', 'last_change': '2024-03-04T10:10:06.853075+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.853075+0000', 'last_undegraded': '2024-03-04T10:10:06.853075+0000', 'last_fullsized': '2024-03-04T10:10:06.853075+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.38 is not active+clean 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851456+0000', 'last_change': '2024-03-04T10:10:06.851456+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851456+0000', 'last_undegraded': '2024-03-04T10:10:06.851456+0000', 'last_fullsized': '2024-03-04T10:10:06.851456+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.850204+0000', 'last_change': '2024-03-04T10:10:06.850204+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.850204+0000', 'last_undegraded': '2024-03-04T10:10:06.850204+0000', 'last_fullsized': '2024-03-04T10:10:06.850204+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.36 is not active+clean 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848505+0000', 'last_change': '2024-03-04T10:10:06.848505+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848505+0000', 'last_undegraded': '2024-03-04T10:10:06.848505+0000', 'last_fullsized': '2024-03-04T10:10:06.848505+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.853200+0000', 'last_change': '2024-03-04T10:10:06.853200+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.853200+0000', 'last_undegraded': '2024-03-04T10:10:06.853200+0000', 'last_fullsized': '2024-03-04T10:10:06.853200+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.34 is not active+clean 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849214+0000', 'last_change': '2024-03-04T10:10:06.849214+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849214+0000', 'last_undegraded': '2024-03-04T10:10:06.849214+0000', 'last_fullsized': '2024-03-04T10:10:06.849214+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:10:11.582 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.33 is not active+clean 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849674+0000', 'last_change': '2024-03-04T10:10:06.849674+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849674+0000', 'last_undegraded': '2024-03-04T10:10:06.849674+0000', 'last_fullsized': '2024-03-04T10:10:06.849674+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.853490+0000', 'last_change': '2024-03-04T10:10:06.853490+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.853490+0000', 'last_undegraded': '2024-03-04T10:10:06.853490+0000', 'last_fullsized': '2024-03-04T10:10:06.853490+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.31 is not active+clean 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848629+0000', 'last_change': '2024-03-04T10:10:06.848629+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848629+0000', 'last_undegraded': '2024-03-04T10:10:06.848629+0000', 'last_fullsized': '2024-03-04T10:10:06.848629+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.30 is not active+clean 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854342+0000', 'last_change': '2024-03-04T10:10:06.854342+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854342+0000', 'last_undegraded': '2024-03-04T10:10:06.854342+0000', 'last_fullsized': '2024-03-04T10:10:06.854342+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2f is not active+clean 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.852040+0000', 'last_change': '2024-03-04T10:10:06.852040+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.852040+0000', 'last_undegraded': '2024-03-04T10:10:06.852040+0000', 'last_fullsized': '2024-03-04T10:10:06.852040+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2e is not active+clean 2024-03-04T10:10:11.583 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.852330+0000', 'last_change': '2024-03-04T10:10:06.852330+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.852330+0000', 'last_undegraded': '2024-03-04T10:10:06.852330+0000', 'last_fullsized': '2024-03-04T10:10:06.852330+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851041+0000', 'last_change': '2024-03-04T10:10:06.851041+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851041+0000', 'last_undegraded': '2024-03-04T10:10:06.851041+0000', 'last_fullsized': '2024-03-04T10:10:06.851041+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2c is not active+clean 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849980+0000', 'last_change': '2024-03-04T10:10:06.849980+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849980+0000', 'last_undegraded': '2024-03-04T10:10:06.849980+0000', 'last_fullsized': '2024-03-04T10:10:06.849980+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851155+0000', 'last_change': '2024-03-04T10:10:06.851155+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851155+0000', 'last_undegraded': '2024-03-04T10:10:06.851155+0000', 'last_fullsized': '2024-03-04T10:10:06.851155+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.853010+0000', 'last_change': '2024-03-04T10:10:06.853010+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.853010+0000', 'last_undegraded': '2024-03-04T10:10:06.853010+0000', 'last_fullsized': '2024-03-04T10:10:06.853010+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-03-04T10:10:11.584 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.855177+0000', 'last_change': '2024-03-04T10:10:06.855177+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.855177+0000', 'last_undegraded': '2024-03-04T10:10:06.855177+0000', 'last_fullsized': '2024-03-04T10:10:06.855177+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.28 is not active+clean 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851940+0000', 'last_change': '2024-03-04T10:10:06.851940+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851940+0000', 'last_undegraded': '2024-03-04T10:10:06.851940+0000', 'last_fullsized': '2024-03-04T10:10:06.851940+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854658+0000', 'last_change': '2024-03-04T10:10:06.854658+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854658+0000', 'last_undegraded': '2024-03-04T10:10:06.854658+0000', 'last_fullsized': '2024-03-04T10:10:06.854658+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.855850+0000', 'last_change': '2024-03-04T10:10:06.855850+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.855850+0000', 'last_undegraded': '2024-03-04T10:10:06.855850+0000', 'last_fullsized': '2024-03-04T10:10:06.855850+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.25 is not active+clean 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848047+0000', 'last_change': '2024-03-04T10:10:06.848047+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848047+0000', 'last_undegraded': '2024-03-04T10:10:06.848047+0000', 'last_fullsized': '2024-03-04T10:10:06.848047+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.24 is not active+clean 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849542+0000', 'last_change': '2024-03-04T10:10:06.849542+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849542+0000', 'last_undegraded': '2024-03-04T10:10:06.849542+0000', 'last_fullsized': '2024-03-04T10:10:06.849542+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-03-04T10:10:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851655+0000', 'last_change': '2024-03-04T10:10:06.851655+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851655+0000', 'last_undegraded': '2024-03-04T10:10:06.851655+0000', 'last_fullsized': '2024-03-04T10:10:06.851655+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.850703+0000', 'last_change': '2024-03-04T10:10:06.850703+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.850703+0000', 'last_undegraded': '2024-03-04T10:10:06.850703+0000', 'last_fullsized': '2024-03-04T10:10:06.850703+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.852821+0000', 'last_change': '2024-03-04T10:10:06.852821+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.852821+0000', 'last_undegraded': '2024-03-04T10:10:06.852821+0000', 'last_fullsized': '2024-03-04T10:10:06.852821+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.20 is not active+clean 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851229+0000', 'last_change': '2024-03-04T10:10:06.851229+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851229+0000', 'last_undegraded': '2024-03-04T10:10:06.851229+0000', 'last_fullsized': '2024-03-04T10:10:06.851229+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1c is not active+clean 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849419+0000', 'last_change': '2024-03-04T10:10:06.849419+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849419+0000', 'last_undegraded': '2024-03-04T10:10:06.849419+0000', 'last_fullsized': '2024-03-04T10:10:06.849419+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1d is not active+clean 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849862+0000', 'last_change': '2024-03-04T10:10:06.849862+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849862+0000', 'last_undegraded': '2024-03-04T10:10:06.849862+0000', 'last_fullsized': '2024-03-04T10:10:06.849862+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-03-04T10:10:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851356+0000', 'last_change': '2024-03-04T10:10:06.851356+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851356+0000', 'last_undegraded': '2024-03-04T10:10:06.851356+0000', 'last_fullsized': '2024-03-04T10:10:06.851356+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1f is not active+clean 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848478+0000', 'last_change': '2024-03-04T10:10:06.848478+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848478+0000', 'last_undegraded': '2024-03-04T10:10:06.848478+0000', 'last_fullsized': '2024-03-04T10:10:06.848478+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851037+0000', 'last_change': '2024-03-04T10:10:06.851037+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851037+0000', 'last_undegraded': '2024-03-04T10:10:06.851037+0000', 'last_fullsized': '2024-03-04T10:10:06.851037+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854753+0000', 'last_change': '2024-03-04T10:10:06.854753+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854753+0000', 'last_undegraded': '2024-03-04T10:10:06.854753+0000', 'last_fullsized': '2024-03-04T10:10:06.854753+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1a is not active+clean 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848193+0000', 'last_change': '2024-03-04T10:10:06.848193+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848193+0000', 'last_undegraded': '2024-03-04T10:10:06.848193+0000', 'last_fullsized': '2024-03-04T10:10:06.848193+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849240+0000', 'last_change': '2024-03-04T10:10:06.849240+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849240+0000', 'last_undegraded': '2024-03-04T10:10:06.849240+0000', 'last_fullsized': '2024-03-04T10:10:06.849240+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849672+0000', 'last_change': '2024-03-04T10:10:06.849672+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849672+0000', 'last_undegraded': '2024-03-04T10:10:06.849672+0000', 'last_fullsized': '2024-03-04T10:10:06.849672+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.853492+0000', 'last_change': '2024-03-04T10:10:06.853492+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.853492+0000', 'last_undegraded': '2024-03-04T10:10:06.853492+0000', 'last_fullsized': '2024-03-04T10:10:06.853492+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851887+0000', 'last_change': '2024-03-04T10:10:06.851887+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851887+0000', 'last_undegraded': '2024-03-04T10:10:06.851887+0000', 'last_fullsized': '2024-03-04T10:10:06.851887+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849865+0000', 'last_change': '2024-03-04T10:10:06.849865+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849865+0000', 'last_undegraded': '2024-03-04T10:10:06.849865+0000', 'last_fullsized': '2024-03-04T10:10:06.849865+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.850421+0000', 'last_change': '2024-03-04T10:10:06.850421+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.850421+0000', 'last_undegraded': '2024-03-04T10:10:06.850421+0000', 'last_fullsized': '2024-03-04T10:10:06.850421+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-04T10:10:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849880+0000', 'last_change': '2024-03-04T10:10:06.849880+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849880+0000', 'last_undegraded': '2024-03-04T10:10:06.849880+0000', 'last_fullsized': '2024-03-04T10:10:06.849880+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848011+0000', 'last_change': '2024-03-04T10:10:06.848011+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848011+0000', 'last_undegraded': '2024-03-04T10:10:06.848011+0000', 'last_fullsized': '2024-03-04T10:10:06.848011+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848785+0000', 'last_change': '2024-03-04T10:10:06.848785+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848785+0000', 'last_undegraded': '2024-03-04T10:10:06.848785+0000', 'last_fullsized': '2024-03-04T10:10:06.848785+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.c is not active+clean 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849037+0000', 'last_change': '2024-03-04T10:10:06.849037+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849037+0000', 'last_undegraded': '2024-03-04T10:10:06.849037+0000', 'last_fullsized': '2024-03-04T10:10:06.849037+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.853333+0000', 'last_change': '2024-03-04T10:10:06.853333+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.853333+0000', 'last_undegraded': '2024-03-04T10:10:06.853333+0000', 'last_fullsized': '2024-03-04T10:10:06.853333+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-03-04T10:10:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851428+0000', 'last_change': '2024-03-04T10:10:06.851428+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851428+0000', 'last_undegraded': '2024-03-04T10:10:06.851428+0000', 'last_fullsized': '2024-03-04T10:10:06.851428+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.8 is not active+clean 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.847765+0000', 'last_change': '2024-03-04T10:10:06.847765+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.847765+0000', 'last_undegraded': '2024-03-04T10:10:06.847765+0000', 'last_fullsized': '2024-03-04T10:10:06.847765+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.852388+0000', 'last_change': '2024-03-04T10:10:06.852388+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.852388+0000', 'last_undegraded': '2024-03-04T10:10:06.852388+0000', 'last_fullsized': '2024-03-04T10:10:06.852388+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851702+0000', 'last_change': '2024-03-04T10:10:06.851702+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851702+0000', 'last_undegraded': '2024-03-04T10:10:06.851702+0000', 'last_fullsized': '2024-03-04T10:10:06.851702+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.b is not active+clean 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849996+0000', 'last_change': '2024-03-04T10:10:06.849996+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849996+0000', 'last_undegraded': '2024-03-04T10:10:06.849996+0000', 'last_fullsized': '2024-03-04T10:10:06.849996+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.856931+0000', 'last_change': '2024-03-04T10:10:06.856931+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.856931+0000', 'last_undegraded': '2024-03-04T10:10:06.856931+0000', 'last_fullsized': '2024-03-04T10:10:06.856931+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.5 is not active+clean 2024-03-04T10:10:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848039+0000', 'last_change': '2024-03-04T10:10:06.848039+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848039+0000', 'last_undegraded': '2024-03-04T10:10:06.848039+0000', 'last_fullsized': '2024-03-04T10:10:06.848039+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.6 is not active+clean 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.848251+0000', 'last_change': '2024-03-04T10:10:06.848251+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.848251+0000', 'last_undegraded': '2024-03-04T10:10:06.848251+0000', 'last_fullsized': '2024-03-04T10:10:06.848251+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854228+0000', 'last_change': '2024-03-04T10:10:06.854228+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854228+0000', 'last_undegraded': '2024-03-04T10:10:06.854228+0000', 'last_fullsized': '2024-03-04T10:10:06.854228+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.847286+0000', 'last_change': '2024-03-04T10:10:06.847286+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.847286+0000', 'last_undegraded': '2024-03-04T10:10:06.847286+0000', 'last_fullsized': '2024-03-04T10:10:06.847286+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851394+0000', 'last_change': '2024-03-04T10:10:06.851394+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851394+0000', 'last_undegraded': '2024-03-04T10:10:06.851394+0000', 'last_fullsized': '2024-03-04T10:10:06.851394+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-03-04T10:10:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.854465+0000', 'last_change': '2024-03-04T10:10:06.854465+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.854465+0000', 'last_undegraded': '2024-03-04T10:10:06.854465+0000', 'last_fullsized': '2024-03-04T10:10:06.854465+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-04T10:10:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3 is not active+clean 2024-03-04T10:10:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.849584+0000', 'last_change': '2024-03-04T10:10:06.849584+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.849584+0000', 'last_undegraded': '2024-03-04T10:10:06.849584+0000', 'last_fullsized': '2024-03-04T10:10:06.849584+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-04T10:10:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-03-04T10:10:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 91, 'state': 'creating+peering', 'last_fresh': '2024-03-04T10:10:06.851418+0000', 'last_change': '2024-03-04T10:10:06.851418+0000', 'last_active': '2024-03-04T10:10:06.837242+0000', 'last_peered': '2024-03-04T10:10:06.837242+0000', 'last_clean': '2024-03-04T10:10:06.837242+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-04T10:10:06.851418+0000', 'last_undegraded': '2024-03-04T10:10:06.851418+0000', 'last_fullsized': '2024-03-04T10:10:06.851418+0000', 'mapping_epoch': 91, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 91, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'last_clean_scrub_stamp': '2024-03-04T10:10:06.837242+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-04T10:10:11.592 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-04T10:10:11.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.714+0000 7fce25e81640 1 Processor -- start 2024-03-04T10:10:11.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.714+0000 7fce25e81640 1 -- start start 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce25e81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce2015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce25e81640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fce2006d3b0 0x7fce2015dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce25e81640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fce2015b410 0x7fce2015b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce25e81640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce200b2720 con 0x7fce2015b410 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce25e81640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2015be00 con 0x7fce2006d3b0 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce25e81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2015e2d0 con 0x7fce2015b030 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fce2015b410 0x7fce2015b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fce2015b410 0x7fce2015b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45590/0 (socket says 172.21.15.99:45590) 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 -- 172.21.15.99:0/2422487366 learned_addr learned my addr 172.21.15.99:0/2422487366 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1effd640 1 --2- 172.21.15.99:0/2422487366 >> v2:172.21.15.162:3300/0 conn(0x7fce2006d3b0 0x7fce2015dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1f7fe640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce2015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 -- 172.21.15.99:0/2422487366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 msgr2=0x7fce2015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce2015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 -- 172.21.15.99:0/2422487366 >> v2:172.21.15.162:3300/0 conn(0x7fce2006d3b0 msgr2=0x7fce2015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 --2- 172.21.15.99:0/2422487366 >> v2:172.21.15.162:3300/0 conn(0x7fce2006d3b0 0x7fce2015dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1ffff640 1 -- 172.21.15.99:0/2422487366 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2006d8b0 con 0x7fce2015b410 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.715+0000 7fce1f7fe640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce2015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce1ffff640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce2015b8c0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fce2006f860 tx=0x7fce1802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2b8832e9482b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce1e7fc640 1 -- 172.21.15.99:0/2422487366 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1802c9e0 con 0x7fce2015b410 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce1e7fc640 1 -- 172.21.15.99:0/2422487366 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1802cb80 con 0x7fce2015b410 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce1e7fc640 1 -- 172.21.15.99:0/2422487366 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce180343f0 con 0x7fce2015b410 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 -- 172.21.15.99:0/2422487366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 msgr2=0x7fce2015b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce2015b8c0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fce2006f860 tx=0x7fce1802be40 comp rx=0 tx=0).stop 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 -- 172.21.15.99:0/2422487366 shutdown_connections 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce2015b8c0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 --2- 172.21.15.99:0/2422487366 >> v2:172.21.15.162:3300/0 conn(0x7fce2006d3b0 0x7fce2015dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 --2- 172.21.15.99:0/2422487366 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce2015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 -- 172.21.15.99:0/2422487366 >> 172.21.15.99:0/2422487366 conn(0x7fce2006acd0 msgr2=0x7fce2006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 -- 172.21.15.99:0/2422487366 shutdown_connections 2024-03-04T10:10:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.716+0000 7fce25e81640 1 -- 172.21.15.99:0/2422487366 wait complete. 2024-03-04T10:10:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 Processor -- start 2024-03-04T10:10:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 -- start start 2024-03-04T10:10:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce2006d3b0 0x7fce201586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce200c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce200ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2006e150 con 0x7fce2015b410 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2006de70 con 0x7fce2006d3b0 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.718+0000 7fce25e81640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2006e690 con 0x7fce2015b030 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce200ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce200ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45596/0 (socket says 172.21.15.99:45596) 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 -- 172.21.15.99:0/833586239 learned_addr learned my addr 172.21.15.99:0/833586239 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1effd640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce200c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 -- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 msgr2=0x7fce200c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce200c61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 -- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce2006d3b0 msgr2=0x7fce201586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce2006d3b0 0x7fce201586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 -- 172.21.15.99:0/833586239 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce200c5140 con 0x7fce2015b410 2024-03-04T10:10:11.724 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce1ffff640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce200ce930 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fce1802c450 tx=0x7fce18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18043020 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1803e450 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce200c5420 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce18009510 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.719+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce200c59e0 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.720+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce20074f60 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.721+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fce18046020 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.721+0000 7fce07fff640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fce0c06b4d0 0x7fce0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.721+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce180acc30 con 0x7fce2015b410 2024-03-04T10:10:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.721+0000 7fce1f7fe640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fce0c06b4d0 0x7fce0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:11.726 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.722+0000 7fce1f7fe640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fce0c06b4d0 0x7fce0c06d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.725+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fce1804d030 con 0x7fce2015b410 2024-03-04T10:10:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.965+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fce20159960 con 0x7fce2015b410 2024-03-04T10:10:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.965+0000 7fce07fff640 1 -- 172.21.15.99:0/833586239 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1454 (secure 0 0 0) 0x7fce1807b920 con 0x7fce2015b410 2024-03-04T10:10:11.968 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:11.968 INFO:teuthology.orchestra.run.smithi099.stdout:{"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":416,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":93,"num_osds":8,"num_up_osds":8,"osd_up_since":1709546606,"num_in_osds":8,"osd_in_since":1709546597,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":248598528,"bytes_avail":772845514752,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":78,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":7644}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-03-04T10:09:57.529739+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.967+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fce0c06b4d0 msgr2=0x7fce0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.967+0000 7fce25e81640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fce0c06b4d0 0x7fce0c06d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10008040 comp rx=0 tx=0).stop 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.967+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 msgr2=0x7fce200ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.967+0000 7fce25e81640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce200ce930 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fce1802c450 tx=0x7fce18004a40 comp rx=0 tx=0).stop 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.967+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 shutdown_connections 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.967+0000 7fce25e81640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fce0c06b4d0 0x7fce0c06d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.968+0000 7fce25e81640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fce2015b410 0x7fce200ce930 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.968+0000 7fce25e81640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fce2015b030 0x7fce200c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.968+0000 7fce25e81640 1 --2- 172.21.15.99:0/833586239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fce2006d3b0 0x7fce201586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.968+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 >> 172.21.15.99:0/833586239 conn(0x7fce2006acd0 msgr2=0x7fce2015a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.968+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 shutdown_connections 2024-03-04T10:10:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:11.968+0000 7fce25e81640 1 -- 172.21.15.99:0/833586239 wait complete. 2024-03-04T10:10:11.980 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-04T10:10:11.981 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:10:12.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.077+0000 7f30a3612640 1 Processor -- start 2024-03-04T10:10:12.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.077+0000 7f30a3612640 1 -- start start 2024-03-04T10:10:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a3612640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c0b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a3612640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f309c0b47c0 0x7f309c0b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a3612640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c0ccad0 con 0x7f309c15b960 2024-03-04T10:10:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a3612640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c0b5190 con 0x7f309c0b47c0 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a3612640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0b10a0 con 0x7f309c0b0700 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a1baf640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f309c0b47c0 0x7f309c0b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a13ae640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f309c15b960 0x7f309c0d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:59642/0 (socket says 172.21.15.99:59642) 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.078+0000 7f30a13ae640 1 -- 172.21.15.99:0/2238047601 learned_addr learned my addr 172.21.15.99:0/2238047601 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f30a0bad640 1 --2- 172.21.15.99:0/2238047601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c0b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f30a1baf640 1 -- 172.21.15.99:0/2238047601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 msgr2=0x7f309c0b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f30a1baf640 1 --2- 172.21.15.99:0/2238047601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c0b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f30a1baf640 1 -- 172.21.15.99:0/2238047601 >> v1:172.21.15.99:6789/0 conn(0x7f309c15b960 legacy=0x7f309c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f3093fff640 1 -- 172.21.15.99:0/2238047601 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509256284 0 0) 0x7f309c0ccad0 con 0x7f309c15b960 2024-03-04T10:10:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f30a1baf640 1 -- 172.21.15.99:0/2238047601 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0cfab0 con 0x7f309c0b47c0 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.079+0000 7f30a1baf640 1 --2- 172.21.15.99:0/2238047601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c0b4c50 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f30940097a0 tx=0x7f309402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba2a9b619df8b62 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f3093fff640 1 -- 172.21.15.99:0/2238047601 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30940315a0 con 0x7f309c0b47c0 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f3093fff640 1 -- 172.21.15.99:0/2238047601 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3094037070 con 0x7f309c0b47c0 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f3093fff640 1 -- 172.21.15.99:0/2238047601 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3094031960 con 0x7f309c0b47c0 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 -- 172.21.15.99:0/2238047601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 msgr2=0x7f309c0b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 --2- 172.21.15.99:0/2238047601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c0b4c50 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f30940097a0 tx=0x7f309402b840 comp rx=0 tx=0).stop 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 -- 172.21.15.99:0/2238047601 shutdown_connections 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 --2- 172.21.15.99:0/2238047601 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c0b4c50 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 --2- 172.21.15.99:0/2238047601 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c0b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 -- 172.21.15.99:0/2238047601 >> 172.21.15.99:0/2238047601 conn(0x7f309c06f5f0 msgr2=0x7f309c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 -- 172.21.15.99:0/2238047601 shutdown_connections 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.080+0000 7f30a3612640 1 -- 172.21.15.99:0/2238047601 wait complete. 2024-03-04T10:10:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.081+0000 7f30a3612640 1 Processor -- start 2024-03-04T10:10:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.081+0000 7f30a3612640 1 -- start start 2024-03-04T10:10:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a3612640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a3612640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c14ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a3612640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f309c15b960 0x7f309c1581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a3612640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0d0100 con 0x7f309c0b47c0 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a3612640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0cf860 con 0x7f309c15b960 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a3612640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c15bd40 con 0x7f309c0b0700 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a0bad640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c14ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a0bad640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c14ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45622/0 (socket says 172.21.15.99:45622) 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a0bad640 1 -- 172.21.15.99:0/2330820663 learned_addr learned my addr 172.21.15.99:0/2330820663 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a1baf640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f309c15b960 0x7f309c1581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.082+0000 7f30a13ae640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a0bad640 1 -- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 msgr2=0x7f309c156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a0bad640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a0bad640 1 -- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f309c15b960 msgr2=0x7f309c1581f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a0bad640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f309c15b960 0x7f309c1581f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a0bad640 1 -- 172.21.15.99:0/2330820663 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c1566b0 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a1baf640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f309c15b960 0x7f309c1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a0bad640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c14ddd0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f308c00d5f0 tx=0x7f308c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0125f0 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c019070 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c156990 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0129d0 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.083+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c156f50 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.084+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c074f60 con 0x7f309c0b47c0 2024-03-04T10:10:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.084+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f308c012b70 con 0x7f309c0b47c0 2024-03-04T10:10:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.085+0000 7f3091ffb640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308006b4d0 0x7f308006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.085+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f308c08a760 con 0x7f309c0b47c0 2024-03-04T10:10:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.085+0000 7f30a13ae640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308006b4d0 0x7f308006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.086+0000 7f30a13ae640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308006b4d0 0x7f308006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f30980012e0 tx=0x7f3098007ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.089+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f308c059450 con 0x7f309c0b47c0 2024-03-04T10:10:12.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.259+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f309c0c7e50 con 0x7f308006b4d0 2024-03-04T10:10:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.266+0000 7f3091ffb640 1 -- 172.21.15.99:0/2330820663 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246767 (secure 0 0 0) 0x7f309c0c7e50 con 0x7f308006b4d0 2024-03-04T10:10:12.270 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:12.272 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308006b4d0 msgr2=0x7f308006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308006b4d0 0x7f308006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f30980012e0 tx=0x7f3098007ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 msgr2=0x7f309c14ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c14ddd0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f308c00d5f0 tx=0x7f308c00dab0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 shutdown_connections 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f308006b4d0 0x7f308006d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f309c15b960 0x7f309c1581f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f309c0b47c0 0x7f309c14ddd0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 --2- 172.21.15.99:0/2330820663 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f309c0b0700 0x7f309c156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 >> 172.21.15.99:0/2330820663 conn(0x7f309c06f5f0 msgr2=0x7f309c15aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 shutdown_connections 2024-03-04T10:10:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:12.268+0000 7f30a3612640 1 -- 172.21.15.99:0/2330820663 wait complete. 2024-03-04T10:10:12.281 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-03-04T10:10:11.532750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":242772,"kb_used_data":9928,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754731948,"statfs":{"total":773094113280,"available":772845514752,"internally_reserved":0,"allocated":10166272,"data_stored":5347640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.695135"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852633+0000","last_change":"2024-03-04T10:10:06.852633+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852633+0000","last_undegraded":"2024-03-04T10:10:06.852633+0000","last_fullsized":"2024-03-04T10:10:06.852633+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854680+0000","last_change":"2024-03-04T10:10:06.854680+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854680+0000","last_undegraded":"2024-03-04T10:10:06.854680+0000","last_fullsized":"2024-03-04T10:10:06.854680+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852344+0000","last_change":"2024-03-04T10:10:06.852344+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852344+0000","last_undegraded":"2024-03-04T10:10:06.852344+0000","last_fullsized":"2024-03-04T10:10:06.852344+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850373+0000","last_change":"2024-03-04T10:10:06.850373+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850373+0000","last_undegraded":"2024-03-04T10:10:06.850373+0000","last_fullsized":"2024-03-04T10:10:06.850373+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850436+0000","last_change":"2024-03-04T10:10:06.850436+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850436+0000","last_undegraded":"2024-03-04T10:10:06.850436+0000","last_fullsized":"2024-03-04T10:10:06.850436+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854406+0000","last_change":"2024-03-04T10:10:06.854406+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854406+0000","last_undegraded":"2024-03-04T10:10:06.854406+0000","last_fullsized":"2024-03-04T10:10:06.854406+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853075+0000","last_change":"2024-03-04T10:10:06.853075+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853075+0000","last_undegraded":"2024-03-04T10:10:06.853075+0000","last_fullsized":"2024-03-04T10:10:06.853075+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851456+0000","last_change":"2024-03-04T10:10:06.851456+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851456+0000","last_undegraded":"2024-03-04T10:10:06.851456+0000","last_fullsized":"2024-03-04T10:10:06.851456+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850204+0000","last_change":"2024-03-04T10:10:06.850204+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850204+0000","last_undegraded":"2024-03-04T10:10:06.850204+0000","last_fullsized":"2024-03-04T10:10:06.850204+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848505+0000","last_change":"2024-03-04T10:10:06.848505+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848505+0000","last_undegraded":"2024-03-04T10:10:06.848505+0000","last_fullsized":"2024-03-04T10:10:06.848505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853200+0000","last_change":"2024-03-04T10:10:06.853200+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853200+0000","last_undegraded":"2024-03-04T10:10:06.853200+0000","last_fullsized":"2024-03-04T10:10:06.853200+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849214+0000","last_change":"2024-03-04T10:10:06.849214+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849214+0000","last_undegraded":"2024-03-04T10:10:06.849214+0000","last_fullsized":"2024-03-04T10:10:06.849214+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849674+0000","last_change":"2024-03-04T10:10:06.849674+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849674+0000","last_undegraded":"2024-03-04T10:10:06.849674+0000","last_fullsized":"2024-03-04T10:10:06.849674+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853490+0000","last_change":"2024-03-04T10:10:06.853490+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853490+0000","last_undegraded":"2024-03-04T10:10:06.853490+0000","last_fullsized":"2024-03-04T10:10:06.853490+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848629+0000","last_change":"2024-03-04T10:10:06.848629+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848629+0000","last_undegraded":"2024-03-04T10:10:06.848629+0000","last_fullsized":"2024-03-04T10:10:06.848629+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854342+0000","last_change":"2024-03-04T10:10:06.854342+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854342+0000","last_undegraded":"2024-03-04T10:10:06.854342+0000","last_fullsized":"2024-03-04T10:10:06.854342+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852040+0000","last_change":"2024-03-04T10:10:06.852040+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852040+0000","last_undegraded":"2024-03-04T10:10:06.852040+0000","last_fullsized":"2024-03-04T10:10:06.852040+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852330+0000","last_change":"2024-03-04T10:10:06.852330+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852330+0000","last_undegraded":"2024-03-04T10:10:06.852330+0000","last_fullsized":"2024-03-04T10:10:06.852330+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851041+0000","last_change":"2024-03-04T10:10:06.851041+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851041+0000","last_undegraded":"2024-03-04T10:10:06.851041+0000","last_fullsized":"2024-03-04T10:10:06.851041+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849980+0000","last_change":"2024-03-04T10:10:06.849980+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849980+0000","last_undegraded":"2024-03-04T10:10:06.849980+0000","last_fullsized":"2024-03-04T10:10:06.849980+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851155+0000","last_change":"2024-03-04T10:10:06.851155+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851155+0000","last_undegraded":"2024-03-04T10:10:06.851155+0000","last_fullsized":"2024-03-04T10:10:06.851155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853010+0000","last_change":"2024-03-04T10:10:06.853010+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853010+0000","last_undegraded":"2024-03-04T10:10:06.853010+0000","last_fullsized":"2024-03-04T10:10:06.853010+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855177+0000","last_change":"2024-03-04T10:10:06.855177+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855177+0000","last_undegraded":"2024-03-04T10:10:06.855177+0000","last_fullsized":"2024-03-04T10:10:06.855177+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851940+0000","last_change":"2024-03-04T10:10:06.851940+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851940+0000","last_undegraded":"2024-03-04T10:10:06.851940+0000","last_fullsized":"2024-03-04T10:10:06.851940+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854658+0000","last_change":"2024-03-04T10:10:06.854658+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854658+0000","last_undegraded":"2024-03-04T10:10:06.854658+0000","last_fullsized":"2024-03-04T10:10:06.854658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.855850+0000","last_change":"2024-03-04T10:10:06.855850+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.855850+0000","last_undegraded":"2024-03-04T10:10:06.855850+0000","last_fullsized":"2024-03-04T10:10:06.855850+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848047+0000","last_change":"2024-03-04T10:10:06.848047+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848047+0000","last_undegraded":"2024-03-04T10:10:06.848047+0000","last_fullsized":"2024-03-04T10:10:06.848047+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849542+0000","last_change":"2024-03-04T10:10:06.849542+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849542+0000","last_undegraded":"2024-03-04T10:10:06.849542+0000","last_fullsized":"2024-03-04T10:10:06.849542+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851655+0000","last_change":"2024-03-04T10:10:06.851655+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851655+0000","last_undegraded":"2024-03-04T10:10:06.851655+0000","last_fullsized":"2024-03-04T10:10:06.851655+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850703+0000","last_change":"2024-03-04T10:10:06.850703+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850703+0000","last_undegraded":"2024-03-04T10:10:06.850703+0000","last_fullsized":"2024-03-04T10:10:06.850703+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852821+0000","last_change":"2024-03-04T10:10:06.852821+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852821+0000","last_undegraded":"2024-03-04T10:10:06.852821+0000","last_fullsized":"2024-03-04T10:10:06.852821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851229+0000","last_change":"2024-03-04T10:10:06.851229+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851229+0000","last_undegraded":"2024-03-04T10:10:06.851229+0000","last_fullsized":"2024-03-04T10:10:06.851229+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849419+0000","last_change":"2024-03-04T10:10:06.849419+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849419+0000","last_undegraded":"2024-03-04T10:10:06.849419+0000","last_fullsized":"2024-03-04T10:10:06.849419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.864395+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.864082+0000","last_became_peered":"2024-03-04T10:10:06.864082+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:45:18.950003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849862+0000","last_change":"2024-03-04T10:10:06.849862+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849862+0000","last_undegraded":"2024-03-04T10:10:06.849862+0000","last_fullsized":"2024-03-04T10:10:06.849862+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861011+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.860571+0000","last_became_peered":"2024-03-04T10:10:06.860571+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:20:35.744190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851356+0000","last_change":"2024-03-04T10:10:06.851356+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851356+0000","last_undegraded":"2024-03-04T10:10:06.851356+0000","last_fullsized":"2024-03-04T10:10:06.851356+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862539+0000","last_change":"2024-03-04T10:10:06.860271+0000","last_active":"2024-03-04T10:10:06.862539+0000","last_peered":"2024-03-04T10:10:06.862539+0000","last_clean":"2024-03-04T10:10:06.862539+0000","last_became_active":"2024-03-04T10:10:06.859984+0000","last_became_peered":"2024-03-04T10:10:06.859984+0000","last_unstale":"2024-03-04T10:10:06.862539+0000","last_undegraded":"2024-03-04T10:10:06.862539+0000","last_fullsized":"2024-03-04T10:10:06.862539+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848478+0000","last_change":"2024-03-04T10:10:06.848478+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848478+0000","last_undegraded":"2024-03-04T10:10:06.848478+0000","last_fullsized":"2024-03-04T10:10:06.848478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870416+0000","last_change":"2024-03-04T10:10:06.865348+0000","last_active":"2024-03-04T10:10:06.870416+0000","last_peered":"2024-03-04T10:10:06.870416+0000","last_clean":"2024-03-04T10:10:06.870416+0000","last_became_active":"2024-03-04T10:10:06.865084+0000","last_became_peered":"2024-03-04T10:10:06.865084+0000","last_unstale":"2024-03-04T10:10:06.870416+0000","last_undegraded":"2024-03-04T10:10:06.870416+0000","last_fullsized":"2024-03-04T10:10:06.870416+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:01:06.838971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851037+0000","last_change":"2024-03-04T10:10:06.851037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851037+0000","last_undegraded":"2024-03-04T10:10:06.851037+0000","last_fullsized":"2024-03-04T10:10:06.851037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867495+0000","last_change":"2024-03-04T10:10:06.863293+0000","last_active":"2024-03-04T10:10:06.867495+0000","last_peered":"2024-03-04T10:10:06.867495+0000","last_clean":"2024-03-04T10:10:06.867495+0000","last_became_active":"2024-03-04T10:10:06.863021+0000","last_became_peered":"2024-03-04T10:10:06.863021+0000","last_unstale":"2024-03-04T10:10:06.867495+0000","last_undegraded":"2024-03-04T10:10:06.867495+0000","last_fullsized":"2024-03-04T10:10:06.867495+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:25:02.877820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854753+0000","last_change":"2024-03-04T10:10:06.854753+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854753+0000","last_undegraded":"2024-03-04T10:10:06.854753+0000","last_fullsized":"2024-03-04T10:10:06.854753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.860442+0000","last_change":"2024-03-04T10:10:06.857045+0000","last_active":"2024-03-04T10:10:06.860442+0000","last_peered":"2024-03-04T10:10:06.860442+0000","last_clean":"2024-03-04T10:10:06.860442+0000","last_became_active":"2024-03-04T10:10:06.856753+0000","last_became_peered":"2024-03-04T10:10:06.856753+0000","last_unstale":"2024-03-04T10:10:06.860442+0000","last_undegraded":"2024-03-04T10:10:06.860442+0000","last_fullsized":"2024-03-04T10:10:06.860442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:51:19.769656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848193+0000","last_change":"2024-03-04T10:10:06.848193+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848193+0000","last_undegraded":"2024-03-04T10:10:06.848193+0000","last_fullsized":"2024-03-04T10:10:06.848193+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866070+0000","last_change":"2024-03-04T10:10:06.862082+0000","last_active":"2024-03-04T10:10:06.866070+0000","last_peered":"2024-03-04T10:10:06.866070+0000","last_clean":"2024-03-04T10:10:06.866070+0000","last_became_active":"2024-03-04T10:10:06.861780+0000","last_became_peered":"2024-03-04T10:10:06.861780+0000","last_unstale":"2024-03-04T10:10:06.866070+0000","last_undegraded":"2024-03-04T10:10:06.866070+0000","last_fullsized":"2024-03-04T10:10:06.866070+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:36.120764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849240+0000","last_change":"2024-03-04T10:10:06.849240+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849240+0000","last_undegraded":"2024-03-04T10:10:06.849240+0000","last_fullsized":"2024-03-04T10:10:06.849240+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870454+0000","last_change":"2024-03-04T10:10:06.865722+0000","last_active":"2024-03-04T10:10:06.870454+0000","last_peered":"2024-03-04T10:10:06.870454+0000","last_clean":"2024-03-04T10:10:06.870454+0000","last_became_active":"2024-03-04T10:10:06.865459+0000","last_became_peered":"2024-03-04T10:10:06.865459+0000","last_unstale":"2024-03-04T10:10:06.870454+0000","last_undegraded":"2024-03-04T10:10:06.870454+0000","last_fullsized":"2024-03-04T10:10:06.870454+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:50.057767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849672+0000","last_change":"2024-03-04T10:10:06.849672+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849672+0000","last_undegraded":"2024-03-04T10:10:06.849672+0000","last_fullsized":"2024-03-04T10:10:06.849672+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867470+0000","last_change":"2024-03-04T10:10:06.860947+0000","last_active":"2024-03-04T10:10:06.867470+0000","last_peered":"2024-03-04T10:10:06.867470+0000","last_clean":"2024-03-04T10:10:06.867470+0000","last_became_active":"2024-03-04T10:10:06.860654+0000","last_became_peered":"2024-03-04T10:10:06.860654+0000","last_unstale":"2024-03-04T10:10:06.867470+0000","last_undegraded":"2024-03-04T10:10:06.867470+0000","last_fullsized":"2024-03-04T10:10:06.867470+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:31:27.491076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853492+0000","last_change":"2024-03-04T10:10:06.853492+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853492+0000","last_undegraded":"2024-03-04T10:10:06.853492+0000","last_fullsized":"2024-03-04T10:10:06.853492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863318+0000","last_change":"2024-03-04T10:10:06.861468+0000","last_active":"2024-03-04T10:10:06.863318+0000","last_peered":"2024-03-04T10:10:06.863318+0000","last_clean":"2024-03-04T10:10:06.863318+0000","last_became_active":"2024-03-04T10:10:06.861177+0000","last_became_peered":"2024-03-04T10:10:06.861177+0000","last_unstale":"2024-03-04T10:10:06.863318+0000","last_undegraded":"2024-03-04T10:10:06.863318+0000","last_fullsized":"2024-03-04T10:10:06.863318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851887+0000","last_change":"2024-03-04T10:10:06.851887+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851887+0000","last_undegraded":"2024-03-04T10:10:06.851887+0000","last_fullsized":"2024-03-04T10:10:06.851887+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870472+0000","last_change":"2024-03-04T10:10:06.866005+0000","last_active":"2024-03-04T10:10:06.870472+0000","last_peered":"2024-03-04T10:10:06.870472+0000","last_clean":"2024-03-04T10:10:06.870472+0000","last_became_active":"2024-03-04T10:10:06.865735+0000","last_became_peered":"2024-03-04T10:10:06.865735+0000","last_unstale":"2024-03-04T10:10:06.870472+0000","last_undegraded":"2024-03-04T10:10:06.870472+0000","last_fullsized":"2024-03-04T10:10:06.870472+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:54:28.683327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849865+0000","last_change":"2024-03-04T10:10:06.849865+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849865+0000","last_undegraded":"2024-03-04T10:10:06.849865+0000","last_fullsized":"2024-03-04T10:10:06.849865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866041+0000","last_change":"2024-03-04T10:10:06.861324+0000","last_active":"2024-03-04T10:10:06.866041+0000","last_peered":"2024-03-04T10:10:06.866041+0000","last_clean":"2024-03-04T10:10:06.866041+0000","last_became_active":"2024-03-04T10:10:06.861027+0000","last_became_peered":"2024-03-04T10:10:06.861027+0000","last_unstale":"2024-03-04T10:10:06.866041+0000","last_undegraded":"2024-03-04T10:10:06.866041+0000","last_fullsized":"2024-03-04T10:10:06.866041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:43.919308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.850421+0000","last_change":"2024-03-04T10:10:06.850421+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.850421+0000","last_undegraded":"2024-03-04T10:10:06.850421+0000","last_fullsized":"2024-03-04T10:10:06.850421+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867748+0000","last_change":"2024-03-04T10:10:06.864064+0000","last_active":"2024-03-04T10:10:06.867748+0000","last_peered":"2024-03-04T10:10:06.867748+0000","last_clean":"2024-03-04T10:10:06.867748+0000","last_became_active":"2024-03-04T10:10:06.863773+0000","last_became_peered":"2024-03-04T10:10:06.863773+0000","last_unstale":"2024-03-04T10:10:06.867748+0000","last_undegraded":"2024-03-04T10:10:06.867748+0000","last_fullsized":"2024-03-04T10:10:06.867748+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:20.480886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849880+0000","last_change":"2024-03-04T10:10:06.849880+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849880+0000","last_undegraded":"2024-03-04T10:10:06.849880+0000","last_fullsized":"2024-03-04T10:10:06.849880+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863554+0000","last_change":"2024-03-04T10:10:06.858647+0000","last_active":"2024-03-04T10:10:06.863554+0000","last_peered":"2024-03-04T10:10:06.863554+0000","last_clean":"2024-03-04T10:10:06.863554+0000","last_became_active":"2024-03-04T10:10:06.858363+0000","last_became_peered":"2024-03-04T10:10:06.858363+0000","last_unstale":"2024-03-04T10:10:06.863554+0000","last_undegraded":"2024-03-04T10:10:06.863554+0000","last_fullsized":"2024-03-04T10:10:06.863554+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:59.513633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848011+0000","last_change":"2024-03-04T10:10:06.848011+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848011+0000","last_undegraded":"2024-03-04T10:10:06.848011+0000","last_fullsized":"2024-03-04T10:10:06.848011+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864363+0000","last_change":"2024-03-04T10:10:06.860311+0000","last_active":"2024-03-04T10:10:06.864363+0000","last_peered":"2024-03-04T10:10:06.864363+0000","last_clean":"2024-03-04T10:10:06.864363+0000","last_became_active":"2024-03-04T10:10:06.860013+0000","last_became_peered":"2024-03-04T10:10:06.860013+0000","last_unstale":"2024-03-04T10:10:06.864363+0000","last_undegraded":"2024-03-04T10:10:06.864363+0000","last_fullsized":"2024-03-04T10:10:06.864363+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:25.671449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848785+0000","last_change":"2024-03-04T10:10:06.848785+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848785+0000","last_undegraded":"2024-03-04T10:10:06.848785+0000","last_fullsized":"2024-03-04T10:10:06.848785+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.864248+0000","last_change":"2024-03-04T10:10:06.859945+0000","last_active":"2024-03-04T10:10:06.864248+0000","last_peered":"2024-03-04T10:10:06.864248+0000","last_clean":"2024-03-04T10:10:06.864248+0000","last_became_active":"2024-03-04T10:10:06.859616+0000","last_became_peered":"2024-03-04T10:10:06.859616+0000","last_unstale":"2024-03-04T10:10:06.864248+0000","last_undegraded":"2024-03-04T10:10:06.864248+0000","last_fullsized":"2024-03-04T10:10:06.864248+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:22.139606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849037+0000","last_change":"2024-03-04T10:10:06.849037+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849037+0000","last_undegraded":"2024-03-04T10:10:06.849037+0000","last_fullsized":"2024-03-04T10:10:06.849037+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865077+0000","last_change":"2024-03-04T10:10:06.862337+0000","last_active":"2024-03-04T10:10:06.865077+0000","last_peered":"2024-03-04T10:10:06.865077+0000","last_clean":"2024-03-04T10:10:06.865077+0000","last_became_active":"2024-03-04T10:10:06.861981+0000","last_became_peered":"2024-03-04T10:10:06.861981+0000","last_unstale":"2024-03-04T10:10:06.865077+0000","last_undegraded":"2024-03-04T10:10:06.865077+0000","last_fullsized":"2024-03-04T10:10:06.865077+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.881925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.853333+0000","last_change":"2024-03-04T10:10:06.853333+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.853333+0000","last_undegraded":"2024-03-04T10:10:06.853333+0000","last_fullsized":"2024-03-04T10:10:06.853333+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.859862+0000","last_change":"2024-03-04T10:10:06.858629+0000","last_active":"2024-03-04T10:10:06.859862+0000","last_peered":"2024-03-04T10:10:06.859862+0000","last_clean":"2024-03-04T10:10:06.859862+0000","last_became_active":"2024-03-04T10:10:06.858362+0000","last_became_peered":"2024-03-04T10:10:06.858362+0000","last_unstale":"2024-03-04T10:10:06.859862+0000","last_undegraded":"2024-03-04T10:10:06.859862+0000","last_fullsized":"2024-03-04T10:10:06.859862+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:28.667220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851428+0000","last_change":"2024-03-04T10:10:06.851428+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851428+0000","last_undegraded":"2024-03-04T10:10:06.851428+0000","last_fullsized":"2024-03-04T10:10:06.851428+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870494+0000","last_change":"2024-03-04T10:10:06.866361+0000","last_active":"2024-03-04T10:10:06.870494+0000","last_peered":"2024-03-04T10:10:06.870494+0000","last_clean":"2024-03-04T10:10:06.870494+0000","last_became_active":"2024-03-04T10:10:06.866083+0000","last_became_peered":"2024-03-04T10:10:06.866083+0000","last_unstale":"2024-03-04T10:10:06.870494+0000","last_undegraded":"2024-03-04T10:10:06.870494+0000","last_fullsized":"2024-03-04T10:10:06.870494+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:57.646340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847765+0000","last_change":"2024-03-04T10:10:06.847765+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847765+0000","last_undegraded":"2024-03-04T10:10:06.847765+0000","last_fullsized":"2024-03-04T10:10:06.847765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867504+0000","last_change":"2024-03-04T10:10:06.861629+0000","last_active":"2024-03-04T10:10:06.867504+0000","last_peered":"2024-03-04T10:10:06.867504+0000","last_clean":"2024-03-04T10:10:06.867504+0000","last_became_active":"2024-03-04T10:10:06.861360+0000","last_became_peered":"2024-03-04T10:10:06.861360+0000","last_unstale":"2024-03-04T10:10:06.867504+0000","last_undegraded":"2024-03-04T10:10:06.867504+0000","last_fullsized":"2024-03-04T10:10:06.867504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:34:32.909646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.852388+0000","last_change":"2024-03-04T10:10:06.852388+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.852388+0000","last_undegraded":"2024-03-04T10:10:06.852388+0000","last_fullsized":"2024-03-04T10:10:06.852388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.861908+0000","last_change":"2024-03-04T10:10:06.857599+0000","last_active":"2024-03-04T10:10:06.861908+0000","last_peered":"2024-03-04T10:10:06.861908+0000","last_clean":"2024-03-04T10:10:06.861908+0000","last_became_active":"2024-03-04T10:10:06.857319+0000","last_became_peered":"2024-03-04T10:10:06.857319+0000","last_unstale":"2024-03-04T10:10:06.861908+0000","last_undegraded":"2024-03-04T10:10:06.861908+0000","last_fullsized":"2024-03-04T10:10:06.861908+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:45:49.470487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851702+0000","last_change":"2024-03-04T10:10:06.851702+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851702+0000","last_undegraded":"2024-03-04T10:10:06.851702+0000","last_fullsized":"2024-03-04T10:10:06.851702+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870325+0000","last_change":"2024-03-04T10:10:06.865016+0000","last_active":"2024-03-04T10:10:06.870325+0000","last_peered":"2024-03-04T10:10:06.870325+0000","last_clean":"2024-03-04T10:10:06.870325+0000","last_became_active":"2024-03-04T10:10:06.864692+0000","last_became_peered":"2024-03-04T10:10:06.864692+0000","last_unstale":"2024-03-04T10:10:06.870325+0000","last_undegraded":"2024-03-04T10:10:06.870325+0000","last_fullsized":"2024-03-04T10:10:06.870325+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:28:10.331623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849996+0000","last_change":"2024-03-04T10:10:06.849996+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849996+0000","last_undegraded":"2024-03-04T10:10:06.849996+0000","last_fullsized":"2024-03-04T10:10:06.849996+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862041+0000","last_change":"2024-03-04T10:10:06.859809+0000","last_active":"2024-03-04T10:10:06.862041+0000","last_peered":"2024-03-04T10:10:06.862041+0000","last_clean":"2024-03-04T10:10:06.862041+0000","last_became_active":"2024-03-04T10:10:06.859499+0000","last_became_peered":"2024-03-04T10:10:06.859499+0000","last_unstale":"2024-03-04T10:10:06.862041+0000","last_undegraded":"2024-03-04T10:10:06.862041+0000","last_fullsized":"2024-03-04T10:10:06.862041+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:17:31.377873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.856931+0000","last_change":"2024-03-04T10:10:06.856931+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.856931+0000","last_undegraded":"2024-03-04T10:10:06.856931+0000","last_fullsized":"2024-03-04T10:10:06.856931+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.862008+0000","last_change":"2024-03-04T10:10:06.859472+0000","last_active":"2024-03-04T10:10:06.862008+0000","last_peered":"2024-03-04T10:10:06.862008+0000","last_clean":"2024-03-04T10:10:06.862008+0000","last_became_active":"2024-03-04T10:10:06.859206+0000","last_became_peered":"2024-03-04T10:10:06.859206+0000","last_unstale":"2024-03-04T10:10:06.862008+0000","last_undegraded":"2024-03-04T10:10:06.862008+0000","last_fullsized":"2024-03-04T10:10:06.862008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:56.057437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848039+0000","last_change":"2024-03-04T10:10:06.848039+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848039+0000","last_undegraded":"2024-03-04T10:10:06.848039+0000","last_fullsized":"2024-03-04T10:10:06.848039+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866014+0000","last_change":"2024-03-04T10:10:06.860264+0000","last_active":"2024-03-04T10:10:06.866014+0000","last_peered":"2024-03-04T10:10:06.866014+0000","last_clean":"2024-03-04T10:10:06.866014+0000","last_became_active":"2024-03-04T10:10:06.859974+0000","last_became_peered":"2024-03-04T10:10:06.859974+0000","last_unstale":"2024-03-04T10:10:06.866014+0000","last_undegraded":"2024-03-04T10:10:06.866014+0000","last_fullsized":"2024-03-04T10:10:06.866014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:28:05.990062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.848251+0000","last_change":"2024-03-04T10:10:06.848251+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.848251+0000","last_undegraded":"2024-03-04T10:10:06.848251+0000","last_fullsized":"2024-03-04T10:10:06.848251+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867523+0000","last_change":"2024-03-04T10:10:06.864593+0000","last_active":"2024-03-04T10:10:06.867523+0000","last_peered":"2024-03-04T10:10:06.867523+0000","last_clean":"2024-03-04T10:10:06.867523+0000","last_became_active":"2024-03-04T10:10:06.864298+0000","last_became_peered":"2024-03-04T10:10:06.864298+0000","last_unstale":"2024-03-04T10:10:06.867523+0000","last_undegraded":"2024-03-04T10:10:06.867523+0000","last_fullsized":"2024-03-04T10:10:06.867523+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:29:44.771552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854228+0000","last_change":"2024-03-04T10:10:06.854228+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854228+0000","last_undegraded":"2024-03-04T10:10:06.854228+0000","last_fullsized":"2024-03-04T10:10:06.854228+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.865410+0000","last_change":"2024-03-04T10:10:06.862581+0000","last_active":"2024-03-04T10:10:06.865410+0000","last_peered":"2024-03-04T10:10:06.865410+0000","last_clean":"2024-03-04T10:10:06.865410+0000","last_became_active":"2024-03-04T10:10:06.862283+0000","last_became_peered":"2024-03-04T10:10:06.862283+0000","last_unstale":"2024-03-04T10:10:06.865410+0000","last_undegraded":"2024-03-04T10:10:06.865410+0000","last_fullsized":"2024-03-04T10:10:06.865410+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:44.951154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.847286+0000","last_change":"2024-03-04T10:10:06.847286+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.847286+0000","last_undegraded":"2024-03-04T10:10:06.847286+0000","last_fullsized":"2024-03-04T10:10:06.847286+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.863815+0000","last_change":"2024-03-04T10:10:06.859084+0000","last_active":"2024-03-04T10:10:06.863815+0000","last_peered":"2024-03-04T10:10:06.863815+0000","last_clean":"2024-03-04T10:10:06.863815+0000","last_became_active":"2024-03-04T10:10:06.858820+0000","last_became_peered":"2024-03-04T10:10:06.858820+0000","last_unstale":"2024-03-04T10:10:06.863815+0000","last_undegraded":"2024-03-04T10:10:06.863815+0000","last_fullsized":"2024-03-04T10:10:06.863815+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:29.284872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851394+0000","last_change":"2024-03-04T10:10:06.851394+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851394+0000","last_undegraded":"2024-03-04T10:10:06.851394+0000","last_fullsized":"2024-03-04T10:10:06.851394+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867553+0000","last_change":"2024-03-04T10:10:06.862727+0000","last_active":"2024-03-04T10:10:06.867553+0000","last_peered":"2024-03-04T10:10:06.867553+0000","last_clean":"2024-03-04T10:10:06.867553+0000","last_became_active":"2024-03-04T10:10:06.861472+0000","last_became_peered":"2024-03-04T10:10:06.861472+0000","last_unstale":"2024-03-04T10:10:06.867553+0000","last_undegraded":"2024-03-04T10:10:06.867553+0000","last_fullsized":"2024-03-04T10:10:06.867553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:33.683923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.854465+0000","last_change":"2024-03-04T10:10:06.854465+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.854465+0000","last_undegraded":"2024-03-04T10:10:06.854465+0000","last_fullsized":"2024-03-04T10:10:06.854465+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.867555+0000","last_change":"2024-03-04T10:10:06.864463+0000","last_active":"2024-03-04T10:10:06.867555+0000","last_peered":"2024-03-04T10:10:06.867555+0000","last_clean":"2024-03-04T10:10:06.867555+0000","last_became_active":"2024-03-04T10:10:06.861931+0000","last_became_peered":"2024-03-04T10:10:06.861931+0000","last_unstale":"2024-03-04T10:10:06.867555+0000","last_undegraded":"2024-03-04T10:10:06.867555+0000","last_fullsized":"2024-03-04T10:10:06.867555+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:15.494976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.849584+0000","last_change":"2024-03-04T10:10:06.849584+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.849584+0000","last_undegraded":"2024-03-04T10:10:06.849584+0000","last_fullsized":"2024-03-04T10:10:06.849584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.870433+0000","last_change":"2024-03-04T10:10:06.865079+0000","last_active":"2024-03-04T10:10:06.870433+0000","last_peered":"2024-03-04T10:10:06.870433+0000","last_clean":"2024-03-04T10:10:06.870433+0000","last_became_active":"2024-03-04T10:10:06.864808+0000","last_became_peered":"2024-03-04T10:10:06.864808+0000","last_unstale":"2024-03-04T10:10:06.870433+0000","last_undegraded":"2024-03-04T10:10:06.870433+0000","last_fullsized":"2024-03-04T10:10:06.870433+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:32.258399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":91,"state":"creating+peering","last_fresh":"2024-03-04T10:10:06.851418+0000","last_change":"2024-03-04T10:10:06.851418+0000","last_active":"2024-03-04T10:10:06.837242+0000","last_peered":"2024-03-04T10:10:06.837242+0000","last_clean":"2024-03-04T10:10:06.837242+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T10:10:06.851418+0000","last_undegraded":"2024-03-04T10:10:06.851418+0000","last_fullsized":"2024-03-04T10:10:06.851418+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.866583+0000","last_change":"2024-03-04T10:10:06.861954+0000","last_active":"2024-03-04T10:10:06.866583+0000","last_peered":"2024-03-04T10:10:06.866583+0000","last_clean":"2024-03-04T10:10:06.866583+0000","last_became_active":"2024-03-04T10:10:06.859757+0000","last_became_peered":"2024-03-04T10:10:06.859757+0000","last_unstale":"2024-03-04T10:10:06.866583+0000","last_undegraded":"2024-03-04T10:10:06.866583+0000","last_fullsized":"2024-03-04T10:10:06.866583+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:58:45.314856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":315,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T10:10:06.854556+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:10:06.854556+0000","last_peered":"2024-03-04T10:10:06.854556+0000","last_clean":"2024-03-04T10:10:06.854556+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:10:06.854556+0000","last_undegraded":"2024-03-04T10:10:06.854556+0000","last_fullsized":"2024-03-04T10:10:06.854556+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574932,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574932,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":520863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28700,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343140,"statfs":{"total":96636764160,"available":96607375360,"internally_reserved":0,"allocated":1716224,"data_stored":1111231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:10:15.287 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:10:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.397+0000 7fe223353640 1 Processor -- start 2024-03-04T10:10:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.397+0000 7fe223353640 1 -- start start 2024-03-04T10:10:15.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe223353640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe21c15b030 0x7fe21c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe223353640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 0x7fe21c15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe223353640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21c15e2d0 con 0x7fe21c15b410 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe223353640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe21c0b2720 con 0x7fe21c15b030 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe223353640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c06d8b0 con 0x7fe21c06d3b0 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe2218f0640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe21c15b410 0x7fe21c15b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57180/0 (socket says 172.21.15.99:57180) 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe2218f0640 1 -- 172.21.15.99:0/1712319777 learned_addr learned my addr 172.21.15.99:0/1712319777 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000728452 0 0) 0x7fe21c15e2d0 con 0x7fe21c15b410 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe208003520 con 0x7fe21c15b410 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe2208ee640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 0x7fe21c15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe2210ef640 1 --2- 172.21.15.99:0/1712319777 >> v2:172.21.15.162:3300/0 conn(0x7fe21c15b030 0x7fe21c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.398+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2170100001 0 0) 0x7fe208003520 con 0x7fe21c15b410 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 msgr2=0x7fe21c15dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 0x7fe21c15dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 >> v2:172.21.15.162:3300/0 conn(0x7fe21c15b030 msgr2=0x7fe21c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 --2- 172.21.15.99:0/1712319777 >> v2:172.21.15.162:3300/0 conn(0x7fe21c15b030 0x7fe21c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c06db90 con 0x7fe21c15b410 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fe2140019f0 con 0x7fe21c15b410 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2210ef640 1 --2- 172.21.15.99:0/1712319777 >> v2:172.21.15.162:3300/0 conn(0x7fe21c15b030 0x7fe21c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 >> v1:172.21.15.99:6789/0 conn(0x7fe21c15b410 legacy=0x7fe21c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2080039b0 0x7fe208003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 0x7fe208004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe208004d40 0x7fe2080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c0b2720 con 0x7fe208004d40 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c06d8b0 con 0x7fe2080039b0 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20800ba70 con 0x7fe21c06d3b0 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2218f0640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe208004d40 0x7fe2080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2210ef640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2080039b0 0x7fe208003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2208ee640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 0x7fe208004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2218f0640 1 -- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 msgr2=0x7fe208004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:15.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2218f0640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c06d3b0 0x7fe208004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2218f0640 1 -- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2080039b0 msgr2=0x7fe208003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:15.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2218f0640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe2080039b0 0x7fe208003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.399+0000 7fe2218f0640 1 -- 172.21.15.99:0/1712319777 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208003520 con 0x7fe208004d40 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe2218f0640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe208004d40 0x7fe2080051c0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe21400a850 tx=0x7fe208005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c2ce8fc406d268 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21400a4a0 con 0x7fe208004d40 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214034650 con 0x7fe208004d40 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe213fff640 1 -- 172.21.15.99:0/1712319777 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214034a40 con 0x7fe208004d40 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe223353640 1 -- 172.21.15.99:0/1712319777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe208004d40 msgr2=0x7fe2080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe223353640 1 --2- 172.21.15.99:0/1712319777 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe208004d40 0x7fe2080051c0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe21400a850 tx=0x7fe208005830 comp rx=0 tx=0).stop 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe2218f0640 1 -- 172.21.15.99:0/1712319777 reap_dead start 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe223353640 1 -- 172.21.15.99:0/1712319777 shutdown_connections 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.400+0000 7fe223353640 1 -- 172.21.15.99:0/1712319777 >> 172.21.15.99:0/1712319777 conn(0x7fe21c06acd0 msgr2=0x7fe21c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.401+0000 7fe223353640 1 -- 172.21.15.99:0/1712319777 shutdown_connections 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.401+0000 7fe223353640 1 -- 172.21.15.99:0/1712319777 wait complete. 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 Processor -- start 2024-03-04T10:10:15.417 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 -- start start 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 0x7fe21c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe21c153670 0x7fe21c15db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c15d200 0x7fe21c14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c06e690 con 0x7fe21c15ce20 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c06e150 con 0x7fe21c153670 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.402+0000 7fe223353640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c06e810 con 0x7fe21c15d200 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 0x7fe21c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 0x7fe21c1f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54808/0 (socket says 172.21.15.99:54808) 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 -- 172.21.15.99:0/1147178214 learned_addr learned my addr 172.21.15.99:0/1147178214 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 -- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c15d200 msgr2=0x7fe21c14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c15d200 0x7fe21c14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 -- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe21c153670 msgr2=0x7fe21c15db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2208ee640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe21c153670 0x7fe21c15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe21c153670 0x7fe21c15db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 -- 172.21.15.99:0/1147178214 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c1541a0 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe2210ef640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 0x7fe21c1f23c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe218011680 tx=0x7fe218011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218019070 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe218004220 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21800f040 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c154480 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.403+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c154a40 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.404+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c074f60 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.405+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe21800f760 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.405+0000 7fe211ffb640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe20006b4d0 0x7fe20006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.405+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe21808a670 con 0x7fe21c15ce20 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.405+0000 7fe2208ee640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe20006b4d0 0x7fe20006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.406+0000 7fe2208ee640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe20006b4d0 0x7fe20006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe21c153ab0 tx=0x7fe20c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:15.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.409+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe218059360 con 0x7fe21c15ce20 2024-03-04T10:10:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.577+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe21c15efe0 con 0x7fe20006b4d0 2024-03-04T10:10:15.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.584+0000 7fe211ffb640 1 -- 172.21.15.99:0/1147178214 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252076 (secure 0 0 0) 0x7fe21c15efe0 con 0x7fe20006b4d0 2024-03-04T10:10:15.589 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:15.590 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe20006b4d0 msgr2=0x7fe20006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe20006b4d0 0x7fe20006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe21c153ab0 tx=0x7fe20c008040 comp rx=0 tx=0).stop 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 msgr2=0x7fe21c1f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 0x7fe21c1f23c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe218011680 tx=0x7fe218011b40 comp rx=0 tx=0).stop 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 shutdown_connections 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe20006b4d0 0x7fe20006d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe21c15d200 0x7fe21c14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe21c153670 0x7fe21c15db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 --2- 172.21.15.99:0/1147178214 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe21c15ce20 0x7fe21c1f23c0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 >> 172.21.15.99:0/1147178214 conn(0x7fe21c06acd0 msgr2=0x7fe21c15a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:15.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 shutdown_connections 2024-03-04T10:10:15.593 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.586+0000 7fe223353640 1 -- 172.21.15.99:0/1147178214 wait complete. 2024-03-04T10:10:15.600 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-04T10:10:15.534042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251340,"kb_used_data":10184,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754723380,"statfs":{"total":773094113280,"available":772836741120,"internally_reserved":0,"allocated":10428416,"data_stored":5463332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002536"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412747+0000","last_change":"2024-03-04T10:10:07.878394+0000","last_active":"2024-03-04T10:10:09.412747+0000","last_peered":"2024-03-04T10:10:09.412747+0000","last_clean":"2024-03-04T10:10:09.412747+0000","last_became_active":"2024-03-04T10:10:07.878112+0000","last_became_peered":"2024-03-04T10:10:07.878112+0000","last_unstale":"2024-03-04T10:10:09.412747+0000","last_undegraded":"2024-03-04T10:10:09.412747+0000","last_fullsized":"2024-03-04T10:10:09.412747+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:44.923877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.413090+0000","last_change":"2024-03-04T10:10:07.869484+0000","last_active":"2024-03-04T10:10:09.413090+0000","last_peered":"2024-03-04T10:10:09.413090+0000","last_clean":"2024-03-04T10:10:09.413090+0000","last_became_active":"2024-03-04T10:10:07.869018+0000","last_became_peered":"2024-03-04T10:10:07.869018+0000","last_unstale":"2024-03-04T10:10:09.413090+0000","last_undegraded":"2024-03-04T10:10:09.413090+0000","last_fullsized":"2024-03-04T10:10:09.413090+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:37:31.841619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412363+0000","last_change":"2024-03-04T10:10:07.868503+0000","last_active":"2024-03-04T10:10:09.412363+0000","last_peered":"2024-03-04T10:10:09.412363+0000","last_clean":"2024-03-04T10:10:09.412363+0000","last_became_active":"2024-03-04T10:10:07.868217+0000","last_became_peered":"2024-03-04T10:10:07.868217+0000","last_unstale":"2024-03-04T10:10:09.412363+0000","last_undegraded":"2024-03-04T10:10:09.412363+0000","last_fullsized":"2024-03-04T10:10:09.412363+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:00.518203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234924+0000","last_change":"2024-03-04T10:10:07.872769+0000","last_active":"2024-03-04T10:10:09.234924+0000","last_peered":"2024-03-04T10:10:09.234924+0000","last_clean":"2024-03-04T10:10:09.234924+0000","last_became_active":"2024-03-04T10:10:07.871908+0000","last_became_peered":"2024-03-04T10:10:07.871908+0000","last_unstale":"2024-03-04T10:10:09.234924+0000","last_undegraded":"2024-03-04T10:10:09.234924+0000","last_fullsized":"2024-03-04T10:10:09.234924+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:29.671789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863538+0000","last_change":"2024-03-04T10:10:07.866811+0000","last_active":"2024-03-04T10:10:08.863538+0000","last_peered":"2024-03-04T10:10:08.863538+0000","last_clean":"2024-03-04T10:10:08.863538+0000","last_became_active":"2024-03-04T10:10:07.865773+0000","last_became_peered":"2024-03-04T10:10:07.865773+0000","last_unstale":"2024-03-04T10:10:08.863538+0000","last_undegraded":"2024-03-04T10:10:08.863538+0000","last_fullsized":"2024-03-04T10:10:08.863538+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:33:44.363183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864361+0000","last_change":"2024-03-04T10:10:07.879427+0000","last_active":"2024-03-04T10:10:08.864361+0000","last_peered":"2024-03-04T10:10:08.864361+0000","last_clean":"2024-03-04T10:10:08.864361+0000","last_became_active":"2024-03-04T10:10:07.879176+0000","last_became_peered":"2024-03-04T10:10:07.879176+0000","last_unstale":"2024-03-04T10:10:08.864361+0000","last_undegraded":"2024-03-04T10:10:08.864361+0000","last_fullsized":"2024-03-04T10:10:08.864361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:04:32.411857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862823+0000","last_change":"2024-03-04T10:10:07.860822+0000","last_active":"2024-03-04T10:10:08.862823+0000","last_peered":"2024-03-04T10:10:08.862823+0000","last_clean":"2024-03-04T10:10:08.862823+0000","last_became_active":"2024-03-04T10:10:07.860566+0000","last_became_peered":"2024-03-04T10:10:07.860566+0000","last_unstale":"2024-03-04T10:10:08.862823+0000","last_undegraded":"2024-03-04T10:10:08.862823+0000","last_fullsized":"2024-03-04T10:10:08.862823+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:18:14.273360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234632+0000","last_change":"2024-03-04T10:10:07.871700+0000","last_active":"2024-03-04T10:10:09.234632+0000","last_peered":"2024-03-04T10:10:09.234632+0000","last_clean":"2024-03-04T10:10:09.234632+0000","last_became_active":"2024-03-04T10:10:07.871244+0000","last_became_peered":"2024-03-04T10:10:07.871244+0000","last_unstale":"2024-03-04T10:10:09.234632+0000","last_undegraded":"2024-03-04T10:10:09.234632+0000","last_fullsized":"2024-03-04T10:10:09.234632+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:23.671502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863847+0000","last_change":"2024-03-04T10:10:07.865666+0000","last_active":"2024-03-04T10:10:08.863847+0000","last_peered":"2024-03-04T10:10:08.863847+0000","last_clean":"2024-03-04T10:10:08.863847+0000","last_became_active":"2024-03-04T10:10:07.864869+0000","last_became_peered":"2024-03-04T10:10:07.864869+0000","last_unstale":"2024-03-04T10:10:08.863847+0000","last_undegraded":"2024-03-04T10:10:08.863847+0000","last_fullsized":"2024-03-04T10:10:08.863847+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:44:14.274680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412628+0000","last_change":"2024-03-04T10:10:07.873055+0000","last_active":"2024-03-04T10:10:09.412628+0000","last_peered":"2024-03-04T10:10:09.412628+0000","last_clean":"2024-03-04T10:10:09.412628+0000","last_became_active":"2024-03-04T10:10:07.872792+0000","last_became_peered":"2024-03-04T10:10:07.872792+0000","last_unstale":"2024-03-04T10:10:09.412628+0000","last_undegraded":"2024-03-04T10:10:09.412628+0000","last_fullsized":"2024-03-04T10:10:09.412628+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:34:23.923307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863950+0000","last_change":"2024-03-04T10:10:07.872616+0000","last_active":"2024-03-04T10:10:08.863950+0000","last_peered":"2024-03-04T10:10:08.863950+0000","last_clean":"2024-03-04T10:10:08.863950+0000","last_became_active":"2024-03-04T10:10:07.872333+0000","last_became_peered":"2024-03-04T10:10:07.872333+0000","last_unstale":"2024-03-04T10:10:08.863950+0000","last_undegraded":"2024-03-04T10:10:08.863950+0000","last_fullsized":"2024-03-04T10:10:08.863950+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:47:29.671789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412699+0000","last_change":"2024-03-04T10:10:07.860254+0000","last_active":"2024-03-04T10:10:09.412699+0000","last_peered":"2024-03-04T10:10:09.412699+0000","last_clean":"2024-03-04T10:10:09.412699+0000","last_became_active":"2024-03-04T10:10:07.859988+0000","last_became_peered":"2024-03-04T10:10:07.859988+0000","last_unstale":"2024-03-04T10:10:09.412699+0000","last_undegraded":"2024-03-04T10:10:09.412699+0000","last_fullsized":"2024-03-04T10:10:09.412699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:14:49.930276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234789+0000","last_change":"2024-03-04T10:10:07.872443+0000","last_active":"2024-03-04T10:10:09.234789+0000","last_peered":"2024-03-04T10:10:09.234789+0000","last_clean":"2024-03-04T10:10:09.234789+0000","last_became_active":"2024-03-04T10:10:07.872150+0000","last_became_peered":"2024-03-04T10:10:07.872150+0000","last_unstale":"2024-03-04T10:10:09.234789+0000","last_undegraded":"2024-03-04T10:10:09.234789+0000","last_fullsized":"2024-03-04T10:10:09.234789+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:32.626643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862838+0000","last_change":"2024-03-04T10:10:07.874049+0000","last_active":"2024-03-04T10:10:08.862838+0000","last_peered":"2024-03-04T10:10:08.862838+0000","last_clean":"2024-03-04T10:10:08.862838+0000","last_became_active":"2024-03-04T10:10:07.873412+0000","last_became_peered":"2024-03-04T10:10:07.873412+0000","last_unstale":"2024-03-04T10:10:08.862838+0000","last_undegraded":"2024-03-04T10:10:08.862838+0000","last_fullsized":"2024-03-04T10:10:08.862838+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:07:34.688492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234668+0000","last_change":"2024-03-04T10:10:07.871825+0000","last_active":"2024-03-04T10:10:09.234668+0000","last_peered":"2024-03-04T10:10:09.234668+0000","last_clean":"2024-03-04T10:10:09.234668+0000","last_became_active":"2024-03-04T10:10:07.871170+0000","last_became_peered":"2024-03-04T10:10:07.871170+0000","last_unstale":"2024-03-04T10:10:09.234668+0000","last_undegraded":"2024-03-04T10:10:09.234668+0000","last_fullsized":"2024-03-04T10:10:09.234668+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:33:44.363183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864311+0000","last_change":"2024-03-04T10:10:07.874339+0000","last_active":"2024-03-04T10:10:08.864311+0000","last_peered":"2024-03-04T10:10:08.864311+0000","last_clean":"2024-03-04T10:10:08.864311+0000","last_became_active":"2024-03-04T10:10:07.874015+0000","last_became_peered":"2024-03-04T10:10:07.874015+0000","last_unstale":"2024-03-04T10:10:08.864311+0000","last_undegraded":"2024-03-04T10:10:08.864311+0000","last_fullsized":"2024-03-04T10:10:08.864311+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:30.724416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864834+0000","last_change":"2024-03-04T10:10:07.876972+0000","last_active":"2024-03-04T10:10:08.864834+0000","last_peered":"2024-03-04T10:10:08.864834+0000","last_clean":"2024-03-04T10:10:08.864834+0000","last_became_active":"2024-03-04T10:10:07.876725+0000","last_became_peered":"2024-03-04T10:10:07.876725+0000","last_unstale":"2024-03-04T10:10:08.864834+0000","last_undegraded":"2024-03-04T10:10:08.864834+0000","last_fullsized":"2024-03-04T10:10:08.864834+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:36:46.063111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412292+0000","last_change":"2024-03-04T10:10:07.867151+0000","last_active":"2024-03-04T10:10:09.412292+0000","last_peered":"2024-03-04T10:10:09.412292+0000","last_clean":"2024-03-04T10:10:09.412292+0000","last_became_active":"2024-03-04T10:10:07.866861+0000","last_became_peered":"2024-03-04T10:10:07.866861+0000","last_unstale":"2024-03-04T10:10:09.412292+0000","last_undegraded":"2024-03-04T10:10:09.412292+0000","last_fullsized":"2024-03-04T10:10:09.412292+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:59:11.495031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863986+0000","last_change":"2024-03-04T10:10:07.871466+0000","last_active":"2024-03-04T10:10:08.863986+0000","last_peered":"2024-03-04T10:10:08.863986+0000","last_clean":"2024-03-04T10:10:08.863986+0000","last_became_active":"2024-03-04T10:10:07.871171+0000","last_became_peered":"2024-03-04T10:10:07.871171+0000","last_unstale":"2024-03-04T10:10:08.863986+0000","last_undegraded":"2024-03-04T10:10:08.863986+0000","last_fullsized":"2024-03-04T10:10:08.863986+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:57.062006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234986+0000","last_change":"2024-03-04T10:10:07.871664+0000","last_active":"2024-03-04T10:10:09.234986+0000","last_peered":"2024-03-04T10:10:09.234986+0000","last_clean":"2024-03-04T10:10:09.234986+0000","last_became_active":"2024-03-04T10:10:07.870853+0000","last_became_peered":"2024-03-04T10:10:07.870853+0000","last_unstale":"2024-03-04T10:10:09.234986+0000","last_undegraded":"2024-03-04T10:10:09.234986+0000","last_fullsized":"2024-03-04T10:10:09.234986+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:58:21.485455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863390+0000","last_change":"2024-03-04T10:10:07.858390+0000","last_active":"2024-03-04T10:10:08.863390+0000","last_peered":"2024-03-04T10:10:08.863390+0000","last_clean":"2024-03-04T10:10:08.863390+0000","last_became_active":"2024-03-04T10:10:07.858135+0000","last_became_peered":"2024-03-04T10:10:07.858135+0000","last_unstale":"2024-03-04T10:10:08.863390+0000","last_undegraded":"2024-03-04T10:10:08.863390+0000","last_fullsized":"2024-03-04T10:10:08.863390+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:04:17.965739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862995+0000","last_change":"2024-03-04T10:10:07.874003+0000","last_active":"2024-03-04T10:10:08.862995+0000","last_peered":"2024-03-04T10:10:08.862995+0000","last_clean":"2024-03-04T10:10:08.862995+0000","last_became_active":"2024-03-04T10:10:07.873065+0000","last_became_peered":"2024-03-04T10:10:07.873065+0000","last_unstale":"2024-03-04T10:10:08.862995+0000","last_undegraded":"2024-03-04T10:10:08.862995+0000","last_fullsized":"2024-03-04T10:10:08.862995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:28:11.336192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412190+0000","last_change":"2024-03-04T10:10:07.873463+0000","last_active":"2024-03-04T10:10:09.412190+0000","last_peered":"2024-03-04T10:10:09.412190+0000","last_clean":"2024-03-04T10:10:09.412190+0000","last_became_active":"2024-03-04T10:10:07.873090+0000","last_became_peered":"2024-03-04T10:10:07.873090+0000","last_unstale":"2024-03-04T10:10:09.412190+0000","last_undegraded":"2024-03-04T10:10:09.412190+0000","last_fullsized":"2024-03-04T10:10:09.412190+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:28:11.336192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412273+0000","last_change":"2024-03-04T10:10:07.868062+0000","last_active":"2024-03-04T10:10:09.412273+0000","last_peered":"2024-03-04T10:10:09.412273+0000","last_clean":"2024-03-04T10:10:09.412273+0000","last_became_active":"2024-03-04T10:10:07.867743+0000","last_became_peered":"2024-03-04T10:10:07.867743+0000","last_unstale":"2024-03-04T10:10:09.412273+0000","last_undegraded":"2024-03-04T10:10:09.412273+0000","last_fullsized":"2024-03-04T10:10:09.412273+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:34:33.226502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412608+0000","last_change":"2024-03-04T10:10:07.870242+0000","last_active":"2024-03-04T10:10:09.412608+0000","last_peered":"2024-03-04T10:10:09.412608+0000","last_clean":"2024-03-04T10:10:09.412608+0000","last_became_active":"2024-03-04T10:10:07.869937+0000","last_became_peered":"2024-03-04T10:10:07.869937+0000","last_unstale":"2024-03-04T10:10:09.412608+0000","last_undegraded":"2024-03-04T10:10:09.412608+0000","last_fullsized":"2024-03-04T10:10:09.412608+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:51.062336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863744+0000","last_change":"2024-03-04T10:10:07.873467+0000","last_active":"2024-03-04T10:10:08.863744+0000","last_peered":"2024-03-04T10:10:08.863744+0000","last_clean":"2024-03-04T10:10:08.863744+0000","last_became_active":"2024-03-04T10:10:07.873152+0000","last_became_peered":"2024-03-04T10:10:07.873152+0000","last_unstale":"2024-03-04T10:10:08.863744+0000","last_undegraded":"2024-03-04T10:10:08.863744+0000","last_fullsized":"2024-03-04T10:10:08.863744+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:18.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412541+0000","last_change":"2024-03-04T10:10:07.866525+0000","last_active":"2024-03-04T10:10:09.412541+0000","last_peered":"2024-03-04T10:10:09.412541+0000","last_clean":"2024-03-04T10:10:09.412541+0000","last_became_active":"2024-03-04T10:10:07.866250+0000","last_became_peered":"2024-03-04T10:10:07.866250+0000","last_unstale":"2024-03-04T10:10:09.412541+0000","last_undegraded":"2024-03-04T10:10:09.412541+0000","last_fullsized":"2024-03-04T10:10:09.412541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:08:26.142632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.235077+0000","last_change":"2024-03-04T10:10:07.872972+0000","last_active":"2024-03-04T10:10:09.235077+0000","last_peered":"2024-03-04T10:10:09.235077+0000","last_clean":"2024-03-04T10:10:09.235077+0000","last_became_active":"2024-03-04T10:10:07.872203+0000","last_became_peered":"2024-03-04T10:10:07.872203+0000","last_unstale":"2024-03-04T10:10:09.235077+0000","last_undegraded":"2024-03-04T10:10:09.235077+0000","last_fullsized":"2024-03-04T10:10:09.235077+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:41:57.062006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863377+0000","last_change":"2024-03-04T10:10:07.858865+0000","last_active":"2024-03-04T10:10:08.863377+0000","last_peered":"2024-03-04T10:10:08.863377+0000","last_clean":"2024-03-04T10:10:08.863377+0000","last_became_active":"2024-03-04T10:10:07.858599+0000","last_became_peered":"2024-03-04T10:10:07.858599+0000","last_unstale":"2024-03-04T10:10:08.863377+0000","last_undegraded":"2024-03-04T10:10:08.863377+0000","last_fullsized":"2024-03-04T10:10:08.863377+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:42:41.350333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863176+0000","last_change":"2024-03-04T10:10:07.867857+0000","last_active":"2024-03-04T10:10:08.863176+0000","last_peered":"2024-03-04T10:10:08.863176+0000","last_clean":"2024-03-04T10:10:08.863176+0000","last_became_active":"2024-03-04T10:10:07.867577+0000","last_became_peered":"2024-03-04T10:10:07.867577+0000","last_unstale":"2024-03-04T10:10:08.863176+0000","last_undegraded":"2024-03-04T10:10:08.863176+0000","last_fullsized":"2024-03-04T10:10:08.863176+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:23.671502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412858+0000","last_change":"2024-03-04T10:10:07.878087+0000","last_active":"2024-03-04T10:10:09.412858+0000","last_peered":"2024-03-04T10:10:09.412858+0000","last_clean":"2024-03-04T10:10:09.412858+0000","last_became_active":"2024-03-04T10:10:07.877811+0000","last_became_peered":"2024-03-04T10:10:07.877811+0000","last_unstale":"2024-03-04T10:10:09.412858+0000","last_undegraded":"2024-03-04T10:10:09.412858+0000","last_fullsized":"2024-03-04T10:10:09.412858+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:30.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864366+0000","last_change":"2024-03-04T10:10:07.874223+0000","last_active":"2024-03-04T10:10:08.864366+0000","last_peered":"2024-03-04T10:10:08.864366+0000","last_clean":"2024-03-04T10:10:08.864366+0000","last_became_active":"2024-03-04T10:10:07.873605+0000","last_became_peered":"2024-03-04T10:10:07.873605+0000","last_unstale":"2024-03-04T10:10:08.864366+0000","last_undegraded":"2024-03-04T10:10:08.864366+0000","last_fullsized":"2024-03-04T10:10:08.864366+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:50:30.838692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864609+0000","last_change":"2024-03-04T10:10:07.876938+0000","last_active":"2024-03-04T10:10:08.864609+0000","last_peered":"2024-03-04T10:10:08.864609+0000","last_clean":"2024-03-04T10:10:08.864609+0000","last_became_active":"2024-03-04T10:10:07.876647+0000","last_became_peered":"2024-03-04T10:10:07.876647+0000","last_unstale":"2024-03-04T10:10:08.864609+0000","last_undegraded":"2024-03-04T10:10:08.864609+0000","last_fullsized":"2024-03-04T10:10:08.864609+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:41:32.082893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"92'4","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412492+0000","last_change":"2024-03-04T10:10:06.864395+0000","last_active":"2024-03-04T10:10:09.412492+0000","last_peered":"2024-03-04T10:10:09.412492+0000","last_clean":"2024-03-04T10:10:09.412492+0000","last_became_active":"2024-03-04T10:10:06.864082+0000","last_became_peered":"2024-03-04T10:10:06.864082+0000","last_unstale":"2024-03-04T10:10:09.412492+0000","last_undegraded":"2024-03-04T10:10:09.412492+0000","last_fullsized":"2024-03-04T10:10:09.412492+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:35:33.665856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864764+0000","last_change":"2024-03-04T10:10:07.872089+0000","last_active":"2024-03-04T10:10:08.864764+0000","last_peered":"2024-03-04T10:10:08.864764+0000","last_clean":"2024-03-04T10:10:08.864764+0000","last_became_active":"2024-03-04T10:10:07.871833+0000","last_became_peered":"2024-03-04T10:10:07.871833+0000","last_unstale":"2024-03-04T10:10:08.864764+0000","last_undegraded":"2024-03-04T10:10:08.864764+0000","last_fullsized":"2024-03-04T10:10:08.864764+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:13:35.914027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863884+0000","last_change":"2024-03-04T10:10:06.861011+0000","last_active":"2024-03-04T10:10:08.863884+0000","last_peered":"2024-03-04T10:10:08.863884+0000","last_clean":"2024-03-04T10:10:08.863884+0000","last_became_active":"2024-03-04T10:10:06.860571+0000","last_became_peered":"2024-03-04T10:10:06.860571+0000","last_unstale":"2024-03-04T10:10:08.863884+0000","last_undegraded":"2024-03-04T10:10:08.863884+0000","last_fullsized":"2024-03-04T10:10:08.863884+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:51:19.769656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862752+0000","last_change":"2024-03-04T10:10:07.871536+0000","last_active":"2024-03-04T10:10:08.862752+0000","last_peered":"2024-03-04T10:10:08.862752+0000","last_clean":"2024-03-04T10:10:08.862752+0000","last_became_active":"2024-03-04T10:10:07.871230+0000","last_became_peered":"2024-03-04T10:10:07.871230+0000","last_unstale":"2024-03-04T10:10:08.862752+0000","last_undegraded":"2024-03-04T10:10:08.862752+0000","last_fullsized":"2024-03-04T10:10:08.862752+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:17:05.090521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862937+0000","last_change":"2024-03-04T10:10:06.860271+0000","last_active":"2024-03-04T10:10:08.862937+0000","last_peered":"2024-03-04T10:10:08.862937+0000","last_clean":"2024-03-04T10:10:08.862937+0000","last_became_active":"2024-03-04T10:10:06.859984+0000","last_became_peered":"2024-03-04T10:10:06.859984+0000","last_unstale":"2024-03-04T10:10:08.862937+0000","last_undegraded":"2024-03-04T10:10:08.862937+0000","last_fullsized":"2024-03-04T10:10:08.862937+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:28:52.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864871+0000","last_change":"2024-03-04T10:10:07.872321+0000","last_active":"2024-03-04T10:10:08.864871+0000","last_peered":"2024-03-04T10:10:08.864871+0000","last_clean":"2024-03-04T10:10:08.864871+0000","last_became_active":"2024-03-04T10:10:07.871450+0000","last_became_peered":"2024-03-04T10:10:07.871450+0000","last_unstale":"2024-03-04T10:10:08.864871+0000","last_undegraded":"2024-03-04T10:10:08.864871+0000","last_fullsized":"2024-03-04T10:10:08.864871+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:28:14.000641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412829+0000","last_change":"2024-03-04T10:10:06.865348+0000","last_active":"2024-03-04T10:10:09.412829+0000","last_peered":"2024-03-04T10:10:09.412829+0000","last_clean":"2024-03-04T10:10:09.412829+0000","last_became_active":"2024-03-04T10:10:06.865084+0000","last_became_peered":"2024-03-04T10:10:06.865084+0000","last_unstale":"2024-03-04T10:10:09.412829+0000","last_undegraded":"2024-03-04T10:10:09.412829+0000","last_fullsized":"2024-03-04T10:10:09.412829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:34:02.895613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234638+0000","last_change":"2024-03-04T10:10:07.880728+0000","last_active":"2024-03-04T10:10:09.234638+0000","last_peered":"2024-03-04T10:10:09.234638+0000","last_clean":"2024-03-04T10:10:09.234638+0000","last_became_active":"2024-03-04T10:10:07.880445+0000","last_became_peered":"2024-03-04T10:10:07.880445+0000","last_unstale":"2024-03-04T10:10:09.234638+0000","last_undegraded":"2024-03-04T10:10:09.234638+0000","last_fullsized":"2024-03-04T10:10:09.234638+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:04:17.965739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863541+0000","last_change":"2024-03-04T10:10:06.863293+0000","last_active":"2024-03-04T10:10:08.863541+0000","last_peered":"2024-03-04T10:10:08.863541+0000","last_clean":"2024-03-04T10:10:08.863541+0000","last_became_active":"2024-03-04T10:10:06.863021+0000","last_became_peered":"2024-03-04T10:10:06.863021+0000","last_unstale":"2024-03-04T10:10:08.863541+0000","last_undegraded":"2024-03-04T10:10:08.863541+0000","last_fullsized":"2024-03-04T10:10:08.863541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:58:45.314856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412425+0000","last_change":"2024-03-04T10:10:07.871816+0000","last_active":"2024-03-04T10:10:09.412425+0000","last_peered":"2024-03-04T10:10:09.412425+0000","last_clean":"2024-03-04T10:10:09.412425+0000","last_became_active":"2024-03-04T10:10:07.871552+0000","last_became_peered":"2024-03-04T10:10:07.871552+0000","last_unstale":"2024-03-04T10:10:09.412425+0000","last_undegraded":"2024-03-04T10:10:09.412425+0000","last_fullsized":"2024-03-04T10:10:09.412425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:04:17.965739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234948+0000","last_change":"2024-03-04T10:10:06.857045+0000","last_active":"2024-03-04T10:10:09.234948+0000","last_peered":"2024-03-04T10:10:09.234948+0000","last_clean":"2024-03-04T10:10:09.234948+0000","last_became_active":"2024-03-04T10:10:06.856753+0000","last_became_peered":"2024-03-04T10:10:06.856753+0000","last_unstale":"2024-03-04T10:10:09.234948+0000","last_undegraded":"2024-03-04T10:10:09.234948+0000","last_fullsized":"2024-03-04T10:10:09.234948+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:24:44.951154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864440+0000","last_change":"2024-03-04T10:10:07.879423+0000","last_active":"2024-03-04T10:10:08.864440+0000","last_peered":"2024-03-04T10:10:08.864440+0000","last_clean":"2024-03-04T10:10:08.864440+0000","last_became_active":"2024-03-04T10:10:07.879160+0000","last_became_peered":"2024-03-04T10:10:07.879160+0000","last_unstale":"2024-03-04T10:10:08.864440+0000","last_undegraded":"2024-03-04T10:10:08.864440+0000","last_fullsized":"2024-03-04T10:10:08.864440+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:00:54.431672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412379+0000","last_change":"2024-03-04T10:10:06.862082+0000","last_active":"2024-03-04T10:10:09.412379+0000","last_peered":"2024-03-04T10:10:09.412379+0000","last_clean":"2024-03-04T10:10:09.412379+0000","last_became_active":"2024-03-04T10:10:06.861780+0000","last_became_peered":"2024-03-04T10:10:06.861780+0000","last_unstale":"2024-03-04T10:10:09.412379+0000","last_undegraded":"2024-03-04T10:10:09.412379+0000","last_fullsized":"2024-03-04T10:10:09.412379+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:32.258399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234937+0000","last_change":"2024-03-04T10:10:07.870545+0000","last_active":"2024-03-04T10:10:09.234937+0000","last_peered":"2024-03-04T10:10:09.234937+0000","last_clean":"2024-03-04T10:10:09.234937+0000","last_became_active":"2024-03-04T10:10:07.870245+0000","last_became_peered":"2024-03-04T10:10:07.870245+0000","last_unstale":"2024-03-04T10:10:09.234937+0000","last_undegraded":"2024-03-04T10:10:09.234937+0000","last_fullsized":"2024-03-04T10:10:09.234937+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:44:14.274680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862783+0000","last_change":"2024-03-04T10:10:06.865722+0000","last_active":"2024-03-04T10:10:08.862783+0000","last_peered":"2024-03-04T10:10:08.862783+0000","last_clean":"2024-03-04T10:10:08.862783+0000","last_became_active":"2024-03-04T10:10:06.865459+0000","last_became_peered":"2024-03-04T10:10:06.865459+0000","last_unstale":"2024-03-04T10:10:08.862783+0000","last_undegraded":"2024-03-04T10:10:08.862783+0000","last_fullsized":"2024-03-04T10:10:08.862783+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:33:42.657260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863671+0000","last_change":"2024-03-04T10:10:07.864242+0000","last_active":"2024-03-04T10:10:08.863671+0000","last_peered":"2024-03-04T10:10:08.863671+0000","last_clean":"2024-03-04T10:10:08.863671+0000","last_became_active":"2024-03-04T10:10:07.863936+0000","last_became_peered":"2024-03-04T10:10:07.863936+0000","last_unstale":"2024-03-04T10:10:08.863671+0000","last_undegraded":"2024-03-04T10:10:08.863671+0000","last_fullsized":"2024-03-04T10:10:08.863671+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:32.626643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412854+0000","last_change":"2024-03-04T10:10:06.860947+0000","last_active":"2024-03-04T10:10:09.412854+0000","last_peered":"2024-03-04T10:10:09.412854+0000","last_clean":"2024-03-04T10:10:09.412854+0000","last_became_active":"2024-03-04T10:10:06.860654+0000","last_became_peered":"2024-03-04T10:10:06.860654+0000","last_unstale":"2024-03-04T10:10:09.412854+0000","last_undegraded":"2024-03-04T10:10:09.412854+0000","last_fullsized":"2024-03-04T10:10:09.412854+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:31.622074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863024+0000","last_change":"2024-03-04T10:10:07.862730+0000","last_active":"2024-03-04T10:10:08.863024+0000","last_peered":"2024-03-04T10:10:08.863024+0000","last_clean":"2024-03-04T10:10:08.863024+0000","last_became_active":"2024-03-04T10:10:07.862423+0000","last_became_peered":"2024-03-04T10:10:07.862423+0000","last_unstale":"2024-03-04T10:10:08.863024+0000","last_undegraded":"2024-03-04T10:10:08.863024+0000","last_fullsized":"2024-03-04T10:10:08.863024+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:48:37.658091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412781+0000","last_change":"2024-03-04T10:10:06.861468+0000","last_active":"2024-03-04T10:10:09.412781+0000","last_peered":"2024-03-04T10:10:09.412781+0000","last_clean":"2024-03-04T10:10:09.412781+0000","last_became_active":"2024-03-04T10:10:06.861177+0000","last_became_peered":"2024-03-04T10:10:06.861177+0000","last_unstale":"2024-03-04T10:10:09.412781+0000","last_undegraded":"2024-03-04T10:10:09.412781+0000","last_fullsized":"2024-03-04T10:10:09.412781+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:50.057767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412690+0000","last_change":"2024-03-04T10:10:07.873968+0000","last_active":"2024-03-04T10:10:09.412690+0000","last_peered":"2024-03-04T10:10:09.412690+0000","last_clean":"2024-03-04T10:10:09.412690+0000","last_became_active":"2024-03-04T10:10:07.873695+0000","last_became_peered":"2024-03-04T10:10:07.873695+0000","last_unstale":"2024-03-04T10:10:09.412690+0000","last_undegraded":"2024-03-04T10:10:09.412690+0000","last_fullsized":"2024-03-04T10:10:09.412690+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:30.289441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"92'3","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864086+0000","last_change":"2024-03-04T10:10:06.866005+0000","last_active":"2024-03-04T10:10:08.864086+0000","last_peered":"2024-03-04T10:10:08.864086+0000","last_clean":"2024-03-04T10:10:08.864086+0000","last_became_active":"2024-03-04T10:10:06.865735+0000","last_became_peered":"2024-03-04T10:10:06.865735+0000","last_unstale":"2024-03-04T10:10:08.864086+0000","last_undegraded":"2024-03-04T10:10:08.864086+0000","last_fullsized":"2024-03-04T10:10:08.864086+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:42.033635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.235094+0000","last_change":"2024-03-04T10:10:07.871819+0000","last_active":"2024-03-04T10:10:09.235094+0000","last_peered":"2024-03-04T10:10:09.235094+0000","last_clean":"2024-03-04T10:10:09.235094+0000","last_became_active":"2024-03-04T10:10:07.871217+0000","last_became_peered":"2024-03-04T10:10:07.871217+0000","last_unstale":"2024-03-04T10:10:09.235094+0000","last_undegraded":"2024-03-04T10:10:09.235094+0000","last_fullsized":"2024-03-04T10:10:09.235094+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:37.125333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"92'6","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864670+0000","last_change":"2024-03-04T10:10:06.861324+0000","last_active":"2024-03-04T10:10:08.864670+0000","last_peered":"2024-03-04T10:10:08.864670+0000","last_clean":"2024-03-04T10:10:08.864670+0000","last_became_active":"2024-03-04T10:10:06.861027+0000","last_became_peered":"2024-03-04T10:10:06.861027+0000","last_unstale":"2024-03-04T10:10:08.864670+0000","last_undegraded":"2024-03-04T10:10:08.864670+0000","last_fullsized":"2024-03-04T10:10:08.864670+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:18:13.268791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234819+0000","last_change":"2024-03-04T10:10:07.871504+0000","last_active":"2024-03-04T10:10:09.234819+0000","last_peered":"2024-03-04T10:10:09.234819+0000","last_clean":"2024-03-04T10:10:09.234819+0000","last_became_active":"2024-03-04T10:10:07.871027+0000","last_became_peered":"2024-03-04T10:10:07.871027+0000","last_unstale":"2024-03-04T10:10:09.234819+0000","last_undegraded":"2024-03-04T10:10:09.234819+0000","last_fullsized":"2024-03-04T10:10:09.234819+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:22:18.123148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863320+0000","last_change":"2024-03-04T10:10:06.864064+0000","last_active":"2024-03-04T10:10:08.863320+0000","last_peered":"2024-03-04T10:10:08.863320+0000","last_clean":"2024-03-04T10:10:08.863320+0000","last_became_active":"2024-03-04T10:10:06.863773+0000","last_became_peered":"2024-03-04T10:10:06.863773+0000","last_unstale":"2024-03-04T10:10:08.863320+0000","last_undegraded":"2024-03-04T10:10:08.863320+0000","last_fullsized":"2024-03-04T10:10:08.863320+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:58:29.539950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862806+0000","last_change":"2024-03-04T10:10:07.866007+0000","last_active":"2024-03-04T10:10:08.862806+0000","last_peered":"2024-03-04T10:10:08.862806+0000","last_clean":"2024-03-04T10:10:08.862806+0000","last_became_active":"2024-03-04T10:10:07.865526+0000","last_became_peered":"2024-03-04T10:10:07.865526+0000","last_unstale":"2024-03-04T10:10:08.862806+0000","last_undegraded":"2024-03-04T10:10:08.862806+0000","last_fullsized":"2024-03-04T10:10:08.862806+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:34:33.914215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864532+0000","last_change":"2024-03-04T10:10:06.858647+0000","last_active":"2024-03-04T10:10:08.864532+0000","last_peered":"2024-03-04T10:10:08.864532+0000","last_clean":"2024-03-04T10:10:08.864532+0000","last_became_active":"2024-03-04T10:10:06.858363+0000","last_became_peered":"2024-03-04T10:10:06.858363+0000","last_unstale":"2024-03-04T10:10:08.864532+0000","last_undegraded":"2024-03-04T10:10:08.864532+0000","last_fullsized":"2024-03-04T10:10:08.864532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:48:36.653522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412319+0000","last_change":"2024-03-04T10:10:07.872995+0000","last_active":"2024-03-04T10:10:09.412319+0000","last_peered":"2024-03-04T10:10:09.412319+0000","last_clean":"2024-03-04T10:10:09.412319+0000","last_became_active":"2024-03-04T10:10:07.872728+0000","last_became_peered":"2024-03-04T10:10:07.872728+0000","last_unstale":"2024-03-04T10:10:09.412319+0000","last_undegraded":"2024-03-04T10:10:09.412319+0000","last_fullsized":"2024-03-04T10:10:09.412319+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:05:53.758242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234747+0000","last_change":"2024-03-04T10:10:06.860311+0000","last_active":"2024-03-04T10:10:09.234747+0000","last_peered":"2024-03-04T10:10:09.234747+0000","last_clean":"2024-03-04T10:10:09.234747+0000","last_became_active":"2024-03-04T10:10:06.860013+0000","last_became_peered":"2024-03-04T10:10:06.860013+0000","last_unstale":"2024-03-04T10:10:09.234747+0000","last_undegraded":"2024-03-04T10:10:09.234747+0000","last_fullsized":"2024-03-04T10:10:09.234747+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:45:18.950003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412727+0000","last_change":"2024-03-04T10:10:07.868738+0000","last_active":"2024-03-04T10:10:09.412727+0000","last_peered":"2024-03-04T10:10:09.412727+0000","last_clean":"2024-03-04T10:10:09.412727+0000","last_became_active":"2024-03-04T10:10:07.868460+0000","last_became_peered":"2024-03-04T10:10:07.868460+0000","last_unstale":"2024-03-04T10:10:09.412727+0000","last_undegraded":"2024-03-04T10:10:09.412727+0000","last_fullsized":"2024-03-04T10:10:09.412727+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:43:41.954257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"92'8","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.234757+0000","last_change":"2024-03-04T10:10:06.859945+0000","last_active":"2024-03-04T10:10:09.234757+0000","last_peered":"2024-03-04T10:10:09.234757+0000","last_clean":"2024-03-04T10:10:09.234757+0000","last_became_active":"2024-03-04T10:10:06.859616+0000","last_became_peered":"2024-03-04T10:10:06.859616+0000","last_unstale":"2024-03-04T10:10:09.234757+0000","last_undegraded":"2024-03-04T10:10:09.234757+0000","last_fullsized":"2024-03-04T10:10:09.234757+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:01:06.838971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864579+0000","last_change":"2024-03-04T10:10:07.878757+0000","last_active":"2024-03-04T10:10:08.864579+0000","last_peered":"2024-03-04T10:10:08.864579+0000","last_clean":"2024-03-04T10:10:08.864579+0000","last_became_active":"2024-03-04T10:10:07.877328+0000","last_became_peered":"2024-03-04T10:10:07.877328+0000","last_unstale":"2024-03-04T10:10:08.864579+0000","last_undegraded":"2024-03-04T10:10:08.864579+0000","last_fullsized":"2024-03-04T10:10:08.864579+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:31.256258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412623+0000","last_change":"2024-03-04T10:10:06.862337+0000","last_active":"2024-03-04T10:10:09.412623+0000","last_peered":"2024-03-04T10:10:09.412623+0000","last_clean":"2024-03-04T10:10:09.412623+0000","last_became_active":"2024-03-04T10:10:06.861981+0000","last_became_peered":"2024-03-04T10:10:06.861981+0000","last_unstale":"2024-03-04T10:10:09.412623+0000","last_undegraded":"2024-03-04T10:10:09.412623+0000","last_fullsized":"2024-03-04T10:10:09.412623+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:10:24.846954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412478+0000","last_change":"2024-03-04T10:10:07.867639+0000","last_active":"2024-03-04T10:10:09.412478+0000","last_peered":"2024-03-04T10:10:09.412478+0000","last_clean":"2024-03-04T10:10:09.412478+0000","last_became_active":"2024-03-04T10:10:07.867347+0000","last_became_peered":"2024-03-04T10:10:07.867347+0000","last_unstale":"2024-03-04T10:10:09.412478+0000","last_undegraded":"2024-03-04T10:10:09.412478+0000","last_fullsized":"2024-03-04T10:10:09.412478+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:33:44.363183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863036+0000","last_change":"2024-03-04T10:10:06.858629+0000","last_active":"2024-03-04T10:10:08.863036+0000","last_peered":"2024-03-04T10:10:08.863036+0000","last_clean":"2024-03-04T10:10:08.863036+0000","last_became_active":"2024-03-04T10:10:06.858362+0000","last_became_peered":"2024-03-04T10:10:06.858362+0000","last_unstale":"2024-03-04T10:10:08.863036+0000","last_undegraded":"2024-03-04T10:10:08.863036+0000","last_fullsized":"2024-03-04T10:10:08.863036+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:37:30.837050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864014+0000","last_change":"2024-03-04T10:10:07.861155+0000","last_active":"2024-03-04T10:10:08.864014+0000","last_peered":"2024-03-04T10:10:08.864014+0000","last_clean":"2024-03-04T10:10:08.864014+0000","last_became_active":"2024-03-04T10:10:07.860854+0000","last_became_peered":"2024-03-04T10:10:07.860854+0000","last_unstale":"2024-03-04T10:10:08.864014+0000","last_undegraded":"2024-03-04T10:10:08.864014+0000","last_fullsized":"2024-03-04T10:10:08.864014+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:28:37.125333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"92'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864165+0000","last_change":"2024-03-04T10:10:06.866361+0000","last_active":"2024-03-04T10:10:08.864165+0000","last_peered":"2024-03-04T10:10:08.864165+0000","last_clean":"2024-03-04T10:10:08.864165+0000","last_became_active":"2024-03-04T10:10:06.866083+0000","last_became_peered":"2024-03-04T10:10:06.866083+0000","last_unstale":"2024-03-04T10:10:08.864165+0000","last_undegraded":"2024-03-04T10:10:08.864165+0000","last_fullsized":"2024-03-04T10:10:08.864165+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:21:25.671449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864461+0000","last_change":"2024-03-04T10:10:07.880062+0000","last_active":"2024-03-04T10:10:08.864461+0000","last_peered":"2024-03-04T10:10:08.864461+0000","last_clean":"2024-03-04T10:10:08.864461+0000","last_became_active":"2024-03-04T10:10:07.879675+0000","last_became_peered":"2024-03-04T10:10:07.879675+0000","last_unstale":"2024-03-04T10:10:08.864461+0000","last_undegraded":"2024-03-04T10:10:08.864461+0000","last_fullsized":"2024-03-04T10:10:08.864461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:31:57.416214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412196+0000","last_change":"2024-03-04T10:10:06.861629+0000","last_active":"2024-03-04T10:10:09.412196+0000","last_peered":"2024-03-04T10:10:09.412196+0000","last_clean":"2024-03-04T10:10:09.412196+0000","last_became_active":"2024-03-04T10:10:06.861360+0000","last_became_peered":"2024-03-04T10:10:06.861360+0000","last_unstale":"2024-03-04T10:10:09.412196+0000","last_undegraded":"2024-03-04T10:10:09.412196+0000","last_fullsized":"2024-03-04T10:10:09.412196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:22.666933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412490+0000","last_change":"2024-03-04T10:10:07.867071+0000","last_active":"2024-03-04T10:10:09.412490+0000","last_peered":"2024-03-04T10:10:09.412490+0000","last_clean":"2024-03-04T10:10:09.412490+0000","last_became_active":"2024-03-04T10:10:07.866775+0000","last_became_peered":"2024-03-04T10:10:07.866775+0000","last_unstale":"2024-03-04T10:10:09.412490+0000","last_undegraded":"2024-03-04T10:10:09.412490+0000","last_fullsized":"2024-03-04T10:10:09.412490+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:17:33.262968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412311+0000","last_change":"2024-03-04T10:10:06.857599+0000","last_active":"2024-03-04T10:10:09.412311+0000","last_peered":"2024-03-04T10:10:09.412311+0000","last_clean":"2024-03-04T10:10:09.412311+0000","last_became_active":"2024-03-04T10:10:06.857319+0000","last_became_peered":"2024-03-04T10:10:06.857319+0000","last_unstale":"2024-03-04T10:10:09.412311+0000","last_undegraded":"2024-03-04T10:10:09.412311+0000","last_fullsized":"2024-03-04T10:10:09.412311+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:13:25.112378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412648+0000","last_change":"2024-03-04T10:10:07.878109+0000","last_active":"2024-03-04T10:10:09.412648+0000","last_peered":"2024-03-04T10:10:09.412648+0000","last_clean":"2024-03-04T10:10:09.412648+0000","last_became_active":"2024-03-04T10:10:07.877805+0000","last_became_peered":"2024-03-04T10:10:07.877805+0000","last_unstale":"2024-03-04T10:10:09.412648+0000","last_undegraded":"2024-03-04T10:10:09.412648+0000","last_fullsized":"2024-03-04T10:10:09.412648+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:59:37.792367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412597+0000","last_change":"2024-03-04T10:10:06.865016+0000","last_active":"2024-03-04T10:10:09.412597+0000","last_peered":"2024-03-04T10:10:09.412597+0000","last_clean":"2024-03-04T10:10:09.412597+0000","last_became_active":"2024-03-04T10:10:06.864692+0000","last_became_peered":"2024-03-04T10:10:06.864692+0000","last_unstale":"2024-03-04T10:10:09.412597+0000","last_undegraded":"2024-03-04T10:10:09.412597+0000","last_fullsized":"2024-03-04T10:10:09.412597+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:04:16.961170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864517+0000","last_change":"2024-03-04T10:10:07.875977+0000","last_active":"2024-03-04T10:10:08.864517+0000","last_peered":"2024-03-04T10:10:08.864517+0000","last_clean":"2024-03-04T10:10:08.864517+0000","last_became_active":"2024-03-04T10:10:07.875702+0000","last_became_peered":"2024-03-04T10:10:07.875702+0000","last_unstale":"2024-03-04T10:10:08.864517+0000","last_undegraded":"2024-03-04T10:10:08.864517+0000","last_fullsized":"2024-03-04T10:10:08.864517+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:23:56.221653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412210+0000","last_change":"2024-03-04T10:10:06.859809+0000","last_active":"2024-03-04T10:10:09.412210+0000","last_peered":"2024-03-04T10:10:09.412210+0000","last_clean":"2024-03-04T10:10:09.412210+0000","last_became_active":"2024-03-04T10:10:06.859499+0000","last_became_peered":"2024-03-04T10:10:06.859499+0000","last_unstale":"2024-03-04T10:10:09.412210+0000","last_undegraded":"2024-03-04T10:10:09.412210+0000","last_fullsized":"2024-03-04T10:10:09.412210+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:46:43.575014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412591+0000","last_change":"2024-03-04T10:10:07.865560+0000","last_active":"2024-03-04T10:10:09.412591+0000","last_peered":"2024-03-04T10:10:09.412591+0000","last_clean":"2024-03-04T10:10:09.412591+0000","last_became_active":"2024-03-04T10:10:07.865231+0000","last_became_peered":"2024-03-04T10:10:07.865231+0000","last_unstale":"2024-03-04T10:10:09.412591+0000","last_undegraded":"2024-03-04T10:10:09.412591+0000","last_fullsized":"2024-03-04T10:10:09.412591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:32.626643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"92'2","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863147+0000","last_change":"2024-03-04T10:10:06.859472+0000","last_active":"2024-03-04T10:10:08.863147+0000","last_peered":"2024-03-04T10:10:08.863147+0000","last_clean":"2024-03-04T10:10:08.863147+0000","last_became_active":"2024-03-04T10:10:06.859206+0000","last_became_peered":"2024-03-04T10:10:06.859206+0000","last_unstale":"2024-03-04T10:10:08.863147+0000","last_undegraded":"2024-03-04T10:10:08.863147+0000","last_fullsized":"2024-03-04T10:10:08.863147+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:40.499353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864955+0000","last_change":"2024-03-04T10:10:07.874403+0000","last_active":"2024-03-04T10:10:08.864955+0000","last_peered":"2024-03-04T10:10:08.864955+0000","last_clean":"2024-03-04T10:10:08.864955+0000","last_became_active":"2024-03-04T10:10:07.873650+0000","last_became_peered":"2024-03-04T10:10:07.873650+0000","last_unstale":"2024-03-04T10:10:08.864955+0000","last_undegraded":"2024-03-04T10:10:08.864955+0000","last_fullsized":"2024-03-04T10:10:08.864955+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:58:24.017092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"93'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864050+0000","last_change":"2024-03-04T10:10:06.860264+0000","last_active":"2024-03-04T10:10:08.864050+0000","last_peered":"2024-03-04T10:10:08.864050+0000","last_clean":"2024-03-04T10:10:08.864050+0000","last_became_active":"2024-03-04T10:10:06.859974+0000","last_became_peered":"2024-03-04T10:10:06.859974+0000","last_unstale":"2024-03-04T10:10:08.864050+0000","last_undegraded":"2024-03-04T10:10:08.864050+0000","last_fullsized":"2024-03-04T10:10:08.864050+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:11:22.139606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864789+0000","last_change":"2024-03-04T10:10:07.873910+0000","last_active":"2024-03-04T10:10:08.864789+0000","last_peered":"2024-03-04T10:10:08.864789+0000","last_clean":"2024-03-04T10:10:08.864789+0000","last_became_active":"2024-03-04T10:10:07.871403+0000","last_became_peered":"2024-03-04T10:10:07.871403+0000","last_unstale":"2024-03-04T10:10:08.864789+0000","last_undegraded":"2024-03-04T10:10:08.864789+0000","last_fullsized":"2024-03-04T10:10:08.864789+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:51:25.763833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864188+0000","last_change":"2024-03-04T10:10:06.864593+0000","last_active":"2024-03-04T10:10:08.864188+0000","last_peered":"2024-03-04T10:10:08.864188+0000","last_clean":"2024-03-04T10:10:08.864188+0000","last_became_active":"2024-03-04T10:10:06.864298+0000","last_became_peered":"2024-03-04T10:10:06.864298+0000","last_unstale":"2024-03-04T10:10:08.864188+0000","last_undegraded":"2024-03-04T10:10:08.864188+0000","last_fullsized":"2024-03-04T10:10:08.864188+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T11:47:59.881925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412751+0000","last_change":"2024-03-04T10:10:07.867064+0000","last_active":"2024-03-04T10:10:09.412751+0000","last_peered":"2024-03-04T10:10:09.412751+0000","last_clean":"2024-03-04T10:10:09.412751+0000","last_became_active":"2024-03-04T10:10:07.866800+0000","last_became_peered":"2024-03-04T10:10:07.866800+0000","last_unstale":"2024-03-04T10:10:09.412751+0000","last_undegraded":"2024-03-04T10:10:09.412751+0000","last_fullsized":"2024-03-04T10:10:09.412751+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:28:53.101723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412829+0000","last_change":"2024-03-04T10:10:06.862581+0000","last_active":"2024-03-04T10:10:09.412829+0000","last_peered":"2024-03-04T10:10:09.412829+0000","last_clean":"2024-03-04T10:10:09.412829+0000","last_became_active":"2024-03-04T10:10:06.862283+0000","last_became_peered":"2024-03-04T10:10:06.862283+0000","last_unstale":"2024-03-04T10:10:09.412829+0000","last_undegraded":"2024-03-04T10:10:09.412829+0000","last_fullsized":"2024-03-04T10:10:09.412829+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:39.261539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412115+0000","last_change":"2024-03-04T10:10:07.872439+0000","last_active":"2024-03-04T10:10:09.412115+0000","last_peered":"2024-03-04T10:10:09.412115+0000","last_clean":"2024-03-04T10:10:09.412115+0000","last_became_active":"2024-03-04T10:10:07.872123+0000","last_became_peered":"2024-03-04T10:10:07.872123+0000","last_unstale":"2024-03-04T10:10:09.412115+0000","last_undegraded":"2024-03-04T10:10:09.412115+0000","last_fullsized":"2024-03-04T10:10:09.412115+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:06:52.395301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864617+0000","last_change":"2024-03-04T10:10:06.859084+0000","last_active":"2024-03-04T10:10:08.864617+0000","last_peered":"2024-03-04T10:10:08.864617+0000","last_clean":"2024-03-04T10:10:08.864617+0000","last_became_active":"2024-03-04T10:10:06.858820+0000","last_became_peered":"2024-03-04T10:10:06.858820+0000","last_unstale":"2024-03-04T10:10:08.864617+0000","last_undegraded":"2024-03-04T10:10:08.864617+0000","last_fullsized":"2024-03-04T10:10:08.864617+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:25.744355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412196+0000","last_change":"2024-03-04T10:10:07.875231+0000","last_active":"2024-03-04T10:10:09.412196+0000","last_peered":"2024-03-04T10:10:09.412196+0000","last_clean":"2024-03-04T10:10:09.412196+0000","last_became_active":"2024-03-04T10:10:07.874787+0000","last_became_peered":"2024-03-04T10:10:07.874787+0000","last_unstale":"2024-03-04T10:10:09.412196+0000","last_undegraded":"2024-03-04T10:10:09.412196+0000","last_fullsized":"2024-03-04T10:10:09.412196+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:41.503922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412679+0000","last_change":"2024-03-04T10:10:06.862727+0000","last_active":"2024-03-04T10:10:09.412679+0000","last_peered":"2024-03-04T10:10:09.412679+0000","last_clean":"2024-03-04T10:10:09.412679+0000","last_became_active":"2024-03-04T10:10:06.861472+0000","last_became_peered":"2024-03-04T10:10:06.861472+0000","last_unstale":"2024-03-04T10:10:09.412679+0000","last_undegraded":"2024-03-04T10:10:09.412679+0000","last_fullsized":"2024-03-04T10:10:09.412679+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:33:43.358614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412302+0000","last_change":"2024-03-04T10:10:07.874180+0000","last_active":"2024-03-04T10:10:09.412302+0000","last_peered":"2024-03-04T10:10:09.412302+0000","last_clean":"2024-03-04T10:10:09.412302+0000","last_became_active":"2024-03-04T10:10:07.873254+0000","last_became_peered":"2024-03-04T10:10:07.873254+0000","last_unstale":"2024-03-04T10:10:09.412302+0000","last_undegraded":"2024-03-04T10:10:09.412302+0000","last_fullsized":"2024-03-04T10:10:09.412302+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:54:23.671502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.412661+0000","last_change":"2024-03-04T10:10:06.864463+0000","last_active":"2024-03-04T10:10:09.412661+0000","last_peered":"2024-03-04T10:10:09.412661+0000","last_clean":"2024-03-04T10:10:09.412661+0000","last_became_active":"2024-03-04T10:10:06.861931+0000","last_became_peered":"2024-03-04T10:10:06.861931+0000","last_unstale":"2024-03-04T10:10:09.412661+0000","last_undegraded":"2024-03-04T10:10:09.412661+0000","last_fullsized":"2024-03-04T10:10:09.412661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:07:10.405012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.864707+0000","last_change":"2024-03-04T10:10:07.878411+0000","last_active":"2024-03-04T10:10:08.864707+0000","last_peered":"2024-03-04T10:10:08.864707+0000","last_clean":"2024-03-04T10:10:08.864707+0000","last_became_active":"2024-03-04T10:10:07.877967+0000","last_became_peered":"2024-03-04T10:10:07.877967+0000","last_unstale":"2024-03-04T10:10:08.864707+0000","last_undegraded":"2024-03-04T10:10:08.864707+0000","last_fullsized":"2024-03-04T10:10:08.864707+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:48:22.610912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862658+0000","last_change":"2024-03-04T10:10:06.865079+0000","last_active":"2024-03-04T10:10:08.862658+0000","last_peered":"2024-03-04T10:10:08.862658+0000","last_clean":"2024-03-04T10:10:08.862658+0000","last_became_active":"2024-03-04T10:10:06.864808+0000","last_became_peered":"2024-03-04T10:10:06.864808+0000","last_unstale":"2024-03-04T10:10:08.862658+0000","last_undegraded":"2024-03-04T10:10:08.862658+0000","last_fullsized":"2024-03-04T10:10:08.862658+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:43:31.215655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.863198+0000","last_change":"2024-03-04T10:10:07.862488+0000","last_active":"2024-03-04T10:10:08.863198+0000","last_peered":"2024-03-04T10:10:08.863198+0000","last_clean":"2024-03-04T10:10:08.863198+0000","last_became_active":"2024-03-04T10:10:07.862234+0000","last_became_peered":"2024-03-04T10:10:07.862234+0000","last_unstale":"2024-03-04T10:10:08.863198+0000","last_undegraded":"2024-03-04T10:10:08.863198+0000","last_fullsized":"2024-03-04T10:10:08.863198+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:06.837242+0000","last_clean_scrub_stamp":"2024-03-04T10:10:06.837242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T10:38:26.748924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:09.235127+0000","last_change":"2024-03-04T10:10:06.861954+0000","last_active":"2024-03-04T10:10:09.235127+0000","last_peered":"2024-03-04T10:10:09.235127+0000","last_clean":"2024-03-04T10:10:09.235127+0000","last_became_active":"2024-03-04T10:10:06.859757+0000","last_became_peered":"2024-03-04T10:10:06.859757+0000","last_unstale":"2024-03-04T10:10:09.235127+0000","last_undegraded":"2024-03-04T10:10:09.235127+0000","last_fullsized":"2024-03-04T10:10:09.235127+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:10:05.832673+0000","last_clean_scrub_stamp":"2024-03-04T10:10:05.832673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:52:15.494976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":319,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T10:10:08.862881+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:10:08.862881+0000","last_peered":"2024-03-04T10:10:08.862881+0000","last_clean":"2024-03-04T10:10:08.862881+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:10:08.862881+0000","last_undegraded":"2024-03-04T10:10:08.862881+0000","last_fullsized":"2024-03-04T10:10:08.862881+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574932,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1748992,"data_stored":1125146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32324,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339516,"statfs":{"total":96636764160,"available":96603664384,"internally_reserved":0,"allocated":1167360,"data_stored":536938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574933,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1146880,"data_stored":534722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574933,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32312,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339528,"statfs":{"total":96636764160,"available":96603676672,"internally_reserved":0,"allocated":1155072,"data_stored":534882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574933,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1146880,"data_stored":534722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574933,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1146880,"data_stored":534722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1150976,"data_stored":534792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574933,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32908,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338932,"statfs":{"total":96636764160,"available":96603066368,"internally_reserved":0,"allocated":1765376,"data_stored":1127408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:10:15.604 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T10:10:15.604 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-04T10:10:15.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 Processor -- start 2024-03-04T10:10:15.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 -- start start 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fad6c0b5700 0x7fad6c0d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b9410 0x7fad6c06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6c0d2400 con 0x7fad6c0d34a0 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6c0d26e0 con 0x7fad6c0b5700 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad71e54640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0d29c0 con 0x7fad6c0b9410 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.745+0000 7fad6b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fad6c0b5700 0x7fad6c0d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fad6c0b5700 0x7fad6c0d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54820/0 (socket says 172.21.15.99:54820) 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6b7fe640 1 -- 172.21.15.99:0/3777152694 learned_addr learned my addr 172.21.15.99:0/3777152694 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6bfff640 1 --2- 172.21.15.99:0/3777152694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b9410 0x7fad6c06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6b7fe640 1 -- 172.21.15.99:0/3777152694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b9410 msgr2=0x7fad6c06c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:15.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6b7fe640 1 --2- 172.21.15.99:0/3777152694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b9410 0x7fad6c06c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6b7fe640 1 -- 172.21.15.99:0/3777152694 >> v1:172.21.15.162:6789/0 conn(0x7fad6c0d34a0 legacy=0x7fad6c06c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.746+0000 7fad6b7fe640 1 -- 172.21.15.99:0/3777152694 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c0d2ca0 con 0x7fad6c0b5700 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad6b7fe640 1 --2- 172.21.15.99:0/3777152694 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0b5700 0x7fad6c0d1b00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009870 tx=0x7fad5c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacd96e0d3a5dee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad6a7fc640 1 -- 172.21.15.99:0/3777152694 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c0317c0 con 0x7fad6c0b5700 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad6a7fc640 1 -- 172.21.15.99:0/3777152694 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c037070 con 0x7fad6c0b5700 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad6a7fc640 1 -- 172.21.15.99:0/3777152694 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c031b80 con 0x7fad6c0b5700 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 -- 172.21.15.99:0/3777152694 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0b5700 msgr2=0x7fad6c0d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 --2- 172.21.15.99:0/3777152694 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0b5700 0x7fad6c0d1b00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009870 tx=0x7fad5c02b860 comp rx=0 tx=0).stop 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 -- 172.21.15.99:0/3777152694 shutdown_connections 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 --2- 172.21.15.99:0/3777152694 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b9410 0x7fad6c06c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 --2- 172.21.15.99:0/3777152694 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0b5700 0x7fad6c0d1b00 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 -- 172.21.15.99:0/3777152694 >> 172.21.15.99:0/3777152694 conn(0x7fad6c0d0490 msgr2=0x7fad6c0d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 -- 172.21.15.99:0/3777152694 shutdown_connections 2024-03-04T10:10:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.747+0000 7fad71e54640 1 -- 172.21.15.99:0/3777152694 wait complete. 2024-03-04T10:10:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.748+0000 7fad71e54640 1 Processor -- start 2024-03-04T10:10:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 -- start start 2024-03-04T10:10:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b5700 0x7fad6c157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad6c0b9410 0x7fad6c15aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 0x7fad6c14ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c06ceb0 con 0x7fad6c0d34a0 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c06cd30 con 0x7fad6c0b9410 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad71e54640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0b5ae0 con 0x7fad6c0b5700 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad6bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 0x7fad6c14ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad6bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 0x7fad6c14ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54830/0 (socket says 172.21.15.99:54830) 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.749+0000 7fad6bfff640 1 -- 172.21.15.99:0/2234393377 learned_addr learned my addr 172.21.15.99:0/2234393377 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6b7fe640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b5700 0x7fad6c157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6affd640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad6c0b9410 0x7fad6c15aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6bfff640 1 -- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b5700 msgr2=0x7fad6c157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6bfff640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b5700 0x7fad6c157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6bfff640 1 -- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad6c0b9410 msgr2=0x7fad6c15aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6bfff640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad6c0b9410 0x7fad6c15aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6bfff640 1 -- 172.21.15.99:0/2234393377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c0cc250 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6b7fe640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b5700 0x7fad6c157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6affd640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad6c0b9410 0x7fad6c15aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad6bfff640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 0x7fad6c14ddc0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fad64007e40 tx=0x7fad6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64014620 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad64019070 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6c0cc530 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64014a20 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.750+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6c1571d0 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.751+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6c074f60 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.752+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fad64014bc0 con 0x7fad6c0d34a0 2024-03-04T10:10:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.752+0000 7fad53fff640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad5806b4d0 0x7fad5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.752+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fad6408abd0 con 0x7fad6c0d34a0 2024-03-04T10:10:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.752+0000 7fad6b7fe640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad5806b4d0 0x7fad5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.753+0000 7fad6b7fe640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad5806b4d0 0x7fad5806d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fad5c036000 tx=0x7fad5c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.756+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fad64059950 con 0x7fad6c0d34a0 2024-03-04T10:10:15.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.996+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fad6c0cf650 con 0x7fad6c0d34a0 2024-03-04T10:10:16.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:15.999+0000 7fad53fff640 1 -- 172.21.15.99:0/2234393377 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7fad64059250 con 0x7fad6c0d34a0 2024-03-04T10:10:16.001 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2024-03-04T10:10:16.001 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBBnOVlOKWpLxAAGt30u6qXdtc02n6/pZk6lQ== 2024-03-04T10:10:16.001 INFO:teuthology.orchestra.run.smithi099.stdout: caps mds = "allow" 2024-03-04T10:10:16.001 INFO:teuthology.orchestra.run.smithi099.stdout: caps mon = "allow r" 2024-03-04T10:10:16.001 INFO:teuthology.orchestra.run.smithi099.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-04T10:10:16.002 INFO:teuthology.orchestra.run.smithi099.stderr:updated caps for client.0 2024-03-04T10:10:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad5806b4d0 msgr2=0x7fad5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad5806b4d0 0x7fad5806d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fad5c036000 tx=0x7fad5c008030 comp rx=0 tx=0).stop 2024-03-04T10:10:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 msgr2=0x7fad6c14ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 0x7fad6c14ddc0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fad64007e40 tx=0x7fad6400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 shutdown_connections 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fad5806b4d0 0x7fad5806d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fad6c0d34a0 0x7fad6c14ddc0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad6c0b9410 0x7fad6c15aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 --2- 172.21.15.99:0/2234393377 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fad6c0b5700 0x7fad6c157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 >> 172.21.15.99:0/2234393377 conn(0x7fad6c0d0490 msgr2=0x7fad6c0b6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 shutdown_connections 2024-03-04T10:10:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.001+0000 7fad71e54640 1 -- 172.21.15.99:0/2234393377 wait complete. 2024-03-04T10:10:16.015 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:16.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.111+0000 7fe9b6759640 1 Processor -- start 2024-03-04T10:10:16.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.111+0000 7fe9b6759640 1 -- start start 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.112+0000 7fe9b6759640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 0x7fe9b0161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.112+0000 7fe9b6759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b006de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.112+0000 7fe9b6759640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b006b500 con 0x7fe9b0158410 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.112+0000 7fe9b6759640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b006b7e0 con 0x7fe9b00b39c0 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.112+0000 7fe9b6759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006bac0 con 0x7fe9b0161ca0 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9af7fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 0x7fe9b0161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9b4cf6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9affff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe9b0158410 0x7fe9b006acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:45092/0 (socket says 172.21.15.99:45092) 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9af7fe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 0x7fe9b0161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:35772/0 (socket says 172.21.15.99:35772) 2024-03-04T10:10:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9affff640 1 -- 172.21.15.99:0/498656831 learned_addr learned my addr 172.21.15.99:0/498656831 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9b4cf6640 1 -- 172.21.15.99:0/498656831 >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 msgr2=0x7fe9b0161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9b4cf6640 1 --2- 172.21.15.99:0/498656831 >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 0x7fe9b0161760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9b4cf6640 1 -- 172.21.15.99:0/498656831 >> v1:172.21.15.162:6789/0 conn(0x7fe9b0158410 legacy=0x7fe9b006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9aeffd640 1 -- 172.21.15.99:0/498656831 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122806768 0 0) 0x7fe9b006b500 con 0x7fe9b0158410 2024-03-04T10:10:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9b4cf6640 1 -- 172.21.15.99:0/498656831 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b006bda0 con 0x7fe9b0161ca0 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.113+0000 7fe9af7fe640 1 --2- 172.21.15.99:0/498656831 >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 0x7fe9b0161760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.114+0000 7fe9b4cf6640 1 --2- 172.21.15.99:0/498656831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b006de50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009d90 tx=0x7fe9a802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9392405c7f95ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.114+0000 7fe9aeffd640 1 -- 172.21.15.99:0/498656831 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8039070 con 0x7fe9b0161ca0 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.114+0000 7fe9aeffd640 1 -- 172.21.15.99:0/498656831 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a802cce0 con 0x7fe9b0161ca0 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.114+0000 7fe9aeffd640 1 -- 172.21.15.99:0/498656831 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8034660 con 0x7fe9b0161ca0 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.114+0000 7fe9b6759640 1 -- 172.21.15.99:0/498656831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 msgr2=0x7fe9b006de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.114+0000 7fe9b6759640 1 --2- 172.21.15.99:0/498656831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b006de50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009d90 tx=0x7fe9a802bf30 comp rx=0 tx=0).stop 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.115+0000 7fe9b6759640 1 -- 172.21.15.99:0/498656831 shutdown_connections 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.115+0000 7fe9b6759640 1 --2- 172.21.15.99:0/498656831 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b006de50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.115+0000 7fe9b6759640 1 --2- 172.21.15.99:0/498656831 >> v2:172.21.15.162:3300/0 conn(0x7fe9b00b39c0 0x7fe9b0161760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.115+0000 7fe9b6759640 1 -- 172.21.15.99:0/498656831 >> 172.21.15.99:0/498656831 conn(0x7fe9b00cfef0 msgr2=0x7fe9b00d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.115+0000 7fe9b6759640 1 -- 172.21.15.99:0/498656831 shutdown_connections 2024-03-04T10:10:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.115+0000 7fe9b6759640 1 -- 172.21.15.99:0/498656831 wait complete. 2024-03-04T10:10:16.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.116+0000 7fe9b6759640 1 Processor -- start 2024-03-04T10:10:16.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.116+0000 7fe9b6759640 1 -- start start 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b00b39c0 0x7fe9b01f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 0x7fe9b014f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b00ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006c5a0 con 0x7fe9b0158410 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00b3e50 con 0x7fe9b00b39c0 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006e390 con 0x7fe9b0161ca0 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 0x7fe9b014f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 0x7fe9b014f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54838/0 (socket says 172.21.15.99:54838) 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 -- 172.21.15.99:0/3595924063 learned_addr learned my addr 172.21.15.99:0/3595924063 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b4cf6640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b00ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9affff640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b00b39c0 0x7fe9b01f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 -- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 msgr2=0x7fe9b00ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b00ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 -- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b00b39c0 msgr2=0x7fe9b01f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b00b39c0 0x7fe9b01f39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 -- 172.21.15.99:0/3595924063 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b01f4020 con 0x7fe9b0158410 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9af7fe640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 0x7fe9b014f040 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400ebe0 tx=0x7fe9a400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4017070 con 0x7fe9b0158410 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a400cc30 con 0x7fe9b0158410 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b01f4300 con 0x7fe9b0158410 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.117+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4013470 con 0x7fe9b0158410 2024-03-04T10:10:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.118+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b01f48c0 con 0x7fe9b0158410 2024-03-04T10:10:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.119+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0074f60 con 0x7fe9b0158410 2024-03-04T10:10:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.119+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe9a401a070 con 0x7fe9b0158410 2024-03-04T10:10:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.119+0000 7fe9acff9640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe99c06b4d0 0x7fe99c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.119+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe9a408a690 con 0x7fe9b0158410 2024-03-04T10:10:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.119+0000 7fe9affff640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe99c06b4d0 0x7fe99c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.120+0000 7fe9affff640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe99c06b4d0 0x7fe99c06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0000f60 tx=0x7fe9a0007b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.124+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe9a4059380 con 0x7fe9b0158410 2024-03-04T10:10:16.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.330+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9b0066e10 con 0x7fe9b0158410 2024-03-04T10:10:16.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.330+0000 7fe9acff9640 1 -- 172.21.15.99:0/3595924063 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7fe9a4058c80 con 0x7fe9b0158410 2024-03-04T10:10:16.333 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:16.333 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":78,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:09.188560+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7644},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7644":{"gid":7644,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3671020073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3671020073},{"type":"v1","addr":"172.21.15.99:6835","nonce":3671020073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:16.333 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 78 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe99c06b4d0 msgr2=0x7fe99c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe99c06b4d0 0x7fe99c06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0000f60 tx=0x7fe9a0007b50 comp rx=0 tx=0).stop 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 msgr2=0x7fe9b014f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 0x7fe9b014f040 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400ebe0 tx=0x7fe9a400c520 comp rx=0 tx=0).stop 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 shutdown_connections 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe99c06b4d0 0x7fe99c06d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe9b0161ca0 0x7fe9b00ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe9b0158410 0x7fe9b014f040 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 --2- 172.21.15.99:0/3595924063 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe9b00b39c0 0x7fe9b01f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 >> 172.21.15.99:0/3595924063 conn(0x7fe9b00cfef0 msgr2=0x7fe9b00aed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 shutdown_connections 2024-03-04T10:10:16.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.332+0000 7fe9b6759640 1 -- 172.21.15.99:0/3595924063 wait complete. 2024-03-04T10:10:16.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:10:07.195925+0000', 'modified': '2024-03-04T10:10:09.188560+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7644}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7644': {'gid': 7644, 'name': 'c', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.99:6835/3671020073', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 3671020073}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 3671020073}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:10:16.752 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:10:16.753 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:10:16.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.852+0000 7fe831569640 1 Processor -- start 2024-03-04T10:10:16.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe831569640 1 -- start start 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe831569640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe82c15de00 0x7fe82c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe831569640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c15e1e0 con 0x7fe82c0b9380 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe831569640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe82c0d1c10 con 0x7fe82c0b5470 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe831569640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe82c0d2820 con 0x7fe82c15de00 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe82a7fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe82c0b9380 0x7fe82c0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57182/0 (socket says 172.21.15.99:57182) 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.853+0000 7fe82a7fc640 1 -- 172.21.15.99:0/3354137609 learned_addr learned my addr 172.21.15.99:0/3354137609 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 --2- 172.21.15.99:0/3354137609 >> v2:172.21.15.162:3300/0 conn(0x7fe82c15de00 0x7fe82c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848652120 0 0) 0x7fe82c15e1e0 con 0x7fe82c0b9380 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814003520 con 0x7fe82c0b9380 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043296939 0 0) 0x7fe814003520 con 0x7fe82c0b9380 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 >> v2:172.21.15.162:3300/0 conn(0x7fe82c15de00 msgr2=0x7fe82c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 --2- 172.21.15.99:0/3354137609 >> v2:172.21.15.162:3300/0 conn(0x7fe82c15de00 0x7fe82c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 >> v1:172.21.15.162:6789/0 conn(0x7fe82c0b5470 legacy=0x7fe82c0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c0d1ef0 con 0x7fe82c0b9380 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fe820001670 con 0x7fe82c0b9380 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 >> v1:172.21.15.99:6789/0 conn(0x7fe82c0b9380 legacy=0x7fe82c0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe8140039b0 0x7fe814003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe814004800 0x7fe814004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8140051c0 0x7fe81400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0d2820 con 0x7fe8140039b0 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81400c060 con 0x7fe8140051c0 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81400c260 con 0x7fe814004800 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe8140039b0 0x7fe814003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8140051c0 0x7fe81400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe814004800 0x7fe814004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 -- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe814004800 msgr2=0x7fe814004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe814004800 0x7fe814004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 -- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8140051c0 msgr2=0x7fe81400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe8140051c0 0x7fe81400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82b7fe640 1 -- 172.21.15.99:0/3354137609 reap_dead start 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82affd640 1 -- 172.21.15.99:0/3354137609 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814003520 con 0x7fe8140039b0 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.854+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe814004800 0x7fe814004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.855+0000 7fe82affd640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe8140039b0 0x7fe814003dd0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009fc0 tx=0x7fe814005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d23ea422d42d2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.855+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c039070 con 0x7fe8140039b0 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.855+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c02cc20 con 0x7fe8140039b0 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.855+0000 7fe829ffb640 1 -- 172.21.15.99:0/3354137609 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c009c10 con 0x7fe8140039b0 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 -- 172.21.15.99:0/3354137609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe8140039b0 msgr2=0x7fe814003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe8140039b0 0x7fe814003dd0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009fc0 tx=0x7fe814005830 comp rx=0 tx=0).stop 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 -- 172.21.15.99:0/3354137609 shutdown_connections 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 --2- 172.21.15.99:0/3354137609 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe8140039b0 0x7fe814003dd0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 -- 172.21.15.99:0/3354137609 >> 172.21.15.99:0/3354137609 conn(0x7fe82c0d0470 msgr2=0x7fe82c0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 -- 172.21.15.99:0/3354137609 shutdown_connections 2024-03-04T10:10:16.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.856+0000 7fe831569640 1 -- 172.21.15.99:0/3354137609 wait complete. 2024-03-04T10:10:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.857+0000 7fe831569640 1 Processor -- start 2024-03-04T10:10:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.857+0000 7fe831569640 1 -- start start 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe831569640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe8140039b0 0x7fe82c1f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe831569640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe82c14a1b0 0x7fe82c1f7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe831569640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 0x7fe82c1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe831569640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0b97e0 con 0x7fe82c0b8520 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe831569640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0d24b0 con 0x7fe82c14a1b0 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe831569640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c0b2e60 con 0x7fe8140039b0 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 0x7fe82c1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 0x7fe82c1579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54856/0 (socket says 172.21.15.99:54856) 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 -- 172.21.15.99:0/2531604393 learned_addr learned my addr 172.21.15.99:0/2531604393 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe82c14a1b0 0x7fe82c1f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82affd640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe8140039b0 0x7fe82c1f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 -- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe8140039b0 msgr2=0x7fe82c1f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe8140039b0 0x7fe82c1f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 -- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe82c14a1b0 msgr2=0x7fe82c1f7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe82c14a1b0 0x7fe82c1f7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 -- 172.21.15.99:0/2531604393 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82c1f78d0 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.858+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 0x7fe82c1579f0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ce210 tx=0x7fe82400f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.859+0000 7fe8137fe640 1 -- 172.21.15.99:0/2531604393 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82401a070 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.859+0000 7fe8137fe640 1 -- 172.21.15.99:0/2531604393 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe824005320 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.859+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c0b71e0 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.859+0000 7fe8137fe640 1 -- 172.21.15.99:0/2531604393 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe824007250 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.859+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c0b7760 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.860+0000 7fe8137fe640 1 -- 172.21.15.99:0/2531604393 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe82401d070 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.860+0000 7fe8137fe640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe81806b4d0 0x7fe81806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.861+0000 7fe8137fe640 1 -- 172.21.15.99:0/2531604393 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe8240893a0 con 0x7fe82c0b8520 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.861+0000 7fe82affd640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe81806b4d0 0x7fe81806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.861+0000 7fe82affd640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe81806b4d0 0x7fe81806d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe81c036000 tx=0x7fe81c02e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.861+0000 7fe834222740 1 Processor -- start 2024-03-04T10:10:16.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 -- start start 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d5e5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d613280 0x557d8d615c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d5e6380 0x557d8d616170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d8d5507c0 con 0x557d8d612ea0 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d8d4f9b50 con 0x557d8d5e6380 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x557d8d54c860 con 0x557d8d613280 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe82affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d5e5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe82affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d5e5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54862/0 (socket says 172.21.15.99:54862) 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.862+0000 7fe82affd640 1 -- 172.21.15.99:0/4081884892 learned_addr learned my addr 172.21.15.99:0/4081884892 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe82affd640 1 -- 172.21.15.99:0/4081884892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d613280 msgr2=0x557d8d615c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe82affd640 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d613280 0x557d8d615c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe82affd640 1 -- 172.21.15.99:0/4081884892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d5e6380 msgr2=0x557d8d616170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe82affd640 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d5e6380 0x557d8d616170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe82affd640 1 -- 172.21.15.99:0/4081884892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d8d50d9a0 con 0x557d8d612ea0 2024-03-04T10:10:16.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe82affd640 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d5e5e40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fe81c02cd00 tx=0x7fe81c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35b7bd528d39654 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe810ff9640 1 -- 172.21.15.99:0/4081884892 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c043040 con 0x557d8d612ea0 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe810ff9640 1 -- 172.21.15.99:0/4081884892 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c03a040 con 0x557d8d612ea0 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe810ff9640 1 -- 172.21.15.99:0/4081884892 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c039050 con 0x557d8d612ea0 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe834222740 1 -- 172.21.15.99:0/4081884892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 msgr2=0x557d8d5e5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.863+0000 7fe834222740 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d5e5e40 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fe81c02cd00 tx=0x7fe81c02e070 comp rx=0 tx=0).stop 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 -- 172.21.15.99:0/4081884892 shutdown_connections 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d5e6380 0x557d8d616170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d613280 0x557d8d615c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 --2- 172.21.15.99:0/4081884892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d5e5e40 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 -- 172.21.15.99:0/4081884892 >> 172.21.15.99:0/4081884892 conn(0x557d8d5e80b0 msgr2=0x557d8d630c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 -- 172.21.15.99:0/4081884892 shutdown_connections 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 -- 172.21.15.99:0/4081884892 wait complete. 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 -- start start 2024-03-04T10:10:16.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.864+0000 7fe834222740 1 Processor -- start 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d612ea0 0x557d8d601710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d618470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 0x557d8d62b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d8d5ea750 con 0x557d8d6189b0 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d8d50cf90 con 0x557d8d613280 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x557d8d5ecf90 con 0x557d8d612ea0 2024-03-04T10:10:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe834222740 10 client.0 _ll_register_callbacks cb 0x557d8d52ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 0x557d8d62b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 0x557d8d62b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54872/0 (socket says 172.21.15.99:54872) 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 -- 172.21.15.99:0/1917731358 learned_addr learned my addr 172.21.15.99:0/1917731358 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d618470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82affd640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d612ea0 0x557d8d601710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 -- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d612ea0 msgr2=0x557d8d601710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d612ea0 0x557d8d601710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 -- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 msgr2=0x557d8d618470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d618470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 -- 172.21.15.99:0/1917731358 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d8d2652d0 con 0x557d8d6189b0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 0x557d8d62b5f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fe8240876e0 tx=0x7fe824057b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e7e8b5012048a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.865+0000 7fe809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe809ffb640 1 -- 172.21.15.99:0/1917731358 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8240073f0 con 0x557d8d6189b0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe809ffb640 1 -- 172.21.15.99:0/1917731358 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe824007590 con 0x557d8d6189b0 2024-03-04T10:10:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81c03a040 con 0x557d8d6189b0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe809ffb640 1 -- 172.21.15.99:0/1917731358 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe824087b60 con 0x557d8d6189b0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 10 client.7818 fetch_fsmap learned FSMap version 78 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x557d8d32f6b0 con 0x557d8d6189b0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe809ffb640 1 -- 172.21.15.99:0/1917731358 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c03a040 con 0x557d8d6189b0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe809ffb640 1 -- 172.21.15.99:0/1917731358 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe824056e90 con 0x557d8d6189b0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 10 client.7818 fetch_fsmap finished waiting for FSMap version 78 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 10 client.7818 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 20 client.7818 populate_metadata read hostname 'smithi099' 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d604480 0x557d8d602710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 4 client.7818 mds_command: new command op to 7644 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 --> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d8d55dd00 con 0x557d8d604480 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe82affd640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d604480 0x557d8d602710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe82affd640 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d604480 0x557d8d602710 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0d61c812e7d02de server_cookie=21facf86aab27688 in_seq=0 out_seq=0 2024-03-04T10:10:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.866+0000 7fe809ffb640 10 client.7818 ms_handle_connect on v2:172.21.15.99:6834/3671020073 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe809ffb640 1 -- 172.21.15.99:0/1917731358 <== mds.0 v2:172.21.15.99:6834/3671020073 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557d8d55dd00 con 0x557d8d604480 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe809ffb640 10 client.7818 handle_command_reply: tid=0 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 client.7818 shutdown 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 msgr2=0x557d8d62b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 0x557d8d62b5f0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fe8240876e0 tx=0x7fe824057b00 comp rx=0 tx=0).stop 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 shutdown_connections 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d604480 0x557d8d602710 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d6189b0 0x557d8d62b5f0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d618470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe809ffb640 0 client.7818 ms_handle_reset on v2:172.21.15.99:6834/3671020073 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 --2- 172.21.15.99:0/1917731358 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d612ea0 0x557d8d601710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 >> 172.21.15.99:0/1917731358 conn(0x557d8d5e6380 msgr2=0x557d8d5e9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 shutdown_connections 2024-03-04T10:10:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 1 -- 172.21.15.99:0/1917731358 wait complete. 2024-03-04T10:10:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.867+0000 7fe834222740 20 client.7818 trim_cache size 0 max 16384 2024-03-04T10:10:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.883+0000 7fe834222740 1 Processor -- start 2024-03-04T10:10:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.883+0000 7fe834222740 1 -- start start 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe834222740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d6038b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d62b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe834222740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c1ffd30 con 0x557d8d612ea0 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c15a9e0 con 0x557d8d613280 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c1ff760 con 0x557d8d6189b0 2024-03-04T10:10:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d6038b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d6038b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54878/0 (socket says 172.21.15.99:54878) 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 -- 172.21.15.99:0/2435027346 learned_addr learned my addr 172.21.15.99:0/2435027346 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d62b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 -- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 msgr2=0x557d8d6541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 -- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 msgr2=0x557d8d62b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d62b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 -- 172.21.15.99:0/2435027346 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d8d5e5b80 con 0x557d8d612ea0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d62b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe82affd640 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d6038b0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe81c037040 tx=0x7fe81c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47640f60d706597 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe80bfff640 1 -- 172.21.15.99:0/2435027346 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c039040 con 0x557d8d612ea0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe80bfff640 1 -- 172.21.15.99:0/2435027346 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c057d00 con 0x557d8d612ea0 2024-03-04T10:10:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.884+0000 7fe80bfff640 1 -- 172.21.15.99:0/2435027346 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c043040 con 0x557d8d612ea0 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 -- 172.21.15.99:0/2435027346 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 msgr2=0x557d8d6038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d6038b0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fe81c037040 tx=0x7fe81c03f000 comp rx=0 tx=0).stop 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 -- 172.21.15.99:0/2435027346 shutdown_connections 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d62b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 --2- 172.21.15.99:0/2435027346 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d6038b0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 -- 172.21.15.99:0/2435027346 >> 172.21.15.99:0/2435027346 conn(0x557d8d5e6380 msgr2=0x557d8d6158a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 -- 172.21.15.99:0/2435027346 shutdown_connections 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 -- 172.21.15.99:0/2435027346 wait complete. 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 -- start start 2024-03-04T10:10:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.885+0000 7fe834222740 1 Processor -- start 2024-03-04T10:10:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d602790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d5ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6003f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d8d604300 con 0x557d8d612ea0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d8d619a80 con 0x557d8d613280 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x557d8d604900 con 0x557d8d6189b0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe834222740 10 client.0 _ll_register_callbacks cb 0x7fe8140036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe82affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d602790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe82affd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d602790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54894/0 (socket says 172.21.15.99:54894) 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe82affd640 1 -- 172.21.15.99:0/656325488 learned_addr learned my addr 172.21.15.99:0/656325488 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe82b7fe640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.886+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d5ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82affd640 1 -- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 msgr2=0x557d8d6003f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82affd640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6003f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82affd640 1 -- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 msgr2=0x557d8d5ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82affd640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d5ffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82affd640 1 -- 172.21.15.99:0/656325488 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d8d5ff990 con 0x557d8d612ea0 2024-03-04T10:10:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d5ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82affd640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d602790 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fe81c003fd0 tx=0x7fe81c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a409bb7a6670b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe809ffb640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe809ffb640 1 -- 172.21.15.99:0/656325488 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c043030 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe809ffb640 1 -- 172.21.15.99:0/656325488 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe81c004690 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe81c057d00 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe809ffb640 1 -- 172.21.15.99:0/656325488 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe81c067050 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 10 client.7824 fetch_fsmap learned FSMap version 78 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x557d8d631920 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe809ffb640 1 -- 172.21.15.99:0/656325488 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c057d00 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe809ffb640 1 -- 172.21.15.99:0/656325488 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 78) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe81c06a020 con 0x557d8d612ea0 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 10 client.7824 fetch_fsmap finished waiting for FSMap version 78 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 10 client.7824 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 20 client.7824 populate_metadata read hostname 'smithi099' 2024-03-04T10:10:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d69c2c0 0x557d8d69c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 4 client.7824 mds_command: new command op to 7644 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 --> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557d8d558050 con 0x557d8d69c2c0 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.887+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d69c2c0 0x557d8d69c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe809ffb640 10 client.7824 ms_handle_connect on v2:172.21.15.99:6834/3671020073 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe82a7fc640 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d69c2c0 0x557d8d69c6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=489335d04f1b630 server_cookie=149251f7d23df071 in_seq=0 out_seq=0 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe809ffb640 1 -- 172.21.15.99:0/656325488 <== mds.0 v2:172.21.15.99:6834/3671020073 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557d8d558050 con 0x557d8d69c2c0 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe809ffb640 10 client.7824 handle_command_reply: tid=0 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 client.7824 shutdown 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 msgr2=0x557d8d602790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d602790 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fe81c003fd0 tx=0x7fe81c004ab0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 shutdown_connections 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:6834/3671020073,v1:172.21.15.99:6835/3671020073] conn(0x557d8d69c2c0 0x557d8d69c6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557d8d6189b0 0x557d8d6003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557d8d613280 0x557d8d5ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe809ffb640 0 client.7824 ms_handle_reset on v2:172.21.15.99:6834/3671020073 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 --2- 172.21.15.99:0/656325488 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557d8d612ea0 0x557d8d602790 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 >> 172.21.15.99:0/656325488 conn(0x557d8d5e6380 msgr2=0x557d8d5e9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 shutdown_connections 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 1 -- 172.21.15.99:0/656325488 wait complete. 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.888+0000 7fe834222740 20 client.7824 trim_cache size 0 max 16384 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 7644, 2024-03-04T10:10:16.898 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 9.0449987660000009, 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 78, 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 93, 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 93, 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 14.068739743 2024-03-04T10:10:16.899 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe81806b4d0 msgr2=0x7fe81806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe81806b4d0 0x7fe81806d990 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe81c036000 tx=0x7fe81c02e000 comp rx=0 tx=0).stop 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 msgr2=0x7fe82c1579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 0x7fe82c1579f0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ce210 tx=0x7fe82400f7d0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 shutdown_connections 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe81806b4d0 0x7fe81806d990 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe82c0b8520 0x7fe82c1579f0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe82c14a1b0 0x7fe82c1f7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 --2- 172.21.15.99:0/2531604393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe8140039b0 0x7fe82c1f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:16.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 >> 172.21.15.99:0/2531604393 conn(0x7fe82c0d0470 msgr2=0x7fe82c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:16.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 shutdown_connections 2024-03-04T10:10:16.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.889+0000 7fe831569640 1 -- 172.21.15.99:0/2531604393 wait complete. 2024-03-04T10:10:16.902 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:10:17.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.999+0000 7f01d52c6640 1 Processor -- start 2024-03-04T10:10:17.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:16.999+0000 7f01d52c6640 1 -- start start 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01d52c6640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f01d006acd0 0x7f01d015c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01d52c6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d006b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01d52c6640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f01d00ce340 0x7f01d00ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01d52c6640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d00d02d0 con 0x7f01d006acd0 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01d52c6640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d00cf700 con 0x7f01d00ce340 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01d52c6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d006ba50 con 0x7f01d006b0b0 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01ceffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f01d006acd0 0x7f01d015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01ceffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f01d006acd0 0x7f01d015c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54896/0 (socket says 172.21.15.99:54896) 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.000+0000 7f01ceffd640 1 -- 172.21.15.99:0/1449958959 learned_addr learned my addr 172.21.15.99:0/1449958959 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01ceffd640 1 -- 172.21.15.99:0/1449958959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 msgr2=0x7f01d006b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01ceffd640 1 --2- 172.21.15.99:0/1449958959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d006b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01ceffd640 1 -- 172.21.15.99:0/1449958959 >> v2:172.21.15.162:3300/0 conn(0x7f01d00ce340 msgr2=0x7f01d00ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01ceffd640 1 --2- 172.21.15.99:0/1449958959 >> v2:172.21.15.162:3300/0 conn(0x7f01d00ce340 0x7f01d00ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01ceffd640 1 -- 172.21.15.99:0/1449958959 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d00cf9a0 con 0x7f01d006acd0 2024-03-04T10:10:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01ceffd640 1 --2- 172.21.15.99:0/1449958959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d006acd0 0x7f01d015c000 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f01d00d1850 tx=0x7f01c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d4e140cccfbfe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.001+0000 7f01cdffb640 1 -- 172.21.15.99:0/1449958959 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c002c9e0 con 0x7f01d006acd0 2024-03-04T10:10:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01cdffb640 1 -- 172.21.15.99:0/1449958959 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c002cb80 con 0x7f01d006acd0 2024-03-04T10:10:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01cdffb640 1 -- 172.21.15.99:0/1449958959 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c00343f0 con 0x7f01d006acd0 2024-03-04T10:10:17.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 -- 172.21.15.99:0/1449958959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d006acd0 msgr2=0x7f01d015c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 --2- 172.21.15.99:0/1449958959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d006acd0 0x7f01d015c000 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f01d00d1850 tx=0x7f01c002be40 comp rx=0 tx=0).stop 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 -- 172.21.15.99:0/1449958959 shutdown_connections 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 --2- 172.21.15.99:0/1449958959 >> v2:172.21.15.162:3300/0 conn(0x7f01d00ce340 0x7f01d00ce7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 --2- 172.21.15.99:0/1449958959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d006b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 --2- 172.21.15.99:0/1449958959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d006acd0 0x7f01d015c000 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 -- 172.21.15.99:0/1449958959 >> 172.21.15.99:0/1449958959 conn(0x7f01d00d0770 msgr2=0x7f01d00d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 -- 172.21.15.99:0/1449958959 shutdown_connections 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.002+0000 7f01d52c6640 1 -- 172.21.15.99:0/1449958959 wait complete. 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.003+0000 7f01d52c6640 1 Processor -- start 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.003+0000 7f01d52c6640 1 -- start start 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01d52c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01d006acd0 0x7f01d00af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01d52c6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01d52c6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 0x7f01d0159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01d52c6640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00d13e0 con 0x7f01d00ce340 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01d52c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d00cff60 con 0x7f01d006acd0 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01d52c6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d0159850 con 0x7f01d006b0b0 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01cf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 0x7f01d0159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01cf7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 0x7f01d0159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54912/0 (socket says 172.21.15.99:54912) 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01cf7fe640 1 -- 172.21.15.99:0/205154959 learned_addr learned my addr 172.21.15.99:0/205154959 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01ceffd640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01d006acd0 0x7f01d00af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.004+0000 7f01ce7fc640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01cf7fe640 1 -- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 msgr2=0x7f01d014ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01cf7fe640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d014ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01cf7fe640 1 -- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01d006acd0 msgr2=0x7f01d00af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01cf7fe640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01d006acd0 0x7f01d00af000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01cf7fe640 1 -- 172.21.15.99:0/205154959 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d015a0c0 con 0x7f01d00ce340 2024-03-04T10:10:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01cf7fe640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 0x7f01d0159b80 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009fd0 tx=0x7f01c800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8011830 con 0x7f01d00ce340 2024-03-04T10:10:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c8016070 con 0x7f01d00ce340 2024-03-04T10:10:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c8011bf0 con 0x7f01d00ce340 2024-03-04T10:10:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d015a360 con 0x7f01d00ce340 2024-03-04T10:10:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.005+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d015a920 con 0x7f01d00ce340 2024-03-04T10:10:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.006+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0074f60 con 0x7f01d00ce340 2024-03-04T10:10:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.006+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f01c801b460 con 0x7f01d00ce340 2024-03-04T10:10:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.007+0000 7f01b77fe640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01bc06b4d0 0x7f01bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.007+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f01c808a060 con 0x7f01d00ce340 2024-03-04T10:10:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.007+0000 7f01ceffd640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01bc06b4d0 0x7f01bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.007+0000 7f01ceffd640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01bc06b4d0 0x7f01bc06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f01c0035000 tx=0x7f01c0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:17.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.011+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f01c8058de0 con 0x7f01d00ce340 2024-03-04T10:10:17.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.216+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f01d015dbf0 con 0x7f01d00ce340 2024-03-04T10:10:17.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.216+0000 7f01b77fe640 1 -- 172.21.15.99:0/205154959 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7f01c80586e0 con 0x7f01d00ce340 2024-03-04T10:10:17.219 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01bc06b4d0 msgr2=0x7f01bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01bc06b4d0 0x7f01bc06d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f01c0035000 tx=0x7f01c0030000 comp rx=0 tx=0).stop 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 msgr2=0x7f01d0159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 0x7f01d0159b80 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009fd0 tx=0x7f01c800b8e0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 shutdown_connections 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01bc06b4d0 0x7f01bc06d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01d00ce340 0x7f01d0159b80 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01d006b0b0 0x7f01d014ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 --2- 172.21.15.99:0/205154959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01d006acd0 0x7f01d00af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 >> 172.21.15.99:0/205154959 conn(0x7f01d00d0770 msgr2=0x7f01d015d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 shutdown_connections 2024-03-04T10:10:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:17.218+0000 7f01d52c6640 1 -- 172.21.15.99:0/205154959 wait complete. 2024-03-04T10:10:17.231 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:10:17.231 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:10:17.231 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:10:17.231 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:10:17.231 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:17.231 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:10:17.232 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:10:17.232 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:10:17.232 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:10:17.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.232 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:10:17.247 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:10:17.247 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:10:17.247 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:10:17.247 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.247 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:10:17.247 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42368sec preferred_lft 42368sec 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:10:17.248 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:10:17.249 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:17.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.250 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:10:17.313 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:10:17.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.314 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:10:17.373 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:10:17.373 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:10:17.373 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:10:17.373 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:10:17.373 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:10:17.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.374 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:10:17.374 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:10:17.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:10:17.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:10:17.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:10:17.374 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:10:17.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.468 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:10:17.483 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:10:17.483 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:10:17.483 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:10:17.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.484 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:10:17.550 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:10:17.551 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:17.606 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:10:17.607 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:10:17.607 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:17.662 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:10:17.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.662 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:10:17.746 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:10:17.747 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:10:17.757 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:10:17.757 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:10:17.757 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:10:17.757 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:10:17.757 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:10:17.758 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:10:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:10:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:10:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:10:17.758 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:10:17.760 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:10:17.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.760 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:17.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.787 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:10:17.844 INFO:teuthology.orchestra.run.smithi099.stdout:client7833 2024-03-04T10:10:17.845 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:17.845 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:17.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:17.901 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-04T10:10:17.959 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:17.959 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:18.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:18.015 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-03-04T10:10:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:6+0 records in 2024-03-04T10:10:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:6+0 records out 2024-03-04T10:10:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0382702 s, 164 MB/s 2024-03-04T10:10:18.110 DEBUG:teuthology.orchestra.run.smithi099:> stdin-killer -- python3 -c ' 2024-03-04T10:10:18.110 DEBUG:teuthology.orchestra.run.smithi099:> import os 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> import stat 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> import json 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> import sys 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> try: 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> except OSError as e: 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> sys.exit(e.errno) 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> print(json.dumps( 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> dict([(a, getattr(s, a)) for a in attrs]), 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> indent=2)) 2024-03-04T10:10:18.111 DEBUG:teuthology.orchestra.run.smithi099:> ' 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_mode": 33188, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_ino": 1099511627777, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_dev": 41, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_nlink": 1, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_uid": 1000, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_gid": 1301, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_size": 6291456, 2024-03-04T10:10:18.178 INFO:teuthology.orchestra.run.smithi099.stdout: "st_atime": 1709547018.0686479, 2024-03-04T10:10:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "st_mtime": 1709547018.0886474, 2024-03-04T10:10:18.179 INFO:teuthology.orchestra.run.smithi099.stdout: "st_ctime": 1709547018.0886474 2024-03-04T10:10:18.179 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:10:18.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T10:10:18.187 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:18.187 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:18.243 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:18.243 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:18.298 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:18.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.435+0000 7f17e4174640 1 Processor -- start 2024-03-04T10:10:18.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.435+0000 7f17e4174640 1 -- start start 2024-03-04T10:10:18.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e4174640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17dc15b960 0x7f17dc0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e4174640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0ccad0 con 0x7f17dc0d0ad0 2024-03-04T10:10:18.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e4174640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0cfab0 con 0x7f17dc0b0e10 2024-03-04T10:10:18.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e4174640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc0cf860 con 0x7f17dc15b960 2024-03-04T10:10:18.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e1f10640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17dc15b960 0x7f17dc0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e170f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f17dc0d0ad0 0x7f17dc0b0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57196/0 (socket says 172.21.15.99:57196) 2024-03-04T10:10:18.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.436+0000 7f17e170f640 1 -- 172.21.15.99:0/3121672319 learned_addr learned my addr 172.21.15.99:0/3121672319 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:18.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.437+0000 7f17e1f10640 1 -- 172.21.15.99:0/3121672319 >> v1:172.21.15.162:6789/0 conn(0x7f17dc0b0e10 legacy=0x7f17dc0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:18.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.437+0000 7f17e1f10640 1 -- 172.21.15.99:0/3121672319 >> v1:172.21.15.99:6789/0 conn(0x7f17dc0d0ad0 legacy=0x7f17dc0b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.437+0000 7f17e1f10640 1 -- 172.21.15.99:0/3121672319 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0cfd90 con 0x7f17dc15b960 2024-03-04T10:10:18.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.437+0000 7f17e0f0e640 1 -- 172.21.15.99:0/3121672319 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770610851 0 0) 0x7f17dc0ccad0 con 0x7f17dc0d0ad0 2024-03-04T10:10:18.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.437+0000 7f17e1f10640 1 --2- 172.21.15.99:0/3121672319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc0d0590 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0cf9e0 tx=0x7f17d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f661fcbcdcab947f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:18.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e0f0e640 1 -- 172.21.15.99:0/3121672319 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80315a0 con 0x7f17dc15b960 2024-03-04T10:10:18.441 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e0f0e640 1 -- 172.21.15.99:0/3121672319 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d8037070 con 0x7f17dc15b960 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e0f0e640 1 -- 172.21.15.99:0/3121672319 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8031960 con 0x7f17dc15b960 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 -- 172.21.15.99:0/3121672319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 msgr2=0x7f17dc0d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 --2- 172.21.15.99:0/3121672319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc0d0590 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0cf9e0 tx=0x7f17d802b840 comp rx=0 tx=0).stop 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 -- 172.21.15.99:0/3121672319 shutdown_connections 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 --2- 172.21.15.99:0/3121672319 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc0d0590 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 -- 172.21.15.99:0/3121672319 >> 172.21.15.99:0/3121672319 conn(0x7f17dc06f5f0 msgr2=0x7f17dc06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 -- 172.21.15.99:0/3121672319 shutdown_connections 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.438+0000 7f17e4174640 1 -- 172.21.15.99:0/3121672319 wait complete. 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.439+0000 7f17e4174640 1 Processor -- start 2024-03-04T10:10:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.439+0000 7f17e4174640 1 -- start start 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e4174640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc0b0e10 0x7f17dc14dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e4174640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17dc0d0ad0 0x7f17dc0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e4174640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc15e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e4174640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0b4ed0 con 0x7f17dc15b960 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e4174640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0d0100 con 0x7f17dc0b0e10 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e4174640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0b5050 con 0x7f17dc0d0ad0 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc15e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc15e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54928/0 (socket says 172.21.15.99:54928) 2024-03-04T10:10:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 -- 172.21.15.99:0/2926784548 learned_addr learned my addr 172.21.15.99:0/2926784548 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e170f640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17dc0d0ad0 0x7f17dc0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e1f10640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc0b0e10 0x7f17dc14dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 -- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17dc0d0ad0 msgr2=0x7f17dc0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17dc0d0ad0 0x7f17dc0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 -- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc0b0e10 msgr2=0x7f17dc14dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc0b0e10 0x7f17dc14dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e2711640 1 -- 172.21.15.99:0/2926784548 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0ad420 con 0x7f17dc15b960 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.440+0000 7f17e1f10640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc0b0e10 0x7f17dc14dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.441+0000 7f17e2711640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc15e340 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f17d400b990 tx=0x7f17d400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.441+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4019070 con 0x7f17dc15b960 2024-03-04T10:10:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.441+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d400cba0 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.441+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc0af110 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.441+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d4014440 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.441+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc0af6d0 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.442+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc074f60 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.442+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f17d401c070 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.443+0000 7f17d27fc640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c006b4d0 0x7f17c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.443+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f17d408a110 con 0x7f17dc15b960 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.443+0000 7f17e1f10640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c006b4d0 0x7f17c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.443+0000 7f17e1f10640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c006b4d0 0x7f17c006d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f17d8036000 tx=0x7f17d8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:18.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.447+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f17d4058e00 con 0x7f17dc15b960 2024-03-04T10:10:18.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.619+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17dc0b6f90 con 0x7f17dc15b960 2024-03-04T10:10:18.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.620+0000 7f17d27fc640 1 -- 172.21.15.99:0/2926784548 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16849 (secure 0 0 0) 0x7f17d4058700 con 0x7f17dc15b960 2024-03-04T10:10:18.623 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:18.623 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":93,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:10:08.853420+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c006b4d0 msgr2=0x7f17c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c006b4d0 0x7f17c006d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f17d8036000 tx=0x7f17d8008030 comp rx=0 tx=0).stop 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 msgr2=0x7f17dc15e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc15e340 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f17d400b990 tx=0x7f17d400be50 comp rx=0 tx=0).stop 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 shutdown_connections 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c006b4d0 0x7f17c006d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17dc15b960 0x7f17dc15e340 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17dc0d0ad0 0x7f17dc0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 --2- 172.21.15.99:0/2926784548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc0b0e10 0x7f17dc14dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 >> 172.21.15.99:0/2926784548 conn(0x7f17dc06f5f0 msgr2=0x7f17dc15af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 shutdown_connections 2024-03-04T10:10:18.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.622+0000 7f17e4174640 1 -- 172.21.15.99:0/2926784548 wait complete. 2024-03-04T10:10:18.635 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:10:05.411440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:10:06.166240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:10:18.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:10:18.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.735+0000 7fe55fbd1640 1 Processor -- start 2024-03-04T10:10:18.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.735+0000 7fe55fbd1640 1 -- start start 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55fbd1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5580cf660 0x7fe5580cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55fbd1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe558162450 con 0x7fe55815da20 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55fbd1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5580cc2d0 con 0x7fe5581616e0 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55fbd1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5580d0000 con 0x7fe5580cf660 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55d16c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe5580cf660 0x7fe5580cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55d96d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe55815da20 0x7fe5580ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57206/0 (socket says 172.21.15.99:57206) 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55d96d640 1 -- 172.21.15.99:0/3508527578 learned_addr learned my addr 172.21.15.99:0/3508527578 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55d16c640 1 -- 172.21.15.99:0/3508527578 >> v1:172.21.15.162:6789/0 conn(0x7fe5581616e0 legacy=0x7fe558161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.736+0000 7fe55d16c640 1 -- 172.21.15.99:0/3508527578 >> v1:172.21.15.99:6789/0 conn(0x7fe55815da20 legacy=0x7fe5580ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.737+0000 7fe55d16c640 1 -- 172.21.15.99:0/3508527578 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580cc5b0 con 0x7fe5580cf660 2024-03-04T10:10:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.737+0000 7fe55c96b640 1 -- 172.21.15.99:0/3508527578 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459657429 0 0) 0x7fe558162450 con 0x7fe55815da20 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.737+0000 7fe55d16c640 1 --2- 172.21.15.99:0/3508527578 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5580cf660 0x7fe5580cfac0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe5480097a0 tx=0x7fe54802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b75e62da4f0128 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.737+0000 7fe55c96b640 1 -- 172.21.15.99:0/3508527578 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5480315a0 con 0x7fe5580cf660 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.737+0000 7fe55c96b640 1 -- 172.21.15.99:0/3508527578 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe548037070 con 0x7fe5580cf660 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.737+0000 7fe55c96b640 1 -- 172.21.15.99:0/3508527578 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548031960 con 0x7fe5580cf660 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 -- 172.21.15.99:0/3508527578 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5580cf660 msgr2=0x7fe5580cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/3508527578 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5580cf660 0x7fe5580cfac0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fe5480097a0 tx=0x7fe54802b840 comp rx=0 tx=0).stop 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 -- 172.21.15.99:0/3508527578 shutdown_connections 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/3508527578 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe5580cf660 0x7fe5580cfac0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 -- 172.21.15.99:0/3508527578 >> 172.21.15.99:0/3508527578 conn(0x7fe5580b33a0 msgr2=0x7fe5580b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 -- 172.21.15.99:0/3508527578 shutdown_connections 2024-03-04T10:10:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.738+0000 7fe55fbd1640 1 -- 172.21.15.99:0/3508527578 wait complete. 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.739+0000 7fe55fbd1640 1 Processor -- start 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.739+0000 7fe55fbd1640 1 -- start start 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55fbd1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5580cf660 0x7fe5581f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55fbd1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 0x7fe55814ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55fbd1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5581616e0 0x7fe5580c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55fbd1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55815de00 con 0x7fe55815da20 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55fbd1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580cc920 con 0x7fe5581616e0 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55fbd1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55806bde0 con 0x7fe5580cf660 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 0x7fe55814ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 0x7fe55814ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54946/0 (socket says 172.21.15.99:54946) 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 -- 172.21.15.99:0/1440467048 learned_addr learned my addr 172.21.15.99:0/1440467048 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 -- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5580cf660 msgr2=0x7fe5581f4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5580cf660 0x7fe5581f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 -- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5581616e0 msgr2=0x7fe5580c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55e16e640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5581616e0 0x7fe5580c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5581616e0 0x7fe5580c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 -- 172.21.15.99:0/1440467048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580b2ff0 con 0x7fe55815da20 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55e16e640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5581616e0 0x7fe5580c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.740+0000 7fe55d16c640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 0x7fe55814ab90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe5480097a0 tx=0x7fe5480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.741+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548032840 con 0x7fe55815da20 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.741+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5480329e0 con 0x7fe55815da20 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.741+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55815f1e0 con 0x7fe55815da20 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.741+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548032dd0 con 0x7fe55815da20 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.741+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55815f7a0 con 0x7fe55815da20 2024-03-04T10:10:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.742+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55815efd0 con 0x7fe55815da20 2024-03-04T10:10:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.742+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe54803b3f0 con 0x7fe55815da20 2024-03-04T10:10:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.742+0000 7fe54e7fc640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe53c06b4d0 0x7fe53c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.743+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe54802f030 con 0x7fe55815da20 2024-03-04T10:10:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.743+0000 7fe55d96d640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe53c06b4d0 0x7fe53c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.743+0000 7fe55d96d640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe53c06b4d0 0x7fe53c06d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe554007fd0 tx=0x7fe554007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:18.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.747+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe548040070 con 0x7fe55815da20 2024-03-04T10:10:18.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.918+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe55806d530 con 0x7fe55815da20 2024-03-04T10:10:18.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.918+0000 7fe54e7fc640 1 -- 172.21.15.99:0/1440467048 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe54807b0b0 con 0x7fe55815da20 2024-03-04T10:10:18.921 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe53c06b4d0 msgr2=0x7fe53c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe53c06b4d0 0x7fe53c06d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe554007fd0 tx=0x7fe554007920 comp rx=0 tx=0).stop 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 msgr2=0x7fe55814ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 0x7fe55814ab90 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe5480097a0 tx=0x7fe5480046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 shutdown_connections 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe53c06b4d0 0x7fe53c06d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5581616e0 0x7fe5580c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe55815da20 0x7fe55814ab90 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 --2- 172.21.15.99:0/1440467048 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5580cf660 0x7fe5581f4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 >> 172.21.15.99:0/1440467048 conn(0x7fe5580b33a0 msgr2=0x7fe5580d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:18.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 shutdown_connections 2024-03-04T10:10:18.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:18.920+0000 7fe55fbd1640 1 -- 172.21.15.99:0/1440467048 wait complete. 2024-03-04T10:10:18.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:10:19.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.033+0000 7f37ae08e640 1 Processor -- start 2024-03-04T10:10:19.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.033+0000 7f37ae08e640 1 -- start start 2024-03-04T10:10:19.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37ae08e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f37a81450e0 0x7f37a80cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37ae08e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f37a80cc510 0x7f37a80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37ae08e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a80cebd0 con 0x7f37a815c9b0 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37ae08e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a80cf7e0 con 0x7f37a80cc510 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37ae08e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a80ceeb0 con 0x7f37a81450e0 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f37a80cc510 0x7f37a80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f37a80cc510 0x7f37a80b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54962/0 (socket says 172.21.15.99:54962) 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 -- 172.21.15.99:0/3496322918 learned_addr learned my addr 172.21.15.99:0/3496322918 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a77fe640 1 --2- 172.21.15.99:0/3496322918 >> v2:172.21.15.162:3300/0 conn(0x7f37a81450e0 0x7f37a80cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 -- 172.21.15.99:0/3496322918 >> v2:172.21.15.162:3300/0 conn(0x7f37a81450e0 msgr2=0x7f37a80cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 --2- 172.21.15.99:0/3496322918 >> v2:172.21.15.162:3300/0 conn(0x7f37a81450e0 0x7f37a80cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 -- 172.21.15.99:0/3496322918 >> v1:172.21.15.162:6789/0 conn(0x7f37a815c9b0 legacy=0x7f37a80cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a7fff640 1 -- 172.21.15.99:0/3496322918 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80cf190 con 0x7f37a80cc510 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.034+0000 7f37a77fe640 1 --2- 172.21.15.99:0/3496322918 >> v2:172.21.15.162:3300/0 conn(0x7f37a81450e0 0x7f37a80cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.035+0000 7f37a7fff640 1 --2- 172.21.15.99:0/3496322918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a80b3250 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f37a80b23d0 tx=0x7f37a002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4803320bf389ebef server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.035+0000 7f37a67fc640 1 -- 172.21.15.99:0/3496322918 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a002ca80 con 0x7f37a80cc510 2024-03-04T10:10:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.035+0000 7f37a67fc640 1 -- 172.21.15.99:0/3496322918 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a002cc20 con 0x7f37a80cc510 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.035+0000 7f37a67fc640 1 -- 172.21.15.99:0/3496322918 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a00343f0 con 0x7f37a80cc510 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 -- 172.21.15.99:0/3496322918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 msgr2=0x7f37a80b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 --2- 172.21.15.99:0/3496322918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a80b3250 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f37a80b23d0 tx=0x7f37a002bee0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 -- 172.21.15.99:0/3496322918 shutdown_connections 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 --2- 172.21.15.99:0/3496322918 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a80b3250 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 --2- 172.21.15.99:0/3496322918 >> v2:172.21.15.162:3300/0 conn(0x7f37a81450e0 0x7f37a80cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 -- 172.21.15.99:0/3496322918 >> 172.21.15.99:0/3496322918 conn(0x7f37a806acd0 msgr2=0x7f37a806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 -- 172.21.15.99:0/3496322918 shutdown_connections 2024-03-04T10:10:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.036+0000 7f37ae08e640 1 -- 172.21.15.99:0/3496322918 wait complete. 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.037+0000 7f37ae08e640 1 Processor -- start 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.037+0000 7f37ae08e640 1 -- start start 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37ae08e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a8139860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37ae08e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37a81450e0 0x7f37a813a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37ae08e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37a815c9b0 0x7f37a8139020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37ae08e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a81454c0 con 0x7f37a80cc510 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37ae08e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815ce60 con 0x7f37a81450e0 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37ae08e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80ccd10 con 0x7f37a815c9b0 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a8139860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a8139860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54966/0 (socket says 172.21.15.99:54966) 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 -- 172.21.15.99:0/949312528 learned_addr learned my addr 172.21.15.99:0/949312528 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a6ffd640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37a81450e0 0x7f37a813a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a7fff640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37a815c9b0 0x7f37a8139020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 -- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37a815c9b0 msgr2=0x7f37a8139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37a815c9b0 0x7f37a8139020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 -- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37a81450e0 msgr2=0x7f37a813a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37a81450e0 0x7f37a813a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 -- 172.21.15.99:0/949312528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a813a870 con 0x7f37a80cc510 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a7fff640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37a815c9b0 0x7f37a8139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.038+0000 7f37a77fe640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a8139860 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f379800c7f0 tx=0x7f379800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.039+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798015070 con 0x7f37a80cc510 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.039+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3798007a30 con 0x7f37a80cc510 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.039+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a813ab50 con 0x7f37a80cc510 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.039+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3798007e00 con 0x7f37a80cc510 2024-03-04T10:10:19.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.039+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a813e9f0 con 0x7f37a80cc510 2024-03-04T10:10:19.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.040+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a813e500 con 0x7f37a80cc510 2024-03-04T10:10:19.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.040+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3798018070 con 0x7f37a80cc510 2024-03-04T10:10:19.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.040+0000 7f378ffff640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f379406b4d0 0x7f379406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.040+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f379808a340 con 0x7f37a80cc510 2024-03-04T10:10:19.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.041+0000 7f37a6ffd640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f379406b4d0 0x7f379406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.041+0000 7f37a6ffd640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f379406b4d0 0x7f379406d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f379c006fd0 tx=0x7f379c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.044+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3798059030 con 0x7f37a80cc510 2024-03-04T10:10:19.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.214+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f37a8074f60 con 0x7f37a80cc510 2024-03-04T10:10:19.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.214+0000 7f378ffff640 1 -- 172.21.15.99:0/949312528 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3798058930 con 0x7f37a80cc510 2024-03-04T10:10:19.217 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f379406b4d0 msgr2=0x7f379406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f379406b4d0 0x7f379406d990 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f379c006fd0 tx=0x7f379c008040 comp rx=0 tx=0).stop 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 msgr2=0x7f37a8139860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a8139860 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f379800c7f0 tx=0x7f379800cbf0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 shutdown_connections 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f379406b4d0 0x7f379406d990 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f37a815c9b0 0x7f37a8139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37a81450e0 0x7f37a813a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 --2- 172.21.15.99:0/949312528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f37a80cc510 0x7f37a8139860 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 >> 172.21.15.99:0/949312528 conn(0x7f37a806acd0 msgr2=0x7f37a806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 shutdown_connections 2024-03-04T10:10:19.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.216+0000 7f37ae08e640 1 -- 172.21.15.99:0/949312528 wait complete. 2024-03-04T10:10:19.230 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:10:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 Processor -- start 2024-03-04T10:10:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 -- start start 2024-03-04T10:10:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 0x7f8db81508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8db806d7e0 0x7f8db8150df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db806dd10 con 0x7f8db8071320 2024-03-04T10:10:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db806dff0 con 0x7f8db806d7e0 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.331+0000 7f8dbdea1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db806e2d0 con 0x7f8db806d3c0 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8db8071320 0x7f8db8071700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57222/0 (socket says 172.21.15.99:57222) 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 -- 172.21.15.99:0/2576994114 learned_addr learned my addr 172.21.15.99:0/2576994114 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269490143 0 0) 0x7f8db806dd10 con 0x7f8db8071320 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db7fff640 1 --2- 172.21.15.99:0/2576994114 >> v2:172.21.15.162:3300/0 conn(0x7f8db806d7e0 0x7f8db8150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da0003520 con 0x7f8db8071320 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db6ffd640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 0x7f8db81508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152815455 0 0) 0x7f8da0003520 con 0x7f8db8071320 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 msgr2=0x7f8db81508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 0x7f8db81508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 >> v2:172.21.15.162:3300/0 conn(0x7f8db806d7e0 msgr2=0x7f8db8150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 --2- 172.21.15.99:0/2576994114 >> v2:172.21.15.162:3300/0 conn(0x7f8db806d7e0 0x7f8db8150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80b2820 con 0x7f8db8071320 2024-03-04T10:10:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8da8001670 con 0x7f8db8071320 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 >> v1:172.21.15.99:6789/0 conn(0x7f8db8071320 legacy=0x7f8db8071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8da00039b0 0x7f8da0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 0x7f8da0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8da0004d40 0x7f8da00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db806dff0 con 0x7f8da00039b0 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db806e2d0 con 0x7f8da0004d40 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da000b9c0 con 0x7f8db806d3c0 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8da00039b0 0x7f8da0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db7fff640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8da0004d40 0x7f8da00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db6ffd640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 0x7f8da0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 -- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 msgr2=0x7f8da0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8db806d3c0 0x7f8da0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 -- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8da0004d40 msgr2=0x7f8da00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8da0004d40 0x7f8da00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.332+0000 7f8db77fe640 1 -- 172.21.15.99:0/2576994114 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da0003520 con 0x7f8da00039b0 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.333+0000 7f8db77fe640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8da00039b0 0x7f8da0003dd0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8da800c240 tx=0x7f8da0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad71edcf6fab220 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.333+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8007d30 con 0x7f8da00039b0 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.333+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da8030070 con 0x7f8da00039b0 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.333+0000 7f8db67fc640 1 -- 172.21.15.99:0/2576994114 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da802cd60 con 0x7f8da00039b0 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8dbdea1640 1 -- 172.21.15.99:0/2576994114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8da00039b0 msgr2=0x7f8da0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/2576994114 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8da00039b0 0x7f8da0003dd0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8da800c240 tx=0x7f8da0005830 comp rx=0 tx=0).stop 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8db7fff640 1 -- 172.21.15.99:0/2576994114 reap_dead start 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8dbdea1640 1 -- 172.21.15.99:0/2576994114 shutdown_connections 2024-03-04T10:10:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8dbdea1640 1 -- 172.21.15.99:0/2576994114 >> 172.21.15.99:0/2576994114 conn(0x7f8db806acd0 msgr2=0x7f8db806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8dbdea1640 1 -- 172.21.15.99:0/2576994114 shutdown_connections 2024-03-04T10:10:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.334+0000 7f8dbdea1640 1 -- 172.21.15.99:0/2576994114 wait complete. 2024-03-04T10:10:19.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.335+0000 7f8dbdea1640 1 Processor -- start 2024-03-04T10:10:19.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.335+0000 7f8dbdea1640 1 -- start start 2024-03-04T10:10:19.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8dbdea1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8da00039b0 0x7f8db8155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8dbdea1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 0x7f8db81efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8dbdea1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8db81f02d0 0x7f8db815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8dbdea1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db80b3fd0 con 0x7f8db8153460 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8dbdea1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db80b3e50 con 0x7f8db81f02d0 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8dbdea1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db81508b0 con 0x7f8da00039b0 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 0x7f8db81efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 0x7f8db81efd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54974/0 (socket says 172.21.15.99:54974) 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 -- 172.21.15.99:0/3300377906 learned_addr learned my addr 172.21.15.99:0/3300377906 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db77fe640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8da00039b0 0x7f8db8155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db7fff640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8db81f02d0 0x7f8db815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 -- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8da00039b0 msgr2=0x7f8db8155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8da00039b0 0x7f8db8155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 -- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8db81f02d0 msgr2=0x7f8db815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8db81f02d0 0x7f8db815c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 -- 172.21.15.99:0/3300377906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db8153940 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db7fff640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8db81f02d0 0x7f8db815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db77fe640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8da00039b0 0x7f8db8155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.336+0000 7f8db6ffd640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 0x7f8db81efd90 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8dac011a10 tx=0x7f8dac011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.337+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac01f070 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.337+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac00ca00 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.337+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db8153c20 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.337+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac00ce10 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.337+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db81544c0 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.338+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db8074f60 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.338+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8dac022070 con 0x7f8db8153460 2024-03-04T10:10:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.338+0000 7f8d9ffff640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8da406b4d0 0x7f8da406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.339+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8dac08bea0 con 0x7f8db8153460 2024-03-04T10:10:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.339+0000 7f8db77fe640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8da406b4d0 0x7f8da406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.339+0000 7f8db77fe640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8da406b4d0 0x7f8da406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8da8034000 tx=0x7f8da8007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.343+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8dac05a540 con 0x7f8db8153460 2024-03-04T10:10:19.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.512+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8db80b0810 con 0x7f8db8153460 2024-03-04T10:10:19.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.512+0000 7f8d9ffff640 1 -- 172.21.15.99:0/3300377906 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8dac05e5e0 con 0x7f8db8153460 2024-03-04T10:10:19.515 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8da406b4d0 msgr2=0x7f8da406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8da406b4d0 0x7f8da406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8da8034000 tx=0x7f8da8007000 comp rx=0 tx=0).stop 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 msgr2=0x7f8db81efd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 0x7f8db81efd90 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8dac011a10 tx=0x7f8dac011ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 shutdown_connections 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8da406b4d0 0x7f8da406d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8db81f02d0 0x7f8db815c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8db8153460 0x7f8db81efd90 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 --2- 172.21.15.99:0/3300377906 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8da00039b0 0x7f8db8155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 >> 172.21.15.99:0/3300377906 conn(0x7f8db806acd0 msgr2=0x7f8db806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 shutdown_connections 2024-03-04T10:10:19.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.514+0000 7f8dbdea1640 1 -- 172.21.15.99:0/3300377906 wait complete. 2024-03-04T10:10:19.528 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:10:19.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.627+0000 7f0aa3634640 1 Processor -- start 2024-03-04T10:10:19.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.627+0000 7f0aa3634640 1 -- start start 2024-03-04T10:10:19.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa3634640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f0a9c06c090 0x7f0a9c06f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa3634640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0a9c06c470 0x7f0a9c0af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa3634640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0ad6c0 con 0x7f0a9c066de0 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa3634640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9c0ad960 con 0x7f0a9c06c470 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa3634640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a9c06ebc0 con 0x7f0a9c06c090 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0a9c06c470 0x7f0a9c0af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f0a9c06c470 0x7f0a9c0af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:54986/0 (socket says 172.21.15.99:54986) 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 -- 172.21.15.99:0/2283076130 learned_addr learned my addr 172.21.15.99:0/2283076130 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa1bd1640 1 --1- 172.21.15.99:0/2283076130 >> v1:172.21.15.99:6789/0 conn(0x7f0a9c066de0 0x7f0a9c15f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57234/0 (socket says 172.21.15.99:57234) 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa13d0640 1 --2- 172.21.15.99:0/2283076130 >> v2:172.21.15.162:3300/0 conn(0x7f0a9c06c090 0x7f0a9c06f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 -- 172.21.15.99:0/2283076130 >> v2:172.21.15.162:3300/0 conn(0x7f0a9c06c090 msgr2=0x7f0a9c06f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 --2- 172.21.15.99:0/2283076130 >> v2:172.21.15.162:3300/0 conn(0x7f0a9c06c090 0x7f0a9c06f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 -- 172.21.15.99:0/2283076130 >> v1:172.21.15.99:6789/0 conn(0x7f0a9c066de0 legacy=0x7f0a9c15f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0aa0bcf640 1 -- 172.21.15.99:0/2283076130 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c06eea0 con 0x7f0a9c06c470 2024-03-04T10:10:19.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.628+0000 7f0a93fff640 1 -- 172.21.15.99:0/2283076130 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119346706 0 0) 0x7f0a9c0ad6c0 con 0x7f0a9c066de0 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.629+0000 7f0aa0bcf640 1 --2- 172.21.15.99:0/2283076130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c06c470 0x7f0a9c0af220 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c06d540 tx=0x7f0a9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374a6e0991d36742 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.629+0000 7f0a93fff640 1 -- 172.21.15.99:0/2283076130 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98035020 con 0x7f0a9c06c470 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.629+0000 7f0a93fff640 1 -- 172.21.15.99:0/2283076130 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a98031870 con 0x7f0a9c06c470 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.629+0000 7f0a93fff640 1 -- 172.21.15.99:0/2283076130 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98031c40 con 0x7f0a9c06c470 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.629+0000 7f0aa3634640 1 -- 172.21.15.99:0/2283076130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c06c470 msgr2=0x7f0a9c0af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.629+0000 7f0aa3634640 1 --2- 172.21.15.99:0/2283076130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c06c470 0x7f0a9c0af220 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c06d540 tx=0x7f0a9802b910 comp rx=0 tx=0).stop 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.630+0000 7f0aa3634640 1 -- 172.21.15.99:0/2283076130 shutdown_connections 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.630+0000 7f0aa3634640 1 --2- 172.21.15.99:0/2283076130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c06c470 0x7f0a9c0af220 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.630+0000 7f0aa3634640 1 --2- 172.21.15.99:0/2283076130 >> v2:172.21.15.162:3300/0 conn(0x7f0a9c06c090 0x7f0a9c06f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.630+0000 7f0aa3634640 1 -- 172.21.15.99:0/2283076130 >> 172.21.15.99:0/2283076130 conn(0x7f0a9c0cfef0 msgr2=0x7f0a9c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.630+0000 7f0aa3634640 1 -- 172.21.15.99:0/2283076130 shutdown_connections 2024-03-04T10:10:19.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.630+0000 7f0aa3634640 1 -- 172.21.15.99:0/2283076130 wait complete. 2024-03-04T10:10:19.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 Processor -- start 2024-03-04T10:10:19.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 -- start start 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a9c066de0 0x7f0a9c1f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a9c06c090 0x7f0a9c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 0x7f0a9c150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0adc40 con 0x7f0a9c0b4860 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0af760 con 0x7f0a9c066de0 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.631+0000 7f0aa3634640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c0addc0 con 0x7f0a9c06c090 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 0x7f0a9c150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 0x7f0a9c150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55000/0 (socket says 172.21.15.99:55000) 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 -- 172.21.15.99:0/1154537898 learned_addr learned my addr 172.21.15.99:0/1154537898 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa13d0640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a9c066de0 0x7f0a9c1f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 -- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a9c06c090 msgr2=0x7f0a9c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa0bcf640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a9c06c090 0x7f0a9c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a9c06c090 0x7f0a9c0b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 -- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a9c066de0 msgr2=0x7f0a9c1f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a9c066de0 0x7f0a9c1f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 -- 172.21.15.99:0/1154537898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0b3660 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa1bd1640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 0x7f0a9c150120 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400c760 tx=0x7f0a9400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9400a640 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9400a800 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9400abf0 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.632+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c0b33a0 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.633+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c1f4060 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.633+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9c074f60 con 0x7f0a9c0b4860 2024-03-04T10:10:19.637 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.634+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0a94007070 con 0x7f0a9c0b4860 2024-03-04T10:10:19.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.634+0000 7f0a91ffb640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a8006b500 0x7f0a8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:19.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.634+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0a9408a1f0 con 0x7f0a9c0b4860 2024-03-04T10:10:19.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.634+0000 7f0aa13d0640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a8006b500 0x7f0a8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:19.638 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.635+0000 7f0aa13d0640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a8006b500 0x7f0a8006d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c006fd0 tx=0x7f0a8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:19.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.638+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f0a94058f70 con 0x7f0a9c0b4860 2024-03-04T10:10:19.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.822+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a9c06def0 con 0x7f0a9c0b4860 2024-03-04T10:10:19.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.822+0000 7f0a91ffb640 1 -- 172.21.15.99:0/1154537898 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v93) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0a94058870 con 0x7f0a9c0b4860 2024-03-04T10:10:19.824 INFO:teuthology.orchestra.run.smithi099.stderr:listed 0 entries 2024-03-04T10:10:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a8006b500 msgr2=0x7f0a8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a8006b500 0x7f0a8006d9c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c006fd0 tx=0x7f0a8c008040 comp rx=0 tx=0).stop 2024-03-04T10:10:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 msgr2=0x7f0a9c150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 0x7f0a9c150120 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f0a9400c760 tx=0x7f0a9400cc20 comp rx=0 tx=0).stop 2024-03-04T10:10:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 shutdown_connections 2024-03-04T10:10:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0a8006b500 0x7f0a8006d9c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0a9c0b4860 0x7f0a9c150120 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f0a9c06c090 0x7f0a9c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 --2- 172.21.15.99:0/1154537898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a9c066de0 0x7f0a9c1f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 >> 172.21.15.99:0/1154537898 conn(0x7f0a9c0cfef0 msgr2=0x7f0a9c15fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 shutdown_connections 2024-03-04T10:10:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:19.824+0000 7f0aa3634640 1 -- 172.21.15.99:0/1154537898 wait complete. 2024-03-04T10:10:19.837 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:10:19.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:19.837 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:19.890 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:10:19.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:19.891 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:10:19.917 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:20.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 Processor -- start 2024-03-04T10:10:20.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 -- start start 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf006d3b0 0x7f8bf00ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8bf00cc500 0x7f8bf00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf00d0140 con 0x7f8bf00cfd60 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf00cf440 con 0x7f8bf00cc500 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.056+0000 7f8bf4fb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf015cbb0 con 0x7f8bf006d3b0 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8bf00cc500 0x7f8bf00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8bf00cc500 0x7f8bf00cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55008/0 (socket says 172.21.15.99:55008) 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 -- 172.21.15.99:0/2290328703 learned_addr learned my addr 172.21.15.99:0/2290328703 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8be7fff640 1 --2- 172.21.15.99:0/2290328703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf006d3b0 0x7f8bf00ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 -- 172.21.15.99:0/2290328703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf006d3b0 msgr2=0x7f8bf00ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 --2- 172.21.15.99:0/2290328703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf006d3b0 0x7f8bf00ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 -- 172.21.15.99:0/2290328703 >> v1:172.21.15.162:6789/0 conn(0x7f8bf00cfd60 legacy=0x7f8bf00cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.057+0000 7f8beed9d640 1 -- 172.21.15.99:0/2290328703 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf015c280 con 0x7f8bf00cc500 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8beed9d640 1 --2- 172.21.15.99:0/2290328703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00cef00 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009e50 tx=0x7f8be802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b08a7511e8842 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bedd9b640 1 -- 172.21.15.99:0/2290328703 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be802ca70 con 0x7f8bf00cc500 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bedd9b640 1 -- 172.21.15.99:0/2290328703 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be802cc10 con 0x7f8bf00cc500 2024-03-04T10:10:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bedd9b640 1 -- 172.21.15.99:0/2290328703 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be80343f0 con 0x7f8bf00cc500 2024-03-04T10:10:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2290328703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 msgr2=0x7f8bf00cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2290328703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00cef00 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009e50 tx=0x7f8be802bed0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2290328703 shutdown_connections 2024-03-04T10:10:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2290328703 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00cef00 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2290328703 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf006d3b0 0x7f8bf00ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2290328703 >> 172.21.15.99:0/2290328703 conn(0x7f8bf006acd0 msgr2=0x7f8bf006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2290328703 shutdown_connections 2024-03-04T10:10:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.058+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2290328703 wait complete. 2024-03-04T10:10:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 Processor -- start 2024-03-04T10:10:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 -- start start 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bf006d3b0 0x7f8bf014b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf00cfd60 0x7f8bf0158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf015bf30 con 0x7f8bf00cc500 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf015c840 con 0x7f8bf006d3b0 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8bf4fb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf015f520 con 0x7f8bf00cfd60 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8be7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8be7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55020/0 (socket says 172.21.15.99:55020) 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.060+0000 7f8be7fff640 1 -- 172.21.15.99:0/2686907975 learned_addr learned my addr 172.21.15.99:0/2686907975 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8bee59c640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bf006d3b0 0x7f8bf014b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8beed9d640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf00cfd60 0x7f8bf0158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be7fff640 1 -- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf00cfd60 msgr2=0x7f8bf0158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be7fff640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf00cfd60 0x7f8bf0158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be7fff640 1 -- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bf006d3b0 msgr2=0x7f8bf014b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be7fff640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bf006d3b0 0x7f8bf014b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be7fff640 1 -- 172.21.15.99:0/2686907975 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf014b570 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be7fff640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00ad010 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800d800 tx=0x7f8bd800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd800b6c0 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd8019070 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf00a88c0 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd800ba80 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.061+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf00a8e80 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.062+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf0074f60 con 0x7f8bf00cc500 2024-03-04T10:10:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.062+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8bd800bc20 con 0x7f8bf00cc500 2024-03-04T10:10:20.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.063+0000 7f8be6ffd640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bd006b4d0 0x7f8bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.063+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8bd808a7b0 con 0x7f8bf00cc500 2024-03-04T10:10:20.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.063+0000 7f8bee59c640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bd006b4d0 0x7f8bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.064+0000 7f8bee59c640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bd006b4d0 0x7f8bd006d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8be0006fd0 tx=0x7f8be0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.068+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8bd8059530 con 0x7f8bf00cc500 2024-03-04T10:10:20.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.275+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bf015baa0 con 0x7f8bf00cc500 2024-03-04T10:10:20.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.275+0000 7f8be6ffd640 1 -- 172.21.15.99:0/2686907975 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f8bd8058e30 con 0x7f8bf00cc500 2024-03-04T10:10:20.278 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:20.278 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":78,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:09.188560+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7644},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7644":{"gid":7644,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.99:6835/3671020073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":3671020073},{"type":"v1","addr":"172.21.15.99:6835","nonce":3671020073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:20.278 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 78 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bd006b4d0 msgr2=0x7f8bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bd006b4d0 0x7f8bd006d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f8be0006fd0 tx=0x7f8be0008040 comp rx=0 tx=0).stop 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 msgr2=0x7f8bf00ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00ad010 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800d800 tx=0x7f8bd800dcc0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 shutdown_connections 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bd006b4d0 0x7f8bd006d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8bf00cfd60 0x7f8bf0158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8bf00cc500 0x7f8bf00ad010 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 --2- 172.21.15.99:0/2686907975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8bf006d3b0 0x7f8bf014b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 >> 172.21.15.99:0/2686907975 conn(0x7f8bf006acd0 msgr2=0x7f8bf015a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 shutdown_connections 2024-03-04T10:10:20.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.277+0000 7f8bf4fb2640 1 -- 172.21.15.99:0/2686907975 wait complete. 2024-03-04T10:10:20.290 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-04T10:10:20.412 INFO:teuthology.orchestra.run.smithi099.stdout:{"message":"","return_code":0} 2024-03-04T10:10:20.412 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-04T10:10:20.413 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:10:20.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 Processor -- start 2024-03-04T10:10:20.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 -- start start 2024-03-04T10:10:20.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f96440cfd60 0x7f96440cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f964406d3b0 0x7f96440cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96440d0140 con 0x7f96440ce9c0 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964415cbb0 con 0x7f96440cfd60 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f964b6bd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964415c280 con 0x7f964406d3b0 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f96440cfd60 0x7f96440cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f96440cfd60 0x7f96440cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55032/0 (socket says 172.21.15.99:55032) 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 -- 172.21.15.99:0/3704463134 learned_addr learned my addr 172.21.15.99:0/3704463134 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 -- 172.21.15.99:0/3704463134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f964406d3b0 msgr2=0x7f96440cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 --2- 172.21.15.99:0/3704463134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f964406d3b0 0x7f96440cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 -- 172.21.15.99:0/3704463134 >> v1:172.21.15.162:6789/0 conn(0x7f96440ce9c0 legacy=0x7f96440cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.512+0000 7f9649459640 1 -- 172.21.15.99:0/3704463134 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964415c560 con 0x7f96440cfd60 2024-03-04T10:10:20.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f9649459640 1 --2- 172.21.15.99:0/3704463134 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440cfd60 0x7f96440cbdf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f964415cd30 tx=0x7f964002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14185fb0480350c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f963bfff640 1 -- 172.21.15.99:0/3704463134 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640035020 con 0x7f96440cfd60 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f963bfff640 1 -- 172.21.15.99:0/3704463134 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640031640 con 0x7f96440cfd60 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f963bfff640 1 -- 172.21.15.99:0/3704463134 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964002d040 con 0x7f96440cfd60 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f964b6bd640 1 -- 172.21.15.99:0/3704463134 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440cfd60 msgr2=0x7f96440cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f964b6bd640 1 --2- 172.21.15.99:0/3704463134 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440cfd60 0x7f96440cbdf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f964415cd30 tx=0x7f964002b8f0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f964b6bd640 1 -- 172.21.15.99:0/3704463134 shutdown_connections 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f964b6bd640 1 --2- 172.21.15.99:0/3704463134 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f964406d3b0 0x7f96440cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f964b6bd640 1 --2- 172.21.15.99:0/3704463134 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440cfd60 0x7f96440cbdf0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.513+0000 7f964b6bd640 1 -- 172.21.15.99:0/3704463134 >> 172.21.15.99:0/3704463134 conn(0x7f964406acd0 msgr2=0x7f964406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.514+0000 7f964b6bd640 1 -- 172.21.15.99:0/3704463134 shutdown_connections 2024-03-04T10:10:20.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.514+0000 7f964b6bd640 1 -- 172.21.15.99:0/3704463134 wait complete. 2024-03-04T10:10:20.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 Processor -- start 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 -- start start 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f964406d3b0 0x7f96440af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 0x7f964414a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f96440cfd60 0x7f96440aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964415f520 con 0x7f96440ce9c0 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964415bf30 con 0x7f964406d3b0 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f964b6bd640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964415f6a0 con 0x7f96440cfd60 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f9648c58640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 0x7f964414a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f9648c58640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 0x7f964414a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55042/0 (socket says 172.21.15.99:55042) 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.515+0000 7f9648c58640 1 -- 172.21.15.99:0/1366777347 learned_addr learned my addr 172.21.15.99:0/1366777347 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9649c5a640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f96440cfd60 0x7f96440aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9649459640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f964406d3b0 0x7f96440af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9648c58640 1 -- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f96440cfd60 msgr2=0x7f96440aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9648c58640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f96440cfd60 0x7f96440aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9648c58640 1 -- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f964406d3b0 msgr2=0x7f96440af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9648c58640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f964406d3b0 0x7f96440af200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9648c58640 1 -- 172.21.15.99:0/1366777347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964414a5f0 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9649459640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f964406d3b0 0x7f96440af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9649c5a640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f96440cfd60 0x7f96440aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9648c58640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 0x7f964414a0b0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f963400b450 tx=0x7f963400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634015020 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9634011890 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96440af7e0 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634011c80 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.516+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96440afda0 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.517+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96440d0510 con 0x7f96440ce9c0 2024-03-04T10:10:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.517+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9634018020 con 0x7f96440ce9c0 2024-03-04T10:10:20.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.518+0000 7f9639ffb640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f962806b4d0 0x7f962806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.518+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f963408a4f0 con 0x7f96440ce9c0 2024-03-04T10:10:20.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.518+0000 7f9649459640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f962806b4d0 0x7f962806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.519+0000 7f9649459640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f962806b4d0 0x7f962806d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9640036000 tx=0x7f964003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.522+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f96440d0510 con 0x7f96440ce9c0 2024-03-04T10:10:20.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.732+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9644074f90 con 0x7f96440ce9c0 2024-03-04T10:10:20.735 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:10:20.733+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:10:20.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.734+0000 7f9639ffb640 1 -- 172.21.15.99:0/1366777347 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v79) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9644074f90 con 0x7f96440ce9c0 2024-03-04T10:10:20.737 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f962806b4d0 msgr2=0x7f962806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f962806b4d0 0x7f962806d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9640036000 tx=0x7f964003c040 comp rx=0 tx=0).stop 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 msgr2=0x7f964414a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 0x7f964414a0b0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f963400b450 tx=0x7f963400b910 comp rx=0 tx=0).stop 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 shutdown_connections 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f962806b4d0 0x7f962806d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f96440cfd60 0x7f96440aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f96440ce9c0 0x7f964414a0b0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 --2- 172.21.15.99:0/1366777347 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f964406d3b0 0x7f96440af200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 >> 172.21.15.99:0/1366777347 conn(0x7f964406acd0 msgr2=0x7f964415b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 shutdown_connections 2024-03-04T10:10:20.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.737+0000 7f964b6bd640 1 -- 172.21.15.99:0/1366777347 wait complete. 2024-03-04T10:10:20.741 INFO:tasks.ceph.mds.c.smithi099.stderr: -3138> 2024-03-04T10:10:09.835+0000 7f30973fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:10:20.751 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata export - 2024-03-04T10:10:20.769 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:10:20.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 Processor -- start 2024-03-04T10:10:20.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 -- start start 2024-03-04T10:10:20.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6adfd7c90 0x55e6ae091040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 0x55e6ae0b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfda9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6ae0b20d0 con 0x55e6ae0941c0 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6adf80b30 con 0x55e6ae0b1670 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10793b7f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6ae0b2370 con 0x55e6adfd7c90 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10766b6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 0x55e6ae0b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10766b6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 0x55e6ae0b2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55072/0 (socket says 172.21.15.99:55072) 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10766b6640 1 -- 172.21.15.99:0/976610587 learned_addr learned my addr 172.21.15.99:0/976610587 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f10776b8640 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfda9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.790+0000 7f1076eb7640 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6adfd7c90 0x55e6ae091040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10766b6640 1 -- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6adfd7c90 msgr2=0x55e6ae091040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10766b6640 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6adfd7c90 0x55e6ae091040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10766b6640 1 -- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 msgr2=0x55e6adfda9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10766b6640 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfda9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10766b6640 1 -- 172.21.15.99:0/976610587 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6adfd9c60 con 0x55e6ae0941c0 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10766b6640 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 0x55e6ae0b2b60 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f106c009780 tx=0x7f106c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3623819d5197cadc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10756b4640 1 -- 172.21.15.99:0/976610587 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c008870 con 0x55e6ae0941c0 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10756b4640 1 -- 172.21.15.99:0/976610587 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c018070 con 0x55e6ae0941c0 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10756b4640 1 -- 172.21.15.99:0/976610587 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c008c30 con 0x55e6ae0941c0 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 -- 172.21.15.99:0/976610587 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 msgr2=0x55e6ae0b2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 0x55e6ae0b2b60 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f106c009780 tx=0x7f106c005a00 comp rx=0 tx=0).stop 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 -- 172.21.15.99:0/976610587 shutdown_connections 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfda9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6ae0941c0 0x55e6ae0b2b60 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 --2- 172.21.15.99:0/976610587 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6adfd7c90 0x55e6ae091040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 -- 172.21.15.99:0/976610587 >> 172.21.15.99:0/976610587 conn(0x55e6adfd9400 msgr2=0x55e6ae0b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 -- 172.21.15.99:0/976610587 shutdown_connections 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.791+0000 7f10793b7f40 1 -- 172.21.15.99:0/976610587 wait complete. 2024-03-04T10:10:20.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.792+0000 7f10793b7f40 1 Processor -- start 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.792+0000 7f10793b7f40 1 -- start start 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 0x55e6ae008d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6ae0941c0 0x55e6ae0aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfdb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6ae0b30a0 con 0x55e6adfd7c90 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6adfdb560 con 0x55e6ae0b1670 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6ae0b43a0 con 0x55e6ae0941c0 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 0x55e6ae008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 0x55e6ae008d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55078/0 (socket says 172.21.15.99:55078) 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 learned_addr learned my addr 172.21.15.99:0/1488554727 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfdb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6ae0941c0 0x55e6ae0aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6ae0941c0 msgr2=0x55e6ae0aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6ae0941c0 0x55e6ae0aaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 msgr2=0x55e6adfdb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6ae0b1670 0x55e6adfdb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6ae098af0 con 0x55e6adfd7c90 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6ae0941c0 0x55e6ae0aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 0x55e6ae008d20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1068001c40 tx=0x7f106800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f105ffff640 1 -- 172.21.15.99:0/1488554727 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106800f650 con 0x55e6adfd7c90 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f105ffff640 1 -- 172.21.15.99:0/1488554727 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106800f7f0 con 0x55e6adfd7c90 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e6ae09a920 con 0x55e6adfd7c90 2024-03-04T10:10:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f105ffff640 1 -- 172.21.15.99:0/1488554727 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106800fbe0 con 0x55e6adfd7c90 2024-03-04T10:10:20.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.793+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6ae098d90 con 0x55e6adfd7c90 2024-03-04T10:10:20.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.795+0000 7f105ffff640 1 -- 172.21.15.99:0/1488554727 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f10680203f0 con 0x55e6adfd7c90 2024-03-04T10:10:20.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.795+0000 7f105ffff640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1064069eb0 0x7f106406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.795+0000 7f105ffff640 1 -- 172.21.15.99:0/1488554727 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1068089810 con 0x55e6adfd7c90 2024-03-04T10:10:20.802 INFO:teuthology.orchestra.run.smithi099.stdout:���� 2024-03-04T10:10:20.802 INFO:teuthology.orchestra.run.smithi099.stdout: �� 2024-03-04T10:10:20.802 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:20.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.795+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1064069eb0 0x7f106406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.795+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55e6ae08cc50 0x55e6ae0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.795+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.0 13:00000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6adfdbe10 con 0x55e6ae08cc50 2024-03-04T10:10:20.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.796+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55e6ae08cc50 0x55e6ae0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.796+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1064069eb0 0x7f106406c370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x55e6ae020fd0 tx=0x7f106c012000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.796+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55e6ae08cc50 0x55e6ae0b3250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.796+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.797+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f10700513e0 0x7f1070051820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.797+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:2 13.10 13:08000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10700513e0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.797+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f10700513e0 0x7f1070051820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.797+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f10700513e0 0x7f1070051820 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.799+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(2 [pgnls start_epoch 94 out=177b] v93'10 uv10 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x7f10680544b0 con 0x7f10700513e0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.799+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f10680580a0 0x7f1068056d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.799+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- osd_op(unknown.0.0:3 13.8 13:10000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680580a0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.799+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f10680580a0 0x7f1068056d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.799+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f10680580a0 0x7f1068056d00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.801+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.2 v2:172.21.15.99:6816/739656799 1 ==== osd_op_reply(3 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f106c008c70 con 0x7f10680580a0 2024-03-04T10:10:20.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.801+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:4 13.18 13:18000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x55e6ae08cc50 2024-03-04T10:10:20.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.801+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== osd_op_reply(4 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.801+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f107005bc40 0x7f107005a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.801+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:5 13.4 13:20000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107005bc40 2024-03-04T10:10:20.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.802+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f107005bc40 0x7f107005a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.802+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f107005bc40 0x7f107005a380 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.804+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(5 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.804+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f107006d410 0x7f107006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.804+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:6 13.14 13:28000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006d410 2024-03-04T10:10:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.804+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f107006d410 0x7f107006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.805+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f107006d410 0x7f107006d830 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.805+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(6 [pgnls start_epoch 94 out=149b] v93'20 uv20 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x7f10680544b0 con 0x7f107006d410 2024-03-04T10:10:20.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.805+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:7 13.c 13:30000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10700513e0 2024-03-04T10:10:20.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.805+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== osd_op_reply(7 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f10680544b0 con 0x7f10700513e0 2024-03-04T10:10:20.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.805+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:8 13.1c 13:38000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107005bc40 2024-03-04T10:10:20.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.806+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== osd_op_reply(8 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.806+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f107006c8b0 0x7f107006b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.806+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:9 13.2 13:40000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006c8b0 2024-03-04T10:10:20.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.806+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f107006c8b0 0x7f107006b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.806+0000 7f10766b6640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f107006c8b0 0x7f107006b330 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(9 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:10 13.12 13:48000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006d410 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== osd_op_reply(10 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f10680544b0 con 0x7f107006d410 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- osd_op(unknown.0.0:11 13.a 13:50000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680580a0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.2 v2:172.21.15.99:6816/739656799 2 ==== osd_op_reply(11 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f106c008c70 con 0x7f10680580a0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:12 13.1a 13:58000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10700513e0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 3 ==== osd_op_reply(12 [pgnls start_epoch 94 out=49b] v92'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f10680600c0 0x7f1068076340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.807+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:13 13.6 13:60000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680600c0 2024-03-04T10:10:20.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.808+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f10680600c0 0x7f1068076340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.808+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f10680600c0 0x7f1068076340 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.808+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(13 [pgnls start_epoch 94 out=77b] v93'3 uv3 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.808+0000 7f10776b8640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f107007d940 0x7f107007c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.808+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- osd_op(unknown.0.0:14 13.16 13:68000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107007d940 2024-03-04T10:10:20.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.808+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f107007d940 0x7f107007c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.809+0000 7f1076eb7640 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f107007d940 0x7f107007c3c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.811+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.3 v2:172.21.15.99:6808/85197782 1 ==== osd_op_reply(14 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068075040 con 0x7f107007d940 2024-03-04T10:10:20.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.811+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:15 13.e 13:70000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x55e6ae08cc50 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.811+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 3 ==== osd_op_reply(15 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.811+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:16 13.1e 13:78000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680600c0 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.811+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== osd_op_reply(16 [pgnls start_epoch 94 out=49b] v92'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.811+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:17 13.1 13:80000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107005bc40 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.812+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 3 ==== osd_op_reply(17 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.812+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:18 13.11 13:88000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10700513e0 2024-03-04T10:10:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.812+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 4 ==== osd_op_reply(18 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.812+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:19 13.9 13:90000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006c8b0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.812+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== osd_op_reply(19 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.812+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- osd_op(unknown.0.0:20 13.19 13:98000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107007d940 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.3 v2:172.21.15.99:6808/85197782 2 ==== osd_op_reply(20 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068075040 con 0x7f107007d940 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:21 13.5 13:a0000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680600c0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 3 ==== osd_op_reply(21 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:22 13.15 13:a8000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680600c0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 4 ==== osd_op_reply(22 [pgnls start_epoch 94 out=97b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:23 13.d 13:b0000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680600c0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 5 ==== osd_op_reply(23 [pgnls start_epoch 94 out=74b] v92'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.813+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:24 13.1d 13:b8000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x55e6ae08cc50 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.814+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 4 ==== osd_op_reply(24 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.814+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:25 13.3 13:c0000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006d410 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.814+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 3 ==== osd_op_reply(25 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.814+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:26 13.13 13:c8000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x55e6ae08cc50 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.814+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 5 ==== osd_op_reply(26 [pgnls start_epoch 94 out=98b] v93'3 uv3 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.814+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:27 13.b 13:d0000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006c8b0 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.815+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 3 ==== osd_op_reply(27 [pgnls start_epoch 94 out=76b] v93'3 uv3 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.815+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:28 13.1b 13:d8000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f10680600c0 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.815+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 6 ==== osd_op_reply(28 [pgnls start_epoch 94 out=49b] v92'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.818 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.815+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:29 13.7 13:e0000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x55e6ae08cc50 2024-03-04T10:10:20.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.816+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 6 ==== osd_op_reply(29 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.816+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:30 13.17 13:e8000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107006c8b0 2024-03-04T10:10:20.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.816+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 4 ==== osd_op_reply(30 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.819 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.816+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:31 13.f 13:f0000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107005bc40 2024-03-04T10:10:20.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.816+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 4 ==== osd_op_reply(31 [pgnls start_epoch 94 out=73b] v92'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.816+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:32 13.1f 13:f8000000:%01:::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x7f107005b390 con 0x7f107005bc40 2024-03-04T10:10:20.820 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.817+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 5 ==== osd_op_reply(32 [pgnls start_epoch 94 out=101b] v93'6 uv6 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.817+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:33 13.10 13:09401a81:::601.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae0b3790 con 0x7f10700513e0 2024-03-04T10:10:20.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.817+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 5 ==== osd_op_reply(33 601.00000000 [read 0~0] v0'0 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.817+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:34 13.10 13:09401a81:::601.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae097a40 con 0x7f10700513e0 2024-03-04T10:10:20.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.817+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 6 ==== osd_op_reply(34 601.00000000 [getxattrs out=124b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.817+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:35 13.10 13:09401a81:::601.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae097de0 con 0x7f10700513e0 2024-03-04T10:10:20.821 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 7 ==== osd_op_reply(35 601.00000000 [omap-get-header out=274b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:36 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae098180 con 0x7f10700513e0 2024-03-04T10:10:20.822 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 8 ==== osd_op_reply(36 601.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:37 13.10 13:0bd6d154:::602.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae098520 con 0x7f10700513e0 2024-03-04T10:10:20.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 9 ==== osd_op_reply(37 602.00000000 [read 0~0] v0'0 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:38 13.10 13:0bd6d154:::602.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae156db0 con 0x7f10700513e0 2024-03-04T10:10:20.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 10 ==== osd_op_reply(38 602.00000000 [getxattrs out=124b] v0'0 uv6 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:39 13.10 13:0bd6d154:::602.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.818+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 11 ==== osd_op_reply(39 602.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:40 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f10700513e0 2024-03-04T10:10:20.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 12 ==== osd_op_reply(40 602.00000000 [omap-get-vals out=5b] v0'0 uv6 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:41 13.10 13:0cf6ad0f:::10000000000.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 13 ==== osd_op_reply(41 10000000000.00000000 [read 0~0] v0'0 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.825 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:42 13.10 13:0cf6ad0f:::10000000000.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 14 ==== osd_op_reply(42 10000000000.00000000 [getxattrs out=124b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+124 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:43 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.826 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 15 ==== osd_op_reply(43 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.827 INFO:teuthology.orchestra.run.smithi099.stdout: ��*$- 601.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray1 ���� �� ((88���� �� ��*$- 602.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray2 ���� �� ((88���� �� ��2,510000000000.00000000�������������������!�������������������������� ���~_layout��������_parent>8subdir ���� ��  ( 2024-03-04T10:10:20.827 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{( 2024-03-04T10:10:20.827 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8` 2024-03-04T10:10:20.827 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.819+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:44 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f10700513e0 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 16 ==== osd_op_reply(44 10000000000.00000000 [omap-get-vals out=586b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+586 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H���� ��OI sixmegs_head1i"� 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H���@@`�������� 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{(8` 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.828 INFO:teuthology.orchestra.run.smithi099.stdout:�������� 2024-03-04T10:10:20.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:45 13.10 13:0d82a743:::600.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 17 ==== osd_op_reply(45 600.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:46 13.10 13:0d82a743:::600.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.829 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 18 ==== osd_op_reply(46 600.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:47 13.10 13:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 19 ==== osd_op_reply(47 600.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.830 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:48 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f10700513e0 2024-03-04T10:10:20.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 20 ==== osd_op_reply(48 600.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.820+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:49 13.10 13:0d89b25e:::603.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 21 ==== osd_op_reply(49 603.00000000 [read 0~0] v0'0 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.831 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:50 13.10 13:0d89b25e:::603.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 22 ==== osd_op_reply(50 603.00000000 [getxattrs out=124b] v0'0 uv8 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:51 13.10 13:0d89b25e:::603.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f10700513e0 2024-03-04T10:10:20.832 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 23 ==== osd_op_reply(51 603.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:52 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f10700513e0 2024-03-04T10:10:20.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.1 v2:172.21.15.99:6824/3369025248 24 ==== osd_op_reply(52 603.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f10700513e0 2024-03-04T10:10:20.833 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.821+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:53 13.14 13:287034c4:::1.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.834 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������ �� ��*$- 600.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray0 ���� �� ((88���� �� ��*$- 603.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray3 ���� �� ((88���� �� ��.(11.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)� ��e���8�C@@�������� ��e���8( ��e���88` 2024-03-04T10:10:20.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 4 ==== osd_op_reply(53 1.00000000.inode [read 0~582 out=582b] v0'0 uv16 ondisk = 0) v8 ==== 160+0+582 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.834 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:54 13.14 13:287034c4:::1.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.834 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8���e��d2���e��d2����NH���e��d2������������ �� 2024-03-04T10:10:20.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 5 ==== osd_op_reply(54 1.00000000.inode [getxattrs out=4b] v0'0 uv16 ondisk = 0) v8 ==== 160+0+4 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.835 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:55 13.14 13:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.835 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 6 ==== osd_op_reply(55 1.00000000.inode [omap-get-header] v0'0 uv16 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:56 13.14 13:287034c4:::1.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f107006d410 2024-03-04T10:10:20.836 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 7 ==== osd_op_reply(56 1.00000000.inode [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 160+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:57 13.14 13:292cf221:::200.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.822+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 8 ==== osd_op_reply(57 200.00000000 [read 0~90 out=90b] v0'0 uv20 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:58 13.14 13:292cf221:::200.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.837 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 9 ==== osd_op_reply(58 200.00000000 [getxattrs out=4b] v0'0 uv20 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.838 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��*$- 200.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@�>@�>@�F@@@ ���� �� 2024-03-04T10:10:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:59 13.14 13:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.838 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 10 ==== osd_op_reply(59 200.00000000 [omap-get-header] v0'0 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:60 13.14 13:292cf221:::200.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f107006d410 2024-03-04T10:10:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 11 ==== osd_op_reply(60 200.00000000 [omap-get-vals out=5b] v0'0 uv20 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.839 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.823+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:61 13.14 13:2e2fa760:::200.00000001:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��*$- 200.00000001�������������������!�������������������������� ���F�F�F�F��XS&YA1e@��XS&YA1h^T���e(ss22 ((88 2 ( 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:( 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:8 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:���eC�o28 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:���eC�o2 ��������@0@��XS&YA1�� 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:� ��esp�,��51!��� entity_id0hostname smithi099kernel_version5.14.0-427.el9.x86_64root/�{ ����@��XS&YA1��� ��e���.n���������@��XS&YA1 ��e���.setattr� C��������� ��e��.�C@@�����������e��d2(8 ��e��.8���e��d2���e��d2����NH���e��d2�����������������X@��XS&YA1�� ��e���8mkdir�p ( ��e���8( ��e���88 ��e���88 ��e���8J 8subdir��������� ��e���8�A����������������� ��e���8 ��e���8(88�������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������2 ((88  C��������� ��e���8�C@@�������� ��e���8( ��e���88 ��e���88���e��d2���e��d2����NH���e��d2��������������������@��XS&YA1� 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:��e9W3openc�p ( ��e���8( ��e���88 2024-03-04T10:10:20.840 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8 ��e���8J 8subdir��������� 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�A����������������� 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ ��e���8( 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� ( 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{( 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{m [sixmegs��������� 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{���@@�������� 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�@(88�������� 2024-03-04T10:10:20.841 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0�������������������@��XS&YA1 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e��� 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:cap update�p ( ��e���8( ��e���88  2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π8 ��e���8J 8subdir��������� 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�A����������������� 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ ��e���8( 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8  2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π8  2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π�������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� ( 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{( 2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8  2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π8  2024-03-04T10:10:20.842 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Πm [sixmegs��������� 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π���@@ �������� 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{��(8  2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π8  2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�Π�������� 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������������@��XS&YA1 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��eb\? 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:cap update�p  ( ��e���8( ��e���88P 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e+�8 ��e���8J 8subdir��������� 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�A����������������� 2024-03-04T10:10:20.843 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ ��e���8( 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8P 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+�8P 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+��������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0��������� ( 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{( 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8P 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+�8P 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+�m [sixmegs��������� 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+����@@P�������� 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+� 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{��(8P 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+�8P 2024-03-04T10:10:20.844 INFO:teuthology.orchestra.run.smithi099.stdout:��e+��������� 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������������ @��XS&YA1] 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:S 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:I 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout: ��e��x3 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:cap update 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:p  ( ��e���8( ��e���88` 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e��HJ 8subdir�������� 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:� 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�A����������������� 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ ��e���8( 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8` 2024-03-04T10:10:20.845 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:�������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0���������  ( 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{( 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8` 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��Hm [sixmegs��������� 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H���@@`�������� 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�(8` 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H�������� 2024-03-04T10:10:20.846 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0�������� C��������� ��e���8�C@@�������� ��e���8( ��e���88` 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8���e��d2���e��d2����NH���e��d2�����������������)@��XS&YA1�� 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:� ��e��3��51!������3@��XS&YA1 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout: ��e��4check_inode_max_size� p  ( ��e���8( ��e���88` 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e��HJ 8subdir�������� � 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�A����������������� 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ ��e���8( 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8` 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H �������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0���������  ( 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{( 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8` 2024-03-04T10:10:20.847 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H� 5sixmegs�������� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H���@@`�������� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{(8` 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:�������� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0�������� 5sixmegs�������� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H���@@`�������� 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.848 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{(8` 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stdout:�������� 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{����security.selinux%unconfined_u:object_r:unlabeled_t:s0����������������34@��XS&YA18.$�a>@��XS&YA1h^T ��e�-2  ( ��e���8( ��e���88` 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 12 ==== osd_op_reply(61 200.00000001 [read 0~18075 out=18075b] v0'0 uv17 ondisk = 0) v8 ==== 156+0+18075 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.849 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H 2 ( 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:( 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:8 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:���eC�o28 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:���eC�o2 ��������@�>@��XS&YA1^TJ ��e���scatter_writebehind2 ( ��e���8( ��e���88` 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H  C��������� ��e���8�C@@�������� ��e���8( ��e���88` 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:62 13.14 13:2e2fa760:::200.00000001:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 13 ==== osd_op_reply(62 200.00000001 [getxattrs out=4b] v0'0 uv17 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.851 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8���e��d2���e��d2����NH���e��d2����������������)B@���� �� 2024-03-04T10:10:20.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:63 13.14 13:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 14 ==== osd_op_reply(63 200.00000001 [omap-get-header] v0'0 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.851 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:64 13.14 13:2e2fa760:::200.00000001:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f107006d410 2024-03-04T10:10:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 15 ==== osd_op_reply(64 200.00000001 [omap-get-vals out=5b] v0'0 uv17 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:65 13.14 13:2f55791f:::606.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 16 ==== osd_op_reply(65 606.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.824+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:66 13.14 13:2f55791f:::606.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.825+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 17 ==== osd_op_reply(66 606.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.825+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:67 13.14 13:2f55791f:::606.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107006d410 2024-03-04T10:10:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.825+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 18 ==== osd_op_reply(67 606.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.825+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:68 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158160 con 0x7f107006d410 2024-03-04T10:10:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.825+0000 7f1076eb7640 1 -- 172.21.15.99:0/1488554727 <== osd.0 v2:172.21.15.99:6800/224245114 19 ==== osd_op_reply(68 606.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1068075040 con 0x7f107006d410 2024-03-04T10:10:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.825+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:69 13.1c 13:3ed09add:::607.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157150 con 0x7f107005bc40 2024-03-04T10:10:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.826+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 6 ==== osd_op_reply(69 607.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.826+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:70 13.1c 13:3ed09add:::607.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae1574f0 con 0x7f107005bc40 2024-03-04T10:10:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.826+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 7 ==== osd_op_reply(70 607.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.826+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:71 13.1c 13:3ed09add:::607.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157890 con 0x7f107005bc40 2024-03-04T10:10:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.826+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 8 ==== osd_op_reply(71 607.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.826+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:72 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158c40 con 0x7f107005bc40 2024-03-04T10:10:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.827+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 9 ==== osd_op_reply(72 607.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.827+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:73 13.6 13:654134d2:::mds0_openfiles.0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae157c30 con 0x7f10680600c0 2024-03-04T10:10:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.827+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 7 ==== osd_op_reply(73 mds0_openfiles.0 [read 0~0] v0'0 uv3 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.827+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:74 13.6 13:654134d2:::mds0_openfiles.0:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158050 con 0x7f10680600c0 2024-03-04T10:10:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.827+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 8 ==== osd_op_reply(74 mds0_openfiles.0 [getxattrs out=4b] v0'0 uv3 ondisk = 0) v8 ==== 160+0+4 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.858 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��*$- 606.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray6 ���� �� ((88���� �� ��*$- 607.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray7 ���� �� ((88���� �� ��.(1mds0_openfiles.0�������������������!�������������������������� �� 2024-03-04T10:10:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.827+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:75 13.6 13:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 9 ==== osd_op_reply(75 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv3 ondisk = 0) v8 ==== 160+0+42 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:76 13.6 13:654134d2:::mds0_openfiles.0:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 10 ==== osd_op_reply(76 mds0_openfiles.0 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 160+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:77 13.e 13:717a0223:::608.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 7 ==== osd_op_reply(77 608.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:78 13.e 13:717a0223:::608.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.829+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 8 ==== osd_op_reply(78 608.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.830+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:79 13.e 13:717a0223:::608.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.830+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 9 ==== osd_op_reply(79 608.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.830+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:80 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.830+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 10 ==== osd_op_reply(80 608.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.830+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:81 13.15 13:a93a17c2:::604.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.831+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 11 ==== osd_op_reply(81 604.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.831+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:82 13.15 13:a93a17c2:::604.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.831+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 12 ==== osd_op_reply(82 604.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.831+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:83 13.15 13:a93a17c2:::604.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.831+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 13 ==== osd_op_reply(83 604.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.831+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:84 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.832+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 14 ==== osd_op_reply(84 604.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.832+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:85 13.15 13:aa448500:::500.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.832+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 15 ==== osd_op_reply(85 500.00000000 [read 0~90 out=90b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.832+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:86 13.15 13:aa448500:::500.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.833+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 16 ==== osd_op_reply(86 500.00000000 [getxattrs out=4b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.868 INFO:teuthology.orchestra.run.smithi099.stdout:���� ��4.*ceph fs volume v011 ���� �� ��*$- 608.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray8 ���� �� ((88���� �� ��*$- 604.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray4 ���� �� ((88���� �� ��*$- 500.00000000�������������������!�������������������������� ��tnZZTceph fs volume v011@@@@@@ ���� �� 2024-03-04T10:10:20.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.833+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:87 13.15 13:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.833+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 17 ==== osd_op_reply(87 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.869 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.833+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:88 13.15 13:aa448500:::500.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.833+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 18 ==== osd_op_reply(88 500.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.833+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:89 13.d 13:b50e409b:::mds_snaptable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 19 ==== osd_op_reply(89 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:90 13.d 13:b50e409b:::mds_snaptable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 20 ==== osd_op_reply(90 mds_snaptable [getxattrs out=4b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+4 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.871 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��+%. mds_snaptable�������������������!�������������������������� ��`ZFF4���� �� 2024-03-04T10:10:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:91 13.d 13:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 21 ==== osd_op_reply(91 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.872 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:92 13.d 13:b50e409b:::mds_snaptable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f10680600c0 2024-03-04T10:10:20.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.4 v2:172.21.15.162:6800/3079837853 22 ==== osd_op_reply(92 mds_snaptable [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f10680600c0 2024-03-04T10:10:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.834+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:93 13.1d 13:b871830b:::605.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.835+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 11 ==== osd_op_reply(93 605.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.835+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:94 13.1d 13:b871830b:::605.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.835+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 12 ==== osd_op_reply(94 605.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.835+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:95 13.1d 13:b871830b:::605.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.835+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 13 ==== osd_op_reply(95 605.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.835+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:96 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.836+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 14 ==== osd_op_reply(96 605.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.836+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:97 13.13 13:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.836+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 15 ==== osd_op_reply(97 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.836+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:98 13.13 13:c91d4a1d:::mds0_inotable:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 16 ==== osd_op_reply(98 mds0_inotable [getxattrs out=4b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+4 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.876 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��*$- 605.00000000�������������������!�������������������������� ���~_layout��������_parent>8stray5 ���� �� ((88���� �� ��+%. mds0_inotable�������������������!�������������������������� ��<6""��������� �� 2024-03-04T10:10:20.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:99 13.13 13:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.877 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 17 ==== osd_op_reply(99 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:100 13.13 13:c91d4a1d:::mds0_inotable:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 18 ==== osd_op_reply(100 mds0_inotable [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+5 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:101 13.13 13:cd5a64a3:::100.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.837+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 19 ==== osd_op_reply(101 100.00000000 [read 0~0] v0'0 uv1 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:102 13.13 13:cd5a64a3:::100.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 20 ==== osd_op_reply(102 100.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.879 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��*$- 100.00000000�������������������!�������������������������� �� 2024-03-04T10:10:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:103 13.13 13:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 21 ==== osd_op_reply(103 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.879 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ( 2024-03-04T10:10:20.879 INFO:teuthology.orchestra.run.smithi099.stdout:( 2024-03-04T10:10:20.880 INFO:teuthology.orchestra.run.smithi099.stdout:8 2024-03-04T10:10:20.880 INFO:teuthology.orchestra.run.smithi099.stdout:���eC�o28 2024-03-04T10:10:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:104 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x55e6ae08cc50 2024-03-04T10:10:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 22 ==== osd_op_reply(104 100.00000000 [omap-get-vals out=5195b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.881 INFO:teuthology.orchestra.run.smithi099.stdout:���eC�o2���� ��PJ 2024-03-04T10:10:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.838+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:105 13.b 13:d0630e4c:::mds0_sessionmap:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158490 con 0x7f107006c8b0 2024-03-04T10:10:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.839+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 5 ==== osd_op_reply(105 mds0_sessionmap [read 0~0] v0'0 uv3 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.839+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:106 13.b 13:d0630e4c:::mds0_sessionmap:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158880 con 0x7f107006c8b0 2024-03-04T10:10:20.881 INFO:teuthology.orchestra.run.smithi099.stdout: stray0_head�i������ef�g2@A�������������������ef�g2(8���ef�g28���ef�g2���ef�g2������������ stray1_head�i������e��h2@A�������������������e��h2(8���e��h28���e��h2���e��h2������������ stray2_head�i������eL�i2@A�������������������eL�i2(8���eL�i28���eL�i2���eL�i2������������ stray3_head�i������e��j2@A�������������������e��j2(8���e��j28���e��j2���e��j2������������ stray4_head�i������e�ik2@A�������������������e�ik2(8���e�ik28���e�ik2���e�ik2������������ stray5_head�i������e�Gl2@A�������������������e�Gl2(8���e�Gl28���e�Gl2���e�Gl2������������ stray6_head�i������e�}m2@A�������������������e�}m2(8���e�}m28���e�}m2���e�}m2������������ stray7_head�i������eH)n2@A�������������������eH)n2(8���eH)n28���eH)n2���eH)n2������������ stray8_head�i������e��n2@A�������������������e��n2(8���e��n28���e��n2���e��n2������������ stray9_head�i��� ���eC�o2@A�������������������eC�o2(8���eC�o28���eC�o2���eC�o2���������������� �� ��-'0mds0_sessionmap�������������������!�������������������������� �� 2024-03-04T10:10:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.839+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 6 ==== osd_op_reply(106 mds0_sessionmap [getxattrs out=4b] v0'0 uv3 ondisk = 0) v8 ==== 159+0+4 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.839+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:107 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae158fe0 con 0x7f107006c8b0 2024-03-04T10:10:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.840+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 7 ==== osd_op_reply(107 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) v8 ==== 159+0+14 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.840+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:108 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae15a390 con 0x7f107006c8b0 2024-03-04T10:10:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.840+0000 7f10766b6640 1 -- 172.21.15.99:0/1488554727 <== osd.5 v2:172.21.15.162:6808/1764948489 8 ==== osd_op_reply(108 mds0_sessionmap [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 159+0+5 (crc 0 0 0) 0x7f106c008c70 con 0x7f107006c8b0 2024-03-04T10:10:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.840+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:109 13.7 13:e0b41d06:::609.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x55e6ae08cc50 2024-03-04T10:10:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.840+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 23 ==== osd_op_reply(109 609.00000000 [read 0~0] v0'0 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.840+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:110 13.7 13:e0b41d06:::609.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x55e6ae08cc50 2024-03-04T10:10:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.841+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 24 ==== osd_op_reply(110 609.00000000 [getxattrs out=124b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+124 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.841+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:111 13.7 13:e0b41d06:::609.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x55e6ae08cc50 2024-03-04T10:10:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.841+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 25 ==== osd_op_reply(111 609.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.841+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:112 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x55e6ae08cc50 2024-03-04T10:10:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.841+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.7 v2:172.21.15.162:6816/2710319333 26 ==== osd_op_reply(112 609.00000000 [omap-get-vals out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x55e6ae08cc50 2024-03-04T10:10:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.841+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:113 13.f 13:f1f69726:::400.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.842+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 10 ==== osd_op_reply(113 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.842+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:114 13.f 13:f1f69726:::400.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.843+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 11 ==== osd_op_reply(114 400.00000000 [getxattrs out=4b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+4 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.887 INFO:teuthology.orchestra.run.smithi099.stdout:���� ������ �� ��*$- 609.00000000�������������������!�������������������������� ���~_layout��������_parent>8 stray9 ���� �� ((88���� �� ��*$- 400.00000000�������������������!�������������������������� ��0*���� �� 2024-03-04T10:10:20.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.843+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:115 13.f 13:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.887 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.843+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 12 ==== osd_op_reply(115 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.843+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:116 13.f 13:f1f69726:::400.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.844+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 13 ==== osd_op_reply(116 400.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.844+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:117 13.1f 13:fe07bba1:::100.00000000.inode:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 14 ==== osd_op_reply(117 100.00000000.inode [read 0~582 out=582b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+582 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.889 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��0*3100.00000000.inode�������������������!�������������������������� ��`ZFFceph fs volume v011)����e�Zg2@A�������������������e�Zg2( 2024-03-04T10:10:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:118 13.1f 13:fe07bba1:::100.00000000.inode:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 15 ==== osd_op_reply(118 100.00000000.inode [getxattrs out=4b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+4 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.890 INFO:teuthology.orchestra.run.smithi099.stdout:8 ���eC�o28 ���eC�o2���e�Zg2����NH������������ �� 2024-03-04T10:10:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:119 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.890 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� 2024-03-04T10:10:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 16 ==== osd_op_reply(119 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:120 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae15a730 con 0x7f107005bc40 2024-03-04T10:10:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 17 ==== osd_op_reply(120 100.00000000.inode [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+5 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.892 INFO:teuthology.orchestra.run.smithi099.stdout:���� �� ��("+ 2024-03-04T10:10:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.846+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:121 13.1f 13:ff5b34d6:::1.00000000:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.847+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 18 ==== osd_op_reply(121 1.00000000 [read 0~0] v0'0 uv6 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.847+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:122 13.1f 13:ff5b34d6:::1.00000000:head [getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.847+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 19 ==== osd_op_reply(122 1.00000000 [getxattrs out=92b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+92 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.847+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:123 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.847+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 20 ==== osd_op_reply(123 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.894 INFO:teuthology.orchestra.run.smithi099.stdout:1.00000000�������������������!�������������������������� ��d^_layout@@_parent ���� ��  ( ��e���8( ��e���88` 2024-03-04T10:10:20.894 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.847+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:124 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55e6ae159380 con 0x7f107005bc40 2024-03-04T10:10:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 <== osd.6 v2:172.21.15.162:6824/1448511037 21 ==== osd_op_reply(124 1.00000000 [omap-get-vals out=585b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+585 (crc 0 0 0) 0x7f1070052960 con 0x7f107005bc40 2024-03-04T10:10:20.895 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H���� ��NH subdir_head1i"� 2024-03-04T10:10:20.895 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{�A����������������� 2024-03-04T10:10:20.895 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{ ��e���8( 2024-03-04T10:10:20.896 INFO:teuthology.orchestra.run.smithi099.stdout:��e�{8` 2024-03-04T10:10:20.896 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H8` 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f107006d410 msgr2=0x7f107006d830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f107006d410 0x7f107006d830 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f10700513e0 msgr2=0x7f1070051820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f10700513e0 0x7f1070051820 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f10680580a0 msgr2=0x7f1068056d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f10680580a0 0x7f1068056d00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f107007d940 msgr2=0x7f107007c3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 reap_dead start 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f107007d940 0x7f107007c3c0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f10680600c0 msgr2=0x7f1068076340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f10680600c0 0x7f1068076340 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f107006c8b0 msgr2=0x7f107006b330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f107006c8b0 0x7f107006b330 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f107005bc40 msgr2=0x7f107005a380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f107005bc40 0x7f107005a380 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55e6ae08cc50 msgr2=0x55e6ae0b3250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55e6ae08cc50 0x55e6ae0b3250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10776b8640 1 -- 172.21.15.99:0/1488554727 reap_dead start 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1064069eb0 msgr2=0x7f106406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1064069eb0 0x7f106406c370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x55e6ae020fd0 tx=0x7f106c012000 comp rx=0 tx=0).stop 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 msgr2=0x55e6ae008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 0x55e6ae008d20 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1068001c40 tx=0x7f106800b6c0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 shutdown_connections 2024-03-04T10:10:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1064069eb0 0x7f106406c370 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 --2- 172.21.15.99:0/1488554727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6adfd7c90 0x55e6ae008d20 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 >> 172.21.15.99:0/1488554727 conn(0x55e6adfd9400 msgr2=0x55e6ae0b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 shutdown_connections 2024-03-04T10:10:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.848+0000 7f10793b7f40 1 -- 172.21.15.99:0/1488554727 wait complete. 2024-03-04T10:10:20.900 INFO:teuthology.orchestra.run.smithi099.stdout:��e��H �������� ��e���8����security.selinux%unconfined_u:object_r:unlabeled_t:s0������������ �� �� 2024-03-04T10:10:20.900 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata ls 2024-03-04T10:10:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.976+0000 7f3c4ca37f40 1 Processor -- start 2024-03-04T10:10:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.976+0000 7f3c4ca37f40 1 -- start start 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4ca37f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 0x55cb859880e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4ca37f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a5f10 0x55cb8595f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4ca37f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb859a62f0 0x55cb85a2be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4ca37f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb859a7d20 con 0x55cb8598cca0 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4ca37f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb85904b30 con 0x55cb859a5f10 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4ca37f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb85a35c20 con 0x55cb859a62f0 2024-03-04T10:10:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 0x55cb859880e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 0x55cb859880e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55100/0 (socket says 172.21.15.99:55100) 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 -- 172.21.15.99:0/3757278125 learned_addr learned my addr 172.21.15.99:0/3757278125 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c494de640 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a5f10 0x55cb8595f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb859a62f0 0x55cb85a2be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 -- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb859a62f0 msgr2=0x55cb85a2be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb859a62f0 0x55cb85a2be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 -- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a5f10 msgr2=0x55cb8595f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a5f10 0x55cb8595f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.977+0000 7f3c49cdf640 1 -- 172.21.15.99:0/3757278125 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb85988c00 con 0x55cb8598cca0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 0x55cb859880e0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c001db0 tx=0x7f3c3c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d60d9676d6bbca server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c3bfff640 1 -- 172.21.15.99:0/3757278125 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c00bde0 con 0x55cb8598cca0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c3bfff640 1 -- 172.21.15.99:0/3757278125 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c00f400 con 0x55cb8598cca0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c3bfff640 1 -- 172.21.15.99:0/3757278125 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c00f7f0 con 0x55cb8598cca0 2024-03-04T10:10:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/3757278125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 msgr2=0x55cb859880e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 0x55cb859880e0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c001db0 tx=0x7f3c3c00b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/3757278125 shutdown_connections 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb859a62f0 0x55cb85a2be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a5f10 0x55cb8595f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/3757278125 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb8598cca0 0x55cb859880e0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/3757278125 >> 172.21.15.99:0/3757278125 conn(0x55cb8595d3e0 msgr2=0x55cb8595d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/3757278125 shutdown_connections 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.978+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/3757278125 wait complete. 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.979+0000 7f3c4ca37f40 1 Processor -- start 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.979+0000 7f3c4ca37f40 1 -- start start 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.979+0000 7f3c4ca37f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb8598cca0 0x55cb85a1b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.979+0000 7f3c4ca37f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb859a5f10 0x55cb85a137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.979+0000 7f3c4ca37f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a62f0 0x55cb8598bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c4ca37f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb8595fff0 con 0x55cb859a5f10 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c4ca37f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb85a3d7b0 con 0x55cb859a62f0 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c4ca37f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb85988ec0 con 0x55cb8598cca0 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb859a5f10 0x55cb85a137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb859a5f10 0x55cb85a137a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55108/0 (socket says 172.21.15.99:55108) 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 learned_addr learned my addr 172.21.15.99:0/1896003644 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb8598cca0 0x55cb85a1b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a62f0 0x55cb8598bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb8598cca0 msgr2=0x55cb85a1b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55cb8598cca0 0x55cb85a1b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a62f0 msgr2=0x55cb8598bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cb859a62f0 0x55cb8598bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb85a1be60 con 0x55cb859a5f10 2024-03-04T10:10:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb859a5f10 0x55cb85a137a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000a6b0 tx=0x7f3c40002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c3affd640 1 -- 172.21.15.99:0/1896003644 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40068600 con 0x55cb859a5f10 2024-03-04T10:10:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c3affd640 1 -- 172.21.15.99:0/1896003644 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c400687a0 con 0x55cb859a5f10 2024-03-04T10:10:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cb85a266d0 con 0x55cb859a5f10 2024-03-04T10:10:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c3affd640 1 -- 172.21.15.99:0/1896003644 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c40068b90 con 0x55cb859a5f10 2024-03-04T10:10:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.980+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb85a26c90 con 0x55cb859a5f10 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c3affd640 1 -- 172.21.15.99:0/1896003644 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3c4005e400 con 0x55cb859a5f10 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c3affd640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2c069ee0 0x7f3c2c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c3affd640 1 -- 172.21.15.99:0/1896003644 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3c400cdda0 con 0x55cb859a5f10 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2c069ee0 0x7f3c2c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55cb85a27490 0x55cb85a27870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x55cb85a27490 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.982+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55cb85a27490 0x55cb85a27870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.983+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2c069ee0 0x7f3c2c06c3a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c001d50 tx=0x7f3c3c009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.983+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55cb85a27490 0x55cb85a27870 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.983+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x55cb85a27490 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.983+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f3c44014040 0x7f3c440133b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.983+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:2 13.10 13:08000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44014040 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.983+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f3c44014040 0x7f3c440133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f3c44014040 0x7f3c440133b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(2 [pgnls start_epoch 94 out=177b] v93'10 uv10 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x7f3c3c00f990 con 0x7f3c44014040 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f3c3c029040 0x7f3c3c029420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- osd_op(unknown.0.0:3 13.8 13:10000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c029040 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f3c3c029040 0x7f3c3c029420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f3c3c029040 0x7f3c3c029420 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.984+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 <== osd.2 v2:172.21.15.99:6816/739656799 1 ==== osd_op_reply(3 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c40065020 con 0x7f3c3c029040 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.985+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:4 13.18 13:18000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x55cb85a27490 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.985+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== osd_op_reply(4 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x55cb85a27490 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.985+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f3c44012340 0x7f3c440127a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.985+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:5 13.4 13:20000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44012340 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.985+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f3c44012340 0x7f3c440127a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.985+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f3c44012340 0x7f3c440127a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(5 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c44012340 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f3c44025040 0x7f3c44022a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:6 13.14 13:28000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025040 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f3c44025040 0x7f3c44022a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f3c44025040 0x7f3c44022a00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(6 [pgnls start_epoch 94 out=149b] v93'20 uv20 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44025040 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.986+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:7 13.c 13:30000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44014040 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.987+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== osd_op_reply(7 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44014040 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.987+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:8 13.1c 13:38000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44012340 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.987+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== osd_op_reply(8 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c44012340 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.987+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f3c44025aa0 0x7f3c440243b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.987+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:9 13.2 13:40000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025aa0 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.987+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f3c44025aa0 0x7f3c440243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c494de640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f3c44025aa0 0x7f3c440243b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(9 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c40065020 con 0x7f3c44025aa0 2024-03-04T10:10:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:10 13.12 13:48000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== osd_op_reply(10 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44025040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- osd_op(unknown.0.0:11 13.a 13:50000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c029040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 <== osd.2 v2:172.21.15.99:6816/739656799 2 ==== osd_op_reply(11 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c40065020 con 0x7f3c3c029040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.988+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:12 13.1a 13:58000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44014040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.989+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.1 v2:172.21.15.99:6824/3369025248 3 ==== osd_op_reply(12 [pgnls start_epoch 94 out=49b] v92'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44014040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.989+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f3c3c039040 0x7f3c3c039aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.989+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:13 13.6 13:60000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c039040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.989+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f3c3c039040 0x7f3c3c039aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.989+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f3c3c039040 0x7f3c3c039aa0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(13 [pgnls start_epoch 94 out=77b] v93'3 uv3 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c3c039040 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c4a4e0640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f3c44024bb0 0x7f3c440343c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- osd_op(unknown.0.0:14 13.16 13:68000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44024bb0 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f3c44024bb0 0x7f3c440343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c49cdf640 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f3c44024bb0 0x7f3c440343c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.3 v2:172.21.15.99:6808/85197782 1 ==== osd_op_reply(14 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44024bb0 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.990+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:15 13.e 13:70000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x55cb85a27490 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.7 v2:172.21.15.162:6816/2710319333 3 ==== osd_op_reply(15 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f3c44002960 con 0x55cb85a27490 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:16 13.1e 13:78000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c039040 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== osd_op_reply(16 [pgnls start_epoch 94 out=49b] v92'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c3c039040 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:17 13.1 13:80000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44012340 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.6 v2:172.21.15.162:6824/1448511037 3 ==== osd_op_reply(17 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c44012340 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:18 13.11 13:88000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44014040 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.1 v2:172.21.15.99:6824/3369025248 4 ==== osd_op_reply(18 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44014040 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.991+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:19 13.9 13:90000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025aa0 2024-03-04T10:10:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.992+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== osd_op_reply(19 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c40065020 con 0x7f3c44025aa0 2024-03-04T10:10:21.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.992+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- osd_op(unknown.0.0:20 13.19 13:98000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44024bb0 2024-03-04T10:10:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.992+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.3 v2:172.21.15.99:6808/85197782 2 ==== osd_op_reply(20 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44024bb0 2024-03-04T10:10:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.992+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:21 13.5 13:a0000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c039040 2024-03-04T10:10:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.992+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.4 v2:172.21.15.162:6800/3079837853 3 ==== osd_op_reply(21 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c3c039040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.992+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:22 13.15 13:a8000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c039040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.4 v2:172.21.15.162:6800/3079837853 4 ==== osd_op_reply(22 [pgnls start_epoch 94 out=97b] v92'3 uv3 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c3c039040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:23 13.d 13:b0000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c039040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.4 v2:172.21.15.162:6800/3079837853 5 ==== osd_op_reply(23 [pgnls start_epoch 94 out=74b] v92'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c3c039040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:24 13.1d 13:b8000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x55cb85a27490 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.7 v2:172.21.15.162:6816/2710319333 4 ==== osd_op_reply(24 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f3c44002960 con 0x55cb85a27490 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:25 13.3 13:c0000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 <== osd.0 v2:172.21.15.99:6800/224245114 3 ==== osd_op_reply(25 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c3c036040 con 0x7f3c44025040 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.993+0000 7f3c49cdf640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:26 13.13 13:c8000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x55cb85a27490 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.994+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.7 v2:172.21.15.162:6816/2710319333 5 ==== osd_op_reply(26 [pgnls start_epoch 94 out=98b] v93'3 uv3 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x7f3c44002960 con 0x55cb85a27490 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.994+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:27 13.b 13:d0000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025aa0 2024-03-04T10:10:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.994+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 <== osd.5 v2:172.21.15.162:6808/1764948489 3 ==== osd_op_reply(27 [pgnls start_epoch 94 out=76b] v93'3 uv3 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x7f3c40065020 con 0x7f3c44025aa0 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.994+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:28 13.1b 13:d8000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c3c039040 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.994+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.4 v2:172.21.15.162:6800/3079837853 6 ==== osd_op_reply(28 [pgnls start_epoch 94 out=49b] v92'1 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c3c039040 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.994+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:29 13.7 13:e0000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x55cb85a27490 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.995+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.7 v2:172.21.15.162:6816/2710319333 6 ==== osd_op_reply(29 [pgnls start_epoch 94 out=73b] v92'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f3c44002960 con 0x55cb85a27490 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.995+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:30 13.17 13:e8000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44025aa0 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.995+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 <== osd.5 v2:172.21.15.162:6808/1764948489 4 ==== osd_op_reply(30 [pgnls start_epoch 94 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f3c40065020 con 0x7f3c44025aa0 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.995+0000 7f3c494de640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:31 13.f 13:f0000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44012340 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.995+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.6 v2:172.21.15.162:6824/1448511037 4 ==== osd_op_reply(31 [pgnls start_epoch 94 out=73b] v92'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c44012340 2024-03-04T10:10:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.995+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:32 13.1f 13:f8000000::::head [pgnls start_epoch 94 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e94) v9 -- 0x55cb85a27db0 con 0x7f3c44012340 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:601.00000000 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:602.00000000 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:10000000000.00000000 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:600.00000000 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:603.00000000 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:1.00000000.inode 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:200.00000000 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:200.00000001 2024-03-04T10:10:21.005 INFO:teuthology.orchestra.run.smithi099.stdout:606.00000000 2024-03-04T10:10:21.006 INFO:teuthology.orchestra.run.smithi099.stdout:607.00000000 2024-03-04T10:10:21.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 <== osd.6 v2:172.21.15.162:6824/1448511037 5 ==== osd_op_reply(32 [pgnls start_epoch 94 out=101b] v93'6 uv6 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x7f3c44002960 con 0x7f3c44012340 2024-03-04T10:10:21.006 INFO:teuthology.orchestra.run.smithi099.stdout:mds0_openfiles.0 2024-03-04T10:10:21.006 INFO:teuthology.orchestra.run.smithi099.stdout:608.00000000 2024-03-04T10:10:21.006 INFO:teuthology.orchestra.run.smithi099.stdout:604.00000000 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:500.00000000 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:mds_snaptable 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:605.00000000 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:mds0_inotable 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:100.00000000 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:mds0_sessionmap 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:609.00000000 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:400.00000000 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:100.00000000.inode 2024-03-04T10:10:21.007 INFO:teuthology.orchestra.run.smithi099.stdout:1.00000000 2024-03-04T10:10:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f3c44025040 msgr2=0x7f3c44022a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f3c44025040 0x7f3c44022a00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f3c44014040 msgr2=0x7f3c440133b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f3c44014040 0x7f3c440133b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f3c3c029040 msgr2=0x7f3c3c029420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f3c3c029040 0x7f3c3c029420 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f3c44024bb0 msgr2=0x7f3c440343c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7f3c44024bb0 0x7f3c440343c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 reap_dead start 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f3c3c039040 msgr2=0x7f3c3c039aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f3c3c039040 0x7f3c3c039aa0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 reap_dead start 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f3c44025aa0 msgr2=0x7f3c440243b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7f3c44025aa0 0x7f3c440243b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f3c44012340 msgr2=0x7f3c440127a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f3c44012340 0x7f3c440127a0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55cb85a27490 msgr2=0x55cb85a27870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55cb85a27490 0x55cb85a27870 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2c069ee0 msgr2=0x7f3c2c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3c2c069ee0 0x7f3c2c06c3a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c001d50 tx=0x7f3c3c009000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb859a5f10 msgr2=0x55cb85a137a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 --2- 172.21.15.99:0/1896003644 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55cb859a5f10 0x55cb85a137a0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000a6b0 tx=0x7f3c40002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4a4e0640 1 -- 172.21.15.99:0/1896003644 reap_dead start 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 shutdown_connections 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 >> 172.21.15.99:0/1896003644 conn(0x55cb8595d3e0 msgr2=0x55cb85989140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 shutdown_connections 2024-03-04T10:10:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:20.996+0000 7f3c4ca37f40 1 -- 172.21.15.99:0/1896003644 wait complete. 2024-03-04T10:10:21.012 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 10000000000.00000000 2024-03-04T10:10:21.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.088+0000 7f4ec5a40f40 1 Processor -- start 2024-03-04T10:10:21.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.088+0000 7f4ec5a40f40 1 -- start start 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec5a40f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 0x555b149c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec5a40f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 0x555b149c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec5a40f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0f310 0x555b14a93a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec5a40f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b14a10d40 con 0x555b14a0ef30 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec5a40f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1496db30 con 0x555b149f5cc0 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec5a40f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b14a9ec40 con 0x555b14a0f310 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 0x555b149c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 0x555b149c7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55126/0 (socket says 172.21.15.99:55126) 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 -- 172.21.15.99:0/2818896419 learned_addr learned my addr 172.21.15.99:0/2818896419 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 -- 172.21.15.99:0/2818896419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0f310 msgr2=0x555b14a93a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0f310 0x555b14a93a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 -- 172.21.15.99:0/2818896419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 msgr2=0x555b149c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 0x555b149c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 -- 172.21.15.99:0/2818896419 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b149c6bc0 con 0x555b14a0ef30 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec257d640 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 0x555b149c7d70 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8001d60 tx=0x7f4eb800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fa0858848e03f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec157b640 1 -- 172.21.15.99:0/2818896419 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb800bd80 con 0x555b14a0ef30 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec157b640 1 -- 172.21.15.99:0/2818896419 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb800f400 con 0x555b14a0ef30 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.089+0000 7f4ec157b640 1 -- 172.21.15.99:0/2818896419 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb800f7f0 con 0x555b14a0ef30 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/2818896419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 msgr2=0x555b149c7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 0x555b149c7d70 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8001d60 tx=0x7f4eb800b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/2818896419 shutdown_connections 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0f310 0x555b14a93a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0ef30 0x555b149c7d70 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/2818896419 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 0x555b149c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/2818896419 >> 172.21.15.99:0/2818896419 conn(0x555b149c6400 msgr2=0x555b14a91a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/2818896419 shutdown_connections 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/2818896419 wait complete. 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 Processor -- start 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.090+0000 7f4ec5a40f40 1 -- start start 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec5a40f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 0x555b14a8ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec5a40f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0ef30 0x555b14a8a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec5a40f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0f310 0x555b14a914d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec5a40f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b14a92240 con 0x555b14a0f310 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec5a40f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b149f09d0 con 0x555b149f5cc0 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec5a40f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b14aa6700 con 0x555b14a0ef30 2024-03-04T10:10:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0f310 0x555b14a914d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0f310 0x555b14a914d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55132/0 (socket says 172.21.15.99:55132) 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 -- 172.21.15.99:0/1387498676 learned_addr learned my addr 172.21.15.99:0/1387498676 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec2d7e640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 0x555b14a8ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec257d640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0ef30 0x555b14a8a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 -- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0ef30 msgr2=0x555b14a8a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b14a0ef30 0x555b14a8a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 -- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 msgr2=0x555b14a8ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b149f5cc0 0x555b14a8ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.091+0000 7f4ec357f640 1 -- 172.21.15.99:0/1387498676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b14a93370 con 0x555b14a0f310 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.092+0000 7f4ec357f640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0f310 0x555b14a914d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc056040 tx=0x7f4ebc00a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.092+0000 7f4eabfff640 1 -- 172.21.15.99:0/1387498676 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc062020 con 0x555b14a0f310 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.092+0000 7f4eabfff640 1 -- 172.21.15.99:0/1387498676 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc05d910 con 0x555b14a0f310 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.092+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555b14a8bca0 con 0x555b14a0f310 2024-03-04T10:10:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.092+0000 7f4eabfff640 1 -- 172.21.15.99:0/1387498676 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc05dd00 con 0x555b14a0f310 2024-03-04T10:10:21.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.092+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b14a8c260 con 0x555b14a0f310 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.093+0000 7f4eabfff640 1 -- 172.21.15.99:0/1387498676 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4ebc065020 con 0x555b14a0f310 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.093+0000 7f4eabfff640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4eb0069eb0 0x7f4eb006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4eabfff640 1 -- 172.21.15.99:0/1387498676 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4ebc0cac60 con 0x555b14a0f310 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec2d7e640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4eb0069eb0 0x7f4eb006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555b149c8310 0x555b14a8d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x555b14a8df20 con 0x555b149c8310 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec257d640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555b149c8310 0x555b14a8d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec257d640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555b149c8310 0x555b14a8d9e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec2d7e640 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4eb0069eb0 0x7f4eb006c370 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4005fd0 tx=0x7f4eb4007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec257d640 1 -- 172.21.15.99:0/1387498676 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-header out=274b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x7f4eb800f630 con 0x555b149c8310 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:header (274 bytes) : 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000000 04 03 0c 01 00 00 0b 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000020 00 00 00 00 0a 9e e5 65 84 7b 17 04 01 00 00 00 |.......e.{......| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000030 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 |................| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000050 00 00 0a 9e e5 65 84 7b 17 04 01 00 00 00 00 00 |.....e.{........| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000060 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 |................| 2024-03-04T10:10:21.101 INFO:teuthology.orchestra.run.smithi099.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:00000080 00 00 60 00 00 00 00 00 01 00 00 00 00 00 00 00 |..`.............| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:000000a0 00 00 00 00 00 00 00 00 0a 9e e5 65 a3 a6 48 05 |...........e..H.| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:000000c0 60 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 |`...............| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:000000e0 00 00 00 00 00 00 0a 9e e5 65 a3 a6 48 05 00 00 |.........e..H...| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:00000110 00 00 |..| 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout:00000112 2024-03-04T10:10:21.102 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555b149c8310 msgr2=0x555b14a8d9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.094+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555b149c8310 0x555b14a8d9e0 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4eb0069eb0 msgr2=0x7f4eb006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4eb0069eb0 0x7f4eb006c370 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4005fd0 tx=0x7f4eb4007040 comp rx=0 tx=0).stop 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0f310 msgr2=0x555b14a914d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 --2- 172.21.15.99:0/1387498676 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b14a0f310 0x555b14a914d0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc056040 tx=0x7f4ebc00a4b0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec357f640 1 -- 172.21.15.99:0/1387498676 reap_dead start 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 shutdown_connections 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 >> 172.21.15.99:0/1387498676 conn(0x555b149c6400 msgr2=0x555b149f55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 shutdown_connections 2024-03-04T10:10:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.095+0000 7f4ec5a40f40 1 -- 172.21.15.99:0/1387498676 wait complete. 2024-03-04T10:10:21.105 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 600.00000000 2024-03-04T10:10:21.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 Processor -- start 2024-03-04T10:10:21.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 -- start start 2024-03-04T10:10:21.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c5386aaf0 0x561c5390a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 0x561c5383e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 0x561c53865970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c53885d40 con 0x561c53883f30 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c537e2b30 con 0x561c53884310 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.184+0000 7fdfb583bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561c53913c40 con 0x561c5386aaf0 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 0x561c5383e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 0x561c5383e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55142/0 (socket says 172.21.15.99:55142) 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 -- 172.21.15.99:0/1862532405 learned_addr learned my addr 172.21.15.99:0/1862532405 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 -- 172.21.15.99:0/1862532405 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c5386aaf0 msgr2=0x561c5390a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c5386aaf0 0x561c5390a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 -- 172.21.15.99:0/1862532405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 msgr2=0x561c53865970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 0x561c53865970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 -- 172.21.15.99:0/1862532405 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c53867fb0 con 0x561c53883f30 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb2396640 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 0x561c5383e9d0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fdfa801db10 tx=0x7fdfa804e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74027305c99310fa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb1394640 1 -- 172.21.15.99:0/1862532405 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa800a150 con 0x561c53883f30 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb1394640 1 -- 172.21.15.99:0/1862532405 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa800a310 con 0x561c53883f30 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb1394640 1 -- 172.21.15.99:0/1862532405 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8008050 con 0x561c53883f30 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1862532405 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 msgr2=0x561c5383e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 0x561c5383e9d0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fdfa801db10 tx=0x7fdfa804e000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1862532405 shutdown_connections 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 0x561c53865970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c53883f30 0x561c5383e9d0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1862532405 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c5386aaf0 0x561c5390a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1862532405 >> 172.21.15.99:0/1862532405 conn(0x561c5383b400 msgr2=0x561c5386a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1862532405 shutdown_connections 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1862532405 wait complete. 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 Processor -- start 2024-03-04T10:10:21.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.185+0000 7fdfb583bf40 1 -- start start 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb583bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c5386aaf0 0x561c53902c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb583bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c53883f30 0x561c538fc170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb583bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 0x561c53912240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb583bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5383cbd0 con 0x561c5386aaf0 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb583bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c5383b980 con 0x561c53884310 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb583bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561c5383ceb0 con 0x561c53883f30 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c5386aaf0 0x561c53902c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c5386aaf0 0x561c53902c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55152/0 (socket says 172.21.15.99:55152) 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 -- 172.21.15.99:0/1480642432 learned_addr learned my addr 172.21.15.99:0/1480642432 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb3398640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 0x561c53912240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2396640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c53883f30 0x561c538fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 -- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c53883f30 msgr2=0x561c538fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561c53883f30 0x561c538fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 -- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 msgr2=0x561c53912240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561c53884310 0x561c53912240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 -- 172.21.15.99:0/1480642432 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c53866fb0 con 0x561c5386aaf0 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.187+0000 7fdfb2b97640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c5386aaf0 0x561c53902c80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009400 tx=0x7fdfa40098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.188+0000 7fdf9bfff640 1 -- 172.21.15.99:0/1480642432 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4014020 con 0x561c5386aaf0 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.188+0000 7fdf9bfff640 1 -- 172.21.15.99:0/1480642432 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa40108e0 con 0x561c5386aaf0 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.188+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561c53867290 con 0x561c5386aaf0 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.188+0000 7fdf9bfff640 1 -- 172.21.15.99:0/1480642432 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4010cd0 con 0x561c5386aaf0 2024-03-04T10:10:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.188+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c53903660 con 0x561c5386aaf0 2024-03-04T10:10:21.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.189+0000 7fdf9bfff640 1 -- 172.21.15.99:0/1480642432 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdfa4017020 con 0x561c5386aaf0 2024-03-04T10:10:21.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.189+0000 7fdf9bfff640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfa0069ee0 0x7fdfa006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.189+0000 7fdf9bfff640 1 -- 172.21.15.99:0/1480642432 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdfa40890b0 con 0x561c5386aaf0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.189+0000 7fdfb2396640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfa0069ee0 0x7fdfa006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x561c538fc6b0 0x561c539019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0d82a743:::600.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x561c53901ef0 con 0x561c538fc6b0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb3398640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x561c538fc6b0 0x561c539019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb3398640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x561c538fc6b0 0x561c539019b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb2396640 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfa0069ee0 0x7fdfa006c3a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8056040 tx=0x7fdfa800a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb3398640 1 -- 172.21.15.99:0/1480642432 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 600.00000000 [omap-get-header out=274b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7fdfac002960 con 0x561c538fc6b0 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stdout:header (274 bytes) : 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.195 INFO:teuthology.orchestra.run.smithi099.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000110 00 00 |..| 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout:00000112 2024-03-04T10:10:21.196 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x561c538fc6b0 msgr2=0x561c539019b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x561c538fc6b0 0x561c539019b0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfa0069ee0 msgr2=0x7fdfa006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdfa0069ee0 0x7fdfa006c3a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8056040 tx=0x7fdfa800a000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c5386aaf0 msgr2=0x561c53902c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 --2- 172.21.15.99:0/1480642432 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561c5386aaf0 0x561c53902c80 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009400 tx=0x7fdfa40098c0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb3398640 1 -- 172.21.15.99:0/1480642432 reap_dead start 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 shutdown_connections 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.190+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 >> 172.21.15.99:0/1480642432 conn(0x561c5383b400 msgr2=0x561c538818c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.191+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 shutdown_connections 2024-03-04T10:10:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.191+0000 7fdfb583bf40 1 -- 172.21.15.99:0/1480642432 wait complete. 2024-03-04T10:10:21.199 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000.inode 2024-03-04T10:10:21.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.277+0000 7f6933247f40 1 Processor -- start 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.277+0000 7f6933247f40 1 -- start start 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6933247f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47863ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6933247f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6933247f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6933247f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4785ae250 con 0x55d47863bf90 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6933247f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4785ae0d0 con 0x55d4785961c0 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6933247f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4785ae3d0 con 0x55d47863c370 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55172/0 (socket says 172.21.15.99:55172) 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 -- 172.21.15.99:0/612107495 learned_addr learned my addr 172.21.15.99:0/612107495 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6930d86640 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6930585640 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47863ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 -- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 msgr2=0x55d4785a79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 -- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 msgr2=0x55d47863ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47863ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 -- 172.21.15.99:0/612107495 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4785a95f0 con 0x55d47863bf90 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f6930d86640 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692fd84640 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a74a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f6918009570 tx=0x7f6918009a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db11ed927fc4150c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692ed82640 1 -- 172.21.15.99:0/612107495 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918017070 con 0x55d47863bf90 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692ed82640 1 -- 172.21.15.99:0/612107495 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918006600 con 0x55d47863bf90 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.278+0000 7f692ed82640 1 -- 172.21.15.99:0/612107495 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180069f0 con 0x55d47863bf90 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 -- 172.21.15.99:0/612107495 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 msgr2=0x55d4785a74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a74a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f6918009570 tx=0x7f6918009a30 comp rx=0 tx=0).stop 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 -- 172.21.15.99:0/612107495 shutdown_connections 2024-03-04T10:10:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a74a0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 --2- 172.21.15.99:0/612107495 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47863ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 -- 172.21.15.99:0/612107495 >> 172.21.15.99:0/612107495 conn(0x55d478566400 msgr2=0x55d478594e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 -- 172.21.15.99:0/612107495 shutdown_connections 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 -- 172.21.15.99:0/612107495 wait complete. 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 Processor -- start 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.279+0000 7f6933247f40 1 -- start start 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6933247f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47861db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6933247f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6933247f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6933247f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d47863ed50 con 0x55d47863bf90 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6933247f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4785956c0 con 0x55d4785961c0 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6933247f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4786341c0 con 0x55d47863c370 2024-03-04T10:10:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f692fd84640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f692fd84640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55184/0 (socket says 172.21.15.99:55184) 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f692fd84640 1 -- 172.21.15.99:0/428300758 learned_addr learned my addr 172.21.15.99:0/428300758 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6930585640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47861db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f6930d86640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f692fd84640 1 -- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 msgr2=0x55d4785a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f692fd84640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d47863c370 0x55d4785a8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.280+0000 7f692fd84640 1 -- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 msgr2=0x55d47861db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f692fd84640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d4785961c0 0x55d47861db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f692fd84640 1 -- 172.21.15.99:0/428300758 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4785a9b90 con 0x55d47863bf90 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f692fd84640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a7d00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f6918009a70 tx=0x7f6918003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f692dd80640 1 -- 172.21.15.99:0/428300758 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918017070 con 0x55d47863bf90 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f692dd80640 1 -- 172.21.15.99:0/428300758 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918018040 con 0x55d47863bf90 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d4785a9e70 con 0x55d47863bf90 2024-03-04T10:10:21.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f692dd80640 1 -- 172.21.15.99:0/428300758 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918012630 con 0x55d47863bf90 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.281+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4785aa430 con 0x55d47863bf90 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.282+0000 7f692dd80640 1 -- 172.21.15.99:0/428300758 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f69180127d0 con 0x55d47863bf90 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.282+0000 7f692dd80640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6910069ee0 0x7f691006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f692dd80640 1 -- 172.21.15.99:0/428300758 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6918089b50 con 0x55d47863bf90 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6930585640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6910069ee0 0x7f691006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6933247f40 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d4785ab140 0x55d4785ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55d4785aba60 con 0x55d4785ab140 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6930d86640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d4785ab140 0x55d4785ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6930d86640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d4785ab140 0x55d4785ab520 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6930585640 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6910069ee0 0x7f691006c3a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f692000b5a0 tx=0x7f6920007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6930d86640 1 -- 172.21.15.99:0/428300758 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-header] v0'0 uv16 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f6928052970 con 0x55d4785ab140 2024-03-04T10:10:21.290 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:21.291 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d4785ab140 msgr2=0x55d4785ab520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.283+0000 7f6933247f40 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d4785ab140 0x55d4785ab520 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6910069ee0 msgr2=0x7f691006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6910069ee0 0x7f691006c3a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f692000b5a0 tx=0x7f6920007040 comp rx=0 tx=0).stop 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 msgr2=0x55d4785a7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 --2- 172.21.15.99:0/428300758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d47863bf90 0x55d4785a7d00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f6918009a70 tx=0x7f6918003020 comp rx=0 tx=0).stop 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6930d86640 1 -- 172.21.15.99:0/428300758 reap_dead start 2024-03-04T10:10:21.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 shutdown_connections 2024-03-04T10:10:21.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 >> 172.21.15.99:0/428300758 conn(0x55d478566400 msgr2=0x55d4785945a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 shutdown_connections 2024-03-04T10:10:21.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.284+0000 7f6933247f40 1 -- 172.21.15.99:0/428300758 wait complete. 2024-03-04T10:10:21.293 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000000 2024-03-04T10:10:21.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.371+0000 7fdf8b63ff40 1 Processor -- start 2024-03-04T10:10:21.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.371+0000 7fdf8b63ff40 1 -- start start 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf8b63ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac3d3c90 0x55d6ac4a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf8b63ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac4a55a0 0x55d6ac4ac5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf8b63ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4ad160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf8b63ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ac37cb30 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf8b63ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ac380a90 con 0x55d6ac3d3c90 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf8b63ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6ac41eab0 con 0x55d6ac4a55a0 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4ad160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4ad160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55198/0 (socket says 172.21.15.99:55198) 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 -- 172.21.15.99:0/3902004817 learned_addr learned my addr 172.21.15.99:0/3902004817 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf880c2640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac4a55a0 0x55d6ac4ac5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf888c3640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac3d3c90 0x55d6ac4a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 -- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac4a55a0 msgr2=0x55d6ac4ac5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac4a55a0 0x55d6ac4ac5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 -- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac3d3c90 msgr2=0x55d6ac4a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac3d3c90 0x55d6ac4a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf890c4640 1 -- 172.21.15.99:0/3902004817 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6ac4057c0 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf888c3640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac3d3c90 0x55d6ac4a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.372+0000 7fdf880c2640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac4a55a0 0x55d6ac4ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf890c4640 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4ad160 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001e90 tx=0x7fdf8000b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e967d397194d3033 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf870c0640 1 -- 172.21.15.99:0/3902004817 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80008070 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf870c0640 1 -- 172.21.15.99:0/3902004817 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8000d500 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf870c0640 1 -- 172.21.15.99:0/3902004817 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000d8d0 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/3902004817 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 msgr2=0x55d6ac4ad160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4ad160 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001e90 tx=0x7fdf8000b6a0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/3902004817 shutdown_connections 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4ad160 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac4a55a0 0x55d6ac4ac5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/3902004817 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac3d3c90 0x55d6ac4a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/3902004817 >> 172.21.15.99:0/3902004817 conn(0x55d6ac3d5400 msgr2=0x55d6ac401770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/3902004817 shutdown_connections 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.373+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/3902004817 wait complete. 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.374+0000 7fdf8b63ff40 1 Processor -- start 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.374+0000 7fdf8b63ff40 1 -- start start 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac3d3c90 0x55d6ac48efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac4a55a0 0x55d6ac401480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4019c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ac41c230 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ac4a3050 con 0x55d6ac4a55a0 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d6ac4a2570 con 0x55d6ac3d3c90 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4019c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55206/0 (socket says 172.21.15.99:55206) 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 -- 172.21.15.99:0/1383404942 learned_addr learned my addr 172.21.15.99:0/1383404942 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf888c3640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac3d3c90 0x55d6ac48efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf880c2640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac4a55a0 0x55d6ac401480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 -- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac3d3c90 msgr2=0x55d6ac48efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55d6ac3d3c90 0x55d6ac48efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 -- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac4a55a0 msgr2=0x55d6ac401480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d6ac4a55a0 0x55d6ac401480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 -- 172.21.15.99:0/1383404942 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6ac41c4d0 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf890c4640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4019c0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b6e0 tx=0x7fdf80000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf860be640 1 -- 172.21.15.99:0/1383404942 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80020070 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf860be640 1 -- 172.21.15.99:0/1383404942 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8001c870 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d6ac404210 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf860be640 1 -- 172.21.15.99:0/1383404942 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8001cc60 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.375+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6ac4047d0 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf860be640 1 -- 172.21.15.99:0/1383404942 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdf80023070 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf860be640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf74069ee0 0x7fdf7406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf860be640 1 -- 172.21.15.99:0/1383404942 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdf8008afb0 con 0x55d6ac4ab5f0 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf888c3640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf74069ee0 0x7fdf7406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d6ac4a6430 0x55d6ac4a6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:292cf221:::200.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55d6ac4a6d50 con 0x55d6ac4a6430 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.377+0000 7fdf880c2640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d6ac4a6430 0x55d6ac4a6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf880c2640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d6ac4a6430 0x55d6ac4a6810 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf888c3640 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf74069ee0 0x7fdf7406c3a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdf78053000 tx=0x7fdf7804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf880c2640 1 -- 172.21.15.99:0/1383404942 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000000 [omap-get-header] v0'0 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fdf7c00a5f0 con 0x55d6ac4a6430 2024-03-04T10:10:21.384 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:21.384 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d6ac4a6430 msgr2=0x55d6ac4a6810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55d6ac4a6430 0x55d6ac4a6810 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf74069ee0 msgr2=0x7fdf7406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf74069ee0 0x7fdf7406c3a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdf78053000 tx=0x7fdf7804e000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 msgr2=0x55d6ac4019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 --2- 172.21.15.99:0/1383404942 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55d6ac4ab5f0 0x55d6ac4019c0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000b6e0 tx=0x7fdf80000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf890c4640 1 -- 172.21.15.99:0/1383404942 reap_dead start 2024-03-04T10:10:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 shutdown_connections 2024-03-04T10:10:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 >> 172.21.15.99:0/1383404942 conn(0x55d6ac3d5400 msgr2=0x55d6ac4ac500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 shutdown_connections 2024-03-04T10:10:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.378+0000 7fdf8b63ff40 1 -- 172.21.15.99:0/1383404942 wait complete. 2024-03-04T10:10:21.386 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 200.00000001 2024-03-04T10:10:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 Processor -- start 2024-03-04T10:10:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 -- start start 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 0x560c60e80a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e84970 0x560c60f0ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60f05a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c60de2b30 con 0x560c60e83970 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c60de6a90 con 0x560c60e84970 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f7655abaf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c60f0b950 con 0x560c60f144a0 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f76535ba640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 0x560c60e80a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f76535ba640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 0x560c60e80a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55216/0 (socket says 172.21.15.99:55216) 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.465+0000 7f76535ba640 1 -- 172.21.15.99:0/1333522888 learned_addr learned my addr 172.21.15.99:0/1333522888 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7653dbb640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60f05a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7652db9640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e84970 0x560c60f0ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f76535ba640 1 -- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 msgr2=0x560c60f05a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f76535ba640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60f05a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f76535ba640 1 -- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e84970 msgr2=0x560c60f0ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f76535ba640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e84970 0x560c60f0ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f76535ba640 1 -- 172.21.15.99:0/1333522888 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c60f067e0 con 0x560c60e83970 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7652db9640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e84970 0x560c60f0ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7653dbb640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60f05a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f76535ba640 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 0x560c60e80a20 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f7644001d40 tx=0x7f764400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169a10f45f365d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7651db7640 1 -- 172.21.15.99:0/1333522888 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400bd80 con 0x560c60e83970 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7651db7640 1 -- 172.21.15.99:0/1333522888 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f764400f400 con 0x560c60e83970 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.466+0000 7f7651db7640 1 -- 172.21.15.99:0/1333522888 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400f7f0 con 0x560c60e83970 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 -- 172.21.15.99:0/1333522888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 msgr2=0x560c60e80a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 0x560c60e80a20 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f7644001d40 tx=0x7f764400b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 -- 172.21.15.99:0/1333522888 shutdown_connections 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60f05a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e84970 0x560c60f0ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 --2- 172.21.15.99:0/1333522888 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e83970 0x560c60e80a20 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 -- 172.21.15.99:0/1333522888 >> 172.21.15.99:0/1333522888 conn(0x560c60e67970 msgr2=0x560c60f0bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 -- 172.21.15.99:0/1333522888 shutdown_connections 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 -- 172.21.15.99:0/1333522888 wait complete. 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 Processor -- start 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.467+0000 7f7655abaf40 1 -- start start 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7655abaf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e83970 0x560c60f08640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7655abaf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e84970 0x560c60e821e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7655abaf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60efc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7655abaf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c60e66de0 con 0x560c60e84970 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7655abaf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c60f0b190 con 0x560c60e83970 2024-03-04T10:10:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7655abaf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c60f0b4f0 con 0x560c60f144a0 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7652db9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e84970 0x560c60e821e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7652db9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e84970 0x560c60e821e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55224/0 (socket says 172.21.15.99:55224) 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7652db9640 1 -- 172.21.15.99:0/4097526102 learned_addr learned my addr 172.21.15.99:0/4097526102 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f7653dbb640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60efc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.468+0000 7f76535ba640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e83970 0x560c60f08640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7652db9640 1 -- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 msgr2=0x560c60efc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7652db9640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560c60f144a0 0x560c60efc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7652db9640 1 -- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e83970 msgr2=0x560c60f08640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7652db9640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c60e83970 0x560c60f08640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7652db9640 1 -- 172.21.15.99:0/4097526102 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c60f06cd0 con 0x560c60e84970 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7652db9640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e84970 0x560c60e821e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f7648055040 tx=0x7f7648002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7650db5640 1 -- 172.21.15.99:0/4097526102 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648062020 con 0x560c60e84970 2024-03-04T10:10:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7650db5640 1 -- 172.21.15.99:0/4097526102 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7648052a10 con 0x560c60e84970 2024-03-04T10:10:21.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560c60f06fb0 con 0x560c60e84970 2024-03-04T10:10:21.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7650db5640 1 -- 172.21.15.99:0/4097526102 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7648057040 con 0x560c60e84970 2024-03-04T10:10:21.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.469+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c60e82720 con 0x560c60e84970 2024-03-04T10:10:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f7650db5640 1 -- 172.21.15.99:0/4097526102 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7648065020 con 0x560c60e84970 2024-03-04T10:10:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f7650db5640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7640069eb0 0x7f764006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f7650db5640 1 -- 172.21.15.99:0/4097526102 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f76480caad0 con 0x560c60e84970 2024-03-04T10:10:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f7655abaf40 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x560c60f0f2c0 0x560c60f05e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f76535ba640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7640069eb0 0x7f764006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:2e2fa760:::200.00000001:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x560c60efcb10 con 0x560c60f0f2c0 2024-03-04T10:10:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.471+0000 7f7653dbb640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x560c60f0f2c0 0x560c60f05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7653dbb640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x560c60f0f2c0 0x560c60f05e20 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f76535ba640 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7640069eb0 0x7f764006c370 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f764400b5f0 tx=0x7f7644009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7653dbb640 1 -- 172.21.15.99:0/4097526102 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000001 [omap-get-header] v0'0 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f764c0058d0 con 0x560c60f0f2c0 2024-03-04T10:10:21.477 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:21.477 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x560c60f0f2c0 msgr2=0x560c60f05e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x560c60f0f2c0 0x560c60f05e20 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7640069eb0 msgr2=0x7f764006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7640069eb0 0x7f764006c370 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f764400b5f0 tx=0x7f7644009000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e84970 msgr2=0x560c60e821e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 --2- 172.21.15.99:0/4097526102 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560c60e84970 0x560c60e821e0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f7648055040 tx=0x7f7648002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7653dbb640 1 -- 172.21.15.99:0/4097526102 reap_dead start 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 shutdown_connections 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 >> 172.21.15.99:0/4097526102 conn(0x560c60e67970 msgr2=0x560c60f0bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 shutdown_connections 2024-03-04T10:10:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.472+0000 7f7655abaf40 1 -- 172.21.15.99:0/4097526102 wait complete. 2024-03-04T10:10:21.480 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_openfiles.0 2024-03-04T10:10:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.559+0000 7f371d840f40 1 Processor -- start 2024-03-04T10:10:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.559+0000 7f371d840f40 1 -- start start 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371d840f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc897cc0 0x556edc894310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371d840f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc937180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371d840f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc8b1310 0x556edc930630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371d840f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x556edc80fb30 con 0x556edc8b0f30 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371d840f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x556edc813a90 con 0x556edc8b1310 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371d840f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x556edc8b2d40 con 0x556edc897cc0 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc937180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc937180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55236/0 (socket says 172.21.15.99:55236) 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 -- 172.21.15.99:0/1612465460 learned_addr learned my addr 172.21.15.99:0/1612465460 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371aaa3640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc897cc0 0x556edc894310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371b2a4640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc8b1310 0x556edc930630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 -- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc897cc0 msgr2=0x556edc894310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc897cc0 0x556edc894310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 -- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc8b1310 msgr2=0x556edc930630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc8b1310 0x556edc930630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 -- 172.21.15.99:0/1612465460 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556edc949300 con 0x556edc8b0f30 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371aaa3640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc897cc0 0x556edc894310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371b2a4640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc8b1310 0x556edc930630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f371a2a2640 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc937180 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3710006f50 tx=0x7f371000b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cbc397da0113e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f37192a0640 1 -- 172.21.15.99:0/1612465460 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710012070 con 0x556edc8b0f30 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f37192a0640 1 -- 172.21.15.99:0/1612465460 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371000e670 con 0x556edc8b0f30 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.560+0000 7f37192a0640 1 -- 172.21.15.99:0/1612465460 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371000eaa0 con 0x556edc8b0f30 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 -- 172.21.15.99:0/1612465460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 msgr2=0x556edc937180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc937180 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3710006f50 tx=0x7f371000b780 comp rx=0 tx=0).stop 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 -- 172.21.15.99:0/1612465460 shutdown_connections 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc8b1310 0x556edc930630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc937180 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 --2- 172.21.15.99:0/1612465460 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc897cc0 0x556edc894310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 -- 172.21.15.99:0/1612465460 >> 172.21.15.99:0/1612465460 conn(0x556edc868400 msgr2=0x556edc869cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 -- 172.21.15.99:0/1612465460 shutdown_connections 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 -- 172.21.15.99:0/1612465460 wait complete. 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 Processor -- start 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.561+0000 7f371d840f40 1 -- start start 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371d840f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc897cc0 0x556edc93c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371d840f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc92c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371d840f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc8b1310 0x556edc92acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371d840f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x556edc894850 con 0x556edc8b0f30 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371d840f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x556edc86aa60 con 0x556edc897cc0 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371d840f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x556edc86a4a0 con 0x556edc8b1310 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc92c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc92c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55244/0 (socket says 172.21.15.99:55244) 2024-03-04T10:10:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 -- 172.21.15.99:0/209557102 learned_addr learned my addr 172.21.15.99:0/209557102 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371aaa3640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc897cc0 0x556edc93c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371b2a4640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc8b1310 0x556edc92acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 -- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc8b1310 msgr2=0x556edc92acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556edc8b1310 0x556edc92acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 -- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc897cc0 msgr2=0x556edc93c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc897cc0 0x556edc93c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371a2a2640 1 -- 172.21.15.99:0/209557102 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556edc92d2c0 con 0x556edc8b0f30 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.562+0000 7f371aaa3640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556edc897cc0 0x556edc93c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.563+0000 7f371a2a2640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc92c8b0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f371000b7c0 tx=0x7f3710000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.563+0000 7f3703fff640 1 -- 172.21.15.99:0/209557102 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710027070 con 0x556edc8b0f30 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.563+0000 7f3703fff640 1 -- 172.21.15.99:0/209557102 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37100179a0 con 0x556edc8b0f30 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.563+0000 7f3703fff640 1 -- 172.21.15.99:0/209557102 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3710012050 con 0x556edc8b0f30 2024-03-04T10:10:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.563+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556edc9d84f0 con 0x556edc8b0f30 2024-03-04T10:10:21.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.563+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556edc9d8ab0 con 0x556edc8b0f30 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.564+0000 7f3703fff640 1 -- 172.21.15.99:0/209557102 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f371002a070 con 0x556edc8b0f30 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.564+0000 7f3703fff640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3708069eb0 0x7f370806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.564+0000 7f3703fff640 1 -- 172.21.15.99:0/209557102 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f371008aa80 con 0x556edc8b0f30 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.565+0000 7f371d840f40 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x556edc92cdf0 0x556edc9d77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.565+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.6 13:654134d2:::mds0_openfiles.0:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x556edc9d7d00 con 0x556edc92cdf0 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.565+0000 7f371aaa3640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3708069eb0 0x7f370806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.565+0000 7f371b2a4640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x556edc92cdf0 0x556edc9d77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.565+0000 7f371aaa3640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3708069eb0 0x7f370806c370 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f370c053000 tx=0x7f370c04e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.565+0000 7f371b2a4640 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x556edc92cdf0 0x556edc9d77c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371b2a4640 1 -- 172.21.15.99:0/209557102 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-header out=42b] v0'0 uv3 ondisk = 0) v8 ==== 160+0+42 (crc 0 0 0) 0x7f371400e470 con 0x556edc92cdf0 2024-03-04T10:10:21.571 INFO:teuthology.orchestra.run.smithi099.stdout:header (42 bytes) : 2024-03-04T10:10:21.571 INFO:teuthology.orchestra.run.smithi099.stdout:00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| 2024-03-04T10:10:21.571 INFO:teuthology.orchestra.run.smithi099.stdout:00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 03 00 00 |me v011.........| 2024-03-04T10:10:21.571 INFO:teuthology.orchestra.run.smithi099.stdout:00000020 00 00 00 00 00 01 00 00 00 00 |..........| 2024-03-04T10:10:21.571 INFO:teuthology.orchestra.run.smithi099.stdout:0000002a 2024-03-04T10:10:21.571 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x556edc92cdf0 msgr2=0x556edc9d77c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x556edc92cdf0 0x556edc9d77c0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3708069eb0 msgr2=0x7f370806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3708069eb0 0x7f370806c370 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f370c053000 tx=0x7f370c04e000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 msgr2=0x556edc92c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 --2- 172.21.15.99:0/209557102 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556edc8b0f30 0x556edc92c8b0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f371000b7c0 tx=0x7f3710000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371b2a4640 1 -- 172.21.15.99:0/209557102 reap_dead start 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 shutdown_connections 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 >> 172.21.15.99:0/209557102 conn(0x556edc868400 msgr2=0x556edc93ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 shutdown_connections 2024-03-04T10:10:21.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.566+0000 7f371d840f40 1 -- 172.21.15.99:0/209557102 wait complete. 2024-03-04T10:10:21.573 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 500.00000000 2024-03-04T10:10:21.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 Processor -- start 2024-03-04T10:10:21.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 -- start start 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3c03c90 0x5639b3c31160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 0x5639b3ccb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cdb5f0 0x5639b3cd43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639b3c4e810 con 0x5639b3cd55a0 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639b3bacb30 con 0x5639b3c03c90 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.653+0000 7f7ccfc40f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639b3c4eab0 con 0x5639b3cdb5f0 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 0x5639b3ccb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 0x5639b3ccb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55252/0 (socket says 172.21.15.99:55252) 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 -- 172.21.15.99:0/2428277477 learned_addr learned my addr 172.21.15.99:0/2428277477 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cccf7e640 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3c03c90 0x5639b3c31160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 -- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cdb5f0 msgr2=0x5639b3cd43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7ccd77f640 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cdb5f0 0x5639b3cd43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cdb5f0 0x5639b3cd43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 -- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3c03c90 msgr2=0x5639b3c31160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3c03c90 0x5639b3c31160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 -- 172.21.15.99:0/2428277477 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639b3cddb80 con 0x5639b3cd55a0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7ccd77f640 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cdb5f0 0x5639b3cd43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc7fff640 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 0x5639b3ccb2c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8055040 tx=0x7f7cb804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4e5ce928336b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc6ffd640 1 -- 172.21.15.99:0/2428277477 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8057040 con 0x5639b3cd55a0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc6ffd640 1 -- 172.21.15.99:0/2428277477 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb8052a10 con 0x5639b3cd55a0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.654+0000 7f7cc6ffd640 1 -- 172.21.15.99:0/2428277477 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8052e00 con 0x5639b3cd55a0 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/2428277477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 msgr2=0x5639b3ccb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 0x5639b3ccb2c0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8055040 tx=0x7f7cb804c000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/2428277477 shutdown_connections 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cdb5f0 0x5639b3cd43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3cd55a0 0x5639b3ccb2c0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/2428277477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3c03c90 0x5639b3c31160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/2428277477 >> 172.21.15.99:0/2428277477 conn(0x5639b3c05400 msgr2=0x5639b3cd1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/2428277477 shutdown_connections 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/2428277477 wait complete. 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 Processor -- start 2024-03-04T10:10:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.655+0000 7f7ccfc40f40 1 -- start start 2024-03-04T10:10:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccfc40f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3c03c90 0x5639b3cd5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccfc40f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cd55a0 0x5639b3cbfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccfc40f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3cdb5f0 0x5639b3c34390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccfc40f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639b3cd3ac0 con 0x5639b3c03c90 2024-03-04T10:10:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccfc40f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639b3cdddf0 con 0x5639b3cdb5f0 2024-03-04T10:10:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccfc40f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5639b3cd3d60 con 0x5639b3cd55a0 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3c03c90 0x5639b3cd5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3c03c90 0x5639b3cd5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55256/0 (socket says 172.21.15.99:55256) 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 -- 172.21.15.99:0/1833726528 learned_addr learned my addr 172.21.15.99:0/1833726528 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cc7fff640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cd55a0 0x5639b3cbfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccd77f640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3cdb5f0 0x5639b3c34390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 -- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cd55a0 msgr2=0x5639b3cbfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5639b3cd55a0 0x5639b3cbfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 -- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3cdb5f0 msgr2=0x5639b3c34390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3cdb5f0 0x5639b3c34390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7cccf7e640 1 -- 172.21.15.99:0/1833726528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639b3c35500 con 0x5639b3c03c90 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.656+0000 7f7ccd77f640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5639b3cdb5f0 0x5639b3c34390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.657+0000 7f7cccf7e640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3c03c90 0x5639b3cd5b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000df50 tx=0x7f7cc000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.657+0000 7f7cc5ffb640 1 -- 172.21.15.99:0/1833726528 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0014020 con 0x5639b3c03c90 2024-03-04T10:10:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.657+0000 7f7cc5ffb640 1 -- 172.21.15.99:0/1833726528 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0010840 con 0x5639b3c03c90 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.657+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5639b3cc17e0 con 0x5639b3c03c90 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.657+0000 7f7cc5ffb640 1 -- 172.21.15.99:0/1833726528 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0010c10 con 0x5639b3c03c90 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.657+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639b3cc1da0 con 0x5639b3c03c90 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.658+0000 7f7cc5ffb640 1 -- 172.21.15.99:0/1833726528 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7cc0017020 con 0x5639b3c03c90 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.658+0000 7f7cc5ffb640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7cb0069eb0 0x7f7cb006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7cc5ffb640 1 -- 172.21.15.99:0/1833726528 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7cc0089430 con 0x5639b3c03c90 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5639b3ccdaf0 0x5639b3cc2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.15 13:aa448500:::500.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x5639b3cc2dc0 con 0x5639b3ccdaf0 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7cc7fff640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7cb0069eb0 0x7f7cb006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7ccd77f640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5639b3ccdaf0 0x5639b3cc2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7cc7fff640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7cb0069eb0 0x7f7cb006c370 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7cb805e000 tx=0x7f7cb8068040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.659+0000 7f7ccd77f640 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5639b3ccdaf0 0x5639b3cc2880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccd77f640 1 -- 172.21.15.99:0/1833726528 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 500.00000000 [omap-get-header] v0'0 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f7cc8011040 con 0x5639b3ccdaf0 2024-03-04T10:10:21.665 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:21.665 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5639b3ccdaf0 msgr2=0x5639b3cc2880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5639b3ccdaf0 0x5639b3cc2880 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7cb0069eb0 msgr2=0x7f7cb006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7cb0069eb0 0x7f7cb006c370 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7cb805e000 tx=0x7f7cb8068040 comp rx=0 tx=0).stop 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3c03c90 msgr2=0x5639b3cd5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 --2- 172.21.15.99:0/1833726528 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5639b3c03c90 0x5639b3cd5b90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000df50 tx=0x7f7cc000b6c0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccd77f640 1 -- 172.21.15.99:0/1833726528 reap_dead start 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 shutdown_connections 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 >> 172.21.15.99:0/1833726528 conn(0x5639b3c05400 msgr2=0x5639b3cd4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 shutdown_connections 2024-03-04T10:10:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.660+0000 7f7ccfc40f40 1 -- 172.21.15.99:0/1833726528 wait complete. 2024-03-04T10:10:21.667 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds_snaptable 2024-03-04T10:10:21.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.745+0000 7fe2e2756f40 1 Processor -- start 2024-03-04T10:10:21.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.745+0000 7fe2e2756f40 1 -- start start 2024-03-04T10:10:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e2756f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e646cc0 0x55f70e6eae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e2756f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e65ff30 0x55f70e6f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e2756f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 0x55f70e6e62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e2756f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f70e661d40 con 0x55f70e660310 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e2756f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f70e5beb30 con 0x55f70e646cc0 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e2756f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f70e6efc40 con 0x55f70e65ff30 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 0x55f70e6e62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 0x55f70e6e62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55278/0 (socket says 172.21.15.99:55278) 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 -- 172.21.15.99:0/2940942669 learned_addr learned my addr 172.21.15.99:0/2940942669 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0254640 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e646cc0 0x55f70e6eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2dfa53640 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e65ff30 0x55f70e6f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 -- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e65ff30 msgr2=0x55f70e6f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e65ff30 0x55f70e6f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 -- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e646cc0 msgr2=0x55f70e6eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e646cc0 0x55f70e6eae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2e0a55640 1 -- 172.21.15.99:0/2940942669 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f70e6eb460 con 0x55f70e660310 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.746+0000 7fe2dfa53640 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e65ff30 0x55f70e6f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e0a55640 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 0x55f70e6e62b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800c4b0 tx=0x7fe2d800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259d5dae666dc82d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2dea51640 1 -- 172.21.15.99:0/2940942669 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d80116b0 con 0x55f70e660310 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2dea51640 1 -- 172.21.15.99:0/2940942669 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d8017070 con 0x55f70e660310 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2dea51640 1 -- 172.21.15.99:0/2940942669 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8011a90 con 0x55f70e660310 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 -- 172.21.15.99:0/2940942669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 msgr2=0x55f70e6e62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 0x55f70e6e62b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800c4b0 tx=0x7fe2d800c970 comp rx=0 tx=0).stop 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 -- 172.21.15.99:0/2940942669 shutdown_connections 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e660310 0x55f70e6e62b0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e65ff30 0x55f70e6f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/2940942669 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e646cc0 0x55f70e6eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 -- 172.21.15.99:0/2940942669 >> 172.21.15.99:0/2940942669 conn(0x55f70e617400 msgr2=0x55f70e6177e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 -- 172.21.15.99:0/2940942669 shutdown_connections 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.747+0000 7fe2e2756f40 1 -- 172.21.15.99:0/2940942669 wait complete. 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.748+0000 7fe2e2756f40 1 Processor -- start 2024-03-04T10:10:21.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.748+0000 7fe2e2756f40 1 -- start start 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.748+0000 7fe2e2756f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e646cc0 0x55f70e6e22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e65ff30 0x55f70e6ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e660310 0x55f70e61a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f70e619e50 con 0x55f70e65ff30 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f70e6eb740 con 0x55f70e660310 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f70e61a130 con 0x55f70e646cc0 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e65ff30 0x55f70e6ddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e65ff30 0x55f70e6ddd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55288/0 (socket says 172.21.15.99:55288) 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 -- 172.21.15.99:0/3506381514 learned_addr learned my addr 172.21.15.99:0/3506381514 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e0254640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e646cc0 0x55f70e6e22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e0a55640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e660310 0x55f70e61a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 -- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e646cc0 msgr2=0x55f70e6e22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f70e646cc0 0x55f70e6e22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 -- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e660310 msgr2=0x55f70e61a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.755 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f70e660310 0x55f70e61a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 -- 172.21.15.99:0/3506381514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f70e6def20 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dfa53640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e65ff30 0x55f70e6ddd10 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00096a0 tx=0x7fe2d0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dda4f640 1 -- 172.21.15.99:0/3506381514 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0006a60 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dda4f640 1 -- 172.21.15.99:0/3506381514 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0006c00 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f70e6e17d0 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2dda4f640 1 -- 172.21.15.99:0/3506381514 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0014440 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.749+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f70e6de420 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.750+0000 7fe2dda4f640 1 -- 172.21.15.99:0/3506381514 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe2d001d440 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.751+0000 7fe2dda4f640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2cc069ee0 0x7fe2cc06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.751+0000 7fe2dda4f640 1 -- 172.21.15.99:0/3506381514 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe2d0089320 con 0x55f70e65ff30 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.751+0000 7fe2e0254640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2cc069ee0 0x7fe2cc06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.751+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f70e787d10 0x55f70e7880f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.751+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.d 13:b50e409b:::mds_snaptable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x55f70e788630 con 0x55f70e787d10 2024-03-04T10:10:21.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.751+0000 7fe2e0a55640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f70e787d10 0x55f70e7880f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e0254640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2cc069ee0 0x7fe2cc06c3a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4055000 tx=0x7fe2d4050000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e0a55640 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f70e787d10 0x55f70e7880f0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e0a55640 1 -- 172.21.15.99:0/3506381514 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 mds_snaptable [omap-get-header] v0'0 uv1 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fe2d8011870 con 0x55f70e787d10 2024-03-04T10:10:21.757 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:21.758 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f70e787d10 msgr2=0x55f70e7880f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f70e787d10 0x55f70e7880f0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2cc069ee0 msgr2=0x7fe2cc06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe2cc069ee0 0x7fe2cc06c3a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe2d4055000 tx=0x7fe2d4050000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e65ff30 msgr2=0x55f70e6ddd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 --2- 172.21.15.99:0/3506381514 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f70e65ff30 0x55f70e6ddd10 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00096a0 tx=0x7fe2d0009b60 comp rx=0 tx=0).stop 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e0a55640 1 -- 172.21.15.99:0/3506381514 reap_dead start 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 shutdown_connections 2024-03-04T10:10:21.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 >> 172.21.15.99:0/3506381514 conn(0x55f70e617400 msgr2=0x55f70e6177e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 shutdown_connections 2024-03-04T10:10:21.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.752+0000 7fe2e2756f40 1 -- 172.21.15.99:0/3506381514 wait complete. 2024-03-04T10:10:21.760 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_inotable 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.839+0000 7f5dbb00ef40 1 Processor -- start 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.839+0000 7f5dbb00ef40 1 -- start start 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5dbb00ef40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 0x555b59d49f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5dbb00ef40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59e1e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5dbb00ef40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59e1eb00 0x555b59e1ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5dbb00ef40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b59e210d0 con 0x555b59d4b1b0 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5dbb00ef40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b59cefb30 con 0x555b59e1eb00 2024-03-04T10:10:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5dbb00ef40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b59e21370 con 0x555b59d4b590 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 0x555b59d49f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 0x555b59d49f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55306/0 (socket says 172.21.15.99:55306) 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 -- 172.21.15.99:0/4284892350 learned_addr learned my addr 172.21.15.99:0/4284892350 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 -- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 msgr2=0x555b59e1e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db775a640 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59e1e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db875c640 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59e1eb00 0x555b59e1ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59e1e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 -- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59e1eb00 msgr2=0x555b59e1ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59e1eb00 0x555b59e1ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 -- 172.21.15.99:0/4284892350 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b59d74a90 con 0x555b59d4b1b0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 0x555b59d49f60 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f5da80097c0 tx=0x7f5da8009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b57317f65a3c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db6758640 1 -- 172.21.15.99:0/4284892350 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da800ebc0 con 0x555b59d4b1b0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db6758640 1 -- 172.21.15.99:0/4284892350 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da800ed60 con 0x555b59d4b1b0 2024-03-04T10:10:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.840+0000 7f5db6758640 1 -- 172.21.15.99:0/4284892350 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da800f670 con 0x555b59d4b1b0 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/4284892350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 msgr2=0x555b59d49f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 0x555b59d49f60 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f5da80097c0 tx=0x7f5da8009c80 comp rx=0 tx=0).stop 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/4284892350 shutdown_connections 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59e1eb00 0x555b59e1ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59e1e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/4284892350 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59d4b1b0 0x555b59d49f60 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/4284892350 >> 172.21.15.99:0/4284892350 conn(0x555b59d48400 msgr2=0x555b59d48810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/4284892350 shutdown_connections 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/4284892350 wait complete. 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.841+0000 7f5dbb00ef40 1 Processor -- start 2024-03-04T10:10:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 -- start start 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59d4b1b0 0x555b59d8c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59eba260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59e1eb00 0x555b59eba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b59dffe40 con 0x555b59e1eb00 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b59d74ff0 con 0x555b59d4b1b0 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5dbb00ef40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555b59d74d10 con 0x555b59d4b590 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5db875c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59e1eb00 0x555b59eba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5db875c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59e1eb00 0x555b59eba7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55318/0 (socket says 172.21.15.99:55318) 2024-03-04T10:10:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5db875c640 1 -- 172.21.15.99:0/1391441654 learned_addr learned my addr 172.21.15.99:0/1391441654 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5db775a640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59eba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.842+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59d4b1b0 0x555b59d8c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db875c640 1 -- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 msgr2=0x555b59eba260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db875c640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59eba260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db875c640 1 -- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59d4b1b0 msgr2=0x555b59d8c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db875c640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59d4b1b0 0x555b59d8c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db875c640 1 -- 172.21.15.99:0/1391441654 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b59eb94a0 con 0x555b59e1eb00 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db775a640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555b59d4b590 0x555b59eba260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555b59d4b1b0 0x555b59d8c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db875c640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59e1eb00 0x555b59eba7a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5db00449b0 tx=0x7f5da0004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db5756640 1 -- 172.21.15.99:0/1391441654 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db000c280 con 0x555b59e1eb00 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db5756640 1 -- 172.21.15.99:0/1391441654 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db0055690 con 0x555b59e1eb00 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5db5756640 1 -- 172.21.15.99:0/1391441654 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0055a60 con 0x555b59e1eb00 2024-03-04T10:10:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555b59eb9780 con 0x555b59e1eb00 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.843+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b59d782f0 con 0x555b59e1eb00 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.844+0000 7f5db5756640 1 -- 172.21.15.99:0/1391441654 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5db0055c00 con 0x555b59e1eb00 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.844+0000 7f5db5756640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5da4069eb0 0x7f5da406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5db5756640 1 -- 172.21.15.99:0/1391441654 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5db00c97e0 con 0x555b59e1eb00 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555b59e092d0 0x555b59ebb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.13 13:c91d4a1d:::mds0_inotable:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94) v9 -- 0x555b59ebb7e0 con 0x555b59e092d0 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5da4069eb0 0x7f5da406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5db775a640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555b59e092d0 0x555b59ebb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5db7f5b640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5da4069eb0 0x7f5da406c370 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5da8001050 tx=0x7f5da801c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.845+0000 7f5db775a640 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555b59e092d0 0x555b59ebb2a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5db775a640 1 -- 172.21.15.99:0/1391441654 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 mds0_inotable [omap-get-header] v0'0 uv3 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f5dac0098f0 con 0x555b59e092d0 2024-03-04T10:10:21.851 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:21.852 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555b59e092d0 msgr2=0x555b59ebb2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555b59e092d0 0x555b59ebb2a0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5da4069eb0 msgr2=0x7f5da406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5da4069eb0 0x7f5da406c370 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5da8001050 tx=0x7f5da801c040 comp rx=0 tx=0).stop 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59e1eb00 msgr2=0x555b59eba7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 --2- 172.21.15.99:0/1391441654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555b59e1eb00 0x555b59eba7a0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f5db00449b0 tx=0x7f5da0004120 comp rx=0 tx=0).stop 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5db875c640 1 -- 172.21.15.99:0/1391441654 reap_dead start 2024-03-04T10:10:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 shutdown_connections 2024-03-04T10:10:21.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 >> 172.21.15.99:0/1391441654 conn(0x555b59d48400 msgr2=0x555b59d48810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 shutdown_connections 2024-03-04T10:10:21.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.846+0000 7f5dbb00ef40 1 -- 172.21.15.99:0/1391441654 wait complete. 2024-03-04T10:10:21.854 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.935+0000 7f55ba20bf40 1 Processor -- start 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.935+0000 7f55ba20bf40 1 -- start start 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55ba20bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 0x5604001ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55ba20bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560400169f30 0x560400124130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55ba20bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56040016a310 0x5604001f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55ba20bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604001f9c40 con 0x560400150af0 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55ba20bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040016bd40 con 0x56040016a310 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55ba20bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604001f9dc0 con 0x560400169f30 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 0x5604001ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 0x5604001ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55336/0 (socket says 172.21.15.99:55336) 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 -- 172.21.15.99:0/3148074935 learned_addr learned my addr 172.21.15.99:0/3148074935 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 -- 172.21.15.99:0/3148074935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560400169f30 msgr2=0x560400124130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560400169f30 0x560400124130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 -- 172.21.15.99:0/3148074935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56040016a310 msgr2=0x5604001f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56040016a310 0x5604001f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 -- 172.21.15.99:0/3148074935 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604001f5900 con 0x560400150af0 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b70df640 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 0x5604001ef270 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f55a80095d0 tx=0x7f55a8009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460f3ed20d8eed2a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b58dc640 1 -- 172.21.15.99:0/3148074935 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8006680 con 0x560400150af0 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b58dc640 1 -- 172.21.15.99:0/3148074935 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a8006820 con 0x560400150af0 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.936+0000 7f55b58dc640 1 -- 172.21.15.99:0/3148074935 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8006c10 con 0x560400150af0 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 -- 172.21.15.99:0/3148074935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 msgr2=0x5604001ef270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 0x5604001ef270 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f55a80095d0 tx=0x7f55a8009a90 comp rx=0 tx=0).stop 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 -- 172.21.15.99:0/3148074935 shutdown_connections 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56040016a310 0x5604001f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x560400169f30 0x560400124130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/3148074935 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400150af0 0x5604001ef270 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 -- 172.21.15.99:0/3148074935 >> 172.21.15.99:0/3148074935 conn(0x560400121400 msgr2=0x560400201380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 -- 172.21.15.99:0/3148074935 shutdown_connections 2024-03-04T10:10:21.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 -- 172.21.15.99:0/3148074935 wait complete. 2024-03-04T10:10:21.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 Processor -- start 2024-03-04T10:10:21.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.937+0000 7f55ba20bf40 1 -- start start 2024-03-04T10:10:21.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55ba20bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560400150af0 0x5604001e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55ba20bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400169f30 0x5604001efd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55ba20bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56040016a310 0x5604001f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55ba20bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604001f5be0 con 0x560400169f30 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55ba20bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604001f5310 con 0x560400150af0 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55ba20bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604001f8f50 con 0x56040016a310 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400169f30 0x5604001efd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400169f30 0x5604001efd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55338/0 (socket says 172.21.15.99:55338) 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 -- 172.21.15.99:0/473957065 learned_addr learned my addr 172.21.15.99:0/473957065 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b70df640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560400150af0 0x5604001e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b78e0640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56040016a310 0x5604001f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 -- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56040016a310 msgr2=0x5604001f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56040016a310 0x5604001f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 -- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560400150af0 msgr2=0x5604001e78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560400150af0 0x5604001e78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 -- 172.21.15.99:0/473957065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560400290250 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b78e0640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x56040016a310 0x5604001f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.938+0000 7f55b68de640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400169f30 0x5604001efd20 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f55ac01cfd0 tx=0x7f55ac005520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.939+0000 7f55b48da640 1 -- 172.21.15.99:0/473957065 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac062070 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.939+0000 7f55b48da640 1 -- 172.21.15.99:0/473957065 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac052c60 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.939+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560400290530 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.939+0000 7f55b48da640 1 -- 172.21.15.99:0/473957065 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac057040 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.939+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560400290af0 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.940+0000 7f55b48da640 1 -- 172.21.15.99:0/473957065 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f55ac065070 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.940+0000 7f55b48da640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f55a4069ee0 0x7f55a406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.940+0000 7f55b48da640 1 -- 172.21.15.99:0/473957065 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f55ac0cae20 con 0x560400169f30 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.940+0000 7f55b70df640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f55a4069ee0 0x7f55a406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.941+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5604001ed770 0x56040011fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.941+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.13 13:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x5604001201d0 con 0x5604001ed770 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.941+0000 7f55b78e0640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5604001ed770 0x56040011fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:21.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.941+0000 7f55b70df640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f55a4069ee0 0x7f55a406c3a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f55a800afd0 tx=0x7f55a800f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.941+0000 7f55b78e0640 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5604001ed770 0x56040011fc90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.941+0000 7f55b78e0640 1 -- 172.21.15.99:0/473957065 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 100.00000000 [omap-get-header out=274b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+274 (crc 0 0 0) 0x7f55b000b6d0 con 0x5604001ed770 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:header (274 bytes) : 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000000 04 03 0c 01 00 00 01 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000030 00 00 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000060 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.947 INFO:teuthology.orchestra.run.smithi099.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:00000090 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:000000a0 00 00 00 00 00 00 00 00 ff 9d e5 65 43 8a 6f 32 |...........eC.o2| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 |................| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:000000e0 00 00 00 00 00 00 ff 9d e5 65 43 8a 6f 32 00 00 |.........eC.o2..| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:00000110 00 00 |..| 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout:00000112 2024-03-04T10:10:21.948 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5604001ed770 msgr2=0x56040011fc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5604001ed770 0x56040011fc90 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f55a4069ee0 msgr2=0x7f55a406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f55a4069ee0 0x7f55a406c3a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f55a800afd0 tx=0x7f55a800f000 comp rx=0 tx=0).stop 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400169f30 msgr2=0x5604001efd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 --2- 172.21.15.99:0/473957065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x560400169f30 0x5604001efd20 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f55ac01cfd0 tx=0x7f55ac005520 comp rx=0 tx=0).stop 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55b78e0640 1 -- 172.21.15.99:0/473957065 reap_dead start 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 shutdown_connections 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 >> 172.21.15.99:0/473957065 conn(0x560400121400 msgr2=0x560400201380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 shutdown_connections 2024-03-04T10:10:21.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:21.942+0000 7f55ba20bf40 1 -- 172.21.15.99:0/473957065 wait complete. 2024-03-04T10:10:21.951 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader mds0_sessionmap 2024-03-04T10:10:22.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.029+0000 7fa273556f40 1 Processor -- start 2024-03-04T10:10:22.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.029+0000 7fa273556f40 1 -- start start 2024-03-04T10:10:22.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa273556f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485df8cc0 0x55e485e9f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa273556f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485dcac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa273556f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485e12310 0x55e485dcc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa273556f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e485d70b30 con 0x55e485e11f30 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa273556f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e485d74a90 con 0x55e485df8cc0 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa273556f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e485e13d40 con 0x55e485e12310 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485dcac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485dcac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55362/0 (socket says 172.21.15.99:55362) 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 -- 172.21.15.99:0/699813028 learned_addr learned my addr 172.21.15.99:0/699813028 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 -- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485e12310 msgr2=0x55e485dcc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485e12310 0x55e485dcc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 -- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485df8cc0 msgr2=0x55e485e9f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa271054640 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485df8cc0 0x55e485e9f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485df8cc0 0x55e485e9f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.030+0000 7fa270853640 1 -- 172.21.15.99:0/699813028 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e485dccc10 con 0x55e485e11f30 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa270853640 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485dcac60 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa268044770 tx=0x7fa26804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548f1065a6c00b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa2637fe640 1 -- 172.21.15.99:0/699813028 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268057040 con 0x55e485e11f30 2024-03-04T10:10:22.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa2637fe640 1 -- 172.21.15.99:0/699813028 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa268062070 con 0x55e485e11f30 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa2637fe640 1 -- 172.21.15.99:0/699813028 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268052e10 con 0x55e485e11f30 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 -- 172.21.15.99:0/699813028 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 msgr2=0x55e485dcac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485dcac60 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fa268044770 tx=0x7fa26804c000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 -- 172.21.15.99:0/699813028 shutdown_connections 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485e12310 0x55e485dcc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485dcac60 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 --2- 172.21.15.99:0/699813028 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485df8cc0 0x55e485e9f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 -- 172.21.15.99:0/699813028 >> 172.21.15.99:0/699813028 conn(0x55e485dc9400 msgr2=0x55e485e93b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 -- 172.21.15.99:0/699813028 shutdown_connections 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.031+0000 7fa273556f40 1 -- 172.21.15.99:0/699813028 wait complete. 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.032+0000 7fa273556f40 1 Processor -- start 2024-03-04T10:10:22.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.032+0000 7fa273556f40 1 -- start start 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.032+0000 7fa273556f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485df8cc0 0x55e485e85d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.032+0000 7fa273556f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485e8b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.032+0000 7fa273556f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485e12310 0x55e485e8bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa273556f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e485e10090 con 0x55e485e11f30 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa273556f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e485dc97e0 con 0x55e485e12310 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa273556f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e485e9fb90 con 0x55e485df8cc0 2024-03-04T10:10:22.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485e8b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485e8b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55374/0 (socket says 172.21.15.99:55374) 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 -- 172.21.15.99:0/4122625945 learned_addr learned my addr 172.21.15.99:0/4122625945 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 -- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485df8cc0 msgr2=0x55e485e85d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e485df8cc0 0x55e485e85d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 -- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485e12310 msgr2=0x55e485e8bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e485e12310 0x55e485e8bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 -- 172.21.15.99:0/4122625945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e485e0e870 con 0x55e485e11f30 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa270853640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485e8b9e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa26805f040 tx=0x7fa268002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa2627fc640 1 -- 172.21.15.99:0/4122625945 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26806c030 con 0x55e485e11f30 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa2627fc640 1 -- 172.21.15.99:0/4122625945 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26805d470 con 0x55e485e11f30 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e485e0eb10 con 0x55e485e11f30 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa2627fc640 1 -- 172.21.15.99:0/4122625945 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268057070 con 0x55e485e11f30 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.033+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e485e0f0d0 con 0x55e485e11f30 2024-03-04T10:10:22.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa2627fc640 1 -- 172.21.15.99:0/4122625945 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa26806f030 con 0x55e485e11f30 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa2627fc640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa254069ee0 0x7fa25406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa2627fc640 1 -- 172.21.15.99:0/4122625945 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fa2680cc580 con 0x55e485e11f30 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa273556f40 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55e485e911f0 0x55e485e8c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa271054640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa254069ee0 0x7fa25406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:1 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x55e485e91710 con 0x55e485e911f0 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.035+0000 7fa271855640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55e485e911f0 0x55e485e8c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa271054640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa254069ee0 0x7fa25406c3a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa264000f60 tx=0x7fa2640013b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa271855640 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55e485e911f0 0x55e485e8c970 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa271855640 1 -- 172.21.15.99:0/4122625945 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-header out=14b] v0'0 uv3 ondisk = 0) v8 ==== 159+0+14 (crc 0 0 0) 0x7fa26c006090 con 0x55e485e911f0 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stdout:header (14 bytes) : 2024-03-04T10:10:22.040 INFO:teuthology.orchestra.run.smithi099.stdout:00000000 01 01 08 00 00 00 04 00 00 00 00 00 00 00 |..............| 2024-03-04T10:10:22.041 INFO:teuthology.orchestra.run.smithi099.stdout:0000000e 2024-03-04T10:10:22.041 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55e485e911f0 msgr2=0x55e485e8c970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55e485e911f0 0x55e485e8c970 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa254069ee0 msgr2=0x7fa25406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa254069ee0 0x7fa25406c3a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa264000f60 tx=0x7fa2640013b0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 msgr2=0x55e485e8b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 --2- 172.21.15.99:0/4122625945 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e485e11f30 0x55e485e8b9e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fa26805f040 tx=0x7fa268002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa271855640 1 -- 172.21.15.99:0/4122625945 reap_dead start 2024-03-04T10:10:22.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.036+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 shutdown_connections 2024-03-04T10:10:22.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.037+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 >> 172.21.15.99:0/4122625945 conn(0x55e485dc9400 msgr2=0x55e485dcac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.037+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 shutdown_connections 2024-03-04T10:10:22.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.037+0000 7fa273556f40 1 -- 172.21.15.99:0/4122625945 wait complete. 2024-03-04T10:10:22.043 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 400.00000000 2024-03-04T10:10:22.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.122+0000 7f80a4435f40 1 Processor -- start 2024-03-04T10:10:22.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.122+0000 7f80a4435f40 1 -- start start 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a4435f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bde733a0 0x5595bde729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a4435f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 0x5595bdeb4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a4435f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdeb5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a4435f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595bdf4b720 con 0x5595bdf49f90 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a4435f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595bde1bb30 con 0x5595bde733a0 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a4435f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5595bdf4b8a0 con 0x5595bdf4a370 2024-03-04T10:10:22.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 0x5595bdeb4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 0x5595bdeb4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55392/0 (socket says 172.21.15.99:55392) 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 -- 172.21.15.99:0/2195175631 learned_addr learned my addr 172.21.15.99:0/2195175631 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a1795640 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bde733a0 0x5595bde729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a1f96640 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdeb5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 -- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 msgr2=0x5595bdeb5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdeb5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 -- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bde733a0 msgr2=0x5595bde729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bde733a0 0x5595bde729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.123+0000 7f80a0f94640 1 -- 172.21.15.99:0/2195175631 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595bdea3900 con 0x5595bdf49f90 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a0f94640 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 0x5595bdeb4f30 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8098001d10 tx=0x7f809800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac638855438ba596 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80937fe640 1 -- 172.21.15.99:0/2195175631 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8098001e60 con 0x5595bdf49f90 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80937fe640 1 -- 172.21.15.99:0/2195175631 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809800bd80 con 0x5595bdf49f90 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80937fe640 1 -- 172.21.15.99:0/2195175631 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809800f640 con 0x5595bdf49f90 2024-03-04T10:10:22.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 -- 172.21.15.99:0/2195175631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 msgr2=0x5595bdeb4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 0x5595bdeb4f30 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f8098001d10 tx=0x7f809800b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 -- 172.21.15.99:0/2195175631 shutdown_connections 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdeb5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bdf49f90 0x5595bdeb4f30 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 --2- 172.21.15.99:0/2195175631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bde733a0 0x5595bde729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 -- 172.21.15.99:0/2195175631 >> 172.21.15.99:0/2195175631 conn(0x5595bde74400 msgr2=0x5595bdf48ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 -- 172.21.15.99:0/2195175631 shutdown_connections 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.124+0000 7f80a4435f40 1 -- 172.21.15.99:0/2195175631 wait complete. 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 Processor -- start 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 -- start start 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bde733a0 0x5595bdfde110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bdf49f90 0x5595bdebae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdfdd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595bdea08b0 con 0x5595bde733a0 2024-03-04T10:10:22.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.125+0000 7f80a4435f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595bdf457a0 con 0x5595bdf49f90 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a4435f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5595bdf454b0 con 0x5595bdf4a370 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bde733a0 0x5595bdfde110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bde733a0 0x5595bdfde110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55406/0 (socket says 172.21.15.99:55406) 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 -- 172.21.15.99:0/1225948630 learned_addr learned my addr 172.21.15.99:0/1225948630 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a0f94640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bdf49f90 0x5595bdebae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1f96640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdfdd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 -- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 msgr2=0x5595bdfdd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdfdd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 -- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bdf49f90 msgr2=0x5595bdebae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5595bdf49f90 0x5595bdebae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 -- 172.21.15.99:0/1225948630 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595bdeb6010 con 0x5595bde733a0 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1f96640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5595bdf4a370 0x5595bdfdd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a1795640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bde733a0 0x5595bdfde110 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f809401b900 tx=0x7f8094003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80927fc640 1 -- 172.21.15.99:0/1225948630 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094061070 con 0x5595bde733a0 2024-03-04T10:10:22.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80927fc640 1 -- 172.21.15.99:0/1225948630 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809405d3f0 con 0x5595bde733a0 2024-03-04T10:10:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5595bdea2f10 con 0x5595bde733a0 2024-03-04T10:10:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80927fc640 1 -- 172.21.15.99:0/1225948630 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094057040 con 0x5595bde733a0 2024-03-04T10:10:22.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.126+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595bdeb5d30 con 0x5595bde733a0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80927fc640 1 -- 172.21.15.99:0/1225948630 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8094064070 con 0x5595bde733a0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80927fc640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8084069eb0 0x7f808406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80927fc640 1 -- 172.21.15.99:0/1225948630 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f80940caeb0 con 0x5595bde733a0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80a4435f40 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5595bdebb3a0 0x5595bdfdbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80a0f94640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8084069eb0 0x7f808406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.f 13:f1f69726:::400.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x5595bdf41040 con 0x5595bdebb3a0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.128+0000 7f80a1f96640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5595bdebb3a0 0x5595bdfdbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.129+0000 7f80a0f94640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8084069eb0 0x7f808406c370 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x5595bdf47d80 tx=0x7f8098009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.129+0000 7f80a1f96640 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5595bdebb3a0 0x5595bdfdbd60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a1f96640 1 -- 172.21.15.99:0/1225948630 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 400.00000000 [omap-get-header] v0'0 uv1 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f809c009870 con 0x5595bdebb3a0 2024-03-04T10:10:22.134 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:22.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5595bdebb3a0 msgr2=0x5595bdfdbd60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5595bdebb3a0 0x5595bdfdbd60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8084069eb0 msgr2=0x7f808406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8084069eb0 0x7f808406c370 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x5595bdf47d80 tx=0x7f8098009000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bde733a0 msgr2=0x5595bdfde110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 --2- 172.21.15.99:0/1225948630 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5595bde733a0 0x5595bdfde110 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f809401b900 tx=0x7f8094003330 comp rx=0 tx=0).stop 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a1f96640 1 -- 172.21.15.99:0/1225948630 reap_dead start 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 shutdown_connections 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 >> 172.21.15.99:0/1225948630 conn(0x5595bde74400 msgr2=0x5595bdf48ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 shutdown_connections 2024-03-04T10:10:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.131+0000 7f80a4435f40 1 -- 172.21.15.99:0/1225948630 wait complete. 2024-03-04T10:10:22.136 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 100.00000000.inode 2024-03-04T10:10:22.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.215+0000 7f77e6932f40 1 Processor -- start 2024-03-04T10:10:22.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.215+0000 7f77e6932f40 1 -- start start 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e6932f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90a57c90 0x558d90b31230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e6932f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b25630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e6932f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90b2f5f0 0x558d90b30640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e6932f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d90a00b30 con 0x558d90b295a0 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e6932f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d90a04a90 con 0x558d90b2f5f0 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e6932f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d90aa2ab0 con 0x558d90a57c90 2024-03-04T10:10:22.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b25630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b25630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55422/0 (socket says 172.21.15.99:55422) 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 -- 172.21.15.99:0/2635859072 learned_addr learned my addr 172.21.15.99:0/2635859072 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 -- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90a57c90 msgr2=0x558d90b31230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90a57c90 0x558d90b31230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 -- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90b2f5f0 msgr2=0x558d90b30640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e4c33640 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90b2f5f0 0x558d90b30640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90b2f5f0 0x558d90b30640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 -- 172.21.15.99:0/2635859072 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d90a899f0 con 0x558d90b295a0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e4c33640 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90b2f5f0 0x558d90b30640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.216+0000 7f77e3c31640 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b25630 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b2e0 tx=0x7f77d800b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6ed65a4fbca141 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e2c2f640 1 -- 172.21.15.99:0/2635859072 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8008070 con 0x558d90b295a0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e2c2f640 1 -- 172.21.15.99:0/2635859072 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d800d500 con 0x558d90b295a0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e2c2f640 1 -- 172.21.15.99:0/2635859072 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d800d8b0 con 0x558d90b295a0 2024-03-04T10:10:22.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 -- 172.21.15.99:0/2635859072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 msgr2=0x558d90b25630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b25630 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f77d800b2e0 tx=0x7f77d800b7a0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 -- 172.21.15.99:0/2635859072 shutdown_connections 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90b2f5f0 0x558d90b30640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b25630 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 --2- 172.21.15.99:0/2635859072 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90a57c90 0x558d90b31230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 -- 172.21.15.99:0/2635859072 >> 172.21.15.99:0/2635859072 conn(0x558d90a59400 msgr2=0x558d90a85980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 -- 172.21.15.99:0/2635859072 shutdown_connections 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.217+0000 7f77e6932f40 1 -- 172.21.15.99:0/2635859072 wait complete. 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 Processor -- start 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 -- start start 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90a57c90 0x558d90b0e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b30500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90b2f5f0 0x558d90b30a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d90b24e60 con 0x558d90b295a0 2024-03-04T10:10:22.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d90a897a0 con 0x558d90a57c90 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e6932f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d90b27ff0 con 0x558d90b2f5f0 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.218+0000 7f77e3c31640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b30500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b30500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55424/0 (socket says 172.21.15.99:55424) 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 -- 172.21.15.99:0/198524815 learned_addr learned my addr 172.21.15.99:0/198524815 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e4432640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90a57c90 0x558d90b0e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e4c33640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90b2f5f0 0x558d90b30a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 -- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90b2f5f0 msgr2=0x558d90b30a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90b2f5f0 0x558d90b30a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 -- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90a57c90 msgr2=0x558d90b0e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90a57c90 0x558d90b0e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 -- 172.21.15.99:0/198524815 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d90b31770 con 0x558d90b295a0 2024-03-04T10:10:22.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e4c33640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x558d90b2f5f0 0x558d90b30a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e4432640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558d90a57c90 0x558d90b0e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e3c31640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b30500 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f77d8000f60 tx=0x7f77d8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e1c2d640 1 -- 172.21.15.99:0/198524815 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8021030 con 0x558d90b295a0 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e1c2d640 1 -- 172.21.15.99:0/198524815 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d801c650 con 0x558d90b295a0 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558d90a874e0 con 0x558d90b295a0 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e1c2d640 1 -- 172.21.15.99:0/198524815 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d801c840 con 0x558d90b295a0 2024-03-04T10:10:22.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.219+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d90a87aa0 con 0x558d90b295a0 2024-03-04T10:10:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.220+0000 7f77e1c2d640 1 -- 172.21.15.99:0/198524815 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f77d8024070 con 0x558d90b295a0 2024-03-04T10:10:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e1c2d640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f77d0069eb0 0x7f77d006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e1c2d640 1 -- 172.21.15.99:0/198524815 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f77d808a5d0 con 0x558d90b295a0 2024-03-04T10:10:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e4432640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f77d0069eb0 0x7f77d006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e6932f40 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x558d90b220d0 0x558d90b1eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x558d90b1f090 con 0x558d90b220d0 2024-03-04T10:10:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e4c33640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x558d90b220d0 0x558d90b1eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.221+0000 7f77e4432640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f77d0069eb0 0x7f77d006c370 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f77d4054000 tx=0x7f77d404c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e4c33640 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x558d90b220d0 0x558d90b1eb50 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e4c33640 1 -- 172.21.15.99:0/198524815 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-header] v0'0 uv3 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f77dc00b6f0 con 0x558d90b220d0 2024-03-04T10:10:22.227 INFO:teuthology.orchestra.run.smithi099.stdout:header (0 bytes) : 2024-03-04T10:10:22.227 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:22.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x558d90b220d0 msgr2=0x558d90b1eb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x558d90b220d0 0x558d90b1eb50 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f77d0069eb0 msgr2=0x7f77d006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f77d0069eb0 0x7f77d006c370 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f77d4054000 tx=0x7f77d404c000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 msgr2=0x558d90b30500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 --2- 172.21.15.99:0/198524815 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x558d90b295a0 0x558d90b30500 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f77d8000f60 tx=0x7f77d8000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e4c33640 1 -- 172.21.15.99:0/198524815 reap_dead start 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 shutdown_connections 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 >> 172.21.15.99:0/198524815 conn(0x558d90a59400 msgr2=0x558d90b31940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 shutdown_connections 2024-03-04T10:10:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.222+0000 7f77e6932f40 1 -- 172.21.15.99:0/198524815 wait complete. 2024-03-04T10:10:22.229 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapheader 1.00000000 2024-03-04T10:10:22.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.309+0000 7f5de6e46f40 1 Processor -- start 2024-03-04T10:10:22.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.309+0000 7f5de6e46f40 1 -- start start 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de6e46f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dcc003ab0 0x564a9cad7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de6e46f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 0x564a9cad9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de6e46f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 0x564a9ca473c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de6e46f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9c9a8b30 con 0x564a9cad5ab0 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de6e46f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9c9aca90 con 0x7f5dcc003ab0 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de6e46f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a9ca4b6d0 con 0x564a9cace190 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 0x564a9cad9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 0x564a9cad9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55440/0 (socket says 172.21.15.99:55440) 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 -- 172.21.15.99:0/3721752359 learned_addr learned my addr 172.21.15.99:0/3721752359 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 -- 172.21.15.99:0/3721752359 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 msgr2=0x564a9ca473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 0x564a9ca473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 -- 172.21.15.99:0/3721752359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dcc003ab0 msgr2=0x564a9cad7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dcc003ab0 0x564a9cad7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 -- 172.21.15.99:0/3721752359 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a9ca2d790 con 0x564a9cad5ab0 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de38c9640 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 0x564a9cad9a80 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80017a0 tx=0x7f5dd80099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126b793dd21c1a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de28c7640 1 -- 172.21.15.99:0/3721752359 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd800e780 con 0x564a9cad5ab0 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de28c7640 1 -- 172.21.15.99:0/3721752359 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd8013030 con 0x564a9cad5ab0 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.310+0000 7f5de28c7640 1 -- 172.21.15.99:0/3721752359 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd800eb80 con 0x564a9cad5ab0 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 -- 172.21.15.99:0/3721752359 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 msgr2=0x564a9cad9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 0x564a9cad9a80 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80017a0 tx=0x7f5dd80099d0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 -- 172.21.15.99:0/3721752359 shutdown_connections 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 0x564a9ca473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x564a9cad5ab0 0x564a9cad9a80 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/3721752359 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5dcc003ab0 0x564a9cad7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 -- 172.21.15.99:0/3721752359 >> 172.21.15.99:0/3721752359 conn(0x564a9ca01400 msgr2=0x564a9ca2cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 -- 172.21.15.99:0/3721752359 shutdown_connections 2024-03-04T10:10:22.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 -- 172.21.15.99:0/3721752359 wait complete. 2024-03-04T10:10:22.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 Processor -- start 2024-03-04T10:10:22.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.311+0000 7f5de6e46f40 1 -- start start 2024-03-04T10:10:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 0x564a9cac8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a9cad5ab0 0x564a9ca476c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5dcc003ab0 0x564a9ca305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ca2dcf0 con 0x7f5dcc003ab0 2024-03-04T10:10:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9cacebd0 con 0x564a9cad5ab0 2024-03-04T10:10:22.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a9cacefb0 con 0x564a9cace190 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5dcc003ab0 0x564a9ca305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5dcc003ab0 0x564a9ca305f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55454/0 (socket says 172.21.15.99:55454) 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 -- 172.21.15.99:0/2186654049 learned_addr learned my addr 172.21.15.99:0/2186654049 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de40ca640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 0x564a9cac8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de38c9640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a9cad5ab0 0x564a9ca476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 -- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 msgr2=0x564a9cac8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x564a9cace190 0x564a9cac8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 -- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a9cad5ab0 msgr2=0x564a9ca476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a9cad5ab0 0x564a9ca476c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 -- 172.21.15.99:0/2186654049 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a9ca465f0 con 0x7f5dcc003ab0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de48cb640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5dcc003ab0 0x564a9ca305f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc005fd0 tx=0x7f5ddc052b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de18c5640 1 -- 172.21.15.99:0/2186654049 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc005a80 con 0x7f5dcc003ab0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de18c5640 1 -- 172.21.15.99:0/2186654049 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc061070 con 0x7f5dcc003ab0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de18c5640 1 -- 172.21.15.99:0/2186654049 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc052e30 con 0x7f5dcc003ab0 2024-03-04T10:10:22.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564a9ca468d0 con 0x7f5dcc003ab0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.313+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a9caca5e0 con 0x7f5dcc003ab0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.315+0000 7f5de18c5640 1 -- 172.21.15.99:0/2186654049 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5ddc05a040 con 0x7f5dcc003ab0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.315+0000 7f5de18c5640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5dd0069eb0 0x7f5dd006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.315+0000 7f5de18c5640 1 -- 172.21.15.99:0/2186654049 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f5ddc0caa20 con 0x7f5dcc003ab0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.315+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x564a9cad2a20 0x564a9cad2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.315+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x564a9cacab00 con 0x564a9cad2a20 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.315+0000 7f5de40ca640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5dd0069eb0 0x7f5dd006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.316+0000 7f5de38c9640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x564a9cad2a20 0x564a9cad2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.316+0000 7f5de40ca640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5dd0069eb0 0x7f5dd006c370 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40011d0 tx=0x7f5dd40013a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.316+0000 7f5de38c9640 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x564a9cad2a20 0x564a9cad2e00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.316+0000 7f5de38c9640 1 -- 172.21.15.99:0/2186654049 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 1.00000000 [omap-get-header out=274b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+274 (crc 0 0 0) 0x7f5dd800e800 con 0x564a9cad2a20 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:header (274 bytes) : 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000000 04 03 0c 01 00 00 0d 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000010 00 00 00 00 00 00 03 02 28 00 00 00 00 00 00 00 |........(.......| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000020 00 00 00 00 09 9e e5 65 97 8d d7 38 00 00 00 00 |.......e...8....| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000030 00 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 |................| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000040 00 00 00 00 03 02 28 00 00 00 00 00 00 00 00 00 |......(.........| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000050 00 00 09 9e e5 65 97 8d d7 38 00 00 00 00 00 00 |.....e...8......| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000060 00 00 01 00 00 00 00 00 00 00 01 00 00 00 00 00 |................| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000070 00 00 03 02 38 00 00 00 00 00 00 00 00 00 00 00 |....8...........| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000080 00 00 60 00 00 00 00 00 01 00 00 00 00 00 00 00 |..`.............| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:00000090 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:22.321 INFO:teuthology.orchestra.run.smithi099.stdout:000000a0 00 00 00 00 00 00 00 00 0a 9e e5 65 a3 a6 48 05 |...........e..H.| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:000000b0 03 02 38 00 00 00 00 00 00 00 00 00 00 00 00 00 |..8.............| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:000000c0 60 00 00 00 00 00 01 00 00 00 00 00 00 00 01 00 |`...............| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:000000e0 00 00 00 00 00 00 0a 9e e5 65 a3 a6 48 05 00 00 |.........e..H...| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:* 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:00000110 00 00 |..| 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout:00000112 2024-03-04T10:10:22.322 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.316+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x564a9cad2a20 msgr2=0x564a9cad2e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.316+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x564a9cad2a20 0x564a9cad2e00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5dd0069eb0 msgr2=0x7f5dd006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5dd0069eb0 0x7f5dd006c370 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40011d0 tx=0x7f5dd40013a0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5dcc003ab0 msgr2=0x564a9ca305f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 --2- 172.21.15.99:0/2186654049 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5dcc003ab0 0x564a9ca305f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc005fd0 tx=0x7f5ddc052b20 comp rx=0 tx=0).stop 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de48cb640 1 -- 172.21.15.99:0/2186654049 reap_dead start 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 shutdown_connections 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 >> 172.21.15.99:0/2186654049 conn(0x564a9ca01400 msgr2=0x564a9ca2cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 shutdown_connections 2024-03-04T10:10:22.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.317+0000 7f5de6e46f40 1 -- 172.21.15.99:0/2186654049 wait complete. 2024-03-04T10:10:22.325 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 10000000000.00000000 2024-03-04T10:10:22.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.404+0000 7f9f6024bf40 1 Processor -- start 2024-03-04T10:10:22.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.404+0000 7f9f6024bf40 1 -- start start 2024-03-04T10:10:22.408 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f6024bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f480022a0 0x55c6e420b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f6024bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 0x55c6e42b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f6024bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6e42b3670 0x55c6e42ab590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f6024bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6e4186b30 con 0x55c6e42b49b0 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f6024bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6e418aa90 con 0x55c6e42b3670 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f6024bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6e4228d50 con 0x7f9f480022a0 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 0x55c6e42b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 0x55c6e42b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55474/0 (socket says 172.21.15.99:55474) 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 -- 172.21.15.99:0/1373854957 learned_addr learned my addr 172.21.15.99:0/1373854957 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5d5a5640 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f480022a0 0x55c6e420b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5dda6640 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6e42b3670 0x55c6e42ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 -- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f480022a0 msgr2=0x55c6e420b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f480022a0 0x55c6e420b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 -- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6e42b3670 msgr2=0x55c6e42ab590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6e42b3670 0x55c6e42ab590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5cda4640 1 -- 172.21.15.99:0/1373854957 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6e420f7e0 con 0x55c6e42b49b0 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.405+0000 7f9f5dda6640 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6e42b3670 0x55c6e42ab590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f5cda4640 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 0x55c6e42b3130 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f9f54000f60 tx=0x7f9f5400dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829a7fcb50102330 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f4f7fe640 1 -- 172.21.15.99:0/1373854957 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5400de40 con 0x55c6e42b49b0 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f4f7fe640 1 -- 172.21.15.99:0/1373854957 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f54014020 con 0x55c6e42b49b0 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f4f7fe640 1 -- 172.21.15.99:0/1373854957 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5400bad0 con 0x55c6e42b49b0 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 -- 172.21.15.99:0/1373854957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 msgr2=0x55c6e42b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 0x55c6e42b3130 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f9f54000f60 tx=0x7f9f5400dbf0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 -- 172.21.15.99:0/1373854957 shutdown_connections 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c6e42b3670 0x55c6e42ab590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b49b0 0x55c6e42b3130 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/1373854957 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9f480022a0 0x55c6e420b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 -- 172.21.15.99:0/1373854957 >> 172.21.15.99:0/1373854957 conn(0x55c6e41df400 msgr2=0x55c6e42ae660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 -- 172.21.15.99:0/1373854957 shutdown_connections 2024-03-04T10:10:22.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.406+0000 7f9f6024bf40 1 -- 172.21.15.99:0/1373854957 wait complete. 2024-03-04T10:10:22.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.407+0000 7f9f6024bf40 1 Processor -- start 2024-03-04T10:10:22.411 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.407+0000 7f9f6024bf40 1 -- start start 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b3670 0x55c6e42b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55c6e42b49b0 0x55c6e429fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f480022a0 0x55c6e42b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6e42aab50 con 0x55c6e42b3670 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6e42b3c90 con 0x7f9f480022a0 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6e42aae90 con 0x55c6e42b49b0 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b3670 0x55c6e42b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b3670 0x55c6e42b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55478/0 (socket says 172.21.15.99:55478) 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 -- 172.21.15.99:0/3983053907 learned_addr learned my addr 172.21.15.99:0/3983053907 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5cda4640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55c6e42b49b0 0x55c6e429fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5dda6640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f480022a0 0x55c6e42b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 -- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55c6e42b49b0 msgr2=0x55c6e429fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55c6e42b49b0 0x55c6e429fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 -- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f480022a0 msgr2=0x55c6e42b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f480022a0 0x55c6e42b7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 -- 172.21.15.99:0/3983053907 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6e42b1600 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5dda6640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9f480022a0 0x55c6e42b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f5d5a5640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b3670 0x55c6e42b2640 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9f50000c00 tx=0x7f9f50005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f4e7fc640 1 -- 172.21.15.99:0/3983053907 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50057040 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f4e7fc640 1 -- 172.21.15.99:0/3983053907 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f500529f0 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c6e42a0180 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f4e7fc640 1 -- 172.21.15.99:0/3983053907 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f50052dc0 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.408+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6e42b1b60 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f4e7fc640 1 -- 172.21.15.99:0/3983053907 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9f5005a050 con 0x55c6e42b3670 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f4e7fc640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f40069eb0 0x7f9f4006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f4e7fc640 1 -- 172.21.15.99:0/3983053907 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f9f500cacb0 con 0x55c6e42b3670 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f5cda4640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f40069eb0 0x7f9f4006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55c6e42a9de0 0x55c6e4351160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x55c6e41de960 con 0x55c6e42a9de0 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.410+0000 7f9f5dda6640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55c6e42a9de0 0x55c6e4351160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f5cda4640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f40069eb0 0x7f9f4006c370 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9f54000f60 tx=0x7f9f54008000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f5dda6640 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55c6e42a9de0 0x55c6e4351160 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f5dda6640 1 -- 172.21.15.99:0/3983053907 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-keys out=21b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+21 (crc 0 0 0) 0x7f9f58006d70 con 0x55c6e42a9de0 2024-03-04T10:10:22.416 INFO:teuthology.orchestra.run.smithi099.stdout:sixmegs_head 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55c6e42a9de0 msgr2=0x55c6e4351160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55c6e42a9de0 0x55c6e4351160 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f40069eb0 msgr2=0x7f9f4006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9f40069eb0 0x7f9f4006c370 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f9f54000f60 tx=0x7f9f54008000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b3670 msgr2=0x55c6e42b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 --2- 172.21.15.99:0/3983053907 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55c6e42b3670 0x55c6e42b2640 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9f50000c00 tx=0x7f9f50005a00 comp rx=0 tx=0).stop 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f5dda6640 1 -- 172.21.15.99:0/3983053907 reap_dead start 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 shutdown_connections 2024-03-04T10:10:22.418 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 >> 172.21.15.99:0/3983053907 conn(0x55c6e41df400 msgr2=0x55c6e42ae660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 shutdown_connections 2024-03-04T10:10:22.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.411+0000 7f9f6024bf40 1 -- 172.21.15.99:0/3983053907 wait complete. 2024-03-04T10:10:22.419 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 600.00000000 2024-03-04T10:10:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 Processor -- start 2024-03-04T10:10:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 -- start start 2024-03-04T10:10:22.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d24af0 0x559897d240b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897cf6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 0x559897cf74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x559897dcdc40 con 0x559897d3e310 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x559897d3fd40 con 0x559897d3df30 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42f77ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x559897dcddc0 con 0x559897d24af0 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42da80640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 0x559897cf74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42da80640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 0x559897cf74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55500/0 (socket says 172.21.15.99:55500) 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.498+0000 7fc42da80640 1 -- 172.21.15.99:0/4041907459 learned_addr learned my addr 172.21.15.99:0/4041907459 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42d27f640 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d24af0 0x559897d240b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42ca7e640 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897cf6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42da80640 1 -- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d24af0 msgr2=0x559897d240b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42da80640 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d24af0 0x559897d240b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42da80640 1 -- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 msgr2=0x559897cf6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42da80640 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897cf6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42da80640 1 -- 172.21.15.99:0/4041907459 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559897cf8570 con 0x559897d3e310 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42ca7e640 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897cf6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42da80640 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 0x559897cf74d0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc42801cef0 tx=0x7fc42804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80962aca03500bd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc41f7fe640 1 -- 172.21.15.99:0/4041907459 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428057070 con 0x559897d3e310 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc41f7fe640 1 -- 172.21.15.99:0/4041907459 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc428061070 con 0x559897d3e310 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc41f7fe640 1 -- 172.21.15.99:0/4041907459 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4280529a0 con 0x559897d3e310 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42f77ff40 1 -- 172.21.15.99:0/4041907459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 msgr2=0x559897cf74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 0x559897cf74d0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc42801cef0 tx=0x7fc42804c000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42f77ff40 1 -- 172.21.15.99:0/4041907459 shutdown_connections 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d3e310 0x559897cf74d0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897cf6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.499+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/4041907459 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d24af0 0x559897d240b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.500+0000 7fc42f77ff40 1 -- 172.21.15.99:0/4041907459 >> 172.21.15.99:0/4041907459 conn(0x559897cf5400 msgr2=0x559897cf57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.500+0000 7fc42f77ff40 1 -- 172.21.15.99:0/4041907459 shutdown_connections 2024-03-04T10:10:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.500+0000 7fc42f77ff40 1 -- 172.21.15.99:0/4041907459 wait complete. 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.500+0000 7fc42f77ff40 1 Processor -- start 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.500+0000 7fc42f77ff40 1 -- start start 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42f77ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d24af0 0x559897dbc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42f77ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897dc33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42f77ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d3e310 0x559897dbbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42f77ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x559897dd51c0 con 0x559897d24af0 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42f77ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x559897cf8850 con 0x559897d3df30 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42f77ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x559897dd4e50 con 0x559897d3e310 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d24af0 0x559897dbc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d24af0 0x559897dbc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55506/0 (socket says 172.21.15.99:55506) 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 -- 172.21.15.99:0/1756172973 learned_addr learned my addr 172.21.15.99:0/1756172973 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42da80640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d3e310 0x559897dbbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42ca7e640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897dc33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 -- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d3e310 msgr2=0x559897dbbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d3e310 0x559897dbbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 -- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 msgr2=0x559897dc33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559897d3df30 0x559897dc33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 -- 172.21.15.99:0/1756172973 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559897dc6b10 con 0x559897d24af0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42da80640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x559897d3e310 0x559897dbbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc42d27f640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d24af0 0x559897dbc640 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc42000f6c0 tx=0x7fc42000fb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.501+0000 7fc41e7fc640 1 -- 172.21.15.99:0/1756172973 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420019020 con 0x559897d24af0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.502+0000 7fc41e7fc640 1 -- 172.21.15.99:0/1756172973 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42000d750 con 0x559897d24af0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.502+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559897dc2900 con 0x559897d24af0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.502+0000 7fc41e7fc640 1 -- 172.21.15.99:0/1756172973 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42000db40 con 0x559897d24af0 2024-03-04T10:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.502+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559897dc2ec0 con 0x559897d24af0 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.503+0000 7fc41e7fc640 1 -- 172.21.15.99:0/1756172973 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc42001c020 con 0x559897d24af0 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.503+0000 7fc41e7fc640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc410069ee0 0x7fc41006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.503+0000 7fc41e7fc640 1 -- 172.21.15.99:0/1756172973 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fc4200897f0 con 0x559897d24af0 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.503+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x559897cf7290 0x559897dbcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0d82a743:::600.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x559897dbd3e0 con 0x559897cf7290 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42ca7e640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc410069ee0 0x7fc41006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42da80640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x559897cf7290 0x559897dbcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42da80640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x559897cf7290 0x559897dbcea0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42ca7e640 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc410069ee0 0x7fc41006c3a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc424005fd0 tx=0x7fc4240055c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42da80640 1 -- 172.21.15.99:0/1756172973 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 600.00000000 [omap-get-keys out=5b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7fc42805a070 con 0x559897cf7290 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x559897cf7290 msgr2=0x559897dbcea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x559897cf7290 0x559897dbcea0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc410069ee0 msgr2=0x7fc41006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc410069ee0 0x7fc41006c3a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc424005fd0 tx=0x7fc4240055c0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d24af0 msgr2=0x559897dbc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 --2- 172.21.15.99:0/1756172973 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x559897d24af0 0x559897dbc640 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc42000f6c0 tx=0x7fc42000fb80 comp rx=0 tx=0).stop 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42da80640 1 -- 172.21.15.99:0/1756172973 reap_dead start 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 shutdown_connections 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 >> 172.21.15.99:0/1756172973 conn(0x559897cf5400 msgr2=0x559897d240b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 shutdown_connections 2024-03-04T10:10:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.504+0000 7fc42f77ff40 1 -- 172.21.15.99:0/1756172973 wait complete. 2024-03-04T10:10:22.512 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000.inode 2024-03-04T10:10:22.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 Processor -- start 2024-03-04T10:10:22.592 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 -- start start 2024-03-04T10:10:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287b7af0 0x55de288680c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 0x55de287b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d1310 0x55de287b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de28860c40 con 0x55de287d0f30 2024-03-04T10:10:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de287d2d40 con 0x55de287b7af0 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b8d5ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de28860dc0 con 0x55de287d1310 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 0x55de287b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 0x55de287b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55528/0 (socket says 172.21.15.99:55528) 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 -- 172.21.15.99:0/1534351765 learned_addr learned my addr 172.21.15.99:0/1534351765 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 -- 172.21.15.99:0/1534351765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d1310 msgr2=0x55de287b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d1310 0x55de287b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 -- 172.21.15.99:0/1534351765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287b7af0 msgr2=0x55de288680c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287b7af0 0x55de288680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.591+0000 7f89b605c640 1 -- 172.21.15.99:0/1534351765 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de287b74d0 con 0x55de287d0f30 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b605c640 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 0x55de287b3c90 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00a770 tx=0x7f89ac00a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9701ff4d2c3adb58 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b505a640 1 -- 172.21.15.99:0/1534351765 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac018070 con 0x55de287d0f30 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b505a640 1 -- 172.21.15.99:0/1534351765 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ac007600 con 0x55de287d0f30 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b505a640 1 -- 172.21.15.99:0/1534351765 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0079b0 con 0x55de287d0f30 2024-03-04T10:10:22.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/1534351765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 msgr2=0x55de287b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 0x55de287b3c90 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00a770 tx=0x7f89ac00a860 comp rx=0 tx=0).stop 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/1534351765 shutdown_connections 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d1310 0x55de287b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287d0f30 0x55de287b3c90 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/1534351765 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287b7af0 0x55de288680c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/1534351765 >> 172.21.15.99:0/1534351765 conn(0x55de28788400 msgr2=0x55de287887e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/1534351765 shutdown_connections 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.592+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/1534351765 wait complete. 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.593+0000 7f89b8d5ff40 1 Processor -- start 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.593+0000 7f89b8d5ff40 1 -- start start 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.593+0000 7f89b8d5ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287b7af0 0x55de2885c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.593+0000 7f89b8d5ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d0f30 0x55de2884cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.593+0000 7f89b8d5ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287d1310 0x55de288f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b8d5ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de28857900 con 0x55de287b7af0 2024-03-04T10:10:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b8d5ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de2878b330 con 0x55de287d1310 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b8d5ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de28857be0 con 0x55de287d0f30 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287b7af0 0x55de2885c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287b7af0 0x55de2885c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55538/0 (socket says 172.21.15.99:55538) 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 -- 172.21.15.99:0/2619309372 learned_addr learned my addr 172.21.15.99:0/2619309372 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 -- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d0f30 msgr2=0x55de2884cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55de287d0f30 0x55de2884cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b705e640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287d1310 0x55de288f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 -- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287d1310 msgr2=0x55de288f6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287d1310 0x55de288f6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 -- 172.21.15.99:0/2619309372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de2885e9e0 con 0x55de287b7af0 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b705e640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55de287d1310 0x55de288f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b685d640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287b7af0 0x55de2885c480 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f89a800a190 tx=0x7f89a8053d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f899ffff640 1 -- 172.21.15.99:0/2619309372 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8061020 con 0x55de287b7af0 2024-03-04T10:10:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f899ffff640 1 -- 172.21.15.99:0/2619309372 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a805d670 con 0x55de287b7af0 2024-03-04T10:10:22.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f899ffff640 1 -- 172.21.15.99:0/2619309372 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8057040 con 0x55de287b7af0 2024-03-04T10:10:22.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55de2885ecc0 con 0x55de287b7af0 2024-03-04T10:10:22.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.594+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de2885f280 con 0x55de287b7af0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.595+0000 7f899ffff640 1 -- 172.21.15.99:0/2619309372 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f89a8064020 con 0x55de287b7af0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f899ffff640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89a4069eb0 0x7f89a406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f899ffff640 1 -- 172.21.15.99:0/2619309372 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f89a80cb0c0 con 0x55de287b7af0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55de288f7530 0x55de2885ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f89b605c640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89a4069eb0 0x7f89a406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x55de2885a300 con 0x55de288f7530 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f89b705e640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55de288f7530 0x55de2885ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f89b605c640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89a4069eb0 0x7f89a406c370 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x55de28788ef0 tx=0x7f89ac00f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.596+0000 7f89b705e640 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55de288f7530 0x55de2885ff90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b705e640 1 -- 172.21.15.99:0/2619309372 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 1.00000000.inode [omap-get-keys out=5b] v0'0 uv16 ondisk = 0) v8 ==== 160+0+5 (crc 0 0 0) 0x7f89b000b630 con 0x55de288f7530 2024-03-04T10:10:22.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55de288f7530 msgr2=0x55de2885ff90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55de288f7530 0x55de2885ff90 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89a4069eb0 msgr2=0x7f89a406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f89a4069eb0 0x7f89a406c370 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x55de28788ef0 tx=0x7f89ac00f000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287b7af0 msgr2=0x55de2885c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 --2- 172.21.15.99:0/2619309372 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55de287b7af0 0x55de2885c480 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f89a800a190 tx=0x7f89a8053d20 comp rx=0 tx=0).stop 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b705e640 1 -- 172.21.15.99:0/2619309372 reap_dead start 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 shutdown_connections 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 >> 172.21.15.99:0/2619309372 conn(0x55de28788400 msgr2=0x55de28856b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 shutdown_connections 2024-03-04T10:10:22.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.597+0000 7f89b8d5ff40 1 -- 172.21.15.99:0/2619309372 wait complete. 2024-03-04T10:10:22.603 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000000 2024-03-04T10:10:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.682+0000 7f0d4a243f40 1 Processor -- start 2024-03-04T10:10:22.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.682+0000 7f0d4a243f40 1 -- start start 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d4a243f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4ffffac90 0x55b500027110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d4a243f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d4a243f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b5000d4670 0x55b5000d6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d4a243f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5000d5610 con 0x55b5000b71c0 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d4a243f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fffa3b30 con 0x55b4ffffac90 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d4a243f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b500045b50 con 0x55b5000d4670 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55552/0 (socket says 172.21.15.99:55552) 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 -- 172.21.15.99:0/1700422914 learned_addr learned my addr 172.21.15.99:0/1700422914 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d47dea640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b5000d4670 0x55b5000d6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d475e9640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4ffffac90 0x55b500027110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 -- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b5000d4670 msgr2=0x55b5000d6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b5000d4670 0x55b5000d6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 -- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4ffffac90 msgr2=0x55b500027110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4ffffac90 0x55b500027110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 -- 172.21.15.99:0/1700422914 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5000b7a70 con 0x55b5000b71c0 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d47dea640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b5000d4670 0x55b5000d6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d475e9640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4ffffac90 0x55b500027110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d46de8640 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb20 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c044790 tx=0x7f0d3c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95850f684f5b529f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.683+0000 7f0d45de6640 1 -- 172.21.15.99:0/1700422914 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c057040 con 0x55b5000b71c0 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d45de6640 1 -- 172.21.15.99:0/1700422914 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c062070 con 0x55b5000b71c0 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d45de6640 1 -- 172.21.15.99:0/1700422914 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c052c50 con 0x55b5000b71c0 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 -- 172.21.15.99:0/1700422914 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 msgr2=0x55b4ffffeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb20 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c044790 tx=0x7f0d3c04c000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 -- 172.21.15.99:0/1700422914 shutdown_connections 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b5000d4670 0x55b5000d6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb20 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/1700422914 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b4ffffac90 0x55b500027110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 -- 172.21.15.99:0/1700422914 >> 172.21.15.99:0/1700422914 conn(0x55b4ffffc400 msgr2=0x55b4ffffc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 -- 172.21.15.99:0/1700422914 shutdown_connections 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.684+0000 7f0d4a243f40 1 -- 172.21.15.99:0/1700422914 wait complete. 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 Processor -- start 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 -- start start 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4ffffac90 0x55b5000421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5000d4670 0x55b5000b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fffff510 con 0x55b5000b71c0 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5000b7820 con 0x55b5000d4670 2024-03-04T10:10:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d4a243f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b4fffff760 con 0x55b4ffffac90 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d46de8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d46de8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55564/0 (socket says 172.21.15.99:55564) 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.685+0000 7f0d46de8640 1 -- 172.21.15.99:0/3294178306 learned_addr learned my addr 172.21.15.99:0/3294178306 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d47dea640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5000d4670 0x55b5000b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d475e9640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4ffffac90 0x55b5000421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d46de8640 1 -- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4ffffac90 msgr2=0x55b5000421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d46de8640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b4ffffac90 0x55b5000421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d46de8640 1 -- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5000d4670 msgr2=0x55b5000b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d46de8640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5000d4670 0x55b5000b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d46de8640 1 -- 172.21.15.99:0/3294178306 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5000d7610 con 0x55b5000b71c0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d47dea640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b5000d4670 0x55b5000b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d46de8640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb60 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c01cfd0 tx=0x7f0d3c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d44de4640 1 -- 172.21.15.99:0/3294178306 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c057040 con 0x55b5000b71c0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d44de4640 1 -- 172.21.15.99:0/3294178306 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c052e40 con 0x55b5000b71c0 2024-03-04T10:10:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b500042a00 con 0x55b5000b71c0 2024-03-04T10:10:22.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d44de4640 1 -- 172.21.15.99:0/3294178306 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c05d6b0 con 0x55b5000b71c0 2024-03-04T10:10:22.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.686+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5000d6e10 con 0x55b5000b71c0 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.687+0000 7f0d44de4640 1 -- 172.21.15.99:0/3294178306 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f0d3c06f020 con 0x55b5000b71c0 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d44de4640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d34069eb0 0x7f0d3406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d44de4640 1 -- 172.21.15.99:0/3294178306 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f0d3c0cc5c0 con 0x55b5000b71c0 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55b4fffff0a0 0x55b5000b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:292cf221:::200.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x55b4ffffe380 con 0x55b4fffff0a0 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d475e9640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d34069eb0 0x7f0d3406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d47dea640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55b4fffff0a0 0x55b5000b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d47dea640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55b4fffff0a0 0x55b5000b8d10 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d475e9640 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d34069eb0 0x7f0d3406c370 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005fd0 tx=0x7f0d38005a00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.688+0000 7f0d47dea640 1 -- 172.21.15.99:0/3294178306 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000000 [omap-get-keys out=5b] v0'0 uv20 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f0d40003450 con 0x55b4fffff0a0 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55b4fffff0a0 msgr2=0x55b5000b8d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55b4fffff0a0 0x55b5000b8d10 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d34069eb0 msgr2=0x7f0d3406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f0d34069eb0 0x7f0d3406c370 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0d38005fd0 tx=0x7f0d38005a00 comp rx=0 tx=0).stop 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 msgr2=0x55b4ffffeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 --2- 172.21.15.99:0/3294178306 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b5000b71c0 0x55b4ffffeb60 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c01cfd0 tx=0x7f0d3c002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d47dea640 1 -- 172.21.15.99:0/3294178306 reap_dead start 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 shutdown_connections 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 >> 172.21.15.99:0/3294178306 conn(0x55b4ffffc400 msgr2=0x55b500026850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 shutdown_connections 2024-03-04T10:10:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.689+0000 7f0d4a243f40 1 -- 172.21.15.99:0/3294178306 wait complete. 2024-03-04T10:10:22.696 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 200.00000001 2024-03-04T10:10:22.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 Processor -- start 2024-03-04T10:10:22.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 -- start start 2024-03-04T10:10:22.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb8e5c90 0x5627cb9a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9a21c0 0x5627cb8e8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 0x5627cb92cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627cb9c0370 con 0x5627cb9bf670 2024-03-04T10:10:22.779 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627cb88eb30 con 0x5627cb8e5c90 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62db048f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627cb9c0610 con 0x5627cb9a21c0 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62d8b32640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 0x5627cb92cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62d8b32640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 0x5627cb92cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55586/0 (socket says 172.21.15.99:55586) 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.774+0000 7f62d8b32640 1 -- 172.21.15.99:0/2713450052 learned_addr learned my addr 172.21.15.99:0/2713450052 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8b32640 1 -- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9a21c0 msgr2=0x5627cb8e8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8b32640 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9a21c0 0x5627cb8e8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8331640 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb8e5c90 0x5627cb9a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8b32640 1 -- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb8e5c90 msgr2=0x5627cb9a86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8b32640 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb8e5c90 0x5627cb9a86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8b32640 1 -- 172.21.15.99:0/2713450052 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627cb8e9ab0 con 0x5627cb9bf670 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d8b32640 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 0x5627cb92cbd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c480 tx=0x7f62d000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965863fab0b9825 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d6b2e640 1 -- 172.21.15.99:0/2713450052 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d00116b0 con 0x5627cb9bf670 2024-03-04T10:10:22.780 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d6b2e640 1 -- 172.21.15.99:0/2713450052 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0017070 con 0x5627cb9bf670 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62d6b2e640 1 -- 172.21.15.99:0/2713450052 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0011a90 con 0x5627cb9bf670 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62db048f40 1 -- 172.21.15.99:0/2713450052 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 msgr2=0x5627cb92cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.775+0000 7f62db048f40 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 0x5627cb92cbd0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c480 tx=0x7f62d000c940 comp rx=0 tx=0).stop 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 -- 172.21.15.99:0/2713450052 shutdown_connections 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb9bf670 0x5627cb92cbd0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9a21c0 0x5627cb8e8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 --2- 172.21.15.99:0/2713450052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb8e5c90 0x5627cb9a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 -- 172.21.15.99:0/2713450052 >> 172.21.15.99:0/2713450052 conn(0x5627cb8e7400 msgr2=0x5627cb9beba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 -- 172.21.15.99:0/2713450052 shutdown_connections 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 -- 172.21.15.99:0/2713450052 wait complete. 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 Processor -- start 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.776+0000 7f62db048f40 1 -- start start 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb8e5c90 0x5627cb99e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb9a21c0 0x5627cb9bc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9bf670 0x5627cb99a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.781 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627cb8e9620 con 0x5627cb8e5c90 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627cb915a40 con 0x5627cb9a21c0 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627cb915760 con 0x5627cb9bf670 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb8e5c90 0x5627cb99e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb8e5c90 0x5627cb99e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55590/0 (socket says 172.21.15.99:55590) 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 -- 172.21.15.99:0/2674769153 learned_addr learned my addr 172.21.15.99:0/2674769153 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d7b30640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb9a21c0 0x5627cb9bc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8b32640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9bf670 0x5627cb99a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 -- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9bf670 msgr2=0x5627cb99a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5627cb9bf670 0x5627cb99a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 -- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb9a21c0 msgr2=0x5627cb9bc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5627cb9a21c0 0x5627cb9bc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 -- 172.21.15.99:0/2674769153 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627cb99e940 con 0x5627cb8e5c90 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d8331640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb8e5c90 0x5627cb99e400 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f62c800cec0 tx=0x7f62c800a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d5b2c640 1 -- 172.21.15.99:0/2674769153 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8014070 con 0x5627cb8e5c90 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d5b2c640 1 -- 172.21.15.99:0/2674769153 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c8010470 con 0x5627cb8e5c90 2024-03-04T10:10:22.782 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62d5b2c640 1 -- 172.21.15.99:0/2674769153 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8010860 con 0x5627cb8e5c90 2024-03-04T10:10:22.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.777+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5627cb9a7a30 con 0x5627cb8e5c90 2024-03-04T10:10:22.783 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.778+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627cb9a7ff0 con 0x5627cb8e5c90 2024-03-04T10:10:22.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.779+0000 7f62d5b2c640 1 -- 172.21.15.99:0/2674769153 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f62c800e040 con 0x5627cb8e5c90 2024-03-04T10:10:22.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.779+0000 7f62d5b2c640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f62c4069ee0 0x7f62c406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.779+0000 7f62d5b2c640 1 -- 172.21.15.99:0/2674769153 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f62c8088c70 con 0x5627cb8e5c90 2024-03-04T10:10:22.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.779+0000 7f62d7b30640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f62c4069ee0 0x7f62c406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.779+0000 7f62db048f40 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5627cb913d30 0x5627cb9c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.779+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:2e2fa760:::200.00000001:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x5627cb916100 con 0x5627cb913d30 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62d8b32640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5627cb913d30 0x5627cb9c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62d8b32640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5627cb913d30 0x5627cb9c1d60 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62d7b30640 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f62c4069ee0 0x7f62c406c3a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f62cc054000 tx=0x7f62cc04c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62d8b32640 1 -- 172.21.15.99:0/2674769153 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000001 [omap-get-keys out=5b] v0'0 uv17 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f62d00118b0 con 0x5627cb913d30 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5627cb913d30 msgr2=0x5627cb9c1d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5627cb913d30 0x5627cb9c1d60 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f62c4069ee0 msgr2=0x7f62c406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f62c4069ee0 0x7f62c406c3a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f62cc054000 tx=0x7f62cc04c000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb8e5c90 msgr2=0x5627cb99e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 --2- 172.21.15.99:0/2674769153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5627cb8e5c90 0x5627cb99e400 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f62c800cec0 tx=0x7f62c800a5a0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62d8b32640 1 -- 172.21.15.99:0/2674769153 reap_dead start 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 shutdown_connections 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 >> 172.21.15.99:0/2674769153 conn(0x5627cb8e7400 msgr2=0x5627cb9beba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.786 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 shutdown_connections 2024-03-04T10:10:22.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.780+0000 7f62db048f40 1 -- 172.21.15.99:0/2674769153 wait complete. 2024-03-04T10:10:22.787 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_openfiles.0 2024-03-04T10:10:22.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.865+0000 7f690ba67f40 1 Processor -- start 2024-03-04T10:10:22.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.865+0000 7f690ba67f40 1 -- start start 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f690ba67f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f690ba67f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2936b0 0x5559ba21d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f690ba67f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 0x5559ba2a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f690ba67f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ba17bb30 con 0x5559ba2b2b30 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f690ba67f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ba17fa90 con 0x5559ba2936b0 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f690ba67f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5559ba2ab6d0 con 0x5559ba21daf0 2024-03-04T10:10:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 0x5559ba2a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 0x5559ba2a39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55606/0 (socket says 172.21.15.99:55606) 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 -- 172.21.15.99:0/3260964343 learned_addr learned my addr 172.21.15.99:0/3260964343 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6908d64640 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2936b0 0x5559ba21d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909565640 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 -- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 msgr2=0x5559ba2a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 -- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2936b0 msgr2=0x5559ba21d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2936b0 0x5559ba21d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6909d66640 1 -- 172.21.15.99:0/3260964343 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559ba21b530 con 0x5559ba2b2b30 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.866+0000 7f6908d64640 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2936b0 0x5559ba21d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f6909d66640 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 0x5559ba2a39e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f6904054040 tx=0x7f690404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a08dd9bd025890 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f68fb7fe640 1 -- 172.21.15.99:0/3260964343 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904063070 con 0x5559ba2b2b30 2024-03-04T10:10:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f68fb7fe640 1 -- 172.21.15.99:0/3260964343 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6904052770 con 0x5559ba2b2b30 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f68fb7fe640 1 -- 172.21.15.99:0/3260964343 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904052b40 con 0x5559ba2b2b30 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 -- 172.21.15.99:0/3260964343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 msgr2=0x5559ba2a39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 0x5559ba2a39e0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f6904054040 tx=0x7f690404c000 comp rx=0 tx=0).stop 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 -- 172.21.15.99:0/3260964343 shutdown_connections 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2b2b30 0x5559ba2a39e0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2936b0 0x5559ba21d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3260964343 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 -- 172.21.15.99:0/3260964343 >> 172.21.15.99:0/3260964343 conn(0x5559ba1d3b80 msgr2=0x5559ba290890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 -- 172.21.15.99:0/3260964343 shutdown_connections 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.867+0000 7f690ba67f40 1 -- 172.21.15.99:0/3260964343 wait complete. 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 Processor -- start 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 -- start start 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2936b0 0x5559ba2a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2b2b30 0x5559ba2a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ba21d610 con 0x5559ba2936b0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ba203490 con 0x5559ba2b2b30 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.868+0000 7f690ba67f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5559ba203080 con 0x5559ba21daf0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2936b0 0x5559ba2a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2936b0 0x5559ba2a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55618/0 (socket says 172.21.15.99:55618) 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 -- 172.21.15.99:0/3855642879 learned_addr learned my addr 172.21.15.99:0/3855642879 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6909565640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6909d66640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2b2b30 0x5559ba2a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 -- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 msgr2=0x5559ba2a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5559ba21daf0 0x5559ba2a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 -- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2b2b30 msgr2=0x5559ba2a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2b2b30 0x5559ba2a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 -- 172.21.15.99:0/3855642879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559ba1d7370 con 0x5559ba2936b0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6909d66640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559ba2b2b30 0x5559ba2a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f6908d64640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2936b0 0x5559ba2a07f0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f6900001730 tx=0x7f690000b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f68fa7fc640 1 -- 172.21.15.99:0/3855642879 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690000f860 con 0x5559ba2936b0 2024-03-04T10:10:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f68fa7fc640 1 -- 172.21.15.99:0/3855642879 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690000fa00 con 0x5559ba2936b0 2024-03-04T10:10:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5559ba2a4b00 con 0x5559ba2936b0 2024-03-04T10:10:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f68fa7fc640 1 -- 172.21.15.99:0/3855642879 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690000fdb0 con 0x5559ba2936b0 2024-03-04T10:10:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.869+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559ba2a0d30 con 0x5559ba2936b0 2024-03-04T10:10:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.870+0000 7f68fa7fc640 1 -- 172.21.15.99:0/3855642879 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f69000203f0 con 0x5559ba2936b0 2024-03-04T10:10:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f68fa7fc640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f68ec069ee0 0x7f68ec06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f68fa7fc640 1 -- 172.21.15.99:0/3855642879 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f6900089a90 con 0x5559ba2936b0 2024-03-04T10:10:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f6909565640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f68ec069ee0 0x7f68ec06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5559ba2a89a0 0x5559ba283860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.6 13:654134d2:::mds0_openfiles.0:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95) v9 -- 0x5559ba283da0 con 0x5559ba2a89a0 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f6909d66640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5559ba2a89a0 0x5559ba283860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.871+0000 7f6909565640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f68ec069ee0 0x7f68ec06c3a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f68fc000c00 tx=0x7f68fc0038b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f6909d66640 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5559ba2a89a0 0x5559ba283860 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f6909d66640 1 -- 172.21.15.99:0/3855642879 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 mds0_openfiles.0 [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) v8 ==== 160+0+5 (crc 0 0 0) 0x7f6904052aa0 con 0x5559ba2a89a0 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5559ba2a89a0 msgr2=0x5559ba283860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5559ba2a89a0 0x5559ba283860 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f68ec069ee0 msgr2=0x7f68ec06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f68ec069ee0 0x7f68ec06c3a0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f68fc000c00 tx=0x7f68fc0038b0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2936b0 msgr2=0x5559ba2a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 --2- 172.21.15.99:0/3855642879 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5559ba2936b0 0x5559ba2a07f0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f6900001730 tx=0x7f690000b780 comp rx=0 tx=0).stop 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f6909d66640 1 -- 172.21.15.99:0/3855642879 reap_dead start 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 shutdown_connections 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 >> 172.21.15.99:0/3855642879 conn(0x5559ba1d3b80 msgr2=0x5559ba2a9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 shutdown_connections 2024-03-04T10:10:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.872+0000 7f690ba67f40 1 -- 172.21.15.99:0/3855642879 wait complete. 2024-03-04T10:10:22.879 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 500.00000000 2024-03-04T10:10:22.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 Processor -- start 2024-03-04T10:10:22.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 -- start start 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d20002850 0x55fdf629df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf620ff90 0x55fdf6213a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 0x55fdf62a0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdf618bb30 con 0x55fdf62bec70 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdf618fa90 con 0x55fdf620ff90 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.957+0000 7f7d37844f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdf62c0840 con 0x7f7d20002850 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 0x55fdf62a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 0x55fdf62a0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55626/0 (socket says 172.21.15.99:55626) 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 -- 172.21.15.99:0/3205355622 learned_addr learned my addr 172.21.15.99:0/3205355622 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 -- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d20002850 msgr2=0x55fdf629df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d34ac7640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d20002850 0x55fdf629df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d342c6640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf620ff90 0x55fdf6213a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d20002850 0x55fdf629df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 -- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf620ff90 msgr2=0x55fdf6213a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf620ff90 0x55fdf6213a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 -- 172.21.15.99:0/3205355622 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdf629fc90 con 0x55fdf62bec70 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d34ac7640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d20002850 0x55fdf629df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d342c6640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf620ff90 0x55fdf6213a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d352c8640 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 0x55fdf62a0990 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c005fd0 tx=0x7f7d2c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf290a62e2310e9a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d332c4640 1 -- 172.21.15.99:0/3205355622 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c007830 con 0x55fdf62bec70 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d332c4640 1 -- 172.21.15.99:0/3205355622 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c0079d0 con 0x55fdf62bec70 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d332c4640 1 -- 172.21.15.99:0/3205355622 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c007d80 con 0x55fdf62bec70 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d37844f40 1 -- 172.21.15.99:0/3205355622 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 msgr2=0x55fdf62a0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.958+0000 7f7d37844f40 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 0x55fdf62a0990 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c005fd0 tx=0x7f7d2c005a00 comp rx=0 tx=0).stop 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 -- 172.21.15.99:0/3205355622 shutdown_connections 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdf62bec70 0x55fdf62a0990 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf620ff90 0x55fdf6213a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 --2- 172.21.15.99:0/3205355622 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d20002850 0x55fdf629df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 -- 172.21.15.99:0/3205355622 >> 172.21.15.99:0/3205355622 conn(0x55fdf61e4400 msgr2=0x55fdf61e79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 -- 172.21.15.99:0/3205355622 shutdown_connections 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 -- 172.21.15.99:0/3205355622 wait complete. 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 Processor -- start 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.959+0000 7f7d37844f40 1 -- start start 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d37844f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdf620ff90 0x55fdf629c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d37844f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf62bec70 0x55fdf620f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d37844f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d20002850 0x55fdf629aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d37844f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdf629e780 con 0x7f7d20002850 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d37844f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdf6210ab0 con 0x55fdf62bec70 2024-03-04T10:10:22.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d37844f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdf629e9e0 con 0x55fdf620ff90 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d20002850 0x55fdf629aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d20002850 0x55fdf629aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55638/0 (socket says 172.21.15.99:55638) 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 -- 172.21.15.99:0/1991006571 learned_addr learned my addr 172.21.15.99:0/1991006571 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d342c6640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf62bec70 0x55fdf620f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d34ac7640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdf620ff90 0x55fdf629c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 -- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdf620ff90 msgr2=0x55fdf629c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdf620ff90 0x55fdf629c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 -- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf62bec70 msgr2=0x55fdf620f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdf62bec70 0x55fdf620f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.960+0000 7f7d352c8640 1 -- 172.21.15.99:0/1991006571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdf629cb70 con 0x7f7d20002850 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.961+0000 7f7d352c8640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d20002850 0x55fdf629aba0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009c30 tx=0x7f7d2c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.961+0000 7f7d322c2640 1 -- 172.21.15.99:0/1991006571 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c015840 con 0x7f7d20002850 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.961+0000 7f7d322c2640 1 -- 172.21.15.99:0/1991006571 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c026070 con 0x7f7d20002850 2024-03-04T10:10:22.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.961+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fdf629ce50 con 0x7f7d20002850 2024-03-04T10:10:22.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.961+0000 7f7d322c2640 1 -- 172.21.15.99:0/1991006571 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c015c20 con 0x7f7d20002850 2024-03-04T10:10:22.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.961+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdf629d410 con 0x7f7d20002850 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.962+0000 7f7d322c2640 1 -- 172.21.15.99:0/1991006571 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7d2c02b460 con 0x7f7d20002850 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.962+0000 7f7d322c2640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d14069eb0 0x7f7d1406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.962+0000 7f7d322c2640 1 -- 172.21.15.99:0/1991006571 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f7d2c08acb0 con 0x7f7d20002850 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d37844f40 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdf6229350 0x55fdf6229730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.15 13:aa448500:::500.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x55fdf6229c70 con 0x55fdf6229350 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d34ac7640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d14069eb0 0x7f7d1406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d342c6640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdf6229350 0x55fdf6229730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d34ac7640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d14069eb0 0x7f7d1406c370 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7d24004290 tx=0x7f7d240068f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d342c6640 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdf6229350 0x55fdf6229730 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.963+0000 7f7d342c6640 1 -- 172.21.15.99:0/1991006571 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 500.00000000 [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7f7d1c0015b0 con 0x55fdf6229350 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdf6229350 msgr2=0x55fdf6229730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdf6229350 0x55fdf6229730 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d14069eb0 msgr2=0x7f7d1406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d14069eb0 0x7f7d1406c370 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7d24004290 tx=0x7f7d240068f0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d20002850 msgr2=0x55fdf629aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 --2- 172.21.15.99:0/1991006571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d20002850 0x55fdf629aba0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009c30 tx=0x7f7d2c002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d352c8640 1 -- 172.21.15.99:0/1991006571 reap_dead start 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 shutdown_connections 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 >> 172.21.15.99:0/1991006571 conn(0x55fdf61e4400 msgr2=0x55fdf62bf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 shutdown_connections 2024-03-04T10:10:22.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:22.964+0000 7f7d37844f40 1 -- 172.21.15.99:0/1991006571 wait complete. 2024-03-04T10:10:22.971 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds_snaptable 2024-03-04T10:10:23.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.048+0000 7f2712044f40 1 Processor -- start 2024-03-04T10:10:23.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.048+0000 7f2712044f40 1 -- start start 2024-03-04T10:10:23.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f2712044f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5b421c0 0x5596e5b553c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f2712044f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5be7f90 0x5596e5b14980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f2712044f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 0x5596e5b14ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f2712044f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e5b5a250 con 0x5596e5be8370 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f2712044f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e5b5a0d0 con 0x5596e5be7f90 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f2712044f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596e5b5a3d0 con 0x5596e5b421c0 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270fc0a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 0x5596e5b14ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270fc0a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 0x5596e5b14ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55654/0 (socket says 172.21.15.99:55654) 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270fc0a640 1 -- 172.21.15.99:0/3510268515 learned_addr learned my addr 172.21.15.99:0/3510268515 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270f409640 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5b421c0 0x5596e5b553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270ec08640 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5be7f90 0x5596e5b14980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270fc0a640 1 -- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5b421c0 msgr2=0x5596e5b553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.049+0000 7f270fc0a640 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5b421c0 0x5596e5b553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270fc0a640 1 -- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5be7f90 msgr2=0x5596e5b14980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270fc0a640 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5be7f90 0x5596e5b14980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270fc0a640 1 -- 172.21.15.99:0/3510268515 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596e5befbd0 con 0x5596e5be8370 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270fc0a640 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 0x5596e5b14ec0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f270800a5b0 tx=0x7f270800a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fb9ea7f2ec7dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270dc06640 1 -- 172.21.15.99:0/3510268515 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708019070 con 0x5596e5be8370 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270dc06640 1 -- 172.21.15.99:0/3510268515 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708007600 con 0x5596e5be8370 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f270dc06640 1 -- 172.21.15.99:0/3510268515 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27080079d0 con 0x5596e5be8370 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 -- 172.21.15.99:0/3510268515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 msgr2=0x5596e5b14ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 0x5596e5b14ec0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f270800a5b0 tx=0x7f270800a6a0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 -- 172.21.15.99:0/3510268515 shutdown_connections 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be8370 0x5596e5b14ec0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5be7f90 0x5596e5b14980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 --2- 172.21.15.99:0/3510268515 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5b421c0 0x5596e5b553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 -- 172.21.15.99:0/3510268515 >> 172.21.15.99:0/3510268515 conn(0x5596e5b12400 msgr2=0x5596e5b40e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 -- 172.21.15.99:0/3510268515 shutdown_connections 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.050+0000 7f2712044f40 1 -- 172.21.15.99:0/3510268515 wait complete. 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.051+0000 7f2712044f40 1 Processor -- start 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.051+0000 7f2712044f40 1 -- start start 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5b421c0 0x5596e5b561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be7f90 0x5596e5b41050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5be8370 0x5596e5bd88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e5be7770 con 0x5596e5be7f90 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e5befeb0 con 0x5596e5b421c0 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5596e5bf0190 con 0x5596e5be8370 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be7f90 0x5596e5b41050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be7f90 0x5596e5b41050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55660/0 (socket says 172.21.15.99:55660) 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 -- 172.21.15.99:0/526637165 learned_addr learned my addr 172.21.15.99:0/526637165 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270f409640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5b421c0 0x5596e5b561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270fc0a640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5be8370 0x5596e5bd88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 -- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5be8370 msgr2=0x5596e5bd88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5be8370 0x5596e5bd88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 -- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5b421c0 msgr2=0x5596e5b561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5b421c0 0x5596e5b561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 -- 172.21.15.99:0/526637165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596e5b53ea0 con 0x5596e5be7f90 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270f409640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5596e5b421c0 0x5596e5b561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270fc0a640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5596e5be8370 0x5596e5bd88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270ec08640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be7f90 0x5596e5b41050 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f2704000c00 tx=0x7f2704002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270cc04640 1 -- 172.21.15.99:0/526637165 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704063070 con 0x5596e5be7f90 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270cc04640 1 -- 172.21.15.99:0/526637165 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27040528a0 con 0x5596e5be7f90 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5596e5b54180 con 0x5596e5be7f90 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.052+0000 7f270cc04640 1 -- 172.21.15.99:0/526637165 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2704057040 con 0x5596e5be7f90 2024-03-04T10:10:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.053+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596e5bd8e00 con 0x5596e5be7f90 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.054+0000 7f270cc04640 1 -- 172.21.15.99:0/526637165 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2704066070 con 0x5596e5be7f90 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.054+0000 7f270cc04640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26fc069eb0 0x7f26fc06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.054+0000 7f270cc04640 1 -- 172.21.15.99:0/526637165 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f27040ca930 con 0x5596e5be7f90 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.054+0000 7f270f409640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26fc069eb0 0x7f26fc06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.054+0000 7f2712044f40 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5596e5be37a0 0x5596e5be3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.054+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.d 13:b50e409b:::mds_snaptable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5596e5be40f0 con 0x5596e5be37a0 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f270fc0a640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5596e5be37a0 0x5596e5be3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f270f409640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26fc069eb0 0x7f26fc06c370 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f270000d7e0 tx=0x7f2700007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f270fc0a640 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5596e5be37a0 0x5596e5be3bb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f270fc0a640 1 -- 172.21.15.99:0/526637165 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 mds_snaptable [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+5 (crc 0 0 0) 0x7f2708007930 con 0x5596e5be37a0 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5596e5be37a0 msgr2=0x5596e5be3bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x5596e5be37a0 0x5596e5be3bb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26fc069eb0 msgr2=0x7f26fc06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f26fc069eb0 0x7f26fc06c370 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f270000d7e0 tx=0x7f2700007040 comp rx=0 tx=0).stop 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be7f90 msgr2=0x5596e5b41050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 --2- 172.21.15.99:0/526637165 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5596e5be7f90 0x5596e5b41050 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f2704000c00 tx=0x7f2704002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f270fc0a640 1 -- 172.21.15.99:0/526637165 reap_dead start 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 shutdown_connections 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.055+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 >> 172.21.15.99:0/526637165 conn(0x5596e5b12400 msgr2=0x5596e5bdbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.056+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 shutdown_connections 2024-03-04T10:10:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.056+0000 7f2712044f40 1 -- 172.21.15.99:0/526637165 wait complete. 2024-03-04T10:10:23.062 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_inotable 2024-03-04T10:10:23.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.140+0000 7ff50fa44f40 1 Processor -- start 2024-03-04T10:10:23.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.140+0000 7ff50fa44f40 1 -- start start 2024-03-04T10:10:23.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50fa44f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238767f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50fa44f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 0x5652387675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50fa44f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387665f0 0x56523875ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50fa44f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x565238637b30 con 0x5652387605a0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50fa44f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x56523863ba90 con 0x56523868ec90 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50fa44f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5652386d9ab0 con 0x5652387665f0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50c50d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 0x5652387675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50c50d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 0x5652387675e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55682/0 (socket says 172.21.15.99:55682) 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50c50d640 1 -- 172.21.15.99:0/3265020670 learned_addr learned my addr 172.21.15.99:0/3265020670 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50cd0e640 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238767f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.141+0000 7ff50d50f640 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387665f0 0x56523875ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50c50d640 1 -- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387665f0 msgr2=0x56523875ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50c50d640 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387665f0 0x56523875ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50c50d640 1 -- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 msgr2=0x565238767f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50c50d640 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238767f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50c50d640 1 -- 172.21.15.99:0/3265020670 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565238767b20 con 0x5652387605a0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50c50d640 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 0x5652387675e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7ff500001dd0 tx=0x7ff50000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2563efdff2a3a067 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50b50b640 1 -- 172.21.15.99:0/3265020670 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50000bde0 con 0x5652387605a0 2024-03-04T10:10:23.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50b50b640 1 -- 172.21.15.99:0/3265020670 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50000f400 con 0x5652387605a0 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50b50b640 1 -- 172.21.15.99:0/3265020670 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50000f7f0 con 0x5652387605a0 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 -- 172.21.15.99:0/3265020670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 msgr2=0x5652387675e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 0x5652387675e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7ff500001dd0 tx=0x7ff50000b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 -- 172.21.15.99:0/3265020670 shutdown_connections 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387665f0 0x56523875ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387605a0 0x5652387675e0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/3265020670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238767f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 -- 172.21.15.99:0/3265020670 >> 172.21.15.99:0/3265020670 conn(0x565238690400 msgr2=0x5652386bc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 -- 172.21.15.99:0/3265020670 shutdown_connections 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.142+0000 7ff50fa44f40 1 -- 172.21.15.99:0/3265020670 wait complete. 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.143+0000 7ff50fa44f40 1 Processor -- start 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.143+0000 7ff50fa44f40 1 -- start start 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238749ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387605a0 0x5652386bfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387665f0 0x565238747160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652386d6dd0 con 0x5652387665f0 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652386d7430 con 0x56523868ec90 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x565238690c10 con 0x5652387605a0 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387665f0 0x565238747160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387665f0 0x565238747160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55698/0 (socket says 172.21.15.99:55698) 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 -- 172.21.15.99:0/657928799 learned_addr learned my addr 172.21.15.99:0/657928799 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50cd0e640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238749ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50c50d640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387605a0 0x5652386bfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 -- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387605a0 msgr2=0x5652386bfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387605a0 0x5652386bfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 -- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 msgr2=0x565238749ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56523868ec90 0x565238749ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 -- 172.21.15.99:0/657928799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652386c02e0 con 0x5652387665f0 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50c50d640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5652387605a0 0x5652386bfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50d50f640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387665f0 0x565238747160 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7ff50400de30 tx=0x7ff50400b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50a509640 1 -- 172.21.15.99:0/657928799 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504014020 con 0x5652387665f0 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50a509640 1 -- 172.21.15.99:0/657928799 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff504010410 con 0x5652387665f0 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56523875ccc0 con 0x5652387665f0 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50a509640 1 -- 172.21.15.99:0/657928799 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504008030 con 0x5652387665f0 2024-03-04T10:10:23.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.144+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56523875d280 con 0x5652387665f0 2024-03-04T10:10:23.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.146+0000 7ff50a509640 1 -- 172.21.15.99:0/657928799 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff504017020 con 0x5652387665f0 2024-03-04T10:10:23.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.146+0000 7ff50a509640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff4f8069ee0 0x7ff4f806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.146+0000 7ff50a509640 1 -- 172.21.15.99:0/657928799 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7ff504088f60 con 0x5652387665f0 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.146+0000 7ff50cd0e640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff4f8069ee0 0x7ff4f806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.146+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5652386be940 0x5652386bed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.146+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.13 13:c91d4a1d:::mds0_inotable:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5652386bf260 con 0x5652386be940 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50c50d640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5652386be940 0x5652386bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50cd0e640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff4f8069ee0 0x7ff4f806c3a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc053000 tx=0x7ff4fc04e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50c50d640 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5652386be940 0x5652386bed20 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50c50d640 1 -- 172.21.15.99:0/657928799 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 mds0_inotable [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+5 (crc 0 0 0) 0x7ff50000f630 con 0x5652386be940 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5652386be940 msgr2=0x5652386bed20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5652386be940 0x5652386bed20 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff4f8069ee0 msgr2=0x7ff4f806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff4f8069ee0 0x7ff4f806c3a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc053000 tx=0x7ff4fc04e000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387665f0 msgr2=0x565238747160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 --2- 172.21.15.99:0/657928799 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5652387665f0 0x565238747160 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7ff50400de30 tx=0x7ff50400b5e0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50d50f640 1 -- 172.21.15.99:0/657928799 reap_dead start 2024-03-04T10:10:23.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 shutdown_connections 2024-03-04T10:10:23.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 >> 172.21.15.99:0/657928799 conn(0x565238690400 msgr2=0x56523875b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 shutdown_connections 2024-03-04T10:10:23.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.147+0000 7ff50fa44f40 1 -- 172.21.15.99:0/657928799 wait complete. 2024-03-04T10:10:23.154 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000 2024-03-04T10:10:23.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.232+0000 7fd052839f40 1 Processor -- start 2024-03-04T10:10:23.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.232+0000 7fd052839f40 1 -- start start 2024-03-04T10:10:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd052839f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x56152198d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd052839f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a59d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd052839f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd052839f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561521a62c40 con 0x5615219d2f30 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd052839f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615219d4d40 con 0x5615219b9af0 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd052839f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561521a62dc0 con 0x5615219d3310 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a59d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a59d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55724/0 (socket says 172.21.15.99:55724) 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 -- 172.21.15.99:0/766147042 learned_addr learned my addr 172.21.15.99:0/766147042 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04fa9c640 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x56152198d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd05029d640 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 -- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 msgr2=0x561521a5a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 -- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 msgr2=0x56152198d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x56152198d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.233+0000 7fd04f29b640 1 -- 172.21.15.99:0/766147042 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561521a5e8c0 con 0x5615219d2f30 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd04f29b640 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a59d70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fd044000c00 tx=0x7fd04404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb4991f1492d790 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd04e299640 1 -- 172.21.15.99:0/766147042 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044057040 con 0x5615219d2f30 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd04e299640 1 -- 172.21.15.99:0/766147042 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044062070 con 0x5615219d2f30 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd04e299640 1 -- 172.21.15.99:0/766147042 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044052b50 con 0x5615219d2f30 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 -- 172.21.15.99:0/766147042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 msgr2=0x561521a59d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a59d70 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fd044000c00 tx=0x7fd04404c000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 -- 172.21.15.99:0/766147042 shutdown_connections 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a59d70 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 --2- 172.21.15.99:0/766147042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x56152198d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 -- 172.21.15.99:0/766147042 >> 172.21.15.99:0/766147042 conn(0x56152198a400 msgr2=0x56152198a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 -- 172.21.15.99:0/766147042 shutdown_connections 2024-03-04T10:10:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.234+0000 7fd052839f40 1 -- 172.21.15.99:0/766147042 wait complete. 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.235+0000 7fd052839f40 1 Processor -- start 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.235+0000 7fd052839f40 1 -- start start 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x561521a4e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a5cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561521a62500 con 0x5615219d2f30 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561521a5eb60 con 0x5615219b9af0 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561521a61a20 con 0x5615219d3310 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a5cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a5cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55728/0 (socket says 172.21.15.99:55728) 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 -- 172.21.15.99:0/3642803284 learned_addr learned my addr 172.21.15.99:0/3642803284 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04fa9c640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x561521a4e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd05029d640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 -- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 msgr2=0x561521a5d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5615219d3310 0x561521a5d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 -- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 msgr2=0x561521a4e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5615219b9af0 0x561521a4e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 -- 172.21.15.99:0/3642803284 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561521a501c0 con 0x5615219d2f30 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04f29b640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a5cc30 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fd04401cfd0 tx=0x7fd044002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04d297640 1 -- 172.21.15.99:0/3642803284 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044057040 con 0x5615219d2f30 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04d297640 1 -- 172.21.15.99:0/3642803284 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd044052d80 con 0x5615219d2f30 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561521a4ef30 con 0x5615219d2f30 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd04d297640 1 -- 172.21.15.99:0/3642803284 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04405d6b0 con 0x5615219d2f30 2024-03-04T10:10:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.236+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561521a5a480 con 0x5615219d2f30 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.238+0000 7fd04d297640 1 -- 172.21.15.99:0/3642803284 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd04406f020 con 0x5615219d2f30 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.238+0000 7fd04d297640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd03c069ee0 0x7fd03c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.238+0000 7fd04d297640 1 -- 172.21.15.99:0/3642803284 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fd0440cc720 con 0x5615219d2f30 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.238+0000 7fd052839f40 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x561521a5d6b0 0x5615219ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.238+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.13 13:cd5a64a3:::100.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x56152198d980 con 0x561521a5d6b0 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.238+0000 7fd04fa9c640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd03c069ee0 0x7fd03c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd05029d640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x561521a5d6b0 0x5615219ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd04fa9c640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd03c069ee0 0x7fd03c06c3a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd0400076a0 tx=0x7fd040005a60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd05029d640 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x561521a5d6b0 0x5615219ce640 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd05029d640 1 -- 172.21.15.99:0/3642803284 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 100.00000000 [omap-get-keys out=155b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+155 (crc 0 0 0) 0x7fd04800e800 con 0x561521a5d6b0 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray0_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray1_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray2_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray3_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray4_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray5_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray6_head 2024-03-04T10:10:23.245 INFO:teuthology.orchestra.run.smithi099.stdout:stray7_head 2024-03-04T10:10:23.246 INFO:teuthology.orchestra.run.smithi099.stdout:stray8_head 2024-03-04T10:10:23.246 INFO:teuthology.orchestra.run.smithi099.stdout:stray9_head 2024-03-04T10:10:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x561521a5d6b0 msgr2=0x5615219ce640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x561521a5d6b0 0x5615219ce640 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd03c069ee0 msgr2=0x7fd03c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd03c069ee0 0x7fd03c06c3a0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd0400076a0 tx=0x7fd040005a60 comp rx=0 tx=0).stop 2024-03-04T10:10:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 msgr2=0x561521a5cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 --2- 172.21.15.99:0/3642803284 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5615219d2f30 0x561521a5cc30 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fd04401cfd0 tx=0x7fd044002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd05029d640 1 -- 172.21.15.99:0/3642803284 reap_dead start 2024-03-04T10:10:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 shutdown_connections 2024-03-04T10:10:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.239+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 >> 172.21.15.99:0/3642803284 conn(0x56152198a400 msgr2=0x56152198c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.240+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 shutdown_connections 2024-03-04T10:10:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.240+0000 7fd052839f40 1 -- 172.21.15.99:0/3642803284 wait complete. 2024-03-04T10:10:23.248 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys mds0_sessionmap 2024-03-04T10:10:23.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.326+0000 7fae0f940f40 1 Processor -- start 2024-03-04T10:10:23.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 -- start start 2024-03-04T10:10:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da2933a0 0x5556da36aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da2c25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da36c670 0x5556da2c2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556da23bb30 con 0x5556da34f1c0 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556da23fa90 con 0x5556da36c670 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0f940f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556da2c49a0 con 0x5556da2933a0 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0cc3f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da2c25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0cc3f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da2c25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55736/0 (socket says 172.21.15.99:55736) 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.327+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2564111053 learned_addr learned my addr 172.21.15.99:0/2564111053 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2564111053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da2933a0 msgr2=0x5556da36aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0cc3f640 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da2933a0 0x5556da36aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2564111053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da36c670 msgr2=0x5556da2c2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0cc3f640 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da36c670 0x5556da2c2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2564111053 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556da2dc040 con 0x5556da34f1c0 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0cc3f640 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da2c25a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fae04001e30 tx=0x7fae0400b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99406f8045b16563 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fadff7fe640 1 -- 172.21.15.99:0/2564111053 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0400be20 con 0x5556da34f1c0 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fadff7fe640 1 -- 172.21.15.99:0/2564111053 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0400f400 con 0x5556da34f1c0 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fadff7fe640 1 -- 172.21.15.99:0/2564111053 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0400f7d0 con 0x5556da34f1c0 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 -- 172.21.15.99:0/2564111053 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 msgr2=0x5556da2c25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da2c25a0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fae04001e30 tx=0x7fae0400b670 comp rx=0 tx=0).stop 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 -- 172.21.15.99:0/2564111053 shutdown_connections 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da36c670 0x5556da2c2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da2c25a0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2564111053 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da2933a0 0x5556da36aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 -- 172.21.15.99:0/2564111053 >> 172.21.15.99:0/2564111053 conn(0x5556da294400 msgr2=0x5556da2947e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 -- 172.21.15.99:0/2564111053 shutdown_connections 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.328+0000 7fae0f940f40 1 -- 172.21.15.99:0/2564111053 wait complete. 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.329+0000 7fae0f940f40 1 Processor -- start 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.329+0000 7fae0f940f40 1 -- start start 2024-03-04T10:10:23.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0f940f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da2933a0 0x5556da34b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0f940f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da369b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0f940f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da36c670 0x5556da2d9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0f940f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556da355a30 con 0x5556da34f1c0 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0f940f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556da2c46c0 con 0x5556da2933a0 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0f940f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556da36f6c0 con 0x5556da36c670 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da369b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da369b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55752/0 (socket says 172.21.15.99:55752) 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2269189417 learned_addr learned my addr 172.21.15.99:0/2269189417 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0d440640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da2933a0 0x5556da34b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0dc41640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da36c670 0x5556da2d9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da36c670 msgr2=0x5556da2d9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5556da36c670 0x5556da2d9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da2933a0 msgr2=0x5556da34b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da2933a0 0x5556da34b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 -- 172.21.15.99:0/2269189417 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556da34b940 con 0x5556da34f1c0 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0d440640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5556da2933a0 0x5556da34b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.330+0000 7fae0cc3f640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da369b00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fae04003040 tx=0x7fae04000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.331+0000 7fadfe7fc640 1 -- 172.21.15.99:0/2269189417 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04020070 con 0x5556da34f1c0 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.331+0000 7fadfe7fc640 1 -- 172.21.15.99:0/2269189417 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0401c640 con 0x5556da34f1c0 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.331+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5556da36a600 con 0x5556da34f1c0 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.331+0000 7fadfe7fc640 1 -- 172.21.15.99:0/2269189417 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0401ca50 con 0x5556da34f1c0 2024-03-04T10:10:23.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.331+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556da34bb70 con 0x5556da34f1c0 2024-03-04T10:10:23.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.332+0000 7fadfe7fc640 1 -- 172.21.15.99:0/2269189417 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fae04023070 con 0x5556da34f1c0 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.332+0000 7fadfe7fc640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fadf0069ee0 0x7fadf006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.332+0000 7fadfe7fc640 1 -- 172.21.15.99:0/2269189417 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fae0408aae0 con 0x5556da34f1c0 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.332+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x5556da2c0d10 0x5556da2da3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.332+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:1 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5556da353a90 con 0x5556da2c0d10 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0d440640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fadf0069ee0 0x7fadf006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0dc41640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x5556da2c0d10 0x5556da2da3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0dc41640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x5556da2c0d10 0x5556da2da3c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0d440640 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fadf0069ee0 0x7fadf006c3a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fae00054000 tx=0x7fae0004c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0dc41640 1 -- 172.21.15.99:0/2269189417 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(1 mds0_sessionmap [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) v8 ==== 159+0+5 (crc 0 0 0) 0x7fae08004540 con 0x5556da2c0d10 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x5556da2c0d10 msgr2=0x5556da2da3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x5556da2c0d10 0x5556da2da3c0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.333+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fadf0069ee0 msgr2=0x7fadf006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fadf0069ee0 0x7fadf006c3a0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fae00054000 tx=0x7fae0004c000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 msgr2=0x5556da369b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 --2- 172.21.15.99:0/2269189417 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5556da34f1c0 0x5556da369b00 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fae04003040 tx=0x7fae04000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0dc41640 1 -- 172.21.15.99:0/2269189417 reap_dead start 2024-03-04T10:10:23.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 shutdown_connections 2024-03-04T10:10:23.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 >> 172.21.15.99:0/2269189417 conn(0x5556da294400 msgr2=0x5556da2947e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 shutdown_connections 2024-03-04T10:10:23.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.334+0000 7fae0f940f40 1 -- 172.21.15.99:0/2269189417 wait complete. 2024-03-04T10:10:23.340 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 400.00000000 2024-03-04T10:10:23.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 Processor -- start 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 -- start start 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 0x563cd9b2b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd9a7a1c0 0x563cd99c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a97670 0x563cd99c11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd9966b30 con 0x563cd99be3a0 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd996aa90 con 0x563cd9a97670 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c757ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563cd9a98610 con 0x563cd9a7a1c0 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c507d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 0x563cd9b2b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c507d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 0x563cd9b2b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55784/0 (socket says 172.21.15.99:55784) 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.419+0000 7fa0c507d640 1 -- 172.21.15.99:0/1958229095 learned_addr learned my addr 172.21.15.99:0/1958229095 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c587e640 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a97670 0x563cd99c11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0bc87c640 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd9a7a1c0 0x563cd99c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c507d640 1 -- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd9a7a1c0 msgr2=0x563cd99c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c507d640 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd9a7a1c0 0x563cd99c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c507d640 1 -- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a97670 msgr2=0x563cd99c11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c507d640 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a97670 0x563cd99c11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c507d640 1 -- 172.21.15.99:0/1958229095 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cd9a91250 con 0x563cd99be3a0 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0bc87c640 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd9a7a1c0 0x563cd99c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c507d640 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 0x563cd9b2b460 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001d80 tx=0x7fa0b8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e618e428bff66b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0bffff640 1 -- 172.21.15.99:0/1958229095 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800bd10 con 0x563cd99be3a0 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0bffff640 1 -- 172.21.15.99:0/1958229095 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b800f400 con 0x563cd99be3a0 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0bffff640 1 -- 172.21.15.99:0/1958229095 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800f7f0 con 0x563cd99be3a0 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1958229095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 msgr2=0x563cd9b2b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 0x563cd9b2b460 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001d80 tx=0x7fa0b8001e70 comp rx=0 tx=0).stop 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1958229095 shutdown_connections 2024-03-04T10:10:23.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a97670 0x563cd99c11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd9a7a1c0 0x563cd99c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1958229095 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd99be3a0 0x563cd9b2b460 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1958229095 >> 172.21.15.99:0/1958229095 conn(0x563cd99bf400 msgr2=0x563cd99bf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.420+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1958229095 shutdown_connections 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.421+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1958229095 wait complete. 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.421+0000 7fa0c757ff40 1 Processor -- start 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.421+0000 7fa0c757ff40 1 -- start start 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd99be3a0 0x563cd9b299d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a7a1c0 0x563cd99c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd9a97670 0x563cd99c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd9a91810 con 0x563cd9a97670 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cd9a91530 con 0x563cd9a7a1c0 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563cd9a91af0 con 0x563cd99be3a0 2024-03-04T10:10:23.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd9a97670 0x563cd99c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd9a97670 0x563cd99c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:55794/0 (socket says 172.21.15.99:55794) 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 -- 172.21.15.99:0/1509917462 learned_addr learned my addr 172.21.15.99:0/1509917462 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0bc87c640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a7a1c0 0x563cd99c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c507d640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd99be3a0 0x563cd9b299d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 -- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd99be3a0 msgr2=0x563cd9b299d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563cd99be3a0 0x563cd9b299d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 -- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a7a1c0 msgr2=0x563cd99c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a7a1c0 0x563cd99c0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 -- 172.21.15.99:0/1509917462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cd9b29f10 con 0x563cd9a97670 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0bc87c640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563cd9a7a1c0 0x563cd99c0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c587e640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd9a97670 0x563cd99c0f90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa0c001b9e0 tx=0x7fa0c0003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0beffd640 1 -- 172.21.15.99:0/1509917462 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c005c810 con 0x563cd9a97670 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0beffd640 1 -- 172.21.15.99:0/1509917462 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c005c9b0 con 0x563cd9a97670 2024-03-04T10:10:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563cd9b2a1b0 con 0x563cd9a97670 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.422+0000 7fa0beffd640 1 -- 172.21.15.99:0/1509917462 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c005cd80 con 0x563cd9a97670 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.423+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cd9b2a770 con 0x563cd9a97670 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.424+0000 7fa0beffd640 1 -- 172.21.15.99:0/1509917462 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa0c00703f0 con 0x563cd9a97670 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.424+0000 7fa0beffd640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa0a8069eb0 0x7fa0a806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.424+0000 7fa0beffd640 1 -- 172.21.15.99:0/1509917462 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fa0c00cbf70 con 0x563cd9a97670 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.424+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563cd9a85090 0x563cd9a85470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.424+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.f 13:f1f69726:::400.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x563cd9a859b0 con 0x563cd9a85090 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.424+0000 7fa0c507d640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa0a8069eb0 0x7fa0a806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.425+0000 7fa0bc87c640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563cd9a85090 0x563cd9a85470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.425+0000 7fa0c507d640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa0a8069eb0 0x7fa0a806c370 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x563cd9a09e90 tx=0x7fa0b8009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.425+0000 7fa0bc87c640 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563cd9a85090 0x563cd9a85470 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0bc87c640 1 -- 172.21.15.99:0/1509917462 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 400.00000000 [omap-get-keys out=5b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x7fa0b000e610 con 0x563cd9a85090 2024-03-04T10:10:23.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563cd9a85090 msgr2=0x563cd9a85470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563cd9a85090 0x563cd9a85470 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa0a8069eb0 msgr2=0x7fa0a806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa0a8069eb0 0x7fa0a806c370 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x563cd9a09e90 tx=0x7fa0b8009000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd9a97670 msgr2=0x563cd99c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 --2- 172.21.15.99:0/1509917462 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563cd9a97670 0x563cd99c0f90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fa0c001b9e0 tx=0x7fa0c0003330 comp rx=0 tx=0).stop 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c587e640 1 -- 172.21.15.99:0/1509917462 reap_dead start 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 shutdown_connections 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 >> 172.21.15.99:0/1509917462 conn(0x563cd99bf400 msgr2=0x563cd99bf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.427+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 shutdown_connections 2024-03-04T10:10:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.428+0000 7fa0c757ff40 1 -- 172.21.15.99:0/1509917462 wait complete. 2024-03-04T10:10:23.434 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 100.00000000.inode 2024-03-04T10:10:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.513+0000 7face66d9f40 1 Processor -- start 2024-03-04T10:10:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.513+0000 7face66d9f40 1 -- start start 2024-03-04T10:10:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face66d9f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7cc2c90 0x5558a7d910f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face66d9f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d945a0 0x5558a7e2e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face66d9f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 0x5558a7e2edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face66d9f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558a7d9b300 con 0x5558a7d9a5f0 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face66d9f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558a7c6bb30 con 0x5558a7cc2c90 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face66d9f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558a7d0d810 con 0x5558a7d945a0 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 0x5558a7e2edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 0x5558a7e2edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37406/0 (socket says 172.21.15.99:37406) 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 -- 172.21.15.99:0/2833798827 learned_addr learned my addr 172.21.15.99:0/2833798827 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face41d9640 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7cc2c90 0x5558a7d910f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face39d8640 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d945a0 0x5558a7e2e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 -- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d945a0 msgr2=0x5558a7e2e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d945a0 0x5558a7e2e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 -- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7cc2c90 msgr2=0x5558a7d910f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7cc2c90 0x5558a7d910f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 -- 172.21.15.99:0/2833798827 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558a7d9d590 con 0x5558a7d9a5f0 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face49da640 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 0x5558a7e2edd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c440 tx=0x7facdc00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3911e9f0aa1f286 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.514+0000 7face29d6640 1 -- 172.21.15.99:0/2833798827 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc010650 con 0x5558a7d9a5f0 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face29d6640 1 -- 172.21.15.99:0/2833798827 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc016070 con 0x5558a7d9a5f0 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face29d6640 1 -- 172.21.15.99:0/2833798827 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc010a50 con 0x5558a7d9a5f0 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 -- 172.21.15.99:0/2833798827 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 msgr2=0x5558a7e2edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 0x5558a7e2edd0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c440 tx=0x7facdc00c900 comp rx=0 tx=0).stop 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 -- 172.21.15.99:0/2833798827 shutdown_connections 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7d9a5f0 0x5558a7e2edd0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d945a0 0x5558a7e2e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 --2- 172.21.15.99:0/2833798827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7cc2c90 0x5558a7d910f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 -- 172.21.15.99:0/2833798827 >> 172.21.15.99:0/2833798827 conn(0x5558a7cc4400 msgr2=0x5558a7d93420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 -- 172.21.15.99:0/2833798827 shutdown_connections 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.515+0000 7face66d9f40 1 -- 172.21.15.99:0/2833798827 wait complete. 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 Processor -- start 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 -- start start 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7cc2c90 0x5558a7d88650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7d945a0 0x5558a7cf2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d9a5f0 0x5558a7cf34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558a7d9c040 con 0x5558a7cc2c90 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558a7cc3960 con 0x5558a7d945a0 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face66d9f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558a7d9bcd0 con 0x5558a7d9a5f0 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.516+0000 7face41d9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7cc2c90 0x5558a7d88650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7cc2c90 0x5558a7d88650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37416/0 (socket says 172.21.15.99:37416) 2024-03-04T10:10:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 -- 172.21.15.99:0/3482267296 learned_addr learned my addr 172.21.15.99:0/3482267296 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face39d8640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7d945a0 0x5558a7cf2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face49da640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d9a5f0 0x5558a7cf34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 -- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d9a5f0 msgr2=0x5558a7cf34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5558a7d9a5f0 0x5558a7cf34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 -- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7d945a0 msgr2=0x5558a7cf2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7d945a0 0x5558a7cf2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 -- 172.21.15.99:0/3482267296 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558a7cf3a30 con 0x5558a7cc2c90 2024-03-04T10:10:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face39d8640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5558a7d945a0 0x5558a7cf2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face41d9640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7cc2c90 0x5558a7d88650 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7facd400df80 tx=0x7facd400b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face19d4640 1 -- 172.21.15.99:0/3482267296 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4014020 con 0x5558a7cc2c90 2024-03-04T10:10:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face19d4640 1 -- 172.21.15.99:0/3482267296 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4010620 con 0x5558a7cc2c90 2024-03-04T10:10:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5558a7d955d0 con 0x5558a7cc2c90 2024-03-04T10:10:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face19d4640 1 -- 172.21.15.99:0/3482267296 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4008030 con 0x5558a7cc2c90 2024-03-04T10:10:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.517+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558a7cf3f80 con 0x5558a7cc2c90 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.518+0000 7face19d4640 1 -- 172.21.15.99:0/3482267296 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7facd4017020 con 0x5558a7cc2c90 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face19d4640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd0069eb0 0x7facd006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face19d4640 1 -- 172.21.15.99:0/3482267296 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7facd4089190 con 0x5558a7cc2c90 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face66d9f40 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5558a7d88b90 0x5558a7d81fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face39d8640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd0069eb0 0x7facd006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5558a7d99830 con 0x5558a7d88b90 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face49da640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5558a7d88b90 0x5558a7d81fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.519+0000 7face39d8640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd0069eb0 0x7facd006c370 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7facd8054000 tx=0x7facd804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face49da640 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5558a7d88b90 0x5558a7d81fe0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face49da640 1 -- 172.21.15.99:0/3482267296 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 100.00000000.inode [omap-get-keys out=5b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+5 (crc 0 0 0) 0x7facdc0106d0 con 0x5558a7d88b90 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5558a7d88b90 msgr2=0x5558a7d81fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5558a7d88b90 0x5558a7d81fe0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd0069eb0 msgr2=0x7facd006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7facd0069eb0 0x7facd006c370 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7facd8054000 tx=0x7facd804e000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7cc2c90 msgr2=0x5558a7d88650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 --2- 172.21.15.99:0/3482267296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5558a7cc2c90 0x5558a7d88650 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7facd400df80 tx=0x7facd400b6a0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face49da640 1 -- 172.21.15.99:0/3482267296 reap_dead start 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 shutdown_connections 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 >> 172.21.15.99:0/3482267296 conn(0x5558a7cc4400 msgr2=0x5558a7d8e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 shutdown_connections 2024-03-04T10:10:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.520+0000 7face66d9f40 1 -- 172.21.15.99:0/3482267296 wait complete. 2024-03-04T10:10:23.529 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata listomapkeys 1.00000000 2024-03-04T10:10:23.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.608+0000 7f8bb18f7f40 1 Processor -- start 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.608+0000 7f8bb18f7f40 1 -- start start 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.608+0000 7f8bb18f7f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304bbf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.608+0000 7f8bb18f7f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 0x563304beaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8bb18f7f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 0x563304c95cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8bb18f7f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563304b63b30 con 0x563304c748e0 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8bb18f7f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563304b67a90 con 0x563304beb1e0 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8bb18f7f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563304c95610 con 0x563304c74cc0 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304bbf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304bbf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37432/0 (socket says 172.21.15.99:37432) 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 -- 172.21.15.99:0/3232809220 learned_addr learned my addr 172.21.15.99:0/3232809220 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 -- 172.21.15.99:0/3232809220 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 msgr2=0x563304beaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 0x563304beaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 -- 172.21.15.99:0/3232809220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 msgr2=0x563304c95cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 0x563304c95cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 -- 172.21.15.99:0/3232809220 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563304bbe530 con 0x563304c748e0 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8baf3f5640 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304bbf2b0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000d6f0 tx=0x7f8ba000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0b1c5374b84d73 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8badbf2640 1 -- 172.21.15.99:0/3232809220 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0013070 con 0x563304c748e0 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8badbf2640 1 -- 172.21.15.99:0/3232809220 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba000b850 con 0x563304c748e0 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.609+0000 7f8badbf2640 1 -- 172.21.15.99:0/3232809220 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba000bc00 con 0x563304c748e0 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/3232809220 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 msgr2=0x563304bbf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304bbf2b0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000d6f0 tx=0x7f8ba000dbb0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/3232809220 shutdown_connections 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 0x563304c95cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 0x563304beaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/3232809220 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304bbf2b0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/3232809220 >> 172.21.15.99:0/3232809220 conn(0x563304bbc400 msgr2=0x563304be8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/3232809220 shutdown_connections 2024-03-04T10:10:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/3232809220 wait complete. 2024-03-04T10:10:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 Processor -- start 2024-03-04T10:10:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.610+0000 7f8bb18f7f40 1 -- start start 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bb18f7f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 0x563304c8fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bb18f7f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304c73b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bb18f7f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 0x563304beb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bb18f7f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563304bbe860 con 0x563304c748e0 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bb18f7f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563304c93fb0 con 0x563304beb1e0 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bb18f7f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563304c93cd0 con 0x563304c74cc0 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304c73b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304c73b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37444/0 (socket says 172.21.15.99:37444) 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 -- 172.21.15.99:0/4204454519 learned_addr learned my addr 172.21.15.99:0/4204454519 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baf3f5640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 0x563304c8fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8bafbf6640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 0x563304beb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 -- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 msgr2=0x563304beb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x563304c74cc0 0x563304beb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 -- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 msgr2=0x563304c8fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563304beb1e0 0x563304c8fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 -- 172.21.15.99:0/4204454519 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563304c8ffa0 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.611+0000 7f8baebf4640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304c73b90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400def0 tx=0x7f8ba400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.612+0000 7f8bacbf0640 1 -- 172.21.15.99:0/4204454519 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40032d0 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.612+0000 7f8bacbf0640 1 -- 172.21.15.99:0/4204454519 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4016070 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.612+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x563304c03260 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.612+0000 7f8bacbf0640 1 -- 172.21.15.99:0/4204454519 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4003cc0 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.612+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563304c03820 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.613+0000 7f8bacbf0640 1 -- 172.21.15.99:0/4204454519 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8ba40016d0 con 0x563304c748e0 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.613+0000 7f8bacbf0640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b9c069ee0 0x7f8b9c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8bacbf0640 1 -- 172.21.15.99:0/4204454519 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f8ba4089020 con 0x563304c748e0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563304c740d0 0x563304c928b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-keys in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x563304c92df0 con 0x563304c740d0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8baf3f5640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b9c069ee0 0x7f8b9c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8bafbf6640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563304c740d0 0x563304c928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8baf3f5640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b9c069ee0 0x7f8b9c06c3a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0001970 tx=0x7f8ba0009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.614+0000 7f8bafbf6640 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563304c740d0 0x563304c928b0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bafbf6640 1 -- 172.21.15.99:0/4204454519 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 1.00000000 [omap-get-keys out=20b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+20 (crc 0 0 0) 0x7f8ba8005ae0 con 0x563304c740d0 2024-03-04T10:10:23.619 INFO:teuthology.orchestra.run.smithi099.stdout:subdir_head 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563304c740d0 msgr2=0x563304c928b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x563304c740d0 0x563304c928b0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b9c069ee0 msgr2=0x7f8b9c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8b9c069ee0 0x7f8b9c06c3a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0001970 tx=0x7f8ba0009000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 msgr2=0x563304c73b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 --2- 172.21.15.99:0/4204454519 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x563304c748e0 0x563304c73b90 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400def0 tx=0x7f8ba400b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bafbf6640 1 -- 172.21.15.99:0/4204454519 reap_dead start 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 shutdown_connections 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 >> 172.21.15.99:0/4204454519 conn(0x563304bbc400 msgr2=0x563304c7a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 shutdown_connections 2024-03-04T10:10:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.615+0000 7f8bb18f7f40 1 -- 172.21.15.99:0/4204454519 wait complete. 2024-03-04T10:10:23.622 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-04T10:10:23.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 Processor -- start 2024-03-04T10:10:23.702 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 -- start start 2024-03-04T10:10:23.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce3fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3cd832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cd83830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3cdcad40 con 0x55bf3cdafaf0 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3cd27b30 con 0x55bf3cdc8f30 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb89846f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf3ce58c40 con 0x55bf3cdc9310 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb86b11640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce3fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb86b11640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce3fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37462/0 (socket says 172.21.15.99:37462) 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.701+0000 7fcb86b11640 1 -- 172.21.15.99:0/3339754665 learned_addr learned my addr 172.21.15.99:0/3339754665 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb86b11640 1 -- 172.21.15.99:0/3339754665 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 msgr2=0x55bf3cd83830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb86b11640 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cd83830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb86b11640 1 -- 172.21.15.99:0/3339754665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 msgr2=0x55bf3cd832f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb86b11640 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3cd832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb86b11640 1 -- 172.21.15.99:0/3339754665 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf3ce4f330 con 0x55bf3cdafaf0 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb86b11640 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce3fca0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fcb7800a770 tx=0x7fcb7800a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11654cb9ddc6ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb8530e640 1 -- 172.21.15.99:0/3339754665 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78018070 con 0x55bf3cdafaf0 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb8530e640 1 -- 172.21.15.99:0/3339754665 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78007600 con 0x55bf3cdafaf0 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb8530e640 1 -- 172.21.15.99:0/3339754665 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb780079b0 con 0x55bf3cdafaf0 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 -- 172.21.15.99:0/3339754665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 msgr2=0x55bf3ce3fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce3fca0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fcb7800a770 tx=0x7fcb7800a860 comp rx=0 tx=0).stop 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 -- 172.21.15.99:0/3339754665 shutdown_connections 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cd83830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3cd832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 --2- 172.21.15.99:0/3339754665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce3fca0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 -- 172.21.15.99:0/3339754665 >> 172.21.15.99:0/3339754665 conn(0x55bf3cd80400 msgr2=0x55bf3cd807e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 -- 172.21.15.99:0/3339754665 shutdown_connections 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.702+0000 7fcb89846f40 1 -- 172.21.15.99:0/3339754665 wait complete. 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.703+0000 7fcb89846f40 1 Processor -- start 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.703+0000 7fcb89846f40 1 -- start start 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce49290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3ce3f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cdaca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3cd819e0 con 0x55bf3cdafaf0 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3cd81cc0 con 0x55bf3cdc8f30 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf3ce606b0 con 0x55bf3cdc9310 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce49290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce49290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37476/0 (socket says 172.21.15.99:37476) 2024-03-04T10:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 -- 172.21.15.99:0/2932626067 learned_addr learned my addr 172.21.15.99:0/2932626067 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86310640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3ce3f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 -- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 msgr2=0x55bf3cdaca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb87312640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cdaca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cdaca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 -- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 msgr2=0x55bf3ce3f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3ce3f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 -- 172.21.15.99:0/2932626067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf3ce3f5f0 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86310640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bf3cdc8f30 0x55bf3ce3f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb87312640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bf3cdc9310 0x55bf3cdaca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb86b11640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce49290 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x55bf3ce4f9f0 tx=0x7fcb78003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb6ffff640 1 -- 172.21.15.99:0/2932626067 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78021020 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb6ffff640 1 -- 172.21.15.99:0/2932626067 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb78019040 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bf3cdacf80 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb6ffff640 1 -- 172.21.15.99:0/2932626067 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb78018070 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.704+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf3ce4ad70 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb6ffff640 1 -- 172.21.15.99:0/2932626067 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcb78024020 con 0x55bf3cdafaf0 2024-03-04T10:10:23.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb6ffff640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcb74069ee0 0x7fcb7406c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb6ffff640 1 -- 172.21.15.99:0/2932626067 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fcb78089fa0 con 0x55bf3cdafaf0 2024-03-04T10:10:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb89846f40 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55bf3cd83650 0x55bf3cdab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x55bf3cdabf00 con 0x55bf3cd83650 2024-03-04T10:10:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb86310640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcb74069ee0 0x7fcb7406c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.706+0000 7fcb87312640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55bf3cd83650 0x55bf3cdab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb87312640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55bf3cd83650 0x55bf3cdab9c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb86310640 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcb74069ee0 0x7fcb7406c3a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c054000 tx=0x7fcb7c04c000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb87312640 1 -- 172.21.15.99:0/2932626067 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7fcb80001710 con 0x55bf3cd83650 2024-03-04T10:10:23.714 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-04T10:10:20.000000+0000, size 0 2024-03-04T10:10:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55bf3cd83650 msgr2=0x55bf3cdab9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55bf3cd83650 0x55bf3cdab9c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.715 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcb74069ee0 msgr2=0x7fcb7406c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcb74069ee0 0x7fcb7406c3a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c054000 tx=0x7fcb7c04c000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 msgr2=0x55bf3ce49290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 --2- 172.21.15.99:0/2932626067 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bf3cdafaf0 0x55bf3ce49290 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x55bf3ce4f9f0 tx=0x7fcb78003020 comp rx=0 tx=0).stop 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb87312640 1 -- 172.21.15.99:0/2932626067 reap_dead start 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 shutdown_connections 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 >> 172.21.15.99:0/2932626067 conn(0x55bf3cd80400 msgr2=0x55bf3cd807e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 shutdown_connections 2024-03-04T10:10:23.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.707+0000 7fcb89846f40 1 -- 172.21.15.99:0/2932626067 wait complete. 2024-03-04T10:10:23.717 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 600.00000000 2024-03-04T10:10:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 Processor -- start 2024-03-04T10:10:23.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 -- start start 2024-03-04T10:10:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9397d9c90 0x55e9398aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9398ab5a0 0x55e9398a7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e939824810 con 0x55e9398b15f0 2024-03-04T10:10:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e939782b30 con 0x55e9397d9c90 2024-03-04T10:10:23.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6428a3bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e939824ab0 con 0x55e9398ab5a0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6426578640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6426578640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398b38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37498/0 (socket says 172.21.15.99:37498) 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.797+0000 7f6426578640 1 -- 172.21.15.99:0/3954383066 learned_addr learned my addr 172.21.15.99:0/3954383066 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6425576640 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9398ab5a0 0x55e9398a7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6425d77640 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9397d9c90 0x55e9398aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6426578640 1 -- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9398ab5a0 msgr2=0x55e9398a7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6426578640 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9398ab5a0 0x55e9398a7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6426578640 1 -- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9397d9c90 msgr2=0x55e9398aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6426578640 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9397d9c90 0x55e9398aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6426578640 1 -- 172.21.15.99:0/3954383066 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9398a2da0 con 0x55e9398b15f0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6425576640 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9398ab5a0 0x55e9398a7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6426578640 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398b38f0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f642000a370 tx=0x7f642000a830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f272682ba06429e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6417fff640 1 -- 172.21.15.99:0/3954383066 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420010070 con 0x55e9398b15f0 2024-03-04T10:10:23.805 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6417fff640 1 -- 172.21.15.99:0/3954383066 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642000c8f0 con 0x55e9398b15f0 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6417fff640 1 -- 172.21.15.99:0/3954383066 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642000cce0 con 0x55e9398b15f0 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 -- 172.21.15.99:0/3954383066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 msgr2=0x55e9398b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398b38f0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f642000a370 tx=0x7f642000a830 comp rx=0 tx=0).stop 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 -- 172.21.15.99:0/3954383066 shutdown_connections 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398b38f0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9398ab5a0 0x55e9398a7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/3954383066 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9397d9c90 0x55e9398aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 -- 172.21.15.99:0/3954383066 >> 172.21.15.99:0/3954383066 conn(0x55e9397db400 msgr2=0x55e939807980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 -- 172.21.15.99:0/3954383066 shutdown_connections 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.798+0000 7f6428a3bf40 1 -- 172.21.15.99:0/3954383066 wait complete. 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.799+0000 7f6428a3bf40 1 Processor -- start 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.799+0000 7f6428a3bf40 1 -- start start 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9397d9c90 0x55e939898640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.806 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9398ab5a0 0x55e9398a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9398a45c0 con 0x55e9398b15f0 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9398b48c0 con 0x55e9398ab5a0 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9398a3040 con 0x55e9397d9c90 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37514/0 (socket says 172.21.15.99:37514) 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 -- 172.21.15.99:0/2863526671 learned_addr learned my addr 172.21.15.99:0/2863526671 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6425576640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9398ab5a0 0x55e9398a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6425d77640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9397d9c90 0x55e939898640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 -- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9397d9c90 msgr2=0x55e939898640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e9397d9c90 0x55e939898640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 -- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9398ab5a0 msgr2=0x55e9398a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e9398ab5a0 0x55e9398a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 -- 172.21.15.99:0/2863526671 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e93980ac70 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6426578640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398accb0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f642000a870 tx=0x7f6420002f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6416ffd640 1 -- 172.21.15.99:0/2863526671 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420025070 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6416ffd640 1 -- 172.21.15.99:0/2863526671 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6420015c10 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.800+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e9398adca0 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.801+0000 7f6416ffd640 1 -- 172.21.15.99:0/2863526671 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420010050 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.801+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e93980af30 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.802+0000 7f6416ffd640 1 -- 172.21.15.99:0/2863526671 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6420028070 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.802+0000 7f6416ffd640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6408069eb0 0x7f640806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.802+0000 7f6416ffd640 1 -- 172.21.15.99:0/2863526671 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f642008a950 con 0x55e9398b15f0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.802+0000 7f6425d77640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6408069eb0 0x7f640806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.802+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e93980b730 0x55e939898c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.802+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0d82a743:::600.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x55e9398991c0 con 0x55e93980b730 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6425576640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e93980b730 0x55e939898c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6425576640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e93980b730 0x55e939898c80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6425d77640 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6408069eb0 0x7f640806c370 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6418052000 tx=0x7f641804d000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6425576640 1 -- 172.21.15.99:0/2863526671 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 600.00000000 [stat out=16b] v0'0 uv2 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x7f641c00a220 con 0x55e93980b730 2024-03-04T10:10:23.809 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/600.00000000 mtime 2024-03-04T10:10:07.000000+0000, size 0 2024-03-04T10:10:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e93980b730 msgr2=0x55e939898c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e93980b730 0x55e939898c80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6408069eb0 msgr2=0x7f640806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6408069eb0 0x7f640806c370 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6418052000 tx=0x7f641804d000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 msgr2=0x55e9398accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 --2- 172.21.15.99:0/2863526671 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e9398b15f0 0x55e9398accb0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f642000a870 tx=0x7f6420002f30 comp rx=0 tx=0).stop 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6426578640 1 -- 172.21.15.99:0/2863526671 reap_dead start 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 shutdown_connections 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 >> 172.21.15.99:0/2863526671 conn(0x55e9397db400 msgr2=0x55e9398a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 shutdown_connections 2024-03-04T10:10:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.803+0000 7f6428a3bf40 1 -- 172.21.15.99:0/2863526671 wait complete. 2024-03-04T10:10:23.811 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000.inode 2024-03-04T10:10:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.891+0000 7fb3c69faf40 1 Processor -- start 2024-03-04T10:10:23.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 -- start start 2024-03-04T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867d01f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867d0c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867d0cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561867d0c300 con 0x561867c33c90 2024-03-04T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561867bdcb30 con 0x561867d0b5f0 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c69faf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561867c7e810 con 0x561867d055a0 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c44f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867d01f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c44f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867d01f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37528/0 (socket says 172.21.15.99:37528) 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c44f8640 1 -- 172.21.15.99:0/998344645 learned_addr learned my addr 172.21.15.99:0/998344645 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c4cf9640 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867d0cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.892+0000 7fb3c3cf7640 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867d0c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c44f8640 1 -- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 msgr2=0x561867d0c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c44f8640 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867d0c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c44f8640 1 -- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 msgr2=0x561867d0cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c44f8640 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867d0cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c44f8640 1 -- 172.21.15.99:0/998344645 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561867d0e7c0 con 0x561867c33c90 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c44f8640 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867d01f60 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4001db0 tx=0x7fb3b400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32a3f330890b66c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c2cf5640 1 -- 172.21.15.99:0/998344645 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400bde0 con 0x561867c33c90 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c2cf5640 1 -- 172.21.15.99:0/998344645 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b400f400 con 0x561867c33c90 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c2cf5640 1 -- 172.21.15.99:0/998344645 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400f7f0 con 0x561867c33c90 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 -- 172.21.15.99:0/998344645 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 msgr2=0x561867d01f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867d01f60 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4001db0 tx=0x7fb3b400b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 -- 172.21.15.99:0/998344645 shutdown_connections 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867d0cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867d0c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/998344645 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867d01f60 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 -- 172.21.15.99:0/998344645 >> 172.21.15.99:0/998344645 conn(0x561867c35400 msgr2=0x561867d0a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 -- 172.21.15.99:0/998344645 shutdown_connections 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.893+0000 7fb3c69faf40 1 -- 172.21.15.99:0/998344645 wait complete. 2024-03-04T10:10:23.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.894+0000 7fb3c69faf40 1 Processor -- start 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.894+0000 7fb3c69faf40 1 -- start start 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867daa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867cff5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867cffb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561867d0e190 con 0x561867c33c90 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561867d0d240 con 0x561867d0b5f0 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561867c7c3c0 con 0x561867d055a0 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867daa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867daa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37544/0 (socket says 172.21.15.99:37544) 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 -- 172.21.15.99:0/2891507638 learned_addr learned my addr 172.21.15.99:0/2891507638 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c4cf9640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867cffb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c3cf7640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867cff5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 -- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 msgr2=0x561867cff5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867cff5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 -- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 msgr2=0x561867cffb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561867d0b5f0 0x561867cffb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 -- 172.21.15.99:0/2891507638 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561867da9d60 con 0x561867c33c90 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c3cf7640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561867d055a0 0x561867cff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c44f8640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867daa3b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4001a70 tx=0x7fb3b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c1cf3640 1 -- 172.21.15.99:0/2891507638 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b4020070 con 0x561867c33c90 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c1cf3640 1 -- 172.21.15.99:0/2891507638 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b401c440 con 0x561867c33c90 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561867cf39a0 con 0x561867c33c90 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c1cf3640 1 -- 172.21.15.99:0/2891507638 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b401c810 con 0x561867c33c90 2024-03-04T10:10:23.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.895+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561867cf3f60 con 0x561867c33c90 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c1cf3640 1 -- 172.21.15.99:0/2891507638 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb3b4023070 con 0x561867c33c90 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c1cf3640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b0069ee0 0x7fb3b006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c1cf3640 1 -- 172.21.15.99:0/2891507638 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb3b408a960 con 0x561867c33c90 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c3cf7640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b0069ee0 0x7fb3b006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561867da6470 0x561867da6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x561867da6dd0 con 0x561867da6470 2024-03-04T10:10:23.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.897+0000 7fb3c4cf9640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561867da6470 0x561867da6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c3cf7640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b0069ee0 0x7fb3b006c3a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8054000 tx=0x7fb3b804e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c4cf9640 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561867da6470 0x561867da6890 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c4cf9640 1 -- 172.21.15.99:0/2891507638 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv16 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x7fb3bc002970 con 0x561867da6470 2024-03-04T10:10:23.904 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/1.00000000.inode mtime 2024-03-04T10:10:20.000000+0000, size 582 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561867da6470 msgr2=0x561867da6890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561867da6470 0x561867da6890 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b0069ee0 msgr2=0x7fb3b006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb3b0069ee0 0x7fb3b006c3a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8054000 tx=0x7fb3b804e000 comp rx=0 tx=0).stop 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 msgr2=0x561867daa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 --2- 172.21.15.99:0/2891507638 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561867c33c90 0x561867daa3b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4001a70 tx=0x7fb3b4000bc0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c4cf9640 1 -- 172.21.15.99:0/2891507638 reap_dead start 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 shutdown_connections 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 >> 172.21.15.99:0/2891507638 conn(0x561867c35400 msgr2=0x561867c64e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 shutdown_connections 2024-03-04T10:10:23.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.898+0000 7fb3c69faf40 1 -- 172.21.15.99:0/2891507638 wait complete. 2024-03-04T10:10:23.907 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000000 2024-03-04T10:10:23.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.985+0000 7fa307bf8f40 1 Processor -- start 2024-03-04T10:10:23.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.985+0000 7fa307bf8f40 1 -- start start 2024-03-04T10:10:23.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa307bf8f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea27766c90 0x55ea27835af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa307bf8f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea278385a0 0x55ea278368c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa307bf8f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 0x55ea27834ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:23.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa307bf8f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea277b1810 con 0x55ea2783e5f0 2024-03-04T10:10:23.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa307bf8f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea2770fb30 con 0x55ea27766c90 2024-03-04T10:10:23.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa307bf8f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea277b1ab0 con 0x55ea278385a0 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa305ef7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 0x55ea27834ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa305ef7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 0x55ea27834ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37560/0 (socket says 172.21.15.99:37560) 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa305ef7640 1 -- 172.21.15.99:0/428879345 learned_addr learned my addr 172.21.15.99:0/428879345 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa3056f6640 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea27766c90 0x55ea27835af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.986+0000 7fa304ef5640 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea278385a0 0x55ea278368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa305ef7640 1 -- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea278385a0 msgr2=0x55ea278368c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa305ef7640 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea278385a0 0x55ea278368c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa305ef7640 1 -- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea27766c90 msgr2=0x55ea27835af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa305ef7640 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea27766c90 0x55ea27835af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa305ef7640 1 -- 172.21.15.99:0/428879345 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea278272a0 con 0x55ea2783e5f0 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa305ef7640 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 0x55ea27834ed0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa300001cb0 tx=0x7fa30000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4cc63b95a4d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa2f77fe640 1 -- 172.21.15.99:0/428879345 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000bcc0 con 0x55ea2783e5f0 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa2f77fe640 1 -- 172.21.15.99:0/428879345 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30000f400 con 0x55ea2783e5f0 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa2f77fe640 1 -- 172.21.15.99:0/428879345 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30000be60 con 0x55ea2783e5f0 2024-03-04T10:10:23.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 -- 172.21.15.99:0/428879345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 msgr2=0x55ea27834ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 0x55ea27834ed0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa300001cb0 tx=0x7fa30000b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 -- 172.21.15.99:0/428879345 shutdown_connections 2024-03-04T10:10:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea2783e5f0 0x55ea27834ed0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea278385a0 0x55ea278368c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/428879345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea27766c90 0x55ea27835af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 -- 172.21.15.99:0/428879345 >> 172.21.15.99:0/428879345 conn(0x55ea27768400 msgr2=0x55ea27841320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 -- 172.21.15.99:0/428879345 shutdown_connections 2024-03-04T10:10:24.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.987+0000 7fa307bf8f40 1 -- 172.21.15.99:0/428879345 wait complete. 2024-03-04T10:10:24.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.988+0000 7fa307bf8f40 1 Processor -- start 2024-03-04T10:10:24.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.988+0000 7fa307bf8f40 1 -- start start 2024-03-04T10:10:24.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea27766c90 0x55ea2783b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea278385a0 0x55ea27823b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea2783e5f0 0x55ea27793ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea2783ea00 con 0x55ea27766c90 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea2783d630 con 0x55ea278385a0 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ea277af490 con 0x55ea2783e5f0 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea27766c90 0x55ea2783b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea27766c90 0x55ea2783b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37570/0 (socket says 172.21.15.99:37570) 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 -- 172.21.15.99:0/2325822078 learned_addr learned my addr 172.21.15.99:0/2325822078 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa304ef5640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea278385a0 0x55ea27823b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa305ef7640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea2783e5f0 0x55ea27793ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 -- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea2783e5f0 msgr2=0x55ea27793ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55ea2783e5f0 0x55ea27793ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 -- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea278385a0 msgr2=0x55ea27823b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ea278385a0 0x55ea27823b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 -- 172.21.15.99:0/2325822078 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea27840c40 con 0x55ea27766c90 2024-03-04T10:10:24.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa3056f6640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea27766c90 0x55ea2783b600 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8054040 tx=0x7fa2f8003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa2f67fc640 1 -- 172.21.15.99:0/2325822078 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8005a80 con 0x55ea27766c90 2024-03-04T10:10:24.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa2f67fc640 1 -- 172.21.15.99:0/2325822078 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f805c810 con 0x55ea27766c90 2024-03-04T10:10:24.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa2f67fc640 1 -- 172.21.15.99:0/2325822078 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f805cc00 con 0x55ea27766c90 2024-03-04T10:10:24.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.989+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ea27822d90 con 0x55ea27766c90 2024-03-04T10:10:24.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.990+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea27823350 con 0x55ea27766c90 2024-03-04T10:10:24.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.991+0000 7fa2f67fc640 1 -- 172.21.15.99:0/2325822078 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa2f8064020 con 0x55ea27766c90 2024-03-04T10:10:24.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.991+0000 7fa2f67fc640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2e8069eb0 0x7fa2e806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.991+0000 7fa2f67fc640 1 -- 172.21.15.99:0/2325822078 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fa2f80cb560 con 0x55ea27766c90 2024-03-04T10:10:24.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa304ef5640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2e8069eb0 0x7fa2e806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55ea27792850 0x55ea278247e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:292cf221:::200.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x55ea27824d20 con 0x55ea27792850 2024-03-04T10:10:24.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa305ef7640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55ea27792850 0x55ea278247e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa305ef7640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55ea27792850 0x55ea278247e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa304ef5640 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2e8069eb0 0x7fa2e806c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x55ea27798a10 tx=0x7fa2fc00c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa305ef7640 1 -- 172.21.15.99:0/2325822078 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000000 [stat out=16b] v0'0 uv20 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x7fa30000f5b0 con 0x55ea27792850 2024-03-04T10:10:24.006 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/200.00000000 mtime 2024-03-04T10:10:20.000000+0000, size 90 2024-03-04T10:10:24.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55ea27792850 msgr2=0x55ea278247e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55ea27792850 0x55ea278247e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2e8069eb0 msgr2=0x7fa2e806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa2e8069eb0 0x7fa2e806c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x55ea27798a10 tx=0x7fa2fc00c040 comp rx=0 tx=0).stop 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea27766c90 msgr2=0x55ea2783b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 --2- 172.21.15.99:0/2325822078 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55ea27766c90 0x55ea2783b600 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8054040 tx=0x7fa2f8003330 comp rx=0 tx=0).stop 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa305ef7640 1 -- 172.21.15.99:0/2325822078 reap_dead start 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 shutdown_connections 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 >> 172.21.15.99:0/2325822078 conn(0x55ea27768400 msgr2=0x55ea27841290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 shutdown_connections 2024-03-04T10:10:24.010 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:23.992+0000 7fa307bf8f40 1 -- 172.21.15.99:0/2325822078 wait complete. 2024-03-04T10:10:24.011 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 200.00000001 2024-03-04T10:10:24.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.088+0000 7f901d5edf40 1 Processor -- start 2024-03-04T10:10:24.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.088+0000 7f901d5edf40 1 -- start start 2024-03-04T10:10:24.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901d5edf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbd08710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901d5edf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdcf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901d5edf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdcfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901d5edf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbd50d40 con 0x5565dbd35cc0 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901d5edf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbcadb30 con 0x5565dbd4ef30 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901d5edf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565dbddec40 con 0x5565dbd4f310 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbd08710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbd08710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37590/0 (socket says 172.21.15.99:37590) 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 -- 172.21.15.99:0/3243928708 learned_addr learned my addr 172.21.15.99:0/3243928708 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b8ee640 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdcfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901a8ec640 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdcf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 -- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 msgr2=0x5565dbdcfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdcfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 -- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 msgr2=0x5565dbdcf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdcf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 -- 172.21.15.99:0/3243928708 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565dbd069a0 con 0x5565dbd35cc0 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.089+0000 7f901b0ed640 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbd08710 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f900c0095d0 tx=0x7f900c009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859e910b4410e2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f90198ea640 1 -- 172.21.15.99:0/3243928708 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c006680 con 0x5565dbd35cc0 2024-03-04T10:10:24.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f90198ea640 1 -- 172.21.15.99:0/3243928708 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c006820 con 0x5565dbd35cc0 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f90198ea640 1 -- 172.21.15.99:0/3243928708 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c006c10 con 0x5565dbd35cc0 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 -- 172.21.15.99:0/3243928708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 msgr2=0x5565dbd08710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbd08710 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f900c0095d0 tx=0x7f900c009a90 comp rx=0 tx=0).stop 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 -- 172.21.15.99:0/3243928708 shutdown_connections 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdcfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdcf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 --2- 172.21.15.99:0/3243928708 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbd08710 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 -- 172.21.15.99:0/3243928708 >> 172.21.15.99:0/3243928708 conn(0x5565dbd06400 msgr2=0x5565dbd079e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 -- 172.21.15.99:0/3243928708 shutdown_connections 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.090+0000 7f901d5edf40 1 -- 172.21.15.99:0/3243928708 wait complete. 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 Processor -- start 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 -- start start 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbdc3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdc5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdc5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbddd0f0 con 0x5565dbd35cc0 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbddce10 con 0x5565dbd4ef30 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901d5edf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565dbddd3d0 con 0x5565dbd4f310 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901b0ed640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbdc3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.091+0000 7f901b0ed640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbdc3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37604/0 (socket says 172.21.15.99:37604) 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 -- 172.21.15.99:0/541118696 learned_addr learned my addr 172.21.15.99:0/541118696 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901a8ec640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdc5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b8ee640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdc5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 -- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 msgr2=0x5565dbdc5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5565dbd4f310 0x5565dbdc5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 -- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 msgr2=0x5565dbdc5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5565dbd4ef30 0x5565dbdc5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 -- 172.21.15.99:0/541118696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565dbdd4f90 con 0x5565dbd35cc0 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901b0ed640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbdc3a30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f900c00afd0 tx=0x7f900c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f90188e8640 1 -- 172.21.15.99:0/541118696 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c020070 con 0x5565dbd35cc0 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f90188e8640 1 -- 172.21.15.99:0/541118696 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f900c01c640 con 0x5565dbd35cc0 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5565dbdd5270 con 0x5565dbd35cc0 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f90188e8640 1 -- 172.21.15.99:0/541118696 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900c01ca30 con 0x5565dbd35cc0 2024-03-04T10:10:24.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.092+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565dbdd5830 con 0x5565dbd35cc0 2024-03-04T10:10:24.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f90188e8640 1 -- 172.21.15.99:0/541118696 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f900c023070 con 0x5565dbd35cc0 2024-03-04T10:10:24.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f90188e8640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9008069eb0 0x7f900806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f90188e8640 1 -- 172.21.15.99:0/541118696 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f900c08aa60 con 0x5565dbd35cc0 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f901a8ec640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9008069eb0 0x7f900806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f901d5edf40 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5565dbdc3f70 0x5565dbdc7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5565dbd4d8d0 con 0x5565dbdc3f70 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f901b8ee640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5565dbdc3f70 0x5565dbdc7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.094+0000 7f901a8ec640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9008069eb0 0x7f900806c370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x5565dbd364c0 tx=0x7f9010005980 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901b8ee640 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5565dbdc3f70 0x5565dbdc7e70 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901b8ee640 1 -- 172.21.15.99:0/541118696 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000001 [stat out=16b] v0'0 uv17 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x7f9014052720 con 0x5565dbdc3f70 2024-03-04T10:10:24.101 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/200.00000001 mtime 2024-03-04T10:10:20.000000+0000, size 18075 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5565dbdc3f70 msgr2=0x5565dbdc7e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5565dbdc3f70 0x5565dbdc7e70 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9008069eb0 msgr2=0x7f900806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9008069eb0 0x7f900806c370 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x5565dbd364c0 tx=0x7f9010005980 comp rx=0 tx=0).stop 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 msgr2=0x5565dbdc3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 --2- 172.21.15.99:0/541118696 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5565dbd35cc0 0x5565dbdc3a30 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f900c00afd0 tx=0x7f900c003020 comp rx=0 tx=0).stop 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901b8ee640 1 -- 172.21.15.99:0/541118696 reap_dead start 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 shutdown_connections 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 >> 172.21.15.99:0/541118696 conn(0x5565dbd06400 msgr2=0x5565dbd09380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 shutdown_connections 2024-03-04T10:10:24.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.095+0000 7f901d5edf40 1 -- 172.21.15.99:0/541118696 wait complete. 2024-03-04T10:10:24.104 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_openfiles.0 2024-03-04T10:10:24.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 Processor -- start 2024-03-04T10:10:24.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 -- start start 2024-03-04T10:10:24.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 0x55fdfa811200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa7f51c0 0x55fdfa769880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 0x55fdfa768370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfa6e1b30 con 0x55fdfa812670 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfa6e5a90 con 0x55fdfa7f51c0 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92b200ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdfa813610 con 0x55fdfa7393a0 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.184+0000 7f92af6e7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 0x55fdfa768370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 0x55fdfa768370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37622/0 (socket says 172.21.15.99:37622) 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 -- 172.21.15.99:0/530223837 learned_addr learned my addr 172.21.15.99:0/530223837 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 -- 172.21.15.99:0/530223837 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 msgr2=0x55fdfa811200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 0x55fdfa811200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 -- 172.21.15.99:0/530223837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa7f51c0 msgr2=0x55fdfa769880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa7f51c0 0x55fdfa769880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 -- 172.21.15.99:0/530223837 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdfa769dc0 con 0x55fdfa812670 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92af6e7640 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 0x55fdfa768370 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f92a801ced0 tx=0x7f92a804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb3bf2f90ce47c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92ad6e3640 1 -- 172.21.15.99:0/530223837 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8057040 con 0x55fdfa812670 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92ad6e3640 1 -- 172.21.15.99:0/530223837 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a8063070 con 0x55fdfa812670 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92ad6e3640 1 -- 172.21.15.99:0/530223837 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8052d20 con 0x55fdfa812670 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92b200ff40 1 -- 172.21.15.99:0/530223837 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 msgr2=0x55fdfa768370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.185+0000 7f92b200ff40 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 0x55fdfa768370 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f92a801ced0 tx=0x7f92a804c000 comp rx=0 tx=0).stop 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 -- 172.21.15.99:0/530223837 shutdown_connections 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa812670 0x55fdfa768370 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa7f51c0 0x55fdfa769880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 --2- 172.21.15.99:0/530223837 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 0x55fdfa811200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 -- 172.21.15.99:0/530223837 >> 172.21.15.99:0/530223837 conn(0x55fdfa73a400 msgr2=0x55fdfa7f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 -- 172.21.15.99:0/530223837 shutdown_connections 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 -- 172.21.15.99:0/530223837 wait complete. 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 Processor -- start 2024-03-04T10:10:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.186+0000 7f92b200ff40 1 -- start start 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92b200ff40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 0x55fdfa810ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92b200ff40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa7f51c0 0x55fdfa7f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92b200ff40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa812670 0x55fdfa7f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92b200ff40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfa7f2450 con 0x55fdfa7f51c0 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92b200ff40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fdfa76a8d0 con 0x55fdfa812670 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92b200ff40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fdfa76a030 con 0x55fdfa7393a0 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa7f51c0 0x55fdfa7f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa7f51c0 0x55fdfa7f7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37638/0 (socket says 172.21.15.99:37638) 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 -- 172.21.15.99:0/1461708665 learned_addr learned my addr 172.21.15.99:0/1461708665 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92aeee6640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 0x55fdfa810ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92af6e7640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa812670 0x55fdfa7f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 -- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 msgr2=0x55fdfa810ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55fdfa7393a0 0x55fdfa810ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 -- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa812670 msgr2=0x55fdfa7f77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdfa812670 0x55fdfa7f77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 -- 172.21.15.99:0/1461708665 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fdfa768c50 con 0x55fdfa7f51c0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.187+0000 7f92ae6e5640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa7f51c0 0x55fdfa7f7260 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f92a400ab30 tx=0x7f92a4005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.188+0000 7f9297fff640 1 -- 172.21.15.99:0/1461708665 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4014470 con 0x55fdfa7f51c0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.188+0000 7f9297fff640 1 -- 172.21.15.99:0/1461708665 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a400f070 con 0x55fdfa7f51c0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.188+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55fdfa7f1790 con 0x55fdfa7f51c0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.188+0000 7f9297fff640 1 -- 172.21.15.99:0/1461708665 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4014850 con 0x55fdfa7f51c0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.188+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fdfa768970 con 0x55fdfa7f51c0 2024-03-04T10:10:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.189+0000 7f9297fff640 1 -- 172.21.15.99:0/1461708665 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f92a401d3f0 con 0x55fdfa7f51c0 2024-03-04T10:10:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.189+0000 7f9297fff640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f929c069eb0 0x7f929c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.189+0000 7f9297fff640 1 -- 172.21.15.99:0/1461708665 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f92a40889e0 con 0x55fdfa7f51c0 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.189+0000 7f92aeee6640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f929c069eb0 0x7f929c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.190+0000 7f92b200ff40 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdfa811000 0x55fdfa8af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.190+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.6 13:654134d2:::mds0_openfiles.0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x55fdfa8094f0 con 0x55fdfa811000 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.190+0000 7f92af6e7640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdfa811000 0x55fdfa8af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.190+0000 7f92aeee6640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f929c069eb0 0x7f929c06c370 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f92a00012e0 tx=0x7f92a00068c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.190+0000 7f92af6e7640 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdfa811000 0x55fdfa8af160 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92af6e7640 1 -- 172.21.15.99:0/1461708665 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 mds0_openfiles.0 [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x7f92a805a040 con 0x55fdfa811000 2024-03-04T10:10:24.197 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/mds0_openfiles.0 mtime 2024-03-04T10:10:20.000000+0000, size 0 2024-03-04T10:10:24.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdfa811000 msgr2=0x55fdfa8af160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55fdfa811000 0x55fdfa8af160 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f929c069eb0 msgr2=0x7f929c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f929c069eb0 0x7f929c06c370 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f92a00012e0 tx=0x7f92a00068c0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa7f51c0 msgr2=0x55fdfa7f7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 --2- 172.21.15.99:0/1461708665 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55fdfa7f51c0 0x55fdfa7f7260 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f92a400ab30 tx=0x7f92a4005f70 comp rx=0 tx=0).stop 2024-03-04T10:10:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92af6e7640 1 -- 172.21.15.99:0/1461708665 reap_dead start 2024-03-04T10:10:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 shutdown_connections 2024-03-04T10:10:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 >> 172.21.15.99:0/1461708665 conn(0x55fdfa73a400 msgr2=0x55fdfa7f2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 shutdown_connections 2024-03-04T10:10:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.191+0000 7f92b200ff40 1 -- 172.21.15.99:0/1461708665 wait complete. 2024-03-04T10:10:24.199 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 500.00000000 2024-03-04T10:10:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.277+0000 7f801f786f40 1 Processor -- start 2024-03-04T10:10:24.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.277+0000 7f801f786f40 1 -- start start 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801f786f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b1041d13a0 0x55b104289eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801f786f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 0x55b1042a8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801f786f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1042aa670 0x55b1042a7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801f786f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1042ab610 con 0x55b10428d1c0 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801f786f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b104179b30 con 0x55b1042aa670 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801f786f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b10421bb50 con 0x55b1041d13a0 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 0x55b1042a8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 0x55b1042a8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37648/0 (socket says 172.21.15.99:37648) 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 -- 172.21.15.99:0/255498782 learned_addr learned my addr 172.21.15.99:0/255498782 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801da87640 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1042aa670 0x55b1042a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f801d286640 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b1041d13a0 0x55b104289eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 -- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b1041d13a0 msgr2=0x55b104289eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b1041d13a0 0x55b104289eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 -- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1042aa670 msgr2=0x55b1042a7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1042aa670 0x55b1042a7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 -- 172.21.15.99:0/255498782 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b10428a3f0 con 0x55b10428d1c0 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.278+0000 7f8014a85640 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 0x55b1042a8380 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f800800b240 tx=0x7f800800b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4d627e3bc64956 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f8017fff640 1 -- 172.21.15.99:0/255498782 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008008070 con 0x55b10428d1c0 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f8017fff640 1 -- 172.21.15.99:0/255498782 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800800d500 con 0x55b10428d1c0 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f8017fff640 1 -- 172.21.15.99:0/255498782 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800800d910 con 0x55b10428d1c0 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 -- 172.21.15.99:0/255498782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 msgr2=0x55b1042a8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 0x55b1042a8380 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f800800b240 tx=0x7f800800b700 comp rx=0 tx=0).stop 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 -- 172.21.15.99:0/255498782 shutdown_connections 2024-03-04T10:10:24.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1042aa670 0x55b1042a7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b10428d1c0 0x55b1042a8380 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 --2- 172.21.15.99:0/255498782 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b1041d13a0 0x55b104289eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 -- 172.21.15.99:0/255498782 >> 172.21.15.99:0/255498782 conn(0x55b1041d2400 msgr2=0x55b104201920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 -- 172.21.15.99:0/255498782 shutdown_connections 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.279+0000 7f801f786f40 1 -- 172.21.15.99:0/255498782 wait complete. 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 Processor -- start 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 -- start start 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1041d13a0 0x55b1042a40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b10428d1c0 0x55b1041d42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b1042aa670 0x55b104200a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1042acd90 con 0x55b1042aa670 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1042a9da0 con 0x55b1041d13a0 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801f786f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b1042a5390 con 0x55b10428d1c0 2024-03-04T10:10:24.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801da87640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b1042aa670 0x55b104200a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801da87640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b1042aa670 0x55b104200a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37652/0 (socket says 172.21.15.99:37652) 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.280+0000 7f801da87640 1 -- 172.21.15.99:0/4112590202 learned_addr learned my addr 172.21.15.99:0/4112590202 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801d286640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1041d13a0 0x55b1042a40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f8014a85640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b10428d1c0 0x55b1041d42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801da87640 1 -- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b10428d1c0 msgr2=0x55b1041d42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801da87640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55b10428d1c0 0x55b1041d42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801da87640 1 -- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1041d13a0 msgr2=0x55b1042a40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801da87640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b1041d13a0 0x55b1042a40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801da87640 1 -- 172.21.15.99:0/4112590202 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b10433c860 con 0x55b1042aa670 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801da87640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b1042aa670 0x55b104200a70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f801801baf0 tx=0x7f8018003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f8016ffd640 1 -- 172.21.15.99:0/4112590202 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8018062070 con 0x55b1042aa670 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f8016ffd640 1 -- 172.21.15.99:0/4112590202 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8018058040 con 0x55b1042aa670 2024-03-04T10:10:24.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b10433e240 con 0x55b1042aa670 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f8016ffd640 1 -- 172.21.15.99:0/4112590202 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801805e830 con 0x55b1042aa670 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.281+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b10433df30 con 0x55b1042aa670 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f8016ffd640 1 -- 172.21.15.99:0/4112590202 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8018065070 con 0x55b1042aa670 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f8016ffd640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8000069eb0 0x7f800006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f8016ffd640 1 -- 172.21.15.99:0/4112590202 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f80180cb430 con 0x55b1042aa670 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f801d286640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8000069eb0 0x7f800006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f801f786f40 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55b10433c3d0 0x55b1042a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.15 13:aa448500:::500.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x55b104200fb0 con 0x55b10433c3d0 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.283+0000 7f8014a85640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55b10433c3d0 0x55b1042a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801d286640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8000069eb0 0x7f800006c370 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8010005fd0 tx=0x7f801000c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f8014a85640 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55b10433c3d0 0x55b1042a56d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f8014a85640 1 -- 172.21.15.99:0/4112590202 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 500.00000000 [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x7f800800d730 con 0x55b10433c3d0 2024-03-04T10:10:24.290 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/500.00000000 mtime 2024-03-04T10:10:07.000000+0000, size 90 2024-03-04T10:10:24.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55b10433c3d0 msgr2=0x55b1042a56d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55b10433c3d0 0x55b1042a56d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8000069eb0 msgr2=0x7f800006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8000069eb0 0x7f800006c370 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8010005fd0 tx=0x7f801000c040 comp rx=0 tx=0).stop 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b1042aa670 msgr2=0x55b104200a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 --2- 172.21.15.99:0/4112590202 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55b1042aa670 0x55b104200a70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f801801baf0 tx=0x7f8018003330 comp rx=0 tx=0).stop 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801da87640 1 -- 172.21.15.99:0/4112590202 reap_dead start 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 shutdown_connections 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 >> 172.21.15.99:0/4112590202 conn(0x55b1041d2400 msgr2=0x55b10428b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 shutdown_connections 2024-03-04T10:10:24.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.284+0000 7f801f786f40 1 -- 172.21.15.99:0/4112590202 wait complete. 2024-03-04T10:10:24.293 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds_snaptable 2024-03-04T10:10:24.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.371+0000 7fefc453cf40 1 Processor -- start 2024-03-04T10:10:24.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.371+0000 7fefc453cf40 1 -- start start 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc453cf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708cccc0 0x563870967230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc453cf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708e5f30 0x563870965d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc453cf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x56387089ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc453cf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638708e7d40 con 0x5638708e6310 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc453cf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x563870844b30 con 0x5638708e5f30 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc453cf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563870975c40 con 0x5638708cccc0 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x56387089ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x56387089ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37670/0 (socket says 172.21.15.99:37670) 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 -- 172.21.15.99:0/1887852586 learned_addr learned my addr 172.21.15.99:0/1887852586 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 -- 172.21.15.99:0/1887852586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708cccc0 msgr2=0x563870967230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708cccc0 0x563870967230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 -- 172.21.15.99:0/1887852586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708e5f30 msgr2=0x563870965d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708e5f30 0x563870965d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.372+0000 7fefc283b640 1 -- 172.21.15.99:0/1887852586 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638708c9e00 con 0x5638708e6310 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc283b640 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x56387089ea50 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fefbc01cf60 tx=0x7fefbc04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6333ce7dafa596c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc0837640 1 -- 172.21.15.99:0/1887852586 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc057040 con 0x5638708e6310 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc0837640 1 -- 172.21.15.99:0/1887852586 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc062070 con 0x5638708e6310 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc0837640 1 -- 172.21.15.99:0/1887852586 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc052b10 con 0x5638708e6310 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 -- 172.21.15.99:0/1887852586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 msgr2=0x56387089ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x56387089ea50 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fefbc01cf60 tx=0x7fefbc04c000 comp rx=0 tx=0).stop 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 -- 172.21.15.99:0/1887852586 shutdown_connections 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x56387089ea50 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708e5f30 0x563870965d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 --2- 172.21.15.99:0/1887852586 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708cccc0 0x563870967230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 -- 172.21.15.99:0/1887852586 >> 172.21.15.99:0/1887852586 conn(0x56387089d400 msgr2=0x5638709749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 -- 172.21.15.99:0/1887852586 shutdown_connections 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.373+0000 7fefc453cf40 1 -- 172.21.15.99:0/1887852586 wait complete. 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.374+0000 7fefc453cf40 1 Processor -- start 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.374+0000 7fefc453cf40 1 -- start start 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708cccc0 0x56387089ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708e5f30 0x563870974c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x563870a0fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x563870966790 con 0x5638708e6310 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638708c9b40 con 0x5638708cccc0 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x563870966a30 con 0x5638708e5f30 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x563870a0fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x563870a0fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37684/0 (socket says 172.21.15.99:37684) 2024-03-04T10:10:24.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 -- 172.21.15.99:0/2554642538 learned_addr learned my addr 172.21.15.99:0/2554642538 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 -- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708e5f30 msgr2=0x563870974c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5638708e5f30 0x563870974c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 -- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708cccc0 msgr2=0x56387089ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5638708cccc0 0x56387089ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 -- 172.21.15.99:0/2554642538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56387089be80 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc283b640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x563870a0fb80 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fefbc01cfd0 tx=0x7fefbc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefb37fe640 1 -- 172.21.15.99:0/2554642538 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc057040 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefb37fe640 1 -- 172.21.15.99:0/2554642538 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc05d470 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefb37fe640 1 -- 172.21.15.99:0/2554642538 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc05d860 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56387089c160 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.375+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56387089c720 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefb37fe640 1 -- 172.21.15.99:0/2554642538 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fefbc06f020 con 0x5638708e6310 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefb37fe640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefa4069eb0 0x7fefa406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefb37fe640 1 -- 172.21.15.99:0/2554642538 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fefbc0ccb70 con 0x5638708e6310 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefc453cf40 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x56387089b9f0 0x563870962140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefc203a640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefa4069eb0 0x7fefa406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:1 13.d 13:b50e409b:::mds_snaptable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x56387089f110 con 0x56387089b9f0 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.377+0000 7fefc1839640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x56387089b9f0 0x563870962140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc203a640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefa4069eb0 0x7fefa406c370 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x563870976380 tx=0x7fefb4007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc1839640 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x56387089b9f0 0x563870962140 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc1839640 1 -- 172.21.15.99:0/2554642538 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(1 mds_snaptable [stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+16 (crc 0 0 0) 0x7fefb800d2c0 con 0x56387089b9f0 2024-03-04T10:10:24.384 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/mds_snaptable mtime 2024-03-04T10:10:07.000000+0000, size 70 2024-03-04T10:10:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x56387089b9f0 msgr2=0x563870962140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x56387089b9f0 0x563870962140 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefa4069eb0 msgr2=0x7fefa406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fefa4069eb0 0x7fefa406c370 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x563870976380 tx=0x7fefb4007040 comp rx=0 tx=0).stop 2024-03-04T10:10:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 msgr2=0x563870a0fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 --2- 172.21.15.99:0/2554642538 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5638708e6310 0x563870a0fb80 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fefbc01cfd0 tx=0x7fefbc002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc283b640 1 -- 172.21.15.99:0/2554642538 reap_dead start 2024-03-04T10:10:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 shutdown_connections 2024-03-04T10:10:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 >> 172.21.15.99:0/2554642538 conn(0x56387089d400 msgr2=0x56387096c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 shutdown_connections 2024-03-04T10:10:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.378+0000 7fefc453cf40 1 -- 172.21.15.99:0/2554642538 wait complete. 2024-03-04T10:10:24.386 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_inotable 2024-03-04T10:10:24.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.465+0000 7fee8c04af40 1 Processor -- start 2024-03-04T10:10:24.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.465+0000 7fee8c04af40 1 -- start start 2024-03-04T10:10:24.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee8c04af40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 0x5644f45021e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee8c04af40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f442e810 0x5644f45059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee8c04af40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644f4505ee0 0x5644f445cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee8c04af40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f43d5b30 con 0x7fee74002c80 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee8c04af40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f43d9a90 con 0x5644f4505ee0 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee8c04af40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644f44786d0 con 0x5644f442e810 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 0x5644f45021e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 0x5644f45021e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37692/0 (socket says 172.21.15.99:37692) 2024-03-04T10:10:24.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 -- 172.21.15.99:0/1234074153 learned_addr learned my addr 172.21.15.99:0/1234074153 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 -- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f442e810 msgr2=0x5644f45059a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f442e810 0x5644f45059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89c14640 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644f4505ee0 0x5644f445cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 -- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644f4505ee0 msgr2=0x5644f445cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644f4505ee0 0x5644f445cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 -- 172.21.15.99:0/1234074153 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644f4474d40 con 0x7fee74002c80 2024-03-04T10:10:24.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee89413640 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 0x5644f45021e0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00aad0 tx=0x7fee7c00af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712545af4770af89 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee7b7fe640 1 -- 172.21.15.99:0/1234074153 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00fcb0 con 0x7fee74002c80 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee7b7fe640 1 -- 172.21.15.99:0/1234074153 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c00fe50 con 0x7fee74002c80 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.466+0000 7fee7b7fe640 1 -- 172.21.15.99:0/1234074153 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c012660 con 0x7fee74002c80 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 -- 172.21.15.99:0/1234074153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 msgr2=0x5644f45021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 0x5644f45021e0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00aad0 tx=0x7fee7c00af90 comp rx=0 tx=0).stop 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 -- 172.21.15.99:0/1234074153 shutdown_connections 2024-03-04T10:10:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644f4505ee0 0x5644f445cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f442e810 0x5644f45059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 --2- 172.21.15.99:0/1234074153 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fee74002c80 0x5644f45021e0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 -- 172.21.15.99:0/1234074153 >> 172.21.15.99:0/1234074153 conn(0x5644f442e400 msgr2=0x5644f442c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 -- 172.21.15.99:0/1234074153 shutdown_connections 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 -- 172.21.15.99:0/1234074153 wait complete. 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 Processor -- start 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.467+0000 7fee8c04af40 1 -- start start 2024-03-04T10:10:24.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee8c04af40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644f4505ee0 0x5644f44f9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee8c04af40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74002c80 0x5644f44731b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee8c04af40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f445a800 0x5644f44736f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee8c04af40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f4502b20 con 0x5644f4505ee0 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee8c04af40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644f44745d0 con 0x7fee74002c80 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee8c04af40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644f4474a60 con 0x5644f445a800 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644f4505ee0 0x5644f44f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644f4505ee0 0x5644f44f9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37704/0 (socket says 172.21.15.99:37704) 2024-03-04T10:10:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 -- 172.21.15.99:0/3652634493 learned_addr learned my addr 172.21.15.99:0/3652634493 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee88c12640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74002c80 0x5644f44731b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 -- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f445a800 msgr2=0x5644f44736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644f445a800 0x5644f44736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 -- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74002c80 msgr2=0x5644f44731b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fee74002c80 0x5644f44731b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 -- 172.21.15.99:0/3652634493 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644f45950d0 con 0x5644f4505ee0 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.468+0000 7fee89413640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644f4505ee0 0x5644f44f9530 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fee7c012ee0 tx=0x7fee7c001320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.469+0000 7fee7a7fc640 1 -- 172.21.15.99:0/3652634493 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00fcb0 con 0x5644f4505ee0 2024-03-04T10:10:24.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.469+0000 7fee7a7fc640 1 -- 172.21.15.99:0/3652634493 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7c023030 con 0x5644f4505ee0 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.469+0000 7fee7a7fc640 1 -- 172.21.15.99:0/3652634493 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c01cab0 con 0x5644f4505ee0 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.469+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5644f445af10 con 0x5644f4505ee0 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.469+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644f4595370 con 0x5644f4505ee0 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.470+0000 7fee7a7fc640 1 -- 172.21.15.99:0/3652634493 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fee7c01cc50 con 0x5644f4505ee0 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.470+0000 7fee7a7fc640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fee6c069eb0 0x7fee6c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.470+0000 7fee7a7fc640 1 -- 172.21.15.99:0/3652634493 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fee7c08dea0 con 0x5644f4505ee0 2024-03-04T10:10:24.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.470+0000 7fee88c12640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fee6c069eb0 0x7fee6c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.471+0000 7fee8c04af40 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5644f445d850 0x5644f45010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.471+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.13 13:c91d4a1d:::mds0_inotable:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5644f459b1a0 con 0x5644f445d850 2024-03-04T10:10:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.471+0000 7fee89c14640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5644f445d850 0x5644f45010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.471+0000 7fee88c12640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fee6c069eb0 0x7fee6c06c370 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fee80054000 tx=0x7fee80002fb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.471+0000 7fee89c14640 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5644f445d850 0x5644f45010b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.471+0000 7fee89c14640 1 -- 172.21.15.99:0/3652634493 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 mds0_inotable [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+16 (crc 0 0 0) 0x7fee84003450 con 0x5644f445d850 2024-03-04T10:10:24.482 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/mds0_inotable mtime 2024-03-04T10:10:20.000000+0000, size 34 2024-03-04T10:10:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5644f445d850 msgr2=0x5644f45010b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5644f445d850 0x5644f45010b0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fee6c069eb0 msgr2=0x7fee6c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fee6c069eb0 0x7fee6c06c370 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fee80054000 tx=0x7fee80002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644f4505ee0 msgr2=0x5644f44f9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 --2- 172.21.15.99:0/3652634493 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644f4505ee0 0x5644f44f9530 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fee7c012ee0 tx=0x7fee7c001320 comp rx=0 tx=0).stop 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee89c14640 1 -- 172.21.15.99:0/3652634493 reap_dead start 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 shutdown_connections 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 >> 172.21.15.99:0/3652634493 conn(0x5644f442e400 msgr2=0x5644f4476d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 shutdown_connections 2024-03-04T10:10:24.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.472+0000 7fee8c04af40 1 -- 172.21.15.99:0/3652634493 wait complete. 2024-03-04T10:10:24.486 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000 2024-03-04T10:10:24.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.563+0000 7f5020a45f40 1 Processor -- start 2024-03-04T10:10:24.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.563+0000 7f5020a45f40 1 -- start start 2024-03-04T10:10:24.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f5020a45f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea3e7cc0 0x5648ea48e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f5020a45f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea400f30 0x5648ea3bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f5020a45f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea486db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f5020a45f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ea402d40 con 0x5648ea401310 2024-03-04T10:10:24.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f5020a45f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ea35fb30 con 0x5648ea400f30 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f5020a45f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648ea490c40 con 0x5648ea3e7cc0 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea486db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea486db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37718/0 (socket says 172.21.15.99:37718) 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 -- 172.21.15.99:0/2215866245 learned_addr learned my addr 172.21.15.99:0/2215866245 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501d5a4640 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea400f30 0x5648ea3bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 -- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea3e7cc0 msgr2=0x5648ea48e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea3e7cc0 0x5648ea48e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 -- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea400f30 msgr2=0x5648ea3bb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea400f30 0x5648ea3bb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.564+0000 7f501e5a6640 1 -- 172.21.15.99:0/2215866245 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648ea3b8ac0 con 0x5648ea401310 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f501e5a6640 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea486db0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f501800e7e0 tx=0x7f501800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f5f9a775dd38f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f500ffff640 1 -- 172.21.15.99:0/2215866245 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018014070 con 0x5648ea401310 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f500ffff640 1 -- 172.21.15.99:0/2215866245 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501800cd00 con 0x5648ea401310 2024-03-04T10:10:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f500ffff640 1 -- 172.21.15.99:0/2215866245 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180196a0 con 0x5648ea401310 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 -- 172.21.15.99:0/2215866245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 msgr2=0x5648ea486db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea486db0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f501800e7e0 tx=0x7f501800eca0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 -- 172.21.15.99:0/2215866245 shutdown_connections 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea486db0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea400f30 0x5648ea3bb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 --2- 172.21.15.99:0/2215866245 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea3e7cc0 0x5648ea48e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 -- 172.21.15.99:0/2215866245 >> 172.21.15.99:0/2215866245 conn(0x5648ea3b8400 msgr2=0x5648ea3e47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 -- 172.21.15.99:0/2215866245 shutdown_connections 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.565+0000 7f5020a45f40 1 -- 172.21.15.99:0/2215866245 wait complete. 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 Processor -- start 2024-03-04T10:10:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 -- start start 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea3e7cc0 0x5648ea491b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea400f30 0x5648ea477a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea3e70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ea3b87e0 con 0x5648ea401310 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ea3b9b60 con 0x5648ea3e7cc0 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.566+0000 7f5020a45f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5648ea4817f0 con 0x5648ea400f30 2024-03-04T10:10:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea3e70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea3e70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37734/0 (socket says 172.21.15.99:37734) 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 -- 172.21.15.99:0/1562310393 learned_addr learned my addr 172.21.15.99:0/1562310393 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501d5a4640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea400f30 0x5648ea477a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501dda5640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea3e7cc0 0x5648ea491b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 -- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea400f30 msgr2=0x5648ea477a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5648ea400f30 0x5648ea477a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 -- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea3e7cc0 msgr2=0x5648ea491b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5648ea3e7cc0 0x5648ea491b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 -- 172.21.15.99:0/1562310393 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648ea4850f0 con 0x5648ea401310 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f501e5a6640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea3e70a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f5018015000 tx=0x7f50180033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f500effd640 1 -- 172.21.15.99:0/1562310393 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501800b030 con 0x5648ea401310 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f500effd640 1 -- 172.21.15.99:0/1562310393 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5018010670 con 0x5648ea401310 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5648ea4853d0 con 0x5648ea401310 2024-03-04T10:10:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f500effd640 1 -- 172.21.15.99:0/1562310393 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5018010a60 con 0x5648ea401310 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.567+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648ea485990 con 0x5648ea401310 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f500effd640 1 -- 172.21.15.99:0/1562310393 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f501802c440 con 0x5648ea401310 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f500effd640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5000069eb0 0x7f500006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f500effd640 1 -- 172.21.15.99:0/1562310393 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f501808b2f0 con 0x5648ea401310 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f501dda5640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5000069eb0 0x7f500006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f5020a45f40 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5648ea477fb0 0x5648ea48be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:1 13.13 13:cd5a64a3:::100.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5648ea48c3c0 con 0x5648ea477fb0 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.569+0000 7f501d5a4640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5648ea477fb0 0x5648ea48be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f501dda5640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5000069eb0 0x7f500006c370 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x5648ea48e520 tx=0x7f50100014f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f501d5a4640 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5648ea477fb0 0x5648ea48be80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f501d5a4640 1 -- 172.21.15.99:0/1562310393 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(1 100.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x7f5014052810 con 0x5648ea477fb0 2024-03-04T10:10:24.574 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/100.00000000 mtime 2024-03-04T10:10:07.000000+0000, size 0 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5648ea477fb0 msgr2=0x5648ea48be80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x5648ea477fb0 0x5648ea48be80 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5000069eb0 msgr2=0x7f500006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5000069eb0 0x7f500006c370 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x5648ea48e520 tx=0x7f50100014f0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 msgr2=0x5648ea3e70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 --2- 172.21.15.99:0/1562310393 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5648ea401310 0x5648ea3e70a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f5018015000 tx=0x7f50180033a0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f501e5a6640 1 -- 172.21.15.99:0/1562310393 reap_dead start 2024-03-04T10:10:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 shutdown_connections 2024-03-04T10:10:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 >> 172.21.15.99:0/1562310393 conn(0x5648ea3b8400 msgr2=0x5648ea3ba220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 shutdown_connections 2024-03-04T10:10:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.570+0000 7f5020a45f40 1 -- 172.21.15.99:0/1562310393 wait complete. 2024-03-04T10:10:24.577 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_sessionmap 2024-03-04T10:10:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.656+0000 7f43bc03af40 1 Processor -- start 2024-03-04T10:10:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.656+0000 7f43bc03af40 1 -- start start 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43bc03af40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385ebfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43bc03af40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385ec00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43bc03af40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385e16110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43bc03af40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x556385e356d0 con 0x556385e1afc0 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43bc03af40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x556385d92b30 con 0x556385ec0e20 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43bc03af40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x556385ec1610 con 0x556385dea3a0 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385ec00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385ec00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37744/0 (socket says 172.21.15.99:37744) 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 -- 172.21.15.99:0/1035030004 learned_addr learned my addr 172.21.15.99:0/1035030004 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b9b47640 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385e16110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b9346640 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385ebfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 -- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 msgr2=0x556385ebfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385ebfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 -- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 msgr2=0x556385e16110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385e16110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 -- 172.21.15.99:0/1035030004 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556385ec4260 con 0x556385e1afc0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b9b47640 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385e16110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43b8b45640 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385ec00c0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f43b000b1f0 tx=0x7f43b000b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc97db69392f37a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43ab7fe640 1 -- 172.21.15.99:0/1035030004 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b0016070 con 0x556385e1afc0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43ab7fe640 1 -- 172.21.15.99:0/1035030004 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b000d6d0 con 0x556385e1afc0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.657+0000 7f43ab7fe640 1 -- 172.21.15.99:0/1035030004 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b000dac0 con 0x556385e1afc0 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 -- 172.21.15.99:0/1035030004 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 msgr2=0x556385ec00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385ec00c0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f43b000b1f0 tx=0x7f43b000b6b0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 -- 172.21.15.99:0/1035030004 shutdown_connections 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385e16110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385ec00c0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 --2- 172.21.15.99:0/1035030004 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385ebfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 -- 172.21.15.99:0/1035030004 >> 172.21.15.99:0/1035030004 conn(0x556385deb400 msgr2=0x556385e31c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 -- 172.21.15.99:0/1035030004 shutdown_connections 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 -- 172.21.15.99:0/1035030004 wait complete. 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.658+0000 7f43bc03af40 1 Processor -- start 2024-03-04T10:10:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 -- start start 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385eaec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385eb0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385eb1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x556385ec44d0 con 0x556385e1afc0 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x556385ec0600 con 0x556385ec0e20 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43bc03af40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x556385deb910 con 0x556385dea3a0 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43b8b45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385eb0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43b8b45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385eb0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37752/0 (socket says 172.21.15.99:37752) 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43b8b45640 1 -- 172.21.15.99:0/2882541509 learned_addr learned my addr 172.21.15.99:0/2882541509 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43b9b47640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385eb1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.659+0000 7f43b9346640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385eaec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43b8b45640 1 -- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 msgr2=0x556385eaec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43b8b45640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x556385dea3a0 0x556385eaec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43b8b45640 1 -- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 msgr2=0x556385eb1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43b8b45640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556385ec0e20 0x556385eb1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43b8b45640 1 -- 172.21.15.99:0/2882541509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556385eace80 con 0x556385e1afc0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43b8b45640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385eb0d00 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f43b000b6f0 tx=0x7f43b0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43aa7fc640 1 -- 172.21.15.99:0/2882541509 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b0016070 con 0x556385e1afc0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43aa7fc640 1 -- 172.21.15.99:0/2882541509 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b0010640 con 0x556385e1afc0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556385ebb120 con 0x556385e1afc0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43aa7fc640 1 -- 172.21.15.99:0/2882541509 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b00109f0 con 0x556385e1afc0 2024-03-04T10:10:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.660+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556385ebb6e0 con 0x556385e1afc0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.661+0000 7f43aa7fc640 1 -- 172.21.15.99:0/2882541509 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f43b0010b90 con 0x556385e1afc0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43aa7fc640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c069eb0 0x7f439c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43aa7fc640 1 -- 172.21.15.99:0/2882541509 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f43b008a340 con 0x556385e1afc0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43b9346640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c069eb0 0x7f439c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43bc03af40 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x556385eaf190 0x556385e9d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:1 13.b 13:d0630e4c:::mds0_sessionmap:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x556385ebacb0 con 0x556385eaf190 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43b9b47640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x556385eaf190 0x556385e9d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.662+0000 7f43b9346640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c069eb0 0x7f439c06c370 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x556385e18100 tx=0x7f43ac007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43b9b47640 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x556385eaf190 0x556385e9d360 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43b9b47640 1 -- 172.21.15.99:0/2882541509 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(1 mds0_sessionmap [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 159+0+16 (crc 0 0 0) 0x7f43b405a040 con 0x556385eaf190 2024-03-04T10:10:24.667 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/mds0_sessionmap mtime 2024-03-04T10:10:20.000000+0000, size 0 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x556385eaf190 msgr2=0x556385e9d360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x556385eaf190 0x556385e9d360 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c069eb0 msgr2=0x7f439c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f439c069eb0 0x7f439c06c370 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x556385e18100 tx=0x7f43ac007040 comp rx=0 tx=0).stop 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 msgr2=0x556385eb0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 --2- 172.21.15.99:0/2882541509 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x556385e1afc0 0x556385eb0d00 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f43b000b6f0 tx=0x7f43b0000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43b9b47640 1 -- 172.21.15.99:0/2882541509 reap_dead start 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 shutdown_connections 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 >> 172.21.15.99:0/2882541509 conn(0x556385deb400 msgr2=0x556385ebfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 shutdown_connections 2024-03-04T10:10:24.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.663+0000 7f43bc03af40 1 -- 172.21.15.99:0/2882541509 wait complete. 2024-03-04T10:10:24.670 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 400.00000000 2024-03-04T10:10:24.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.750+0000 7f50f4566f40 1 Processor -- start 2024-03-04T10:10:24.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.750+0000 7f50f4566f40 1 -- start start 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f4566f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b01b33a0 0x5644b01e35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f4566f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f4566f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b0289e20 0x5644b028d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f4566f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b015bb30 con 0x5644b01e3fc0 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f4566f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b015fa90 con 0x5644b01b33a0 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f4566f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644b01fe6d0 con 0x5644b0289e20 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37772/0 (socket says 172.21.15.99:37772) 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 -- 172.21.15.99:0/343148585 learned_addr learned my addr 172.21.15.99:0/343148585 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f2064640 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b01b33a0 0x5644b01e35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f2865640 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b0289e20 0x5644b028d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.756 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 -- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b0289e20 msgr2=0x5644b028d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b0289e20 0x5644b028d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 -- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b01b33a0 msgr2=0x5644b01e35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b01b33a0 0x5644b01e35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 -- 172.21.15.99:0/343148585 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644b0289020 con 0x5644b01e3fc0 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50e9863640 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cc00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f50dc001dd0 tx=0x7f50dc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb59ee342ce78ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f1062640 1 -- 172.21.15.99:0/343148585 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00bde0 con 0x5644b01e3fc0 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f1062640 1 -- 172.21.15.99:0/343148585 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc00f400 con 0x5644b01e3fc0 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.751+0000 7f50f1062640 1 -- 172.21.15.99:0/343148585 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00f830 con 0x5644b01e3fc0 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 -- 172.21.15.99:0/343148585 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 msgr2=0x5644b028cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cc00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f50dc001dd0 tx=0x7f50dc00b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 -- 172.21.15.99:0/343148585 shutdown_connections 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b0289e20 0x5644b028d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cc00 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 --2- 172.21.15.99:0/343148585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b01b33a0 0x5644b01e35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.757 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 -- 172.21.15.99:0/343148585 >> 172.21.15.99:0/343148585 conn(0x5644b01b4400 msgr2=0x5644b01b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 -- 172.21.15.99:0/343148585 shutdown_connections 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 -- 172.21.15.99:0/343148585 wait complete. 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 Processor -- start 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.752+0000 7f50f4566f40 1 -- start start 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f4566f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b01b33a0 0x5644b01f4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f4566f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f4566f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b0289e20 0x5644b02842b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f4566f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b0288dd0 con 0x5644b01e3fc0 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f4566f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644b0281ea0 con 0x5644b0289e20 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f4566f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5644b01b3780 con 0x5644b01b33a0 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37778/0 (socket says 172.21.15.99:37778) 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 -- 172.21.15.99:0/1231885795 learned_addr learned my addr 172.21.15.99:0/1231885795 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f2064640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b01b33a0 0x5644b01f4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f2865640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b0289e20 0x5644b02842b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 -- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b01b33a0 msgr2=0x5644b01f4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5644b01b33a0 0x5644b01f4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 -- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b0289e20 msgr2=0x5644b02842b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b0289e20 0x5644b02842b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50e9863640 1 -- 172.21.15.99:0/1231885795 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644b0280f20 con 0x5644b01e3fc0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.753+0000 7f50f2865640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644b0289e20 0x5644b02842b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.754+0000 7f50e9863640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cdf0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x5644b01dfaf0 tx=0x7f50dc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.754+0000 7f50ebfff640 1 -- 172.21.15.99:0/1231885795 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc01c440 con 0x5644b01e3fc0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.754+0000 7f50ebfff640 1 -- 172.21.15.99:0/1231885795 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc021020 con 0x5644b01e3fc0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.754+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5644b028bd20 con 0x5644b01e3fc0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.754+0000 7f50ebfff640 1 -- 172.21.15.99:0/1231885795 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc01c620 con 0x5644b01e3fc0 2024-03-04T10:10:24.759 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.754+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644b02811c0 con 0x5644b01e3fc0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.755+0000 7f50ebfff640 1 -- 172.21.15.99:0/1231885795 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f50dc017070 con 0x5644b01e3fc0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.755+0000 7f50ebfff640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d4069eb0 0x7f50d406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.755+0000 7f50ebfff640 1 -- 172.21.15.99:0/1231885795 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f50dc08ad00 con 0x5644b01e3fc0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.756+0000 7f50f4566f40 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5644b0284d00 0x5644b0324590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.756+0000 7f50f2064640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d4069eb0 0x7f50d406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.756+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.f 13:f1f69726:::400.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x5644b028d330 con 0x5644b0284d00 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.756+0000 7f50f2865640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5644b0284d00 0x5644b0324590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.756+0000 7f50f2064640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d4069eb0 0x7f50d406c370 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f50e4054000 tx=0x7f50e404e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.756+0000 7f50f2865640 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5644b0284d00 0x5644b0324590 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f2865640 1 -- 172.21.15.99:0/1231885795 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 400.00000000 [stat out=16b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x7f50ec0065b0 con 0x5644b0284d00 2024-03-04T10:10:24.762 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/400.00000000 mtime 2024-03-04T10:10:07.000000+0000, size 22 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5644b0284d00 msgr2=0x5644b0324590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x5644b0284d00 0x5644b0324590 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d4069eb0 msgr2=0x7f50d406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f50d4069eb0 0x7f50d406c370 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f50e4054000 tx=0x7f50e404e000 comp rx=0 tx=0).stop 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 msgr2=0x5644b028cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 --2- 172.21.15.99:0/1231885795 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5644b01e3fc0 0x5644b028cdf0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x5644b01dfaf0 tx=0x7f50dc000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f2865640 1 -- 172.21.15.99:0/1231885795 reap_dead start 2024-03-04T10:10:24.763 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 shutdown_connections 2024-03-04T10:10:24.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 >> 172.21.15.99:0/1231885795 conn(0x5644b01b4400 msgr2=0x5644b01b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 shutdown_connections 2024-03-04T10:10:24.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.757+0000 7f50f4566f40 1 -- 172.21.15.99:0/1231885795 wait complete. 2024-03-04T10:10:24.764 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 100.00000000.inode 2024-03-04T10:10:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.842+0000 7f6677ed3f40 1 Processor -- start 2024-03-04T10:10:24.844 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.842+0000 7f6677ed3f40 1 -- start start 2024-03-04T10:10:24.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f6677ed3f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f1913aaf0 0x555f19136380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f6677ed3f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f19153f30 0x555f1910b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f6677ed3f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1910ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f6677ed3f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f19155d40 con 0x555f19154310 2024-03-04T10:10:24.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f6677ed3f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f190b2b30 con 0x555f1913aaf0 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f6677ed3f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f191e3c40 con 0x555f19153f30 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1910ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1910ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37808/0 (socket says 172.21.15.99:37808) 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 -- 172.21.15.99:0/762269014 learned_addr learned my addr 172.21.15.99:0/762269014 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66759d1640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f1913aaf0 0x555f19136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66751d0640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f19153f30 0x555f1910b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 -- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f19153f30 msgr2=0x555f1910b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f19153f30 0x555f1910b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 -- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f1913aaf0 msgr2=0x555f19136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f1913aaf0 0x555f19136380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66761d2640 1 -- 172.21.15.99:0/762269014 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f191e1d50 con 0x555f19154310 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66751d0640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f19153f30 0x555f1910b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.843+0000 7f66759d1640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f1913aaf0 0x555f19136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f66761d2640 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1910ca50 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f6670002d90 tx=0x7f667000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e363df8dd647c2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6667fff640 1 -- 172.21.15.99:0/762269014 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667000f400 con 0x555f19154310 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6667fff640 1 -- 172.21.15.99:0/762269014 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670014070 con 0x555f19154310 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6667fff640 1 -- 172.21.15.99:0/762269014 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667000f800 con 0x555f19154310 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 -- 172.21.15.99:0/762269014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 msgr2=0x555f1910ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1910ca50 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f6670002d90 tx=0x7f667000b5b0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 -- 172.21.15.99:0/762269014 shutdown_connections 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1910ca50 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f19153f30 0x555f1910b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/762269014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f1913aaf0 0x555f19136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 -- 172.21.15.99:0/762269014 >> 172.21.15.99:0/762269014 conn(0x555f1910b400 msgr2=0x555f1910e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 -- 172.21.15.99:0/762269014 shutdown_connections 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.844+0000 7f6677ed3f40 1 -- 172.21.15.99:0/762269014 wait complete. 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.845+0000 7f6677ed3f40 1 Processor -- start 2024-03-04T10:10:24.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.845+0000 7f6677ed3f40 1 -- start start 2024-03-04T10:10:24.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f1913aaf0 0x555f191cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f19153f30 0x555f191eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1914ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1910bd20 con 0x555f19154310 2024-03-04T10:10:24.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f191e3040 con 0x555f19153f30 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f191d0fa0 con 0x555f1913aaf0 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1914ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1914ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37818/0 (socket says 172.21.15.99:37818) 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 -- 172.21.15.99:0/2934981091 learned_addr learned my addr 172.21.15.99:0/2934981091 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66751d0640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f19153f30 0x555f191eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66759d1640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f1913aaf0 0x555f191cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 -- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f1913aaf0 msgr2=0x555f191cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555f1913aaf0 0x555f191cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 -- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f19153f30 msgr2=0x555f191eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555f19153f30 0x555f191eb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 -- 172.21.15.99:0/2934981091 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f191d1280 con 0x555f19154310 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f66761d2640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1914ea70 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f6670000c00 tx=0x7f6670001fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6666ffd640 1 -- 172.21.15.99:0/2934981091 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670002320 con 0x555f19154310 2024-03-04T10:10:24.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6666ffd640 1 -- 172.21.15.99:0/2934981091 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66700196a0 con 0x555f19154310 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555f191d1560 con 0x555f19154310 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6666ffd640 1 -- 172.21.15.99:0/2934981091 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670019a90 con 0x555f19154310 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.846+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f191d1b20 con 0x555f19154310 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f6666ffd640 1 -- 172.21.15.99:0/2934981091 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f667000e070 con 0x555f19154310 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f6666ffd640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6658069eb0 0x7f665806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f6666ffd640 1 -- 172.21.15.99:0/2934981091 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f667008aa40 con 0x555f19154310 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f66759d1640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6658069eb0 0x7f665806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555f191d2550 0x555f191d2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.1f 13:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x555f191d32c0 con 0x555f191d2550 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.848+0000 7f66751d0640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555f191d2550 0x555f191d2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f66759d1640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6658069eb0 0x7f665806c370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x555f191d40e0 tx=0x7f6668007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f66751d0640 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555f191d2550 0x555f191d2d80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f66751d0640 1 -- 172.21.15.99:0/2934981091 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x7f666c008740 con 0x555f191d2550 2024-03-04T10:10:24.854 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/100.00000000.inode mtime 2024-03-04T10:10:07.000000+0000, size 582 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555f191d2550 msgr2=0x555f191d2d80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555f191d2550 0x555f191d2d80 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6658069eb0 msgr2=0x7f665806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6658069eb0 0x7f665806c370 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x555f191d40e0 tx=0x7f6668007040 comp rx=0 tx=0).stop 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 msgr2=0x555f1914ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 --2- 172.21.15.99:0/2934981091 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555f19154310 0x555f1914ea70 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f6670000c00 tx=0x7f6670001fe0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f66761d2640 1 -- 172.21.15.99:0/2934981091 reap_dead start 2024-03-04T10:10:24.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 shutdown_connections 2024-03-04T10:10:24.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 >> 172.21.15.99:0/2934981091 conn(0x555f1910b400 msgr2=0x555f191d69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 shutdown_connections 2024-03-04T10:10:24.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.849+0000 7f6677ed3f40 1 -- 172.21.15.99:0/2934981091 wait complete. 2024-03-04T10:10:24.856 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 1.00000000 2024-03-04T10:10:24.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.935+0000 7f75c8e36f40 1 Processor -- start 2024-03-04T10:10:24.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.935+0000 7f75c8e36f40 1 -- start start 2024-03-04T10:10:24.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c8e36f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75b40022d0 0x555a419d7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c8e36f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 0x555a419da6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c8e36f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a419be580 0x555a419dabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c8e36f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a418a7b30 con 0x555a419de850 2024-03-04T10:10:24.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c8e36f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a418aba90 con 0x555a419be580 2024-03-04T10:10:24.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c8e36f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a419d8e60 con 0x7f75b40022d0 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 0x555a419da6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 0x555a419da6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37832/0 (socket says 172.21.15.99:37832) 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 -- 172.21.15.99:0/1008815756 learned_addr learned my addr 172.21.15.99:0/1008815756 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c69ce640 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a419be580 0x555a419dabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c61cd640 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75b40022d0 0x555a419d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 -- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75b40022d0 msgr2=0x555a419d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75b40022d0 0x555a419d7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 -- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a419be580 msgr2=0x555a419dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a419be580 0x555a419dabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 -- 172.21.15.99:0/1008815756 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a419b9070 con 0x555a419de850 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75c61cd640 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75b40022d0 0x555a419d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.936+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 0x555a419da6b0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009810 tx=0x7f75b0005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e828582fc1046e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c51cb640 1 -- 172.21.15.99:0/1008815756 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b0007800 con 0x555a419de850 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c51cb640 1 -- 172.21.15.99:0/1008815756 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b0019070 con 0x555a419de850 2024-03-04T10:10:24.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c51cb640 1 -- 172.21.15.99:0/1008815756 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b0007ba0 con 0x555a419de850 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1008815756 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 msgr2=0x555a419da6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 0x555a419da6b0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f75b0009810 tx=0x7f75b0005a00 comp rx=0 tx=0).stop 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1008815756 shutdown_connections 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555a419be580 0x555a419dabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419de850 0x555a419da6b0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1008815756 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f75b40022d0 0x555a419d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1008815756 >> 172.21.15.99:0/1008815756 conn(0x555a41900400 msgr2=0x555a4192fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1008815756 shutdown_connections 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.937+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1008815756 wait complete. 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 Processor -- start 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 -- start start 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419be580 0x555a419dc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555a419de850 0x555a419d6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75b40022d0 0x555a41902ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a419d79d0 con 0x555a419be580 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a419bc0e0 con 0x7f75b40022d0 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c8e36f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a419d7ed0 con 0x555a419de850 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c61cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419be580 0x555a419dc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c61cd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419be580 0x555a419dc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37844/0 (socket says 172.21.15.99:37844) 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c61cd640 1 -- 172.21.15.99:0/1127293276 learned_addr learned my addr 172.21.15.99:0/1127293276 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.938+0000 7f75c69ce640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75b40022d0 0x555a41902ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555a419de850 0x555a419d6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c61cd640 1 -- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555a419de850 msgr2=0x555a419d6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c61cd640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555a419de850 0x555a419d6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c61cd640 1 -- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75b40022d0 msgr2=0x555a41902ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c61cd640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f75b40022d0 0x555a41902ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c61cd640 1 -- 172.21.15.99:0/1127293276 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a419d6530 con 0x555a419be580 2024-03-04T10:10:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555a419de850 0x555a419d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c61cd640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419be580 0x555a419dc090 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000c00 tx=0x7f75b4004120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75bffff640 1 -- 172.21.15.99:0/1127293276 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8057040 con 0x555a419be580 2024-03-04T10:10:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75bffff640 1 -- 172.21.15.99:0/1127293276 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b805d3f0 con 0x555a419be580 2024-03-04T10:10:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555a419de540 con 0x555a419be580 2024-03-04T10:10:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75bffff640 1 -- 172.21.15.99:0/1127293276 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8001ce0 con 0x555a419be580 2024-03-04T10:10:24.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.939+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a419dc8e0 con 0x555a419be580 2024-03-04T10:10:24.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.940+0000 7f75bffff640 1 -- 172.21.15.99:0/1127293276 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f75b805d7c0 con 0x555a419be580 2024-03-04T10:10:24.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75bffff640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75a8069eb0 0x7f75a806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75bffff640 1 -- 172.21.15.99:0/1127293276 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f75b80cac60 con 0x555a419be580 2024-03-04T10:10:24.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75a8069eb0 0x7f75a806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555a419b5620 0x555a419bea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:1 13.1f 13:ff5b34d6:::1.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v9 -- 0x555a419bef80 con 0x555a419b5620 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75c69ce640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555a419b5620 0x555a419bea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.941+0000 7f75bd9cc640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75a8069eb0 0x7f75a806c370 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f75b0000c00 tx=0x7f75b0013000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c69ce640 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555a419b5620 0x555a419bea40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c69ce640 1 -- 172.21.15.99:0/1127293276 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(1 1.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) v8 ==== 154+0+16 (crc 0 0 0) 0x7f75c0003630 con 0x555a419b5620 2024-03-04T10:10:24.947 INFO:teuthology.orchestra.run.smithi099.stdout:cephfs_metadata/1.00000000 mtime 2024-03-04T10:10:20.000000+0000, size 0 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555a419b5620 msgr2=0x555a419bea40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555a419b5620 0x555a419bea40 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75a8069eb0 msgr2=0x7f75a806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f75a8069eb0 0x7f75a806c370 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f75b0000c00 tx=0x7f75b0013000 comp rx=0 tx=0).stop 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419be580 msgr2=0x555a419dc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 --2- 172.21.15.99:0/1127293276 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555a419be580 0x555a419dc090 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000c00 tx=0x7f75b4004120 comp rx=0 tx=0).stop 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c69ce640 1 -- 172.21.15.99:0/1127293276 reap_dead start 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 shutdown_connections 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 >> 172.21.15.99:0/1127293276 conn(0x555a41900400 msgr2=0x555a41900810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:24.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 shutdown_connections 2024-03-04T10:10:24.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:24.942+0000 7f75c8e36f40 1 -- 172.21.15.99:0/1127293276 wait complete. 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '10000000000.00000000' will be 'no damage' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '600.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000.inode' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '200.00000001' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_openfiles.0' will be 'no damage' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '500.00000000' will be 'no damage' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds_snaptable' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_inotable' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.950 INFO:tasks.cephfs.test_damage:Expectation on rm 'mds0_sessionmap' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on rm '400.00000000' will be 'no damage' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on rm '100.00000000.inode' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on rm '1.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '10000000000.00000000' will be 'no damage' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '600.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_openfiles.0' will be 'no damage' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '100.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on corrupt header 'mds0_sessionmap' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Expectation on corrupt header '1.00000000' will be 'damaged_on_start' 2024-03-04T10:10:24.951 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 10000000000.00000000' 2024-03-04T10:10:24.951 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:24.951 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:25.005 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:10:25.005 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:10:25.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:25.005 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:10:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:10:25.060 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-04T10:10:25.060 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:10:25.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.199+0000 7f64ea052640 1 Processor -- start 2024-03-04T10:10:25.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.199+0000 7f64ea052640 1 -- start start 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64ea052640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f64e40b0700 0x7f64e40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64ea052640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64e40b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64ea052640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e40ccad0 con 0x7f64e415b960 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64ea052640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e40b5190 con 0x7f64e40b0700 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64ea052640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40cf860 con 0x7f64e40b47c0 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e37fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f64e415b960 0x7f64e40d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:43304/0 (socket says 172.21.15.99:43304) 2024-03-04T10:10:25.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e37fe640 1 -- 172.21.15.99:0/1546571508 learned_addr learned my addr 172.21.15.99:0/1546571508 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540054087 0 0) 0x7f64e40ccad0 con 0x7f64e415b960 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e2ffd640 1 --2- 172.21.15.99:0/1546571508 >> v2:172.21.15.162:3300/0 conn(0x7f64e40b0700 0x7f64e40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e3fff640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64e40b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64cc003520 con 0x7f64e415b960 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.200+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233137385 0 0) 0x7f64cc003520 con 0x7f64e415b960 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 msgr2=0x7f64e40b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64e40b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 >> v2:172.21.15.162:3300/0 conn(0x7f64e40b0700 msgr2=0x7f64e40b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 --2- 172.21.15.99:0/1546571508 >> v2:172.21.15.162:3300/0 conn(0x7f64e40b0700 0x7f64e40b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e40cfab0 con 0x7f64e415b960 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f64d4001670 con 0x7f64e415b960 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 >> v1:172.21.15.99:6789/0 conn(0x7f64e415b960 legacy=0x7f64e40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64cc004d40 0x7f64cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40b5190 con 0x7f64cc004d40 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40cf860 con 0x7f64cc0039b0 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc00b9f0 con 0x7f64e40b47c0 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e3fff640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64cc004d40 0x7f64cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e37fe640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e2ffd640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e3fff640 1 -- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 msgr2=0x7f64cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e3fff640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e3fff640 1 -- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64cc0039b0 msgr2=0x7f64cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e3fff640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e3fff640 1 -- 172.21.15.99:0/1546571508 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc003520 con 0x7f64cc004d40 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e37fe640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64cc0039b0 0x7f64cc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.201+0000 7f64e2ffd640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e40b47c0 0x7f64cc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64e3fff640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64cc004d40 0x7f64cc0051c0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009fc0 tx=0x7f64cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b3686a2c4c2983 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc039070 con 0x7f64cc004d40 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc02cc20 con 0x7f64cc004d40 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64e27fc640 1 -- 172.21.15.99:0/1546571508 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc009c10 con 0x7f64cc004d40 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64ea052640 1 -- 172.21.15.99:0/1546571508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64cc004d40 msgr2=0x7f64cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64ea052640 1 --2- 172.21.15.99:0/1546571508 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64cc004d40 0x7f64cc0051c0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009fc0 tx=0x7f64cc005830 comp rx=0 tx=0).stop 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64e3fff640 1 -- 172.21.15.99:0/1546571508 reap_dead start 2024-03-04T10:10:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64ea052640 1 -- 172.21.15.99:0/1546571508 shutdown_connections 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64ea052640 1 -- 172.21.15.99:0/1546571508 >> 172.21.15.99:0/1546571508 conn(0x7f64e406f5f0 msgr2=0x7f64e406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64ea052640 1 -- 172.21.15.99:0/1546571508 shutdown_connections 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.202+0000 7f64ea052640 1 -- 172.21.15.99:0/1546571508 wait complete. 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 Processor -- start 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 -- start start 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e4155fa0 0x7f64e40c4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e40c4c50 0x7f64e41f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 0x7f64e41f7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e415bd40 con 0x7f64e4156380 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e40d0100 con 0x7f64e40c4c50 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.204+0000 7f64ea052640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e415bec0 con 0x7f64e4155fa0 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 0x7f64e41f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 0x7f64e41f7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37854/0 (socket says 172.21.15.99:37854) 2024-03-04T10:10:25.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 -- 172.21.15.99:0/2862987494 learned_addr learned my addr 172.21.15.99:0/2862987494 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e37fe640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e4155fa0 0x7f64e40c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e2ffd640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e40c4c50 0x7f64e41f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 -- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e4155fa0 msgr2=0x7f64e40c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e4155fa0 0x7f64e40c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 -- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e40c4c50 msgr2=0x7f64e41f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e40c4c50 0x7f64e41f6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 -- 172.21.15.99:0/2862987494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e41f36d0 con 0x7f64e4156380 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64e3fff640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 0x7f64e41f7420 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f64dc002230 tx=0x7f64dc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc039030 con 0x7f64e4156380 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc043070 con 0x7f64e4156380 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e41f39b0 con 0x7f64e4156380 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64dc009c10 con 0x7f64e4156380 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.205+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e41f3f70 con 0x7f64e4156380 2024-03-04T10:10:25.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.206+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e4074f60 con 0x7f64e4156380 2024-03-04T10:10:25.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.207+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f64dc005d10 con 0x7f64e4156380 2024-03-04T10:10:25.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.207+0000 7f64cbfff640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64d006b4d0 0x7f64d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:25.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.207+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f64dc0ac580 con 0x7f64e4156380 2024-03-04T10:10:25.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.207+0000 7f64e37fe640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64d006b4d0 0x7f64d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:25.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.208+0000 7f64e37fe640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64d006b4d0 0x7f64d006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f64d4000c00 tx=0x7f64d4007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:25.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.212+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f64dc07ab60 con 0x7f64e4156380 2024-03-04T10:10:25.419 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.418+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f64e415aff0 con 0x7f64e4156380 2024-03-04T10:10:25.929 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.927+0000 7f64cbfff640 1 -- 172.21.15.99:0/2862987494 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v81) v1 ==== 161+0+0 (secure 0 0 0) 0x7f64dc07ec00 con 0x7f64e4156380 2024-03-04T10:10:25.929 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64d006b4d0 msgr2=0x7f64d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64d006b4d0 0x7f64d006d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f64d4000c00 tx=0x7f64d4007040 comp rx=0 tx=0).stop 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 msgr2=0x7f64e41f7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 0x7f64e41f7420 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f64dc002230 tx=0x7f64dc009750 comp rx=0 tx=0).stop 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 shutdown_connections 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64d006b4d0 0x7f64d006d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e4156380 0x7f64e41f7420 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e40c4c50 0x7f64e41f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 --2- 172.21.15.99:0/2862987494 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e4155fa0 0x7f64e40c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 >> 172.21.15.99:0/2862987494 conn(0x7f64e406f5f0 msgr2=0x7f64e40b67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 shutdown_connections 2024-03-04T10:10:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:25.930+0000 7f64ea052640 1 -- 172.21.15.99:0/2862987494 wait complete. 2024-03-04T10:10:25.943 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-03-04T10:10:26.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.043+0000 7f28a6124640 1 Processor -- start 2024-03-04T10:10:26.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.043+0000 7f28a6124640 1 -- start start 2024-03-04T10:10:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f28a6124640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f28a0154a80 0x7f28a015e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f28a6124640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f28a00cfae0 0x7f28a0161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f28a6124640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00cff70 con 0x7f28a00cf650 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f28a6124640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a0071df0 con 0x7f28a00cfae0 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f28a6124640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a00cd010 con 0x7f28a0154a80 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f28a00cfae0 0x7f28a0161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f28a00cfae0 0x7f28a0161dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37866/0 (socket says 172.21.15.99:37866) 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289ffff640 1 -- 172.21.15.99:0/3754596498 learned_addr learned my addr 172.21.15.99:0/3754596498 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289f7fe640 1 --2- 172.21.15.99:0/3754596498 >> v2:172.21.15.162:3300/0 conn(0x7f28a0154a80 0x7f28a015e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289ffff640 1 -- 172.21.15.99:0/3754596498 >> v2:172.21.15.162:3300/0 conn(0x7f28a0154a80 msgr2=0x7f28a015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289ffff640 1 --2- 172.21.15.99:0/3754596498 >> v2:172.21.15.162:3300/0 conn(0x7f28a0154a80 0x7f28a015e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.044+0000 7f289ffff640 1 -- 172.21.15.99:0/3754596498 >> v1:172.21.15.162:6789/0 conn(0x7f28a00cf650 legacy=0x7f28a01616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.045+0000 7f289ffff640 1 -- 172.21.15.99:0/3754596498 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a00cdaa0 con 0x7f28a00cfae0 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.045+0000 7f289f7fe640 1 --2- 172.21.15.99:0/3754596498 >> v2:172.21.15.162:3300/0 conn(0x7f28a0154a80 0x7f28a015e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.045+0000 7f289ffff640 1 --2- 172.21.15.99:0/3754596498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a00cfae0 0x7f28a0161dd0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f28a006e970 tx=0x7f289802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418d32ef33e24fec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.045+0000 7f289e7fc640 1 -- 172.21.15.99:0/3754596498 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289802ca80 con 0x7f28a00cfae0 2024-03-04T10:10:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.045+0000 7f289e7fc640 1 -- 172.21.15.99:0/3754596498 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289802cc20 con 0x7f28a00cfae0 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.045+0000 7f289e7fc640 1 -- 172.21.15.99:0/3754596498 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980343f0 con 0x7f28a00cfae0 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 -- 172.21.15.99:0/3754596498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a00cfae0 msgr2=0x7f28a0161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 --2- 172.21.15.99:0/3754596498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a00cfae0 0x7f28a0161dd0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f28a006e970 tx=0x7f289802bee0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 -- 172.21.15.99:0/3754596498 shutdown_connections 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 --2- 172.21.15.99:0/3754596498 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a00cfae0 0x7f28a0161dd0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 --2- 172.21.15.99:0/3754596498 >> v2:172.21.15.162:3300/0 conn(0x7f28a0154a80 0x7f28a015e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 -- 172.21.15.99:0/3754596498 >> 172.21.15.99:0/3754596498 conn(0x7f28a006f5f0 msgr2=0x7f28a006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 -- 172.21.15.99:0/3754596498 shutdown_connections 2024-03-04T10:10:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.046+0000 7f28a6124640 1 -- 172.21.15.99:0/3754596498 wait complete. 2024-03-04T10:10:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.047+0000 7f28a6124640 1 Processor -- start 2024-03-04T10:10:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.047+0000 7f28a6124640 1 -- start start 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f28a6124640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28a00cf650 0x7f28a00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f28a6124640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28a00cfae0 0x7f28a015a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f28a6124640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 0x7f28a015ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f28a6124640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00ce340 con 0x7f28a0154a80 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f28a6124640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00ce060 con 0x7f28a00cfae0 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f28a6124640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00ce620 con 0x7f28a00cf650 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 0x7f28a015ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 0x7f28a015ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37868/0 (socket says 172.21.15.99:37868) 2024-03-04T10:10:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 -- 172.21.15.99:0/3256378839 learned_addr learned my addr 172.21.15.99:0/3256378839 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289f7fe640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28a00cf650 0x7f28a00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289effd640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28a00cfae0 0x7f28a015a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 -- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28a00cf650 msgr2=0x7f28a00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28a00cf650 0x7f28a00b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 -- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28a00cfae0 msgr2=0x7f28a015a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28a00cfae0 0x7f28a015a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.048+0000 7f289ffff640 1 -- 172.21.15.99:0/3256378839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a00b4ba0 con 0x7f28a0154a80 2024-03-04T10:10:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.049+0000 7f289ffff640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 0x7f28a015ad40 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f2898009f40 tx=0x7f2898004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.049+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898043020 con 0x7f28a0154a80 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.049+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289803e450 con 0x7f28a0154a80 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.049+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a01e9cb0 con 0x7f28a0154a80 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.049+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2898009510 con 0x7f28a0154a80 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.049+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a013eaa0 con 0x7f28a0154a80 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.050+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a0074f60 con 0x7f28a0154a80 2024-03-04T10:10:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.050+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2898046020 con 0x7f28a0154a80 2024-03-04T10:10:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.051+0000 7f2887fff640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f288c06b4d0 0x7f288c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.051+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f28980acd20 con 0x7f28a0154a80 2024-03-04T10:10:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.051+0000 7f289f7fe640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f288c06b4d0 0x7f288c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.051+0000 7f289f7fe640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f288c06b4d0 0x7f288c06d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f289000a950 tx=0x7f2890008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.055+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f289804d030 con 0x7f28a0154a80 2024-03-04T10:10:26.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.291+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) v1 -- 0x7f28a00cd250 con 0x7f28a0154a80 2024-03-04T10:10:26.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.934+0000 7f2887fff640 1 -- 172.21.15.99:0/3256378839 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v82) v1 ==== 107+0+0 (secure 0 0 0) 0x7f289807b950 con 0x7f28a0154a80 2024-03-04T10:10:26.935 INFO:teuthology.orchestra.run.smithi099.stderr:nothing to do: rank is not damaged 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f288c06b4d0 msgr2=0x7f288c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f288c06b4d0 0x7f288c06d990 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f289000a950 tx=0x7f2890008040 comp rx=0 tx=0).stop 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 msgr2=0x7f28a015ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 0x7f28a015ad40 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f2898009f40 tx=0x7f2898004a40 comp rx=0 tx=0).stop 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 shutdown_connections 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f288c06b4d0 0x7f288c06d990 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f28a0154a80 0x7f28a015ad40 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f28a00cfae0 0x7f28a015a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 --2- 172.21.15.99:0/3256378839 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f28a00cf650 0x7f28a00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 >> 172.21.15.99:0/3256378839 conn(0x7f28a006f5f0 msgr2=0x7f28a006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 shutdown_connections 2024-03-04T10:10:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.936+0000 7f28a6124640 1 -- 172.21.15.99:0/3256378839 wait complete. 2024-03-04T10:10:26.950 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-03-04T10:10:26.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 Processor -- start 2024-03-04T10:10:26.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 -- start start 2024-03-04T10:10:26.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552c34c90 0x55f552d03230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 0x55f552d02550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552d0c5f0 0x55f552c7c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f552d0d300 con 0x55f552d065a0 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f552bddb30 con 0x55f552c34c90 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a5638f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f552c7f810 con 0x55f552d0c5f0 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a20bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 0x55f552d02550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.991+0000 7fb8a20bb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 0x55f552d02550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37880/0 (socket says 172.21.15.99:37880) 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1981137239 learned_addr learned my addr 172.21.15.99:0/1981137239 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1981137239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552d0c5f0 msgr2=0x55f552c7c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552d0c5f0 0x55f552c7c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1981137239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552c34c90 msgr2=0x55f552d03230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552c34c90 0x55f552d03230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1981137239 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f552d01610 con 0x55f552d065a0 2024-03-04T10:10:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 0x55f552d02550 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fb898000c00 tx=0x7fb89804c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e26f92b68c60c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a10b9640 1 -- 172.21.15.99:0/1981137239 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898057040 con 0x55f552d065a0 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a10b9640 1 -- 172.21.15.99:0/1981137239 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb898062070 con 0x55f552d065a0 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.992+0000 7fb8a10b9640 1 -- 172.21.15.99:0/1981137239 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898052c50 con 0x55f552d065a0 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1981137239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 msgr2=0x55f552d02550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 0x55f552d02550 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fb898000c00 tx=0x7fb89804c000 comp rx=0 tx=0).stop 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1981137239 shutdown_connections 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552d0c5f0 0x55f552c7c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d065a0 0x55f552d02550 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1981137239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552c34c90 0x55f552d03230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1981137239 >> 172.21.15.99:0/1981137239 conn(0x55f552c36400 msgr2=0x55f552d05420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1981137239 shutdown_connections 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1981137239 wait complete. 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 Processor -- start 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.993+0000 7fb8a5638f40 1 -- start start 2024-03-04T10:10:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a5638f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552c34c90 0x55f552c7c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a5638f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552d065a0 0x55f552cf4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a5638f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 0x55f552d0a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a5638f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f552d05b30 con 0x55f552d0c5f0 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a5638f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f552d05d80 con 0x55f552d065a0 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a5638f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f552d0d970 con 0x55f552c34c90 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 0x55f552d0a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 0x55f552d0a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37886/0 (socket says 172.21.15.99:37886) 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 learned_addr learned my addr 172.21.15.99:0/1021160700 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552d065a0 0x55f552cf4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552c34c90 0x55f552c7c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552c34c90 msgr2=0x55f552c7c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55f552c34c90 0x55f552c7c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552d065a0 msgr2=0x55f552cf4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f552d065a0 0x55f552cf4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.994+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f552d0abf0 con 0x55f552d0c5f0 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.995+0000 7fb8a30bd640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 0x55f552d0a6b0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009da0 tx=0x7fb89c006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.995+0000 7fb88bfff640 1 -- 172.21.15.99:0/1021160700 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c017070 con 0x55f552d0c5f0 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.995+0000 7fb88bfff640 1 -- 172.21.15.99:0/1021160700 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c005e40 con 0x55f552d0c5f0 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.995+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f552d0aec0 con 0x55f552d0c5f0 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.995+0000 7fb88bfff640 1 -- 172.21.15.99:0/1021160700 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c006e60 con 0x55f552d0c5f0 2024-03-04T10:10:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.995+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f552d0b480 con 0x55f552d0c5f0 2024-03-04T10:10:27.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.996+0000 7fb88bfff640 1 -- 172.21.15.99:0/1021160700 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb89c01a070 con 0x55f552d0c5f0 2024-03-04T10:10:27.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb88bfff640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb890069eb0 0x7fb89006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb88bfff640 1 -- 172.21.15.99:0/1021160700 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb89c088020 con 0x55f552d0c5f0 2024-03-04T10:10:27.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb890069eb0 0x7fb89006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.001 INFO:teuthology.orchestra.run.smithi099.stdout:Importing pool 2024-03-04T10:10:27.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55f552c64c60 0x55f552c65040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552c65580 con 0x55f552c64c60 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55f552c64c60 0x55f552c65040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55f552c64c60 0x55f552c65040 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.997+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb890069eb0 0x7fb89006c370 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb894006fd0 tx=0x7fb894006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.998+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 601.00000000 [create] v96'11 uv4 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.998+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:2 13.10 13:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552c65950 con 0x55f552c64c60 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.999+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== osd_op_reply(2 601.00000000 [delete] v96'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:26.999+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:3 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3560 con 0x55f552c64c60 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.000+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 3 ==== osd_op_reply(3 601.00000000 [create] v96'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.003 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-03-04T10:10:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.000+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:4 13.10 13:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db4910 con 0x55f552c64c60 2024-03-04T10:10:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.001+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v96'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.001+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:5 13.10 13:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db4cb0 con 0x55f552c64c60 2024-03-04T10:10:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.002+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v96'15 uv15 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.002+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:6 13.10 13:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3900 con 0x55f552c64c60 2024-03-04T10:10:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.005+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v96'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.005+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:7 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.007+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 7 ==== osd_op_reply(7 602.00000000 [create] v96'17 uv6 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.007+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:8 13.10 13:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.010+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 8 ==== osd_op_reply(8 602.00000000 [delete] v96'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.010+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:9 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.014 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-03-04T10:10:27.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.013+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 9 ==== osd_op_reply(9 602.00000000 [create] v96'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.013+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:10 13.10 13:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.015+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v96'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.015+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:11 13.10 13:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.018+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v96'21 uv21 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.019 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.018+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:12 13.10 13:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.021+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v96'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.021+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:13 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.024 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.023+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 13 ==== osd_op_reply(13 10000000000.00000000 [create] v96'23 uv10 ondisk = -17 ((17) File exists)) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.025 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.023+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:14 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.026+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v96'24 uv24 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.026+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:15 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.029+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 15 ==== osd_op_reply(15 10000000000.00000000 [create] v96'25 uv25 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.030 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-03-04T10:10:27.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.029+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:16 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.032+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v96'26 uv26 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.032+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:17 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.034+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v96'27 uv27 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.034+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:18 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3ca0 con 0x55f552c64c60 2024-03-04T10:10:27.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.037+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v96'28 uv28 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.037+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:19 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db4060 con 0x55f552c64c60 2024-03-04T10:10:27.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.040+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v96'29 uv29 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.040+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:20 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db4060 con 0x55f552c64c60 2024-03-04T10:10:27.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.043+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 20 ==== osd_op_reply(20 600.00000000 [create] v96'30 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.043+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:21 13.10 13:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db44d0 con 0x55f552c64c60 2024-03-04T10:10:27.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.045+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 21 ==== osd_op_reply(21 600.00000000 [delete] v96'31 uv31 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.045+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:22 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.049 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-03-04T10:10:27.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.048+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 22 ==== osd_op_reply(22 600.00000000 [create] v96'32 uv32 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.048+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:23 13.10 13:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.049+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v96'33 uv33 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.050 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.049+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:24 13.10 13:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.050+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v96'34 uv34 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.051 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.050+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:25 13.10 13:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.051+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v96'35 uv35 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.051+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:26 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.051+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 26 ==== osd_op_reply(26 603.00000000 [create] v96'36 uv8 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.052+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:27 13.10 13:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.052+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 27 ==== osd_op_reply(27 603.00000000 [delete] v96'37 uv37 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.052+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:28 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.053+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 28 ==== osd_op_reply(28 603.00000000 [create] v96'38 uv38 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.055 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-03-04T10:10:27.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.053+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:29 13.10 13:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.054+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v96'39 uv39 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.054+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:30 13.10 13:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.055+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v96'40 uv40 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.055+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:31 13.10 13:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db3f50 con 0x55f552c64c60 2024-03-04T10:10:27.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.056+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.1 v2:172.21.15.99:6824/3369025248 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v96'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552c64c60 2024-03-04T10:10:27.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.056+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55f552db3f50 0x55f552db7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.056+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:32 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db75a0 con 0x55f552db3f50 2024-03-04T10:10:27.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.056+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55f552db3f50 0x55f552db7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.056+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55f552db3f50 0x55f552db7060 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.058+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(32 1.00000000.inode [create] v96'21 uv16 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.058+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:33 13.14 13:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db7940 con 0x55f552db3f50 2024-03-04T10:10:27.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.059+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== osd_op_reply(33 1.00000000.inode [delete] v96'22 uv22 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.059+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:34 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db7ce0 con 0x55f552db3f50 2024-03-04T10:10:27.063 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-03-04T10:10:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.062+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 3 ==== osd_op_reply(34 1.00000000.inode [create] v96'23 uv23 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.062+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:35 13.14 13:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db8530 con 0x55f552db3f50 2024-03-04T10:10:27.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.064+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 4 ==== osd_op_reply(35 1.00000000.inode [write 0~582] v96'24 uv24 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.065 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.064+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:36 13.14 13:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db88d0 con 0x55f552db3f50 2024-03-04T10:10:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.066+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 5 ==== osd_op_reply(36 1.00000000.inode [omap-set-header] v96'25 uv25 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.066+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:37 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db8cc0 con 0x55f552db3f50 2024-03-04T10:10:27.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.069+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 6 ==== osd_op_reply(37 200.00000000 [create] v96'26 uv20 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.069+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:38 13.14 13:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db90b0 con 0x55f552db3f50 2024-03-04T10:10:27.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.070+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 7 ==== osd_op_reply(38 200.00000000 [delete] v96'27 uv27 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.070+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:39 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db94a0 con 0x55f552db3f50 2024-03-04T10:10:27.074 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-03-04T10:10:27.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.073+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 8 ==== osd_op_reply(39 200.00000000 [create] v96'28 uv28 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.073+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:40 13.14 13:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db94a0 con 0x55f552db3f50 2024-03-04T10:10:27.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.076+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 9 ==== osd_op_reply(40 200.00000000 [write 0~90] v96'29 uv29 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.077 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.076+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:41 13.14 13:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.077+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 10 ==== osd_op_reply(41 200.00000000 [omap-set-header] v96'30 uv30 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.077+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:42 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.080+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 11 ==== osd_op_reply(42 200.00000001 [create] v96'31 uv17 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.080+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:43 13.14 13:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.082+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 12 ==== osd_op_reply(43 200.00000001 [delete] v96'32 uv32 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.082+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:44 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.085 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-03-04T10:10:27.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.083+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 13 ==== osd_op_reply(44 200.00000001 [create] v96'33 uv33 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.084+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:45 13.14 13:2e2fa760:::200.00000001:head [write 0~18075 in=18075b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.086+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 14 ==== osd_op_reply(45 200.00000001 [write 0~18075] v96'34 uv34 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.086+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:46 13.14 13:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.090 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.089+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 15 ==== osd_op_reply(46 200.00000001 [omap-set-header] v96'35 uv35 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.089+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:47 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.091+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 16 ==== osd_op_reply(47 606.00000000 [create] v96'36 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.091+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:48 13.14 13:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.094+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 17 ==== osd_op_reply(48 606.00000000 [delete] v96'37 uv37 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.094+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:49 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.098 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-03-04T10:10:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.096+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 18 ==== osd_op_reply(49 606.00000000 [create] v96'38 uv38 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.096+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:50 13.14 13:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dba460 con 0x55f552db3f50 2024-03-04T10:10:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.098+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 19 ==== osd_op_reply(50 606.00000000 [setxattr (30)] v96'39 uv39 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.098+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:51 13.14 13:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dba460 con 0x55f552db3f50 2024-03-04T10:10:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.099+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 20 ==== osd_op_reply(51 606.00000000 [setxattr (62)] v96'40 uv40 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.099+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:52 13.14 13:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552db9450 con 0x55f552db3f50 2024-03-04T10:10:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.100+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.0 v2:172.21.15.99:6800/224245114 21 ==== osd_op_reply(52 606.00000000 [omap-set-header] v96'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552db3f50 2024-03-04T10:10:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.100+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55f552db9ad0 0x55f552db9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.100+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:53 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbc870 con 0x55f552db9ad0 2024-03-04T10:10:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.100+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55f552db9ad0 0x55f552db9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.100+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55f552db9ad0 0x55f552db9ed0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.102+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(53 607.00000000 [create] v96'3 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.102+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:54 13.1c 13:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbc870 con 0x55f552db9ad0 2024-03-04T10:10:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.103+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== osd_op_reply(54 607.00000000 [delete] v96'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.103+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:55 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbc870 con 0x55f552db9ad0 2024-03-04T10:10:27.105 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-03-04T10:10:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.104+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 3 ==== osd_op_reply(55 607.00000000 [create] v96'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.104+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:56 13.1c 13:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbdc70 con 0x55f552db9ad0 2024-03-04T10:10:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.105+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 4 ==== osd_op_reply(56 607.00000000 [setxattr (30)] v96'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.105+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:57 13.1c 13:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbdc70 con 0x55f552db9ad0 2024-03-04T10:10:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.106+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 5 ==== osd_op_reply(57 607.00000000 [setxattr (62)] v96'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.106+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:58 13.1c 13:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbcc60 con 0x55f552db9ad0 2024-03-04T10:10:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.107+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 6 ==== osd_op_reply(58 607.00000000 [omap-set-header] v96'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.107+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f552dbd170 0x55f552dbf560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.107+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:59 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dbfbc0 con 0x55f552dbd170 2024-03-04T10:10:27.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.107+0000 7fb8a30bd640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f552dbd170 0x55f552dbf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.108+0000 7fb8a30bd640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f552dbd170 0x55f552dbf560 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.109+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(59 mds0_openfiles.0 [create] v96'4 uv3 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.109+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:60 13.6 13:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc00b0 con 0x55f552dbd170 2024-03-04T10:10:27.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.110+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== osd_op_reply(60 mds0_openfiles.0 [delete] v96'5 uv5 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.110+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:61 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc05b0 con 0x55f552dbd170 2024-03-04T10:10:27.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.112+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 3 ==== osd_op_reply(61 mds0_openfiles.0 [create] v96'6 uv6 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.113 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-03-04T10:10:27.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.112+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:62 13.6 13:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc0e70 con 0x55f552dbd170 2024-03-04T10:10:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.113+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 4 ==== osd_op_reply(62 mds0_openfiles.0 [omap-set-header] v96'7 uv7 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.113+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55f552dc1720 0x55f552dc3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.113+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:63 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4050 con 0x55f552dc1720 2024-03-04T10:10:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.113+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55f552dc1720 0x55f552dc3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.114+0000 7fb8a28bc640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55f552dc1720 0x55f552dc3b10 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.115+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(63 608.00000000 [create] v96'3 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.115+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:64 13.e 13:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4110 con 0x55f552dc1720 2024-03-04T10:10:27.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.116+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== osd_op_reply(64 608.00000000 [delete] v96'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.116+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:65 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4530 con 0x55f552dc1720 2024-03-04T10:10:27.118 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-03-04T10:10:27.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.117+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 3 ==== osd_op_reply(65 608.00000000 [create] v96'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.117+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:66 13.e 13:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5540 con 0x55f552dc1720 2024-03-04T10:10:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.119+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 4 ==== osd_op_reply(66 608.00000000 [setxattr (30)] v96'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.119+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:67 13.e 13:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5540 con 0x55f552dc1720 2024-03-04T10:10:27.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.120+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 5 ==== osd_op_reply(67 608.00000000 [setxattr (62)] v96'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.121+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:68 13.e 13:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4530 con 0x55f552dc1720 2024-03-04T10:10:27.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.122+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 6 ==== osd_op_reply(68 608.00000000 [omap-set-header] v96'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.122+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:69 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4530 con 0x55f552dbd170 2024-03-04T10:10:27.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.123+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 5 ==== osd_op_reply(69 604.00000000 [create] v96'4 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.123+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:70 13.15 13:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc48d0 con 0x55f552dbd170 2024-03-04T10:10:27.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.124+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 6 ==== osd_op_reply(70 604.00000000 [delete] v96'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.124+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:71 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.125+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 7 ==== osd_op_reply(71 604.00000000 [create] v96'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.127 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-03-04T10:10:27.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.125+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:72 13.15 13:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5c80 con 0x55f552dbd170 2024-03-04T10:10:27.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.126+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 8 ==== osd_op_reply(72 604.00000000 [setxattr (30)] v96'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.126+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:73 13.15 13:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5c80 con 0x55f552dbd170 2024-03-04T10:10:27.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.127+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 9 ==== osd_op_reply(73 604.00000000 [setxattr (62)] v96'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.127+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:74 13.15 13:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.129+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 10 ==== osd_op_reply(74 604.00000000 [omap-set-header] v96'9 uv9 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.129+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:75 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.130+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 11 ==== osd_op_reply(75 500.00000000 [create] v96'10 uv3 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.130+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:76 13.15 13:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.131+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 12 ==== osd_op_reply(76 500.00000000 [delete] v96'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.131+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:77 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.132+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 13 ==== osd_op_reply(77 500.00000000 [create] v96'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.133 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-03-04T10:10:27.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.132+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:78 13.15 13:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.133+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 14 ==== osd_op_reply(78 500.00000000 [write 0~90] v96'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.133+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:79 13.15 13:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.134+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 15 ==== osd_op_reply(79 500.00000000 [omap-set-header] v96'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.134+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:80 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.136+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 16 ==== osd_op_reply(80 mds_snaptable [create] v96'2 uv1 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.136+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:81 13.d 13:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.137+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 17 ==== osd_op_reply(81 mds_snaptable [delete] v96'3 uv3 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.137+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:82 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.138+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 18 ==== osd_op_reply(82 mds_snaptable [create] v96'4 uv4 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.139 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-03-04T10:10:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.138+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:83 13.d 13:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.139+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 19 ==== osd_op_reply(83 mds_snaptable [write 0~70] v96'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.139+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:84 13.d 13:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dbd170 2024-03-04T10:10:27.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.141+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 <== osd.4 v2:172.21.15.162:6800/3079837853 20 ==== osd_op_reply(84 mds_snaptable [omap-set-header] v96'6 uv6 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb89c019030 con 0x55f552dbd170 2024-03-04T10:10:27.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.141+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:85 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.142+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 7 ==== osd_op_reply(85 605.00000000 [create] v96'3 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.142+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:86 13.1d 13:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.144+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 8 ==== osd_op_reply(86 605.00000000 [delete] v96'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.144+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:87 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.147+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 9 ==== osd_op_reply(87 605.00000000 [create] v96'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.148 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-03-04T10:10:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.147+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:88 13.1d 13:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.150+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 10 ==== osd_op_reply(88 605.00000000 [setxattr (30)] v96'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.150+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:89 13.1d 13:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.155+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 11 ==== osd_op_reply(89 605.00000000 [setxattr (62)] v96'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.155+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:90 13.1d 13:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.156+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 12 ==== osd_op_reply(90 605.00000000 [omap-set-header] v96'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.156+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:91 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.157+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 13 ==== osd_op_reply(91 mds0_inotable [create] v96'4 uv3 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.157+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:92 13.13 13:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.162+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 14 ==== osd_op_reply(92 mds0_inotable [delete] v96'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.162+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:93 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc4c70 con 0x55f552dc1720 2024-03-04T10:10:27.164 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-03-04T10:10:27.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.163+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 15 ==== osd_op_reply(93 mds0_inotable [create] v96'6 uv6 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.163+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:94 13.13 13:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5030 con 0x55f552dc1720 2024-03-04T10:10:27.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.166+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 16 ==== osd_op_reply(94 mds0_inotable [write 0~34] v96'7 uv7 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.167+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:95 13.13 13:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5030 con 0x55f552dc1720 2024-03-04T10:10:27.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.168+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 17 ==== osd_op_reply(95 mds0_inotable [omap-set-header] v96'8 uv8 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.168+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:96 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5030 con 0x55f552dc1720 2024-03-04T10:10:27.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.171+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 18 ==== osd_op_reply(96 100.00000000 [create] v96'9 uv1 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.171+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:97 13.13 13:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5030 con 0x55f552dc1720 2024-03-04T10:10:27.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.173+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 19 ==== osd_op_reply(97 100.00000000 [delete] v96'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.173+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:98 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5030 con 0x55f552dc1720 2024-03-04T10:10:27.177 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-03-04T10:10:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.176+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 20 ==== osd_op_reply(98 100.00000000 [create] v96'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.176+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:99 13.13 13:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc5030 con 0x55f552dc1720 2024-03-04T10:10:27.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.179+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 21 ==== osd_op_reply(99 100.00000000 [omap-set-header] v96'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.179+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:100 13.13 13:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc54e0 con 0x55f552dc1720 2024-03-04T10:10:27.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.182+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 22 ==== osd_op_reply(100 100.00000000 [omap-set-vals] v96'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.182+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55f552dc5ba0 0x55f552dc7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.182+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:101 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc5ba0 2024-03-04T10:10:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.182+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55f552dc5ba0 0x55f552dc7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.183+0000 7fb8a20bb640 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55f552dc5ba0 0x55f552dc7f90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.183+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(101 mds0_sessionmap [create] v96'4 uv3 ondisk = -17 ((17) File exists)) v8 ==== 159+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552dc5ba0 2024-03-04T10:10:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.184+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:102 13.b 13:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc5ba0 2024-03-04T10:10:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.188+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== osd_op_reply(102 mds0_sessionmap [delete] v96'5 uv5 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552dc5ba0 2024-03-04T10:10:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.188+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:103 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc5ba0 2024-03-04T10:10:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.189+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.5 v2:172.21.15.162:6808/1764948489 3 ==== osd_op_reply(103 mds0_sessionmap [create] v96'6 uv6 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552dc5ba0 2024-03-04T10:10:27.191 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-03-04T10:10:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.189+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:104 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc5ba0 2024-03-04T10:10:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.192+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.5 v2:172.21.15.162:6808/1764948489 4 ==== osd_op_reply(104 mds0_sessionmap [omap-set-header] v96'7 uv7 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552dc5ba0 2024-03-04T10:10:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.192+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:105 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc1720 2024-03-04T10:10:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.193+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 23 ==== osd_op_reply(105 609.00000000 [create] v96'3 uv2 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.193+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:106 13.7 13:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc1720 2024-03-04T10:10:27.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.194+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 24 ==== osd_op_reply(106 609.00000000 [delete] v96'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.194+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:107 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc1720 2024-03-04T10:10:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.196+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 25 ==== osd_op_reply(107 609.00000000 [create] v96'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.198 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-03-04T10:10:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.196+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:108 13.7 13:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc94e0 con 0x55f552dc1720 2024-03-04T10:10:27.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.198+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 26 ==== osd_op_reply(108 609.00000000 [setxattr (30)] v96'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.198+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:109 13.7 13:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc94e0 con 0x55f552dc1720 2024-03-04T10:10:27.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.201+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 27 ==== osd_op_reply(109 609.00000000 [setxattr (62)] v96'7 uv7 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.201+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:110 13.7 13:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552dc1720 2024-03-04T10:10:27.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.204+0000 7fb8a28bc640 1 -- 172.21.15.99:0/1021160700 <== osd.7 v2:172.21.15.162:6816/2710319333 28 ==== osd_op_reply(110 609.00000000 [omap-set-header] v96'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb8940030b0 con 0x55f552dc1720 2024-03-04T10:10:27.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.204+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:111 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.207+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 7 ==== osd_op_reply(111 400.00000000 [create] v96'2 uv1 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.207+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:112 13.f 13:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.208+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 8 ==== osd_op_reply(112 400.00000000 [delete] v96'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.209+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:113 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.210 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-03-04T10:10:27.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.209+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 9 ==== osd_op_reply(113 400.00000000 [create] v96'4 uv4 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.209+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:114 13.f 13:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.211+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 10 ==== osd_op_reply(114 400.00000000 [write 0~22] v96'5 uv5 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.211+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:115 13.f 13:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.212+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 11 ==== osd_op_reply(115 400.00000000 [omap-set-header] v96'6 uv6 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.212+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:116 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.214+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 12 ==== osd_op_reply(116 100.00000000.inode [create] v96'7 uv3 ondisk = -17 ((17) File exists)) v8 ==== 162+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.214+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:117 13.1f 13:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.218+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 13 ==== osd_op_reply(117 100.00000000.inode [delete] v96'8 uv8 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.218+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:118 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.218+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 14 ==== osd_op_reply(118 100.00000000.inode [create] v96'9 uv9 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.220 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-03-04T10:10:27.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.219+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:119 13.1f 13:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.221+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 15 ==== osd_op_reply(119 100.00000000.inode [write 0~582] v96'10 uv10 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.221+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:120 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.224+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 16 ==== osd_op_reply(120 100.00000000.inode [omap-set-header] v96'11 uv11 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.224+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:121 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.227+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 17 ==== osd_op_reply(121 1.00000000 [create] v96'12 uv6 ondisk = -17 ((17) File exists)) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.227+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:122 13.1f 13:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.229+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 18 ==== osd_op_reply(122 1.00000000 [delete] v96'13 uv13 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.229+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:123 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.232+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 19 ==== osd_op_reply(123 1.00000000 [create] v96'14 uv14 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.233 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-03-04T10:10:27.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.232+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:124 13.1f 13:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc94e0 con 0x55f552db9ad0 2024-03-04T10:10:27.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.235+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 20 ==== osd_op_reply(124 1.00000000 [setxattr (30)] v96'15 uv15 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.235+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:125 13.1f 13:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc94e0 con 0x55f552db9ad0 2024-03-04T10:10:27.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.237+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 21 ==== osd_op_reply(125 1.00000000 [setxattr (30)] v96'16 uv16 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.237+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:126 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc84d0 con 0x55f552db9ad0 2024-03-04T10:10:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.240+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 22 ==== osd_op_reply(126 1.00000000 [omap-set-header] v96'17 uv17 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.240+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:127 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v9 -- 0x55f552dc8980 con 0x55f552db9ad0 2024-03-04T10:10:27.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a20bb640 1 -- 172.21.15.99:0/1021160700 <== osd.6 v2:172.21.15.162:6824/1448511037 23 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v96'18 uv18 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7fb89805a040 con 0x55f552db9ad0 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55f552db3f50 msgr2=0x55f552db7060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55f552db3f50 0x55f552db7060 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55f552c64c60 msgr2=0x55f552c65040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55f552c64c60 0x55f552c65040 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f552dbd170 msgr2=0x55f552dbf560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55f552dbd170 0x55f552dbf560 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55f552dc5ba0 msgr2=0x55f552dc7f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55f552dc5ba0 0x55f552dc7f90 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55f552db9ad0 msgr2=0x55f552db9ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55f552db9ad0 0x55f552db9ed0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55f552dc1720 msgr2=0x55f552dc3b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55f552dc1720 0x55f552dc3b10 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 reap_dead start 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 reap_dead start 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a30bd640 1 -- 172.21.15.99:0/1021160700 reap_dead start 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb890069eb0 msgr2=0x7fb89006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb890069eb0 0x7fb89006c370 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb894006fd0 tx=0x7fb894006a20 comp rx=0 tx=0).stop 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 msgr2=0x55f552d0a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 0x55f552d0a6b0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009da0 tx=0x7fb89c006750 comp rx=0 tx=0).stop 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.242+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 shutdown_connections 2024-03-04T10:10:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.243+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb890069eb0 0x7fb89006c370 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.243+0000 7fb8a5638f40 1 --2- 172.21.15.99:0/1021160700 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55f552d0c5f0 0x55f552d0a6b0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.243+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 >> 172.21.15.99:0/1021160700 conn(0x55f552c36400 msgr2=0x55f552c367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:27.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.243+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 shutdown_connections 2024-03-04T10:10:27.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.243+0000 7fb8a5638f40 1 -- 172.21.15.99:0/1021160700 wait complete. 2024-03-04T10:10:27.248 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-03-04T10:10:27.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.327+0000 7f58d3240f40 1 Processor -- start 2024-03-04T10:10:27.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.327+0000 7f58d3240f40 1 -- start start 2024-03-04T10:10:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d3240f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc265cc0 0x55edbc30d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d3240f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc39f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d3240f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc27f310 0x55edbc39fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d3240f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edbc280d40 con 0x55edbc27ef30 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d3240f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edbc1ddb30 con 0x55edbc27f310 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d3240f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edbc30ec40 con 0x55edbc265cc0 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc39f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc39f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37902/0 (socket says 172.21.15.99:37902) 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 -- 172.21.15.99:0/3838994653 learned_addr learned my addr 172.21.15.99:0/3838994653 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d0e29640 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc27f310 0x55edbc39fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58d0628640 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc265cc0 0x55edbc30d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 -- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc265cc0 msgr2=0x55edbc30d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc265cc0 0x55edbc30d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 -- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc27f310 msgr2=0x55edbc39fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc27f310 0x55edbc39fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 -- 172.21.15.99:0/3838994653 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edbc27d330 con 0x55edbc27ef30 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.328+0000 7f58c7e27640 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc39f8c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f58b800b240 tx=0x7f58b800b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe66f3ba3894450 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58cf626640 1 -- 172.21.15.99:0/3838994653 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b8008070 con 0x55edbc27ef30 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58cf626640 1 -- 172.21.15.99:0/3838994653 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b800d500 con 0x55edbc27ef30 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58cf626640 1 -- 172.21.15.99:0/3838994653 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b800d8f0 con 0x55edbc27ef30 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 -- 172.21.15.99:0/3838994653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 msgr2=0x55edbc39f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc39f8c0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f58b800b240 tx=0x7f58b800b700 comp rx=0 tx=0).stop 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 -- 172.21.15.99:0/3838994653 shutdown_connections 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc27f310 0x55edbc39fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc39f8c0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 --2- 172.21.15.99:0/3838994653 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc265cc0 0x55edbc30d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 -- 172.21.15.99:0/3838994653 >> 172.21.15.99:0/3838994653 conn(0x55edbc236400 msgr2=0x55edbc239ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 -- 172.21.15.99:0/3838994653 shutdown_connections 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.329+0000 7f58d3240f40 1 -- 172.21.15.99:0/3838994653 wait complete. 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 Processor -- start 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 -- start start 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc265cc0 0x55edbc30a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc2fa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc27f310 0x55edbc300ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edbc27d090 con 0x55edbc27ef30 2024-03-04T10:10:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edbc305ef0 con 0x55edbc265cc0 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58d3240f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55edbc302430 con 0x55edbc27f310 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.330+0000 7f58c7e27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc2fa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc2fa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37906/0 (socket says 172.21.15.99:37906) 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 -- 172.21.15.99:0/2901239564 learned_addr learned my addr 172.21.15.99:0/2901239564 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58d0628640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc265cc0 0x55edbc30a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58d0e29640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc27f310 0x55edbc300ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 -- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc27f310 msgr2=0x55edbc300ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55edbc27f310 0x55edbc300ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 -- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc265cc0 msgr2=0x55edbc30a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edbc265cc0 0x55edbc30a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 -- 172.21.15.99:0/2901239564 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edbc30e3b0 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58c7e27640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc2fa9a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f58b800b740 tx=0x7f58b8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58ce624640 1 -- 172.21.15.99:0/2901239564 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b8021030 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58ce624640 1 -- 172.21.15.99:0/2901239564 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b801c650 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55edbc30e0d0 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58ce624640 1 -- 172.21.15.99:0/2901239564 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b801c860 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.331+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55edbc30a020 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.332+0000 7f58ce624640 1 -- 172.21.15.99:0/2901239564 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f58b8024070 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58ce624640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58b0069eb0 0x7f58b006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58ce624640 1 -- 172.21.15.99:0/2901239564 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f58b808a2a0 con 0x55edbc27ef30 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58d3240f40 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55edbc316670 0x55edbc2feeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v9 -- 0x55edbc2ff3f0 con 0x55edbc316670 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58d0628640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58b0069eb0 0x7f58b006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58d0e29640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55edbc316670 0x55edbc2feeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.333+0000 7f58d0e29640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55edbc316670 0x55edbc2feeb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.334+0000 7f58d0628640 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58b0069eb0 0x7f58b006c370 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x55edbc234c60 tx=0x7f58c000c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d0e29640 1 -- 172.21.15.99:0/2901239564 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v96'42 uv42 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f58c80525c0 con 0x55edbc316670 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55edbc316670 msgr2=0x55edbc2feeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55edbc316670 0x55edbc2feeb0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58b0069eb0 msgr2=0x7f58b006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58b0069eb0 0x7f58b006c370 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x55edbc234c60 tx=0x7f58c000c040 comp rx=0 tx=0).stop 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 msgr2=0x55edbc2fa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 --2- 172.21.15.99:0/2901239564 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55edbc27ef30 0x55edbc2fa9a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f58b800b740 tx=0x7f58b8000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d0e29640 1 -- 172.21.15.99:0/2901239564 reap_dead start 2024-03-04T10:10:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 shutdown_connections 2024-03-04T10:10:27.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 >> 172.21.15.99:0/2901239564 conn(0x55edbc236400 msgr2=0x55edbc2367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:27.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 shutdown_connections 2024-03-04T10:10:27.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.335+0000 7f58d3240f40 1 -- 172.21.15.99:0/2901239564 wait complete. 2024-03-04T10:10:27.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:10:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.484+0000 7f8cacec0640 1 Processor -- start 2024-03-04T10:10:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.484+0000 7f8cacec0640 1 -- start start 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8cacec0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8cacec0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8ca815c2e0 0x7f8ca80ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8cacec0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca815ca50 con 0x7f8ca80cecf0 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8cacec0640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca815d2f0 con 0x7f8ca815c2e0 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8cacec0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca815d6d0 con 0x7f8ca80ccbd0 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca5d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8ca815c2e0 0x7f8ca80ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca6d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8ca80cecf0 0x7f8ca80b0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:43326/0 (socket says 172.21.15.99:43326) 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca6d9d640 1 -- 172.21.15.99:0/2602011706 learned_addr learned my addr 172.21.15.99:0/2602011706 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca659c640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca5d9b640 1 -- 172.21.15.99:0/2602011706 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 msgr2=0x7f8ca80b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca5d9b640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca5d9b640 1 -- 172.21.15.99:0/2602011706 >> v1:172.21.15.99:6789/0 conn(0x7f8ca80cecf0 legacy=0x7f8ca80b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca559a640 1 -- 172.21.15.99:0/2602011706 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804437536 0 0) 0x7f8ca815ca50 con 0x7f8ca80cecf0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca5d9b640 1 -- 172.21.15.99:0/2602011706 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca815cd30 con 0x7f8ca815c2e0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.485+0000 7f8ca659c640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.486+0000 7f8ca5d9b640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca80ce7b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0097a0 tx=0x7f8c9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af5aa5c0130afe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.486+0000 7f8ca559a640 1 -- 172.21.15.99:0/2602011706 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c035020 con 0x7f8ca815c2e0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.486+0000 7f8ca559a640 1 -- 172.21.15.99:0/2602011706 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c031830 con 0x7f8ca815c2e0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.486+0000 7f8ca559a640 1 -- 172.21.15.99:0/2602011706 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c031c00 con 0x7f8ca815c2e0 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.486+0000 7f8cacec0640 1 -- 172.21.15.99:0/2602011706 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 msgr2=0x7f8ca80ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.486+0000 7f8cacec0640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca80ce7b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0097a0 tx=0x7f8c9c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.487+0000 7f8cacec0640 1 -- 172.21.15.99:0/2602011706 shutdown_connections 2024-03-04T10:10:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.487+0000 7f8cacec0640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca80ce7b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.487+0000 7f8cacec0640 1 --2- 172.21.15.99:0/2602011706 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.487+0000 7f8cacec0640 1 -- 172.21.15.99:0/2602011706 >> 172.21.15.99:0/2602011706 conn(0x7f8ca8066dc0 msgr2=0x7f8ca80671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.487+0000 7f8cacec0640 1 -- 172.21.15.99:0/2602011706 shutdown_connections 2024-03-04T10:10:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.487+0000 7f8cacec0640 1 -- 172.21.15.99:0/2602011706 wait complete. 2024-03-04T10:10:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.488+0000 7f8cacec0640 1 Processor -- start 2024-03-04T10:10:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.488+0000 7f8cacec0640 1 -- start start 2024-03-04T10:10:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ca80cecf0 0x7f8ca8148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca8149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80ccfb0 con 0x7f8ca815c2e0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca815c760 con 0x7f8ca80cecf0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca80b0c70 con 0x7f8ca80ccbd0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca8149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca8149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37924/0 (socket says 172.21.15.99:37924) 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 -- 172.21.15.99:0/382554496 learned_addr learned my addr 172.21.15.99:0/382554496 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca659c640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca5d9b640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ca80cecf0 0x7f8ca8148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 -- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 msgr2=0x7f8ca80c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 -- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ca80cecf0 msgr2=0x7f8ca8148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ca80cecf0 0x7f8ca8148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 -- 172.21.15.99:0/382554496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca80c6290 con 0x7f8ca815c2e0 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca5d9b640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ca80cecf0 0x7f8ca8148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8ca6d9d640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca8149470 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000c7a0 tx=0x7f8ca000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0018070 con 0x7f8ca815c2e0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca000a640 con 0x7f8ca815c2e0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca8156b40 con 0x7f8ca815c2e0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.489+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca000aab0 con 0x7f8ca815c2e0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.490+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca8157100 con 0x7f8ca815c2e0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.491+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca8074f60 con 0x7f8ca815c2e0 2024-03-04T10:10:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.491+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8ca001b070 con 0x7f8ca815c2e0 2024-03-04T10:10:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.491+0000 7f8c96ffd640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c8806b500 0x7f8c8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.491+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f8ca00899c0 con 0x7f8ca815c2e0 2024-03-04T10:10:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.492+0000 7f8ca659c640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c8806b500 0x7f8c8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.492+0000 7f8ca659c640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c8806b500 0x7f8c8806d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8c98006fd0 tx=0x7f8c98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.496+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8ca008be20 con 0x7f8ca815c2e0 2024-03-04T10:10:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.701+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f8ca80b0770 con 0x7f8ca815c2e0 2024-03-04T10:10:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.939+0000 7f8c96ffd640 1 -- 172.21.15.99:0/382554496 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v83) v1 ==== 162+0+0 (secure 0 0 0) 0x7f8ca0058680 con 0x7f8ca815c2e0 2024-03-04T10:10:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:10:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c8806b500 msgr2=0x7f8c8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c8806b500 0x7f8c8806d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8c98006fd0 tx=0x7f8c98008040 comp rx=0 tx=0).stop 2024-03-04T10:10:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 msgr2=0x7f8ca8149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca8149470 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f8ca000c7a0 tx=0x7f8ca000cc60 comp rx=0 tx=0).stop 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 shutdown_connections 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8c8806b500 0x7f8c8806d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8ca815c2e0 0x7f8ca8149470 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8ca80cecf0 0x7f8ca8148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 --2- 172.21.15.99:0/382554496 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8ca80ccbd0 0x7f8ca80c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 >> 172.21.15.99:0/382554496 conn(0x7f8ca8066dc0 msgr2=0x7f8ca80b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 shutdown_connections 2024-03-04T10:10:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:27.941+0000 7f8cacec0640 1 -- 172.21.15.99:0/382554496 wait complete. 2024-03-04T10:10:28.357 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.463+0000 7f63bc59d640 1 Processor -- start 2024-03-04T10:10:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.463+0000 7f63bc59d640 1 -- start start 2024-03-04T10:10:28.465 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63bc59d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f63b40cfd60 0x7f63b40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63bc59d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f63b406d3b0 0x7f63b40cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63bc59d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40d0140 con 0x7f63b40ce9c0 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63bc59d640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b415cbb0 con 0x7f63b406d3b0 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63bc59d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b415c280 con 0x7f63b40cfd60 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f63b406d3b0 0x7f63b40cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f63b406d3b0 0x7f63b40cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37934/0 (socket says 172.21.15.99:37934) 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 -- 172.21.15.99:0/1363789002 learned_addr learned my addr 172.21.15.99:0/1363789002 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63ba339640 1 --2- 172.21.15.99:0/1363789002 >> v2:172.21.15.162:3300/0 conn(0x7f63b40cfd60 0x7f63b40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 -- 172.21.15.99:0/1363789002 >> v2:172.21.15.162:3300/0 conn(0x7f63b40cfd60 msgr2=0x7f63b40cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 --2- 172.21.15.99:0/1363789002 >> v2:172.21.15.162:3300/0 conn(0x7f63b40cfd60 0x7f63b40cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 -- 172.21.15.99:0/1363789002 >> v1:172.21.15.162:6789/0 conn(0x7f63b40ce9c0 legacy=0x7f63b40cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.464+0000 7f63b9b38640 1 -- 172.21.15.99:0/1363789002 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b415c560 con 0x7f63b406d3b0 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.465+0000 7f63b9b38640 1 --2- 172.21.15.99:0/1363789002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b406d3b0 0x7f63b40cc330 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f63b415cd30 tx=0x7f63a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87b870be9d2e77b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.465+0000 7f63b9337640 1 -- 172.21.15.99:0/1363789002 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4035020 con 0x7f63b406d3b0 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.465+0000 7f63b9337640 1 -- 172.21.15.99:0/1363789002 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a4031870 con 0x7f63b406d3b0 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.465+0000 7f63b9337640 1 -- 172.21.15.99:0/1363789002 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4031c40 con 0x7f63b406d3b0 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.465+0000 7f63bc59d640 1 -- 172.21.15.99:0/1363789002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b406d3b0 msgr2=0x7f63b40cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.465+0000 7f63bc59d640 1 --2- 172.21.15.99:0/1363789002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b406d3b0 0x7f63b40cc330 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f63b415cd30 tx=0x7f63a402b910 comp rx=0 tx=0).stop 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.466+0000 7f63bc59d640 1 -- 172.21.15.99:0/1363789002 shutdown_connections 2024-03-04T10:10:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.466+0000 7f63bc59d640 1 --2- 172.21.15.99:0/1363789002 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b406d3b0 0x7f63b40cc330 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.466+0000 7f63bc59d640 1 --2- 172.21.15.99:0/1363789002 >> v2:172.21.15.162:3300/0 conn(0x7f63b40cfd60 0x7f63b40cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.466+0000 7f63bc59d640 1 -- 172.21.15.99:0/1363789002 >> 172.21.15.99:0/1363789002 conn(0x7f63b406acd0 msgr2=0x7f63b406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.466+0000 7f63bc59d640 1 -- 172.21.15.99:0/1363789002 shutdown_connections 2024-03-04T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.466+0000 7f63bc59d640 1 -- 172.21.15.99:0/1363789002 wait complete. 2024-03-04T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 Processor -- start 2024-03-04T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 -- start start 2024-03-04T10:10:28.471 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63b406d3b0 0x7f63b4144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 0x7f63b4158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b40cfd60 0x7f63b40ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b415f520 con 0x7f63b40ce9c0 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b415bf30 con 0x7f63b40cfd60 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63bc59d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b415f6a0 con 0x7f63b406d3b0 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.467+0000 7f63b9b38640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 0x7f63b4158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 0x7f63b4158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37940/0 (socket says 172.21.15.99:37940) 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 -- 172.21.15.99:0/458990686 learned_addr learned my addr 172.21.15.99:0/458990686 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63bab3a640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b40cfd60 0x7f63b40ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63ba339640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63b406d3b0 0x7f63b4144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 -- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63b406d3b0 msgr2=0x7f63b4144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63b406d3b0 0x7f63b4144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 -- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b40cfd60 msgr2=0x7f63b40ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b40cfd60 0x7f63b40ae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 -- 172.21.15.99:0/458990686 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4158f70 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63ba339640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63b406d3b0 0x7f63b4144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63b9b38640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 0x7f63b4158a30 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f63a402b950 tx=0x7f63a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4040070 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a4030d00 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b4157990 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4035040 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.468+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b4157f50 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.469+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b4074f60 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.469+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f63a4038020 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.470+0000 7f63aaffd640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f639806b500 0x7f639806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.470+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f63a40ac2d0 con 0x7f63b40ce9c0 2024-03-04T10:10:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.470+0000 7f63ba339640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f639806b500 0x7f639806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.471+0000 7f63ba339640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f639806b500 0x7f639806d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f63b0006fd0 tx=0x7f63b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:28.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.474+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f63a407ae80 con 0x7f63b40ce9c0 2024-03-04T10:10:28.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.678+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63b4156360 con 0x7f63b40ce9c0 2024-03-04T10:10:28.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.679+0000 7f63aaffd640 1 -- 172.21.15.99:0/458990686 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f63a407a780 con 0x7f63b40ce9c0 2024-03-04T10:10:28.680 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:28.680 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":84,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:27.941792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"c","rank":0,"incarnation":84,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/932198368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":932198368},{"type":"v1","addr":"172.21.15.99:6835","nonce":932198368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 84 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f639806b500 msgr2=0x7f639806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f639806b500 0x7f639806d9c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f63b0006fd0 tx=0x7f63b0008040 comp rx=0 tx=0).stop 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 msgr2=0x7f63b4158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 0x7f63b4158a30 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f63a402b950 tx=0x7f63a40046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 shutdown_connections 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f639806b500 0x7f639806d9c0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f63b40cfd60 0x7f63b40ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63b40ce9c0 0x7f63b4158a30 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 --2- 172.21.15.99:0/458990686 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f63b406d3b0 0x7f63b4144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 >> 172.21.15.99:0/458990686 conn(0x7f63b406acd0 msgr2=0x7f63b4159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 shutdown_connections 2024-03-04T10:10:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:28.681+0000 7f63bc59d640 1 -- 172.21.15.99:0/458990686 wait complete. 2024-03-04T10:10:28.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:10:07.195925+0000', 'modified': '2024-03-04T10:10:27.941792+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 94, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7890}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7890': {'gid': 7890, 'name': 'c', 'rank': 0, 'incarnation': 84, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.99:6835/932198368', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 932198368}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 932198368}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:10:28.693 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7890:up:replay 2024-03-04T10:10:28.693 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:10:31.743 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:31.742+0000 7f23292e4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:10:34.098 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:34.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.209+0000 7fb8c35c4640 1 Processor -- start 2024-03-04T10:10:34.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.209+0000 7fb8c35c4640 1 -- start start 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c35c4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb8bc15dd70 0x7fb8bc0d1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c35c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0b9710 0x7fb8bc0ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c35c4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc0d2630 con 0x7fb8bc0ced90 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c35c4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc15e150 con 0x7fb8bc15dd70 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c35c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d2910 con 0x7fb8bc0b9710 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c1360640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb8bc15dd70 0x7fb8bc0d1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c1b61640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb8bc0ced90 0x7fb8bc06c1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58804/0 (socket says 172.21.15.99:58804) 2024-03-04T10:10:34.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.210+0000 7fb8c1b61640 1 -- 172.21.15.99:0/2947503257 learned_addr learned my addr 172.21.15.99:0/2947503257 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:34.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8c0b5f640 1 --2- 172.21.15.99:0/2947503257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0b9710 0x7fb8bc0ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8c1360640 1 -- 172.21.15.99:0/2947503257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0b9710 msgr2=0x7fb8bc0ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8c1360640 1 --2- 172.21.15.99:0/2947503257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0b9710 0x7fb8bc0ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8c1360640 1 -- 172.21.15.99:0/2947503257 >> v1:172.21.15.99:6789/0 conn(0x7fb8bc0ced90 legacy=0x7fb8bc06c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8b3fff640 1 -- 172.21.15.99:0/2947503257 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300385311 0 0) 0x7fb8bc0d2630 con 0x7fb8bc0ced90 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8c1360640 1 -- 172.21.15.99:0/2947503257 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0d2bf0 con 0x7fb8bc15dd70 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.211+0000 7fb8c1360640 1 --2- 172.21.15.99:0/2947503257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc0d1de0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0097a0 tx=0x7fb8ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608dc46f7481405e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8b3fff640 1 -- 172.21.15.99:0/2947503257 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac035020 con 0x7fb8bc15dd70 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8b3fff640 1 -- 172.21.15.99:0/2947503257 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac031830 con 0x7fb8bc15dd70 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8b3fff640 1 -- 172.21.15.99:0/2947503257 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac031c00 con 0x7fb8bc15dd70 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 -- 172.21.15.99:0/2947503257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc15dd70 msgr2=0x7fb8bc0d1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/2947503257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc0d1de0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0097a0 tx=0x7fb8ac02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 -- 172.21.15.99:0/2947503257 shutdown_connections 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/2947503257 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0b9710 0x7fb8bc0ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/2947503257 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc0d1de0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 -- 172.21.15.99:0/2947503257 >> 172.21.15.99:0/2947503257 conn(0x7fb8bc072a60 msgr2=0x7fb8bc0d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 -- 172.21.15.99:0/2947503257 shutdown_connections 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.212+0000 7fb8c35c4640 1 -- 172.21.15.99:0/2947503257 wait complete. 2024-03-04T10:10:34.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.213+0000 7fb8c35c4640 1 Processor -- start 2024-03-04T10:10:34.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 -- start start 2024-03-04T10:10:34.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 0x7fb8bc15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0ced90 0x7fb8bc153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d2320 con 0x7fb8bc0b9710 2024-03-04T10:10:34.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d31b0 con 0x7fb8bc15dd70 2024-03-04T10:10:34.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c35c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0d36d0 con 0x7fb8bc0ced90 2024-03-04T10:10:34.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c1360640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 0x7fb8bc15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c1360640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 0x7fb8bc15c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37624/0 (socket says 172.21.15.99:37624) 2024-03-04T10:10:34.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c1360640 1 -- 172.21.15.99:0/1857007035 learned_addr learned my addr 172.21.15.99:0/1857007035 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c0b5f640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0ced90 0x7fb8bc153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.214+0000 7fb8c1b61640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c1360640 1 -- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0ced90 msgr2=0x7fb8bc153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c1360640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0ced90 0x7fb8bc153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c1360640 1 -- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8bc15dd70 msgr2=0x7fb8bc149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c1360640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c1360640 1 -- 172.21.15.99:0/1857007035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc15e7d0 con 0x7fb8bc0b9710 2024-03-04T10:10:34.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c1360640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 0x7fb8bc15c0c0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac02bee0 tx=0x7fb8ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac041070 con 0x7fb8bc0b9710 2024-03-04T10:10:34.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac03b920 con 0x7fb8bc0b9710 2024-03-04T10:10:34.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc15eab0 con 0x7fb8bc0b9710 2024-03-04T10:10:34.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac035040 con 0x7fb8bc0b9710 2024-03-04T10:10:34.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.215+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc15f070 con 0x7fb8bc0b9710 2024-03-04T10:10:34.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.216+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc074f60 con 0x7fb8bc0b9710 2024-03-04T10:10:34.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.217+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb8ac04c060 con 0x7fb8bc0b9710 2024-03-04T10:10:34.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.217+0000 7fb8b1ffb640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8a006b4d0 0x7fb8a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.217+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb8ac0abdd0 con 0x7fb8bc0b9710 2024-03-04T10:10:34.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.217+0000 7fb8c0b5f640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8a006b4d0 0x7fb8a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.218+0000 7fb8c0b5f640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8a006b4d0 0x7fb8a006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0b5770 tx=0x7fb8b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.222+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb8ac07aa10 con 0x7fb8bc0b9710 2024-03-04T10:10:34.428 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.427+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8bc0d4630 con 0x7fb8bc0b9710 2024-03-04T10:10:34.429 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.428+0000 7fb8b1ffb640 1 -- 172.21.15.99:0/1857007035 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fb8ac07a310 con 0x7fb8bc0b9710 2024-03-04T10:10:34.429 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:34.429 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":87,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:29.953978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":94,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":5,"addr":"172.21.15.99:6835/932198368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":932198368},{"type":"v1","addr":"172.21.15.99:6835","nonce":932198368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:34.430 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 87 2024-03-04T10:10:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.429+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8a006b4d0 msgr2=0x7fb8a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.429+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8a006b4d0 0x7fb8a006d990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0b5770 tx=0x7fb8b8008040 comp rx=0 tx=0).stop 2024-03-04T10:10:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 msgr2=0x7fb8bc15c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 0x7fb8bc15c0c0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac02bee0 tx=0x7fb8ac0046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 shutdown_connections 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb8a006b4d0 0x7fb8a006d990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8bc15dd70 0x7fb8bc149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb8bc0ced90 0x7fb8bc153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 --2- 172.21.15.99:0/1857007035 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb8bc0b9710 0x7fb8bc15c0c0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 >> 172.21.15.99:0/1857007035 conn(0x7fb8bc072a60 msgr2=0x7fb8bc0d3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 shutdown_connections 2024-03-04T10:10:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.430+0000 7fb8c35c4640 1 -- 172.21.15.99:0/1857007035 wait complete. 2024-03-04T10:10:34.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T10:10:07.195925+0000', 'modified': '2024-03-04T10:10:29.953978+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 94, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7890}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7890': {'gid': 7890, 'name': 'c', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.99:6835/932198368', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.99:6834', 'nonce': 932198368}, {'type': 'v1', 'addr': '172.21.15.99:6835', 'nonce': 932198368}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T10:10:34.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T10:10:34.443 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-04T10:10:34.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.543+0000 7fc6467bf640 1 Processor -- start 2024-03-04T10:10:34.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.543+0000 7fc6467bf640 1 -- start start 2024-03-04T10:10:34.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc6467bf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc6400b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc6467bf640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc6400b47c0 0x7fc6400b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc6467bf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6400ccad0 con 0x7fc64015b960 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc6467bf640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6400b5190 con 0x7fc6400b47c0 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc6467bf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400b10a0 con 0x7fc6400b0700 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc6400b47c0 0x7fc6400b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc6400b47c0 0x7fc6400b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37626/0 (socket says 172.21.15.99:37626) 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 -- 172.21.15.99:0/3025184424 learned_addr learned my addr 172.21.15.99:0/3025184424 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3025184424 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc6400b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 -- 172.21.15.99:0/3025184424 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 msgr2=0x7fc6400b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 --2- 172.21.15.99:0/3025184424 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc6400b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 -- 172.21.15.99:0/3025184424 >> v1:172.21.15.162:6789/0 conn(0x7fc64015b960 legacy=0x7fc6400d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.544+0000 7fc644d5c640 1 -- 172.21.15.99:0/3025184424 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400cfab0 con 0x7fc6400b47c0 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc644d5c640 1 --2- 172.21.15.99:0/3025184424 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6400b47c0 0x7fc6400b4c50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc638009e50 tx=0x7fc63802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17462d31a369fffb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc63effd640 1 -- 172.21.15.99:0/3025184424 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63802c9e0 con 0x7fc6400b47c0 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc63effd640 1 -- 172.21.15.99:0/3025184424 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63802cb80 con 0x7fc6400b47c0 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc63effd640 1 -- 172.21.15.99:0/3025184424 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6380343f0 con 0x7fc6400b47c0 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 -- 172.21.15.99:0/3025184424 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6400b47c0 msgr2=0x7fc6400b4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 --2- 172.21.15.99:0/3025184424 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6400b47c0 0x7fc6400b4c50 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc638009e50 tx=0x7fc63802be40 comp rx=0 tx=0).stop 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 -- 172.21.15.99:0/3025184424 shutdown_connections 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 --2- 172.21.15.99:0/3025184424 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc6400b47c0 0x7fc6400b4c50 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 --2- 172.21.15.99:0/3025184424 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc6400b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 -- 172.21.15.99:0/3025184424 >> 172.21.15.99:0/3025184424 conn(0x7fc64006f5f0 msgr2=0x7fc64006fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.548 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 -- 172.21.15.99:0/3025184424 shutdown_connections 2024-03-04T10:10:34.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.545+0000 7fc6467bf640 1 -- 172.21.15.99:0/3025184424 wait complete. 2024-03-04T10:10:34.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 Processor -- start 2024-03-04T10:10:34.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 -- start start 2024-03-04T10:10:34.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc640152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6400b47c0 0x7fc6400ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 0x7fc6400aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400d0100 con 0x7fc64015b960 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400cf860 con 0x7fc6400b47c0 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.547+0000 7fc6467bf640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64015bd40 con 0x7fc6400b0700 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 0x7fc6400aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 0x7fc6400aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37636/0 (socket says 172.21.15.99:37636) 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 -- 172.21.15.99:0/2853258760 learned_addr learned my addr 172.21.15.99:0/2853258760 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6400b47c0 0x7fc6400ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc63ffff640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc640152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 -- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 msgr2=0x7fc640152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc640152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 -- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6400b47c0 msgr2=0x7fc6400ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6400b47c0 0x7fc6400ad070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 -- 172.21.15.99:0/2853258760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400afc00 con 0x7fc64015b960 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc644d5c640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 0x7fc6400aec60 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc638009f70 tx=0x7fc638004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc63cff9640 1 -- 172.21.15.99:0/2853258760 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638043020 con 0x7fc64015b960 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc63cff9640 1 -- 172.21.15.99:0/2853258760 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63803e450 con 0x7fc64015b960 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc63cff9640 1 -- 172.21.15.99:0/2853258760 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638009510 con 0x7fc64015b960 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6400afee0 con 0x7fc64015b960 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.548+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6400b04a0 con 0x7fc64015b960 2024-03-04T10:10:34.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.550+0000 7fc63cff9640 1 -- 172.21.15.99:0/2853258760 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc638046020 con 0x7fc64015b960 2024-03-04T10:10:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.550+0000 7fc63cff9640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc62c06b4d0 0x7fc62c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.550+0000 7fc63cff9640 1 -- 172.21.15.99:0/2853258760 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fc6380aca40 con 0x7fc64015b960 2024-03-04T10:10:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.550+0000 7fc63ffff640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc62c06b4d0 0x7fc62c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.551+0000 7fc63ffff640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc62c06b4d0 0x7fc62c06d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc6400c7db0 tx=0x7fc630007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.551+0000 7fc649395740 1 Processor -- start 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.551+0000 7fc649395740 1 -- start start 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6efea0 0x55bdaf6c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc649395740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf6f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6c3380 0x55bdaf6f3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc649395740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf62d7c0 con 0x55bdaf6f0280 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf5d6b50 con 0x55bdaf6efea0 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf629860 con 0x55bdaf6c3380 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf6f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf6f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37650/0 (socket says 172.21.15.99:37650) 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3561537627 learned_addr learned my addr 172.21.15.99:0/3561537627 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63ffff640 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6efea0 0x55bdaf6c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc644d5c640 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6c3380 0x55bdaf6f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6c3380 msgr2=0x55bdaf6f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6c3380 0x55bdaf6f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6efea0 msgr2=0x55bdaf6c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6efea0 0x55bdaf6c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3561537627 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdaf5ea9a0 con 0x55bdaf6f0280 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63ffff640 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6efea0 0x55bdaf6c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf6f2c30 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc6340078f0 tx=0x7fc634007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64e309a157b924b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc625ffb640 1 -- 172.21.15.99:0/3561537627 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634010aa0 con 0x55bdaf6f0280 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc625ffb640 1 -- 172.21.15.99:0/3561537627 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc634010c40 con 0x55bdaf6f0280 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.552+0000 7fc625ffb640 1 -- 172.21.15.99:0/3561537627 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634004030 con 0x55bdaf6f0280 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 -- 172.21.15.99:0/3561537627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 msgr2=0x55bdaf6f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf6f2c30 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc6340078f0 tx=0x7fc634007db0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 -- 172.21.15.99:0/3561537627 shutdown_connections 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6c3380 0x55bdaf6f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf6f2c30 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 --2- 172.21.15.99:0/3561537627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6efea0 0x55bdaf6c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 -- 172.21.15.99:0/3561537627 >> 172.21.15.99:0/3561537627 conn(0x55bdaf6c50b0 msgr2=0x55bdaf70dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 -- 172.21.15.99:0/3561537627 shutdown_connections 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 -- 172.21.15.99:0/3561537627 wait complete. 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.553+0000 7fc649395740 1 -- start start 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 Processor -- start 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf722340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf70c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf6f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf6c7750 con 0x55bdaf6efea0 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf5e9f90 con 0x55bdaf6f68d0 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf6c9f90 con 0x55bdaf6f0280 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc649395740 10 client.0 _ll_register_callbacks cb 0x55bdaf60bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc63ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf722340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc63ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf722340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37656/0 (socket says 172.21.15.99:37656) 2024-03-04T10:10:34.565 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.554+0000 7fc63ffff640 1 -- 172.21.15.99:0/741341075 learned_addr learned my addr 172.21.15.99:0/741341075 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc644d5c640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf6f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf70c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63ffff640 1 -- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 msgr2=0x55bdaf70c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63ffff640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf70c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63ffff640 1 -- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 msgr2=0x55bdaf6f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63ffff640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf6f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63ffff640 1 -- 172.21.15.99:0/741341075 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdaf3422d0 con 0x55bdaf6efea0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf70c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc63ffff640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf722340 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc6300147b0 tx=0x7fc630000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf3839cd0bf195c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc61effd640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc61effd640 1 -- 172.21.15.99:0/741341075 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc63001b070 con 0x55bdaf6efea0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc61effd640 1 -- 172.21.15.99:0/741341075 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fc6300050d0 con 0x55bdaf6efea0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc634010c40 con 0x55bdaf6efea0 2024-03-04T10:10:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc61effd640 1 -- 172.21.15.99:0/741341075 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc63002d050 con 0x55bdaf6efea0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc649395740 10 client.8379 fetch_fsmap learned FSMap version 87 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55bdaf40c6b0 con 0x55bdaf6efea0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc61effd640 1 -- 172.21.15.99:0/741341075 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc634010c40 con 0x55bdaf6efea0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc61effd640 1 -- 172.21.15.99:0/741341075 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 87) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc630030020 con 0x55bdaf6efea0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.555+0000 7fc649395740 10 client.8379 fetch_fsmap finished waiting for FSMap version 87 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc649395740 10 client.8379 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc649395740 20 client.8379 populate_metadata read hostname 'smithi099' 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc649395740 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf72a2e0 0x55bdaf72c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc649395740 4 client.8379 mds_command: new command op to 7890 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 --> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdaf63ad00 con 0x55bdaf72a2e0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf72a2e0 0x55bdaf72c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf72a2e0 0x55bdaf72c6d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8ca6bc57fd79a9 server_cookie=d7255abbce3ddb85 in_seq=0 out_seq=0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc61effd640 10 client.8379 ms_handle_connect on v2:172.21.15.99:6834/932198368 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc61effd640 1 -- 172.21.15.99:0/741341075 <== mds.0 v2:172.21.15.99:6834/932198368 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bdaf63ad00 con 0x55bdaf72a2e0 2024-03-04T10:10:34.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc61effd640 10 client.8379 handle_command_reply: tid=0 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.556+0000 7fc649395740 1 client.8379 shutdown 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 msgr2=0x55bdaf722340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf722340 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fc6300147b0 tx=0x7fc630000ed0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 shutdown_connections 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf72a2e0 0x55bdaf72c6d0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf6f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc61effd640 0 client.8379 ms_handle_reset on v2:172.21.15.99:6834/932198368 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf70c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 --2- 172.21.15.99:0/741341075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf722340 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 >> 172.21.15.99:0/741341075 conn(0x55bdaf6c3380 msgr2=0x55bdaf6c6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 shutdown_connections 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 1 -- 172.21.15.99:0/741341075 wait complete. 2024-03-04T10:10:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.557+0000 7fc649395740 20 client.8379 trim_cache size 0 max 16384 2024-03-04T10:10:34.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.572+0000 7fc649395740 1 Processor -- start 2024-03-04T10:10:34.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.572+0000 7fc649395740 1 -- start start 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc649395740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf6eb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf773d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf774270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc649395740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401ffa10 con 0x55bdaf6efea0 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401fee40 con 0x55bdaf6f68d0 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401ff410 con 0x55bdaf6f0280 2024-03-04T10:10:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf6eb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf6eb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37662/0 (socket says 172.21.15.99:37662) 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 -- 172.21.15.99:0/115450186 learned_addr learned my addr 172.21.15.99:0/115450186 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 -- 172.21.15.99:0/115450186 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 msgr2=0x55bdaf773d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf773d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 -- 172.21.15.99:0/115450186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 msgr2=0x55bdaf774270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf774270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.573+0000 7fc63ffff640 1 -- 172.21.15.99:0/115450186 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdaf6c2b80 con 0x55bdaf6efea0 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc63ffff640 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf6eb580 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc630001eb0 tx=0x7fc630020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b4b44e2a837231 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc6257fa640 1 -- 172.21.15.99:0/115450186 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630036070 con 0x55bdaf6efea0 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc6257fa640 1 -- 172.21.15.99:0/115450186 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63001c040 con 0x55bdaf6efea0 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc6257fa640 1 -- 172.21.15.99:0/115450186 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630015c10 con 0x55bdaf6efea0 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 -- 172.21.15.99:0/115450186 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 msgr2=0x55bdaf6eb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf6eb580 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc630001eb0 tx=0x7fc630020000 comp rx=0 tx=0).stop 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 -- 172.21.15.99:0/115450186 shutdown_connections 2024-03-04T10:10:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf774270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6f0280 0x55bdaf773d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 --2- 172.21.15.99:0/115450186 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6efea0 0x55bdaf6eb580 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 -- 172.21.15.99:0/115450186 >> 172.21.15.99:0/115450186 conn(0x55bdaf6c3380 msgr2=0x55bdaf6be4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 -- 172.21.15.99:0/115450186 shutdown_connections 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.574+0000 7fc649395740 1 -- 172.21.15.99:0/115450186 wait complete. 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.575+0000 7fc649395740 1 -- start start 2024-03-04T10:10:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.575+0000 7fc649395740 1 Processor -- start 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6efea0 0x55bdaf6c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf70d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf75f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf71ef30 con 0x55bdaf6f0280 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf6f2570 con 0x55bdaf6f68d0 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdaf6f4ab0 con 0x55bdaf6efea0 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 10 client.0 _ll_register_callbacks cb 0x7fc634007530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf70d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf70d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37676/0 (socket says 172.21.15.99:37676) 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3175689950 learned_addr learned my addr 172.21.15.99:0/3175689950 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc644d5c640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf75f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63ffff640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6efea0 0x55bdaf6c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6efea0 msgr2=0x55bdaf6c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6efea0 0x55bdaf6c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 msgr2=0x55bdaf75f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf75f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 -- 172.21.15.99:0/3175689950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdaf70dc30 con 0x55bdaf6f0280 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc644d5c640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf75f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc63f7fe640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf70d170 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc6340078c0 tx=0x7fc6340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a5f818f9f1eee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc61effd640 10 client.0 ms_handle_connect on v2:172.21.15.99:3300/0 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc61effd640 1 -- 172.21.15.99:0/3175689950 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634030060 con 0x55bdaf6f0280 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc61effd640 1 -- 172.21.15.99:0/3175689950 <== mon.0 v2:172.21.15.99:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fc634005990 con 0x55bdaf6f0280 2024-03-04T10:10:34.583 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc63001c040 con 0x55bdaf6f0280 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc61effd640 1 -- 172.21.15.99:0/3175689950 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc634036050 con 0x55bdaf6f0280 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 10 client.8385 fetch_fsmap learned FSMap version 87 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.576+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55bdaf70d710 con 0x55bdaf6f0280 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc61effd640 1 -- 172.21.15.99:0/3175689950 <== mon.0 v2:172.21.15.99:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63001c040 con 0x55bdaf6f0280 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc61effd640 1 -- 172.21.15.99:0/3175689950 <== mon.0 v2:172.21.15.99:3300/0 5 ==== fsmap(e 87) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc634039020 con 0x55bdaf6f0280 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 10 client.8385 fetch_fsmap finished waiting for FSMap version 87 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 10 client.8385 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 20 client.8385 populate_metadata read hostname 'smithi099' 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf6e4d30 0x55bdaf70b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 4 client.8385 mds_command: new command op to 7890 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 --> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bdaf635050 con 0x55bdaf6e4d30 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc63ffff640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf6e4d30 0x55bdaf70b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc61effd640 10 client.8385 ms_handle_connect on v2:172.21.15.99:6834/932198368 2024-03-04T10:10:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc63ffff640 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf6e4d30 0x55bdaf70b880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a0efd5b13fda9d server_cookie=1a2b5798be5af9b9 in_seq=0 out_seq=0 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc61effd640 1 -- 172.21.15.99:0/3175689950 <== mds.0 v2:172.21.15.99:6834/932198368 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55bdaf635050 con 0x55bdaf6e4d30 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc61effd640 10 client.8385 handle_command_reply: tid=0 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 1 client.8385 shutdown 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 msgr2=0x55bdaf70d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.577+0000 7fc649395740 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf70d170 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fc6340078c0 tx=0x7fc6340046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 shutdown_connections 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:6834/932198368,v1:172.21.15.99:6835/932198368] conn(0x55bdaf6e4d30 0x55bdaf70b880 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bdaf6f68d0 0x55bdaf75f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55bdaf6f0280 0x55bdaf70d170 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 --2- 172.21.15.99:0/3175689950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55bdaf6efea0 0x55bdaf6c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc61effd640 0 client.8385 ms_handle_reset on v2:172.21.15.99:6834/932198368 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 >> 172.21.15.99:0/3175689950 conn(0x55bdaf6c3380 msgr2=0x55bdaf710e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 shutdown_connections 2024-03-04T10:10:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 1 -- 172.21.15.99:0/3175689950 wait complete. 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc649395740 20 client.8385 trim_cache size 0 max 16384 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "cluster_fsid": "8fdf3d83-d8a6-45f1-acf1-cb71ab619f47", 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "whoami": 0, 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "id": 7890, 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "want_state": "up:active", 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "state": "up:active", 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "fs_name": "cephfs", 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "rank_uptime": 6.6310621860000003, 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "mdsmap_epoch": 87, 2024-03-04T10:10:34.586 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch": 96, 2024-03-04T10:10:34.587 INFO:teuthology.orchestra.run.smithi099.stdout: "osdmap_epoch_barrier": 96, 2024-03-04T10:10:34.587 INFO:teuthology.orchestra.run.smithi099.stdout: "uptime": 13.808528834000001 2024-03-04T10:10:34.587 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:10:34.587 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc62c06b4d0 msgr2=0x7fc62c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc62c06b4d0 0x7fc62c06d990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc6400c7db0 tx=0x7fc630007e30 comp rx=0 tx=0).stop 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 msgr2=0x7fc6400aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 0x7fc6400aec60 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc638009f70 tx=0x7fc638004a40 comp rx=0 tx=0).stop 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 shutdown_connections 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc62c06b4d0 0x7fc62c06d990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc64015b960 0x7fc6400aec60 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6400b47c0 0x7fc6400ad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 --2- 172.21.15.99:0/2853258760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc6400b0700 0x7fc640152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 >> 172.21.15.99:0/2853258760 conn(0x7fc64006f5f0 msgr2=0x7fc6400b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 shutdown_connections 2024-03-04T10:10:34.588 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.578+0000 7fc6467bf640 1 -- 172.21.15.99:0/2853258760 wait complete. 2024-03-04T10:10:34.591 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-04T10:10:34.591 INFO:tasks.cephfs.test_damage:Daemons came up after mutation 'Delete 10000000000.00000000', proceeding to ls 2024-03-04T10:10:34.591 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-04T10:10:34.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.690+0000 7f41b0a4b640 1 Processor -- start 2024-03-04T10:10:34.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.690+0000 7f41b0a4b640 1 -- start start 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41b0a4b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f41ac0aeb20 0x7f41ac071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41b0a4b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac15da40 con 0x7f41ac15d660 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41b0a4b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac0af090 con 0x7f41ac071890 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41b0a4b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ac071ce0 con 0x7f41ac0aeb20 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41a3fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f41ac0aeb20 0x7f41ac071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41aa59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f41ac15d660 0x7f41ac15e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58816/0 (socket says 172.21.15.99:58816) 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41aa59c640 1 -- 172.21.15.99:0/2342092460 learned_addr learned my addr 172.21.15.99:0/2342092460 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41a3fff640 1 -- 172.21.15.99:0/2342092460 >> v1:172.21.15.162:6789/0 conn(0x7f41ac071890 legacy=0x7f41ac150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41a3fff640 1 -- 172.21.15.99:0/2342092460 >> v1:172.21.15.99:6789/0 conn(0x7f41ac15d660 legacy=0x7f41ac15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.691+0000 7f41a3fff640 1 -- 172.21.15.99:0/2342092460 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac15a9f0 con 0x7f41ac0aeb20 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.692+0000 7f41a3fff640 1 --2- 172.21.15.99:0/2342092460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac0aeb20 0x7f41ac071350 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f41940097a0 tx=0x7f419402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89184d35908b0a22 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.692+0000 7f41a9d9b640 1 -- 172.21.15.99:0/2342092460 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194036020 con 0x7f41ac0aeb20 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.692+0000 7f41a9d9b640 1 -- 172.21.15.99:0/2342092460 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4194032810 con 0x7f41ac0aeb20 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.692+0000 7f41a9d9b640 1 -- 172.21.15.99:0/2342092460 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4194032be0 con 0x7f41ac0aeb20 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2342092460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac0aeb20 msgr2=0x7f41ac071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2342092460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac0aeb20 0x7f41ac071350 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f41940097a0 tx=0x7f419402c8b0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2342092460 shutdown_connections 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2342092460 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac0aeb20 0x7f41ac071350 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2342092460 >> 172.21.15.99:0/2342092460 conn(0x7f41ac06acd0 msgr2=0x7f41ac06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2342092460 shutdown_connections 2024-03-04T10:10:34.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.693+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2342092460 wait complete. 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.694+0000 7f41b0a4b640 1 Processor -- start 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.694+0000 7f41b0a4b640 1 -- start start 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41b0a4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41ac071890 0x7f41ac1430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41b0a4b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41ac0aeb20 0x7f41ac157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41b0a4b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 0x7f41ac0b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41b0a4b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac15afb0 con 0x7f41ac15d660 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41b0a4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac15cbc0 con 0x7f41ac071890 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41b0a4b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac15a660 con 0x7f41ac0aeb20 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 0x7f41ac0b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 0x7f41ac0b1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37686/0 (socket says 172.21.15.99:37686) 2024-03-04T10:10:34.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 -- 172.21.15.99:0/2776277068 learned_addr learned my addr 172.21.15.99:0/2776277068 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aa59c640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41ac071890 0x7f41ac1430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41a3fff640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41ac0aeb20 0x7f41ac157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 -- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41ac0aeb20 msgr2=0x7f41ac157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41ac0aeb20 0x7f41ac157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 -- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41ac071890 msgr2=0x7f41ac1430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41ac071890 0x7f41ac1430e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 -- 172.21.15.99:0/2776277068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac154d80 con 0x7f41ac15d660 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41a3fff640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41ac0aeb20 0x7f41ac157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aa59c640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41ac071890 0x7f41ac1430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.695+0000 7f41aad9d640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 0x7f41ac0b1400 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c780 tx=0x7f41a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.696+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a4017070 con 0x7f41ac15d660 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.696+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a4003da0 con 0x7f41ac15d660 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.696+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ac158420 con 0x7f41ac15d660 2024-03-04T10:10:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.696+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a40052b0 con 0x7f41ac15d660 2024-03-04T10:10:34.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.696+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ac155320 con 0x7f41ac15d660 2024-03-04T10:10:34.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.697+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ac074f60 con 0x7f41ac15d660 2024-03-04T10:10:34.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.697+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f41a401a070 con 0x7f41ac15d660 2024-03-04T10:10:34.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.697+0000 7f41a2ffd640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f418c06b4d0 0x7f418c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:34.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.698+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f41a4089ad0 con 0x7f41ac15d660 2024-03-04T10:10:34.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.698+0000 7f41aa59c640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f418c06b4d0 0x7f418c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:34.701 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.698+0000 7f41aa59c640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f418c06b4d0 0x7f418c06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f419c000f60 tx=0x7f419c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:34.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.702+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f41a4058700 con 0x7f41ac15d660 2024-03-04T10:10:34.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.911+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f41ac072460 con 0x7f41ac15d660 2024-03-04T10:10:34.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.911+0000 7f41a2ffd640 1 -- 172.21.15.99:0/2776277068 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) v1 ==== 53+0+73 (secure 0 0 0) 0x7f41a4058000 con 0x7f41ac15d660 2024-03-04T10:10:34.913 INFO:teuthology.orchestra.run.smithi099.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f418c06b4d0 msgr2=0x7f418c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f418c06b4d0 0x7f418c06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f419c000f60 tx=0x7f419c009040 comp rx=0 tx=0).stop 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 msgr2=0x7f41ac0b1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 0x7f41ac0b1400 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f41a400c780 tx=0x7f41a400cc40 comp rx=0 tx=0).stop 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 shutdown_connections 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f418c06b4d0 0x7f418c06d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f41ac15d660 0x7f41ac0b1400 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f41ac0aeb20 0x7f41ac157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 --2- 172.21.15.99:0/2776277068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f41ac071890 0x7f41ac1430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:34.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 >> 172.21.15.99:0/2776277068 conn(0x7f41ac06acd0 msgr2=0x7f41ac154200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:34.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 shutdown_connections 2024-03-04T10:10:34.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:34.913+0000 7f41b0a4b640 1 -- 172.21.15.99:0/2776277068 wait complete. 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi099.front.sepia.ceph.com 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T10:10:34.926 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:10:34.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:34.927 DEBUG:teuthology.orchestra.run.smithi099:> ip addr 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 ::1/128 scope host 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:10:34.942 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:10 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f0 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:d8:a2:11 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: altname enp7s0f1 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:02 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether 0c:c4:7a:8f:d8:03 brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: inet 172.21.15.99/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft 42351sec preferred_lft 42351sec 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::ec4:7aff:fe8f:d803/64 scope link 2024-03-04T10:10:34.943 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::46c:84ff:fea1:d341/64 scope link 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether c2:ee:d4:d0:a9:0c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::c0ee:d4ff:fed0:a90c/64 scope link 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: link/ether e6:ee:45:73:d6:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:10:34.944 INFO:teuthology.orchestra.run.smithi099.stdout: inet6 fe80::e4ee:45ff:fe73:d633/64 scope link 2024-03-04T10:10:34.945 INFO:teuthology.orchestra.run.smithi099.stdout: valid_lft forever preferred_lft forever 2024-03-04T10:10:34.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:34.945 DEBUG:teuthology.orchestra.run.smithi099:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T10:10:35.008 INFO:teuthology.orchestra.run.smithi099.stdout:1 2024-03-04T10:10:35.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.010 DEBUG:teuthology.orchestra.run.smithi099:> route 2024-03-04T10:10:35.066 INFO:teuthology.orchestra.run.smithi099.stdout:Kernel IP routing table 2024-03-04T10:10:35.066 INFO:teuthology.orchestra.run.smithi099.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T10:10:35.066 INFO:teuthology.orchestra.run.smithi099.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-04T10:10:35.066 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-04T10:10:35.066 INFO:teuthology.orchestra.run.smithi099.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T10:10:35.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.066 DEBUG:teuthology.orchestra.run.smithi099:> (cd / && exec bash -c ' 2024-03-04T10:10:35.067 DEBUG:teuthology.orchestra.run.smithi099:> set -e 2024-03-04T10:10:35.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-04T10:10:35.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-04T10:10:35.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-04T10:10:35.067 DEBUG:teuthology.orchestra.run.smithi099:> ') 2024-03-04T10:10:35.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.162 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:10:35.177 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:10:35.177 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:10:35.177 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:10:35.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.177 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link set brx.0 up 2024-03-04T10:10:35.244 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:10:35.244 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:35.299 INFO:teuthology.orchestra.run.smithi099.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T10:10:35.299 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T10:10:35.299 DEBUG:teuthology.orchestra.run.smithi099:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:35.353 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-03-04T10:10:35.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.354 DEBUG:teuthology.orchestra.run.smithi099:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:10:35.438 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:10:35.438 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:10:35.445 INFO:teuthology.orchestra.run.smithi099.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-03-04T10:10:35.445 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: resolved to: "172.21.15.99:3300,172.21.15.162:3300,172.21.15.162:3301" 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: trying mount with new device syntax: 0@8fdf3d83-d8a6-45f1-acf1-cb71ab619f47.cephfs=/ 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.99:3300/172.21.15.162:3300/172.21.15.162:3301" will pass to kernel 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: You just mounted a file system that supports labels which does not 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T10:10:35.446 INFO:teuthology.orchestra.run.smithi099.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T10:10:35.447 INFO:teuthology.orchestra.run.smithi099.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T10:10:35.447 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T10:10:35.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.448 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T10:10:35.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.474 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T10:10:35.530 INFO:teuthology.orchestra.run.smithi099.stdout:client8397 2024-03-04T10:10:35.531 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:35.531 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:35.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:35.587 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -R .) 2024-03-04T10:10:35.629 INFO:teuthology.orchestra.run:waiting for 20 2024-03-04T10:10:35.643 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:35.641+0000 7f23252dc640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-03-04T10:10:35.643 INFO:teuthology.orchestra.run.smithi099.stdout:.: 2024-03-04T10:10:35.644 INFO:teuthology.orchestra.run.smithi099.stdout:subdir 2024-03-04T10:10:35.644 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:35.644 INFO:teuthology.orchestra.run.smithi099.stdout:./subdir: 2024-03-04T10:10:35.644 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 10000000000.00000000' did not caused DAMAGED state 2024-03-04T10:10:35.644 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 600.00000000' 2024-03-04T10:10:35.644 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:35.644 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:35.699 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:35.699 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:35.754 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:10:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.892+0000 7fb999924640 1 Processor -- start 2024-03-04T10:10:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.892+0000 7fb999924640 1 -- start start 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb999924640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb9940b2e60 0x7fb9940b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb999924640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb9940b5470 0x7fb9940b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb999924640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99415e1e0 con 0x7fb99415de00 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb999924640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9940b98c0 con 0x7fb9940b5470 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb999924640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9940b32c0 con 0x7fb9940b2e60 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9937fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb9940b5470 0x7fb9940b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb992ffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fb99415de00 0x7fb9940b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58826/0 (socket says 172.21.15.99:58826) 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb992ffd640 1 -- 172.21.15.99:0/3727158980 learned_addr learned my addr 172.21.15.99:0/3727158980 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9937fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb9940b5470 0x7fb9940b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37694/0 (socket says 172.21.15.99:37694) 2024-03-04T10:10:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9927fc640 1 --2- 172.21.15.99:0/3727158980 >> v2:172.21.15.162:3300/0 conn(0x7fb9940b2e60 0x7fb9940b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9937fe640 1 -- 172.21.15.99:0/3727158980 >> v2:172.21.15.162:3300/0 conn(0x7fb9940b2e60 msgr2=0x7fb9940b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9937fe640 1 --2- 172.21.15.99:0/3727158980 >> v2:172.21.15.162:3300/0 conn(0x7fb9940b2e60 0x7fb9940b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9937fe640 1 -- 172.21.15.99:0/3727158980 >> v1:172.21.15.99:6789/0 conn(0x7fb99415de00 legacy=0x7fb9940b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb9937fe640 1 -- 172.21.15.99:0/3727158980 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9940d1c10 con 0x7fb9940b5470 2024-03-04T10:10:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.893+0000 7fb991ffb640 1 -- 172.21.15.99:0/3727158980 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051751523 0 0) 0x7fb99415e1e0 con 0x7fb99415de00 2024-03-04T10:10:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.894+0000 7fb9937fe640 1 --2- 172.21.15.99:0/3727158980 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b5900 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fb994161830 tx=0x7fb98402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f0e796f425dfd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.894+0000 7fb991ffb640 1 -- 172.21.15.99:0/3727158980 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984035020 con 0x7fb9940b5470 2024-03-04T10:10:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.894+0000 7fb991ffb640 1 -- 172.21.15.99:0/3727158980 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb984031830 con 0x7fb9940b5470 2024-03-04T10:10:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.894+0000 7fb991ffb640 1 -- 172.21.15.99:0/3727158980 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984031c00 con 0x7fb9940b5470 2024-03-04T10:10:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.894+0000 7fb999924640 1 -- 172.21.15.99:0/3727158980 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 msgr2=0x7fb9940b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.894+0000 7fb999924640 1 --2- 172.21.15.99:0/3727158980 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b5900 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fb994161830 tx=0x7fb98402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.895+0000 7fb999924640 1 -- 172.21.15.99:0/3727158980 shutdown_connections 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.895+0000 7fb999924640 1 --2- 172.21.15.99:0/3727158980 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b5900 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.895+0000 7fb999924640 1 --2- 172.21.15.99:0/3727158980 >> v2:172.21.15.162:3300/0 conn(0x7fb9940b2e60 0x7fb9940b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.895+0000 7fb999924640 1 -- 172.21.15.99:0/3727158980 >> 172.21.15.99:0/3727158980 conn(0x7fb9940d0470 msgr2=0x7fb9940d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.895+0000 7fb999924640 1 -- 172.21.15.99:0/3727158980 shutdown_connections 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.895+0000 7fb999924640 1 -- 172.21.15.99:0/3727158980 wait complete. 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.896+0000 7fb999924640 1 Processor -- start 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.896+0000 7fb999924640 1 -- start start 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9940b2e60 0x7fb99415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb99415de00 0x7fb994153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9940d21d0 con 0x7fb9940b5470 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9940d2820 con 0x7fb99415de00 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9940d24b0 con 0x7fb9940b2e60 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37708/0 (socket says 172.21.15.99:37708) 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 -- 172.21.15.99:0/1381589752 learned_addr learned my addr 172.21.15.99:0/1381589752 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9937fe640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb99415de00 0x7fb994153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb992ffd640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9940b2e60 0x7fb99415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 -- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9940b2e60 msgr2=0x7fb99415a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9940b2e60 0x7fb99415a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 -- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb99415de00 msgr2=0x7fb994153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb99415de00 0x7fb994153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 -- 172.21.15.99:0/1381589752 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb994157730 con 0x7fb9940b5470 2024-03-04T10:10:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb992ffd640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9940b2e60 0x7fb99415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb9927fc640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b72e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00d800 tx=0x7fb97c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00b6c0 con 0x7fb9940b5470 2024-03-04T10:10:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c019070 con 0x7fb9940b5470 2024-03-04T10:10:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb994157a10 con 0x7fb9940b5470 2024-03-04T10:10:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c00bac0 con 0x7fb9940b5470 2024-03-04T10:10:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.897+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb994161120 con 0x7fb9940b5470 2024-03-04T10:10:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.898+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb994074f60 con 0x7fb9940b5470 2024-03-04T10:10:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.899+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb97c00bc60 con 0x7fb9940b5470 2024-03-04T10:10:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.899+0000 7fb98bfff640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb97406b4d0 0x7fb97406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.899+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fb97c08a880 con 0x7fb9940b5470 2024-03-04T10:10:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.899+0000 7fb992ffd640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb97406b4d0 0x7fb97406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.900+0000 7fb992ffd640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb97406b4d0 0x7fb97406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb98c007fd0 tx=0x7fb98c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:35.903+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb97c059540 con 0x7fb9940b5470 2024-03-04T10:10:36.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.078+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9940b6b50 con 0x7fb9940b5470 2024-03-04T10:10:36.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.079+0000 7fb98bfff640 1 -- 172.21.15.99:0/1381589752 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+17172 (secure 0 0 0) 0x7fb97c058e40 con 0x7fb9940b5470 2024-03-04T10:10:36.080 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:36.081 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":96,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:10:22.909263+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.31","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.99:6835/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6834/3671020073":"2024-03-05T10:10:20.734907+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:10:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb97406b4d0 msgr2=0x7fb97406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb97406b4d0 0x7fb97406d990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb98c007fd0 tx=0x7fb98c009040 comp rx=0 tx=0).stop 2024-03-04T10:10:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 msgr2=0x7fb9940b72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b72e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00d800 tx=0x7fb97c00dcc0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 shutdown_connections 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb97406b4d0 0x7fb97406d990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb99415de00 0x7fb994153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb9940b5470 0x7fb9940b72e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 --2- 172.21.15.99:0/1381589752 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb9940b2e60 0x7fb99415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 >> 172.21.15.99:0/1381589752 conn(0x7fb9940d0470 msgr2=0x7fb9940b9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 shutdown_connections 2024-03-04T10:10:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.081+0000 7fb999924640 1 -- 172.21.15.99:0/1381589752 wait complete. 2024-03-04T10:10:36.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:10:05.411440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:10:06.166240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:10:36.094 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:10:36.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ecab640 1 Processor -- start 2024-03-04T10:10:36.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ecab640 1 -- start start 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ecab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 0x7f4928161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ecab640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280d2400 con 0x7f49280b5700 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ecab640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280d26e0 con 0x7f49280b9410 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ecab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49280d29c0 con 0x7f49280d34a0 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.196+0000 7f492ca47640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f49280b5700 0x7f49280d1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58832/0 (socket says 172.21.15.99:58832) 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f492ca47640 1 -- 172.21.15.99:0/2650857672 learned_addr learned my addr 172.21.15.99:0/2650857672 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365365093 0 0) 0x7f49280d2400 con 0x7f49280b5700 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4918003520 con 0x7f49280b5700 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f4927fff640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 0x7f4928161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726496357 0 0) 0x7f4918003520 con 0x7f49280b5700 2024-03-04T10:10:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 msgr2=0x7f4928161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 0x7f4928161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 >> v1:172.21.15.162:6789/0 conn(0x7f49280b9410 legacy=0x7f4928162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280d2ca0 con 0x7f49280b5700 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f491c001670 con 0x7f49280b5700 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 >> v1:172.21.15.99:6789/0 conn(0x7f49280b5700 legacy=0x7f49280d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 0x7f49180039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4918003f60 0x7f4918004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4918004d40 0x7f49180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49280d29c0 con 0x7f4918004d40 2024-03-04T10:10:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491800bb50 con 0x7f4918003f60 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491800bd50 con 0x7f49280d34a0 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f492d248640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4918004d40 0x7f49180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f492ca47640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 0x7f49180039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.197+0000 7f4927fff640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4918003f60 0x7f4918004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f492d248640 1 -- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 msgr2=0x7f49180039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f492d248640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f49280d34a0 0x7f49180039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f492d248640 1 -- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4918003f60 msgr2=0x7f4918004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f492d248640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4918003f60 0x7f4918004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f492d248640 1 -- 172.21.15.99:0/2650857672 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4918003520 con 0x7f4918004d40 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f492d248640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4918004d40 0x7f49180051c0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f4920001e50 tx=0x7f4918005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1613845dde60b578 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920001a70 con 0x7f4918004d40 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920003b70 con 0x7f4918004d40 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.198+0000 7f49277fe640 1 -- 172.21.15.99:0/2650857672 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200050d0 con 0x7f4918004d40 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492ecab640 1 -- 172.21.15.99:0/2650857672 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4918004d40 msgr2=0x7f49180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492ecab640 1 --2- 172.21.15.99:0/2650857672 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4918004d40 0x7f49180051c0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f4920001e50 tx=0x7f4918005830 comp rx=0 tx=0).stop 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492d248640 1 -- 172.21.15.99:0/2650857672 reap_dead start 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492ecab640 1 -- 172.21.15.99:0/2650857672 shutdown_connections 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492ecab640 1 -- 172.21.15.99:0/2650857672 >> 172.21.15.99:0/2650857672 conn(0x7f49280d0490 msgr2=0x7f49280d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492ecab640 1 -- 172.21.15.99:0/2650857672 shutdown_connections 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.199+0000 7f492ecab640 1 -- 172.21.15.99:0/2650857672 wait complete. 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.200+0000 7f492ecab640 1 Processor -- start 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.200+0000 7f492ecab640 1 -- start start 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ecab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4928155ba0 0x7f4928157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ecab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 0x7f49280cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ecab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49280cba30 0x7f49280cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ecab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49280b5c60 con 0x7f4928155f80 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ecab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49280b5ae0 con 0x7f49280cba30 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ecab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49280b5de0 con 0x7f4928155ba0 2024-03-04T10:10:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 0x7f49280cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 0x7f49280cb4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37724/0 (socket says 172.21.15.99:37724) 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 -- 172.21.15.99:0/1645533407 learned_addr learned my addr 172.21.15.99:0/1645533407 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492ca47640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4928155ba0 0x7f4928157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492d248640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49280cba30 0x7f49280cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 -- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4928155ba0 msgr2=0x7f4928157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4928155ba0 0x7f4928157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 -- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49280cba30 msgr2=0x7f49280cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49280cba30 0x7f49280cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f4927fff640 1 -- 172.21.15.99:0/1645533407 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4928158180 con 0x7f4928155f80 2024-03-04T10:10:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.201+0000 7f492d248640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49280cba30 0x7f49280cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.202+0000 7f4927fff640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 0x7f49280cb4f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f49140119a0 tx=0x7f4914011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.202+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491400faf0 con 0x7f4928155f80 2024-03-04T10:10:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.202+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491400fc90 con 0x7f4928155f80 2024-03-04T10:10:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.202+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914014440 con 0x7f4928155f80 2024-03-04T10:10:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.202+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49280cbf40 con 0x7f4928155f80 2024-03-04T10:10:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.202+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49280c7a10 con 0x7f4928155f80 2024-03-04T10:10:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.203+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49280674d0 con 0x7f4928155f80 2024-03-04T10:10:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.203+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4914012040 con 0x7f4928155f80 2024-03-04T10:10:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.203+0000 7f49257fa640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f490c06b500 0x7f490c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.203+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f491408a9f0 con 0x7f4928155f80 2024-03-04T10:10:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.204+0000 7f492ca47640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f490c06b500 0x7f490c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.204+0000 7f492ca47640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f490c06b500 0x7f490c06d9c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f49281625c0 tx=0x7f491c0078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.208+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f49280674d0 con 0x7f4928155f80 2024-03-04T10:10:36.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.381+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4928074f60 con 0x7f4928155f80 2024-03-04T10:10:36.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.381+0000 7f49257fa640 1 -- 172.21.15.99:0/1645533407 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4914019030 con 0x7f4928155f80 2024-03-04T10:10:36.383 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:10:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f490c06b500 msgr2=0x7f490c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f490c06b500 0x7f490c06d9c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f49281625c0 tx=0x7f491c0078f0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 msgr2=0x7f49280cb4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 0x7f49280cb4f0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f49140119a0 tx=0x7f4914011e60 comp rx=0 tx=0).stop 2024-03-04T10:10:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 shutdown_connections 2024-03-04T10:10:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f490c06b500 0x7f490c06d9c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49280cba30 0x7f49280cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4928155f80 0x7f49280cb4f0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 --2- 172.21.15.99:0/1645533407 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4928155ba0 0x7f4928157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.383+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 >> 172.21.15.99:0/1645533407 conn(0x7f49280d0490 msgr2=0x7f492806c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.384+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 shutdown_connections 2024-03-04T10:10:36.387 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.384+0000 7f492ecab640 1 -- 172.21.15.99:0/1645533407 wait complete. 2024-03-04T10:10:36.396 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:10:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.497+0000 7fcc5380b640 1 Processor -- start 2024-03-04T10:10:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.498+0000 7fcc5380b640 1 -- start start 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.498+0000 7fcc5380b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcc4c0cd110 0x7fcc4c0d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.498+0000 7fcc5380b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0cc350 con 0x7fcc4c158230 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.498+0000 7fcc5380b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c0cc630 con 0x7fcc4c0cfdf0 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc5380b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4c0cc910 con 0x7fcc4c0cd110 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc515a7640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fcc4c158230 0x7fcc4c0cf6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58846/0 (socket says 172.21.15.99:58846) 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc515a7640 1 -- 172.21.15.99:0/237452219 learned_addr learned my addr 172.21.15.99:0/237452219 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc51da8640 1 --2- 172.21.15.99:0/237452219 >> v2:172.21.15.162:3300/0 conn(0x7fcc4c0cd110 0x7fcc4c0d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629006513 0 0) 0x7fcc4c0cc350 con 0x7fcc4c158230 2024-03-04T10:10:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc38003520 con 0x7fcc4c158230 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2235664968 0 0) 0x7fcc38003520 con 0x7fcc4c158230 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 >> v2:172.21.15.162:3300/0 conn(0x7fcc4c0cd110 msgr2=0x7fcc4c0d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 --2- 172.21.15.99:0/237452219 >> v2:172.21.15.162:3300/0 conn(0x7fcc4c0cd110 0x7fcc4c0d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 >> v1:172.21.15.162:6789/0 conn(0x7fcc4c0cfdf0 legacy=0x7fcc4c0d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0ccbf0 con 0x7fcc4c158230 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fcc40001670 con 0x7fcc4c158230 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 >> v1:172.21.15.99:6789/0 conn(0x7fcc4c158230 legacy=0x7fcc4c0cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc380039b0 0x7fcc38003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc38004800 0x7fcc38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc380051c0 0x7fcc3800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0cc910 con 0x7fcc38004800 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c060 con 0x7fcc380051c0 2024-03-04T10:10:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3800c260 con 0x7fcc380039b0 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc51da8640 1 --2- 172.21.15.99:0/237452219 >> v2:172.21.15.162:3300/0 conn(0x7fcc4c0cd110 0x7fcc4c0d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc50da6640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc38004800 0x7fcc38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc515a7640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc380039b0 0x7fcc38003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.499+0000 7fcc51da8640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc380051c0 0x7fcc3800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc50da6640 1 -- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc380039b0 msgr2=0x7fcc38003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc50da6640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc380039b0 0x7fcc38003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc50da6640 1 -- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc380051c0 msgr2=0x7fcc3800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc50da6640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc380051c0 0x7fcc3800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc51da8640 1 -- 172.21.15.99:0/237452219 reap_dead start 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc50da6640 1 -- 172.21.15.99:0/237452219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc38003520 con 0x7fcc38004800 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc515a7640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc380039b0 0x7fcc38003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc50da6640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc38004800 0x7fcc38004c80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fcc48001750 tx=0x7fcc38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9ccb0dd53ca62a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48001370 con 0x7fcc38004800 2024-03-04T10:10:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc480052d0 con 0x7fcc38004800 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.500+0000 7fcc3ffff640 1 -- 172.21.15.99:0/237452219 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48003830 con 0x7fcc38004800 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 -- 172.21.15.99:0/237452219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc38004800 msgr2=0x7fcc38004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc38004800 0x7fcc38004c80 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fcc48001750 tx=0x7fcc38005830 comp rx=0 tx=0).stop 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 -- 172.21.15.99:0/237452219 shutdown_connections 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 --2- 172.21.15.99:0/237452219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc38004800 0x7fcc38004c80 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 -- 172.21.15.99:0/237452219 >> 172.21.15.99:0/237452219 conn(0x7fcc4c0b3330 msgr2=0x7fcc4c0b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 -- 172.21.15.99:0/237452219 shutdown_connections 2024-03-04T10:10:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.501+0000 7fcc5380b640 1 -- 172.21.15.99:0/237452219 wait complete. 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.502+0000 7fcc5380b640 1 Processor -- start 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.502+0000 7fcc5380b640 1 -- start start 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc5380b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc38004800 0x7fcc4c153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc5380b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc4c1f54e0 0x7fcc4c0b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc5380b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 0x7fcc4c0b2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc5380b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0cc0f0 con 0x7fcc4c1f5960 2024-03-04T10:10:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc5380b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0cbf70 con 0x7fcc38004800 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc5380b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0cf6e0 con 0x7fcc4c1f54e0 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 0x7fcc4c0b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 0x7fcc4c0b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37742/0 (socket says 172.21.15.99:37742) 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 -- 172.21.15.99:0/1380773354 learned_addr learned my addr 172.21.15.99:0/1380773354 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc50da6640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc4c1f54e0 0x7fcc4c0b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc515a7640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc38004800 0x7fcc4c153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 -- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc4c1f54e0 msgr2=0x7fcc4c0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc4c1f54e0 0x7fcc4c0b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 -- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc38004800 msgr2=0x7fcc4c153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc38004800 0x7fcc4c153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 -- 172.21.15.99:0/1380773354 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0b1170 con 0x7fcc4c1f5960 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc50da6640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc4c1f54e0 0x7fcc4c0b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.503+0000 7fcc51da8640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 0x7fcc4c0b2710 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0d02f0 tx=0x7fcc44009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.504+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44011da0 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.504+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4401b070 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.504+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c0b1410 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.504+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44003990 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.504+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c0b19d0 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.505+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4c0cea00 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.505+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcc44005d10 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.506+0000 7fcc3dffb640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.506+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fcc4408aa30 con 0x7fcc4c1f5960 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.506+0000 7fcc515a7640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.506+0000 7fcc515a7640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc3006b4d0 0x7fcc3006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcc40001030 tx=0x7fcc40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.510+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcc4c0cea00 con 0x7fcc4c1f5960 2024-03-04T10:10:36.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.679+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcc4c074f60 con 0x7fcc4c1f5960 2024-03-04T10:10:36.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.680+0000 7fcc3dffb640 1 -- 172.21.15.99:0/1380773354 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcc44014350 con 0x7fcc4c1f5960 2024-03-04T10:10:36.681 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:10:36.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc3006b4d0 msgr2=0x7fcc3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc3006b4d0 0x7fcc3006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcc40001030 tx=0x7fcc40009040 comp rx=0 tx=0).stop 2024-03-04T10:10:36.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 msgr2=0x7fcc4c0b2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 0x7fcc4c0b2710 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c0d02f0 tx=0x7fcc44009f90 comp rx=0 tx=0).stop 2024-03-04T10:10:36.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 shutdown_connections 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcc3006b4d0 0x7fcc3006d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc4c1f5960 0x7fcc4c0b2710 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc4c1f54e0 0x7fcc4c0b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 --2- 172.21.15.99:0/1380773354 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc38004800 0x7fcc4c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 >> 172.21.15.99:0/1380773354 conn(0x7fcc4c0b3330 msgr2=0x7fcc4c15f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 shutdown_connections 2024-03-04T10:10:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.682+0000 7fcc5380b640 1 -- 172.21.15.99:0/1380773354 wait complete. 2024-03-04T10:10:36.695 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:10:36.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.796+0000 7f3fff559640 1 Processor -- start 2024-03-04T10:10:36.798 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.797+0000 7f3fff559640 1 -- start start 2024-03-04T10:10:36.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.797+0000 7f3fff559640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff80ccbd0 0x7f3ff80b7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.797+0000 7f3fff559640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff815ca50 con 0x7f3ff80ceec0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.797+0000 7f3fff559640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ff815cd30 con 0x7f3ff815c2e0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.797+0000 7f3fff559640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff815d2f0 con 0x7f3ff80ccbd0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffdaf6640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3ff80ceec0 0x7f3ff80b0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58854/0 (socket says 172.21.15.99:58854) 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffdaf6640 1 -- 172.21.15.99:0/781005867 learned_addr learned my addr 172.21.15.99:0/781005867 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff80ccbd0 0x7f3ff80b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238432418 0 0) 0x7f3ff815ca50 con 0x7f3ff80ceec0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe4003520 con 0x7f3ff80ceec0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987896220 0 0) 0x7f3ff815cd30 con 0x7f3ff815c2e0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff815ca50 con 0x7f3ff815c2e0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 741606116 0 0) 0x7f3fe4003520 con 0x7f3ff80ceec0 2024-03-04T10:10:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff80ccbd0 msgr2=0x7f3ff80b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff80ccbd0 0x7f3ff80b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 >> v1:172.21.15.162:6789/0 conn(0x7f3ff815c2e0 legacy=0x7f3ff80ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff815d010 con 0x7f3ff80ceec0 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3ff00019f0 con 0x7f3ff80ceec0 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 >> v1:172.21.15.99:6789/0 conn(0x7f3ff80ceec0 legacy=0x7f3ff80b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe4003b70 0x7f3fe4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe4004800 0x7f3fe4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe40051c0 0x7f3fe400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe400bfb0 con 0x7f3fe4004800 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe400c130 con 0x7f3fe4003b70 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe400c330 con 0x7f3fe40051c0 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3ff80ccbd0 0x7f3ff80b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:10:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffcaf4640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe4004800 0x7f3fe4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffdaf6640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe40051c0 0x7f3fe400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe4003b70 0x7f3fe4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffcaf4640 1 -- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe40051c0 msgr2=0x7f3fe400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffcaf4640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe40051c0 0x7f3fe400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffcaf4640 1 -- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe4003b70 msgr2=0x7f3fe4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffcaf4640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe4003b70 0x7f3fe4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffcaf4640 1 -- 172.21.15.99:0/781005867 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4003520 con 0x7f3fe4004800 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.798+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3fe4003b70 0x7f3fe4003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.799+0000 7f3ffcaf4640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe4004800 0x7f3fe4004c80 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f3fe80018c0 tx=0x7f3fe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf39c93b38dcbbd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.799+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8037070 con 0x7f3fe4004800 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.799+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe80052f0 con 0x7f3fe4004800 2024-03-04T10:10:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.799+0000 7f3feffff640 1 -- 172.21.15.99:0/781005867 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe80014e0 con 0x7f3fe4004800 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3fff559640 1 -- 172.21.15.99:0/781005867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe4004800 msgr2=0x7f3fe4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3fff559640 1 --2- 172.21.15.99:0/781005867 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3fe4004800 0x7f3fe4004c80 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f3fe80018c0 tx=0x7f3fe4005830 comp rx=0 tx=0).stop 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3ffdaf6640 1 -- 172.21.15.99:0/781005867 reap_dead start 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3fff559640 1 -- 172.21.15.99:0/781005867 shutdown_connections 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3fff559640 1 -- 172.21.15.99:0/781005867 >> 172.21.15.99:0/781005867 conn(0x7f3ff8066dc0 msgr2=0x7f3ff80671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3fff559640 1 -- 172.21.15.99:0/781005867 shutdown_connections 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.800+0000 7f3fff559640 1 -- 172.21.15.99:0/781005867 wait complete. 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.801+0000 7f3fff559640 1 Processor -- start 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.801+0000 7f3fff559640 1 -- start start 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe4004800 0x7f3ff8153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff81548b0 0x7f3ff8154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 0x7f3ff80c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff80ccfb0 con 0x7f3ff80c6960 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff815d6d0 con 0x7f3ff81548b0 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff815eb70 con 0x7f3fe4004800 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 0x7f3ff80c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 0x7f3ff80c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37748/0 (socket says 172.21.15.99:37748) 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 -- 172.21.15.99:0/3085217688 learned_addr learned my addr 172.21.15.99:0/3085217688 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffcaf4640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff81548b0 0x7f3ff8154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe4004800 0x7f3ff8153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 -- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe4004800 msgr2=0x7f3ff8153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe4004800 0x7f3ff8153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 -- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff81548b0 msgr2=0x7f3ff8154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff81548b0 0x7f3ff8154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 -- 172.21.15.99:0/3085217688 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80c9810 con 0x7f3ff80c6960 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffcaf4640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff81548b0 0x7f3ff8154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3ffdaf6640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 0x7f3ff80c6df0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80b47b0 tx=0x7f3ff000e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff000ebd0 con 0x7f3ff80c6960 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff0018070 con 0x7f3ff80c6960 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0013420 con 0x7f3ff80c6960 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.802+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff80c9af0 con 0x7f3ff80c6960 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.803+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff8157870 con 0x7f3ff80c6960 2024-03-04T10:10:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.804+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff8074f60 con 0x7f3ff80c6960 2024-03-04T10:10:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.804+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3ff00135c0 con 0x7f3ff80c6960 2024-03-04T10:10:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.804+0000 7f3fedffb640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fdc06b4d0 0x7f3fdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.805+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f3ff008a800 con 0x7f3ff80c6960 2024-03-04T10:10:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.805+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fdc06b4d0 0x7f3fdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.805+0000 7f3ffd2f5640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fdc06b4d0 0x7f3fdc06d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80b38b0 tx=0x7f3ff4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.809+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3ff00593a0 con 0x7f3ff80c6960 2024-03-04T10:10:36.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.979+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ff8157a40 con 0x7f3ff80c6960 2024-03-04T10:10:36.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.979+0000 7f3fedffb640 1 -- 172.21.15.99:0/3085217688 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3ff0058ca0 con 0x7f3ff80c6960 2024-03-04T10:10:36.980 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:10:36.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fdc06b4d0 msgr2=0x7f3fdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fdc06b4d0 0x7f3fdc06d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80b38b0 tx=0x7f3ff4008040 comp rx=0 tx=0).stop 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 msgr2=0x7f3ff80c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 0x7f3ff80c6df0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80b47b0 tx=0x7f3ff000e160 comp rx=0 tx=0).stop 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 shutdown_connections 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3fdc06b4d0 0x7f3fdc06d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3ff80c6960 0x7f3ff80c6df0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ff81548b0 0x7f3ff8154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 --2- 172.21.15.99:0/3085217688 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3fe4004800 0x7f3ff8153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 >> 172.21.15.99:0/3085217688 conn(0x7f3ff8066dc0 msgr2=0x7f3ff815b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 shutdown_connections 2024-03-04T10:10:36.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:36.981+0000 7f3fff559640 1 -- 172.21.15.99:0/3085217688 wait complete. 2024-03-04T10:10:36.993 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T10:10:37.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.092+0000 7f445b49a640 1 Processor -- start 2024-03-04T10:10:37.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.092+0000 7f445b49a640 1 -- start start 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f445b49a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540d0050 0x7f44541624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f445b49a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f44540ce360 0x7f44540ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f445b49a640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540cf730 con 0x7f445415de20 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f445b49a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44540cfcb0 con 0x7f44540ce360 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f445b49a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44540cf9d0 con 0x7f44540d0050 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f4459a37640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f44540ce360 0x7f44540ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f4459236640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f445415de20 0x7f445415ed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58866/0 (socket says 172.21.15.99:58866) 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.093+0000 7f4459236640 1 -- 172.21.15.99:0/3799782852 learned_addr learned my addr 172.21.15.99:0/3799782852 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f4458a35640 1 --2- 172.21.15.99:0/3799782852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540d0050 0x7f44541624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f4459a37640 1 -- 172.21.15.99:0/3799782852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540d0050 msgr2=0x7f44541624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f4459a37640 1 --2- 172.21.15.99:0/3799782852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540d0050 0x7f44541624e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f4459a37640 1 -- 172.21.15.99:0/3799782852 >> v1:172.21.15.99:6789/0 conn(0x7f445415de20 legacy=0x7f445415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f444bfff640 1 -- 172.21.15.99:0/3799782852 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612085769 0 0) 0x7f44540cf730 con 0x7f445415de20 2024-03-04T10:10:37.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f4459a37640 1 -- 172.21.15.99:0/3799782852 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454161a00 con 0x7f44540ce360 2024-03-04T10:10:37.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.094+0000 7f4459a37640 1 --2- 172.21.15.99:0/3799782852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44540ce360 0x7f44540ce810 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51d6ca2691c5c09 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f444bfff640 1 -- 172.21.15.99:0/3799782852 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c035020 con 0x7f44540ce360 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f444bfff640 1 -- 172.21.15.99:0/3799782852 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c031830 con 0x7f44540ce360 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f444bfff640 1 -- 172.21.15.99:0/3799782852 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c031c00 con 0x7f44540ce360 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 -- 172.21.15.99:0/3799782852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44540ce360 msgr2=0x7f44540ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 --2- 172.21.15.99:0/3799782852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44540ce360 0x7f44540ce810 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 -- 172.21.15.99:0/3799782852 shutdown_connections 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 --2- 172.21.15.99:0/3799782852 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f44540ce360 0x7f44540ce810 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 --2- 172.21.15.99:0/3799782852 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540d0050 0x7f44541624e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 -- 172.21.15.99:0/3799782852 >> 172.21.15.99:0/3799782852 conn(0x7f445406f6e0 msgr2=0x7f445406faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 -- 172.21.15.99:0/3799782852 shutdown_connections 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.095+0000 7f445b49a640 1 -- 172.21.15.99:0/3799782852 wait complete. 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.096+0000 7f445b49a640 1 Processor -- start 2024-03-04T10:10:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.096+0000 7f445b49a640 1 -- start start 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f445b49a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540ce360 0x7f4454157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f445b49a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 0x7f44540c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f445b49a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44540c7180 0x7f44541f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f445b49a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445406c050 con 0x7f445415de20 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f445b49a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4454161fc0 con 0x7f44540c7180 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f445b49a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445406c1d0 con 0x7f44540ce360 2024-03-04T10:10:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 0x7f44540c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 0x7f44540c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37774/0 (socket says 172.21.15.99:37774) 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 -- 172.21.15.99:0/4058328110 learned_addr learned my addr 172.21.15.99:0/4058328110 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4459236640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540ce360 0x7f4454157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4459a37640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44540c7180 0x7f44541f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 -- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540ce360 msgr2=0x7f4454157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540ce360 0x7f4454157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 -- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44540c7180 msgr2=0x7f44541f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44540c7180 0x7f44541f6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.097+0000 7f4458a35640 1 -- 172.21.15.99:0/4058328110 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540c5980 con 0x7f445415de20 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.098+0000 7f4458a35640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 0x7f44540c5440 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f444400d800 tx=0x7f444400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.098+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444400b6c0 con 0x7f445415de20 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.098+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4444019070 con 0x7f445415de20 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.098+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445415c4c0 con 0x7f445415de20 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.098+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444400ba80 con 0x7f445415de20 2024-03-04T10:10:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.098+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445415ca80 con 0x7f445415de20 2024-03-04T10:10:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.099+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454074f60 con 0x7f445415de20 2024-03-04T10:10:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.099+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f444400bc20 con 0x7f445415de20 2024-03-04T10:10:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.099+0000 7f4449ffb640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f443806b500 0x7f443806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.100+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7f444408a8b0 con 0x7f445415de20 2024-03-04T10:10:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.100+0000 7f4459236640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f443806b500 0x7f443806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.100+0000 7f4459236640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f443806b500 0x7f443806d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4450007ed0 tx=0x7f4450007e60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.104+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f444401c360 con 0x7f445415de20 2024-03-04T10:10:37.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.285+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f44540b2da0 con 0x7f445415de20 2024-03-04T10:10:37.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.285+0000 7f4449ffb640 1 -- 172.21.15.99:0/4058328110 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v96) v1 ==== 80+0+122 (secure 0 0 0) 0x7f44440594e0 con 0x7f445415de20 2024-03-04T10:10:37.287 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6835/3671020073 2024-03-05T10:10:20.734907+0000 2024-03-04T10:10:37.287 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99:6834/3671020073 2024-03-05T10:10:20.734907+0000 2024-03-04T10:10:37.287 INFO:teuthology.orchestra.run.smithi099.stderr:listed 2 entries 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f443806b500 msgr2=0x7f443806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f443806b500 0x7f443806d9c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f4450007ed0 tx=0x7f4450007e60 comp rx=0 tx=0).stop 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 msgr2=0x7f44540c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 0x7f44540c5440 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f444400d800 tx=0x7f444400dcc0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 shutdown_connections 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f443806b500 0x7f443806d9c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f44540c7180 0x7f44541f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f445415de20 0x7f44540c5440 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 --2- 172.21.15.99:0/4058328110 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f44540ce360 0x7f4454157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 >> 172.21.15.99:0/4058328110 conn(0x7f445406f6e0 msgr2=0x7f4454152c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 shutdown_connections 2024-03-04T10:10:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.287+0000 7f445b49a640 1 -- 172.21.15.99:0/4058328110 wait complete. 2024-03-04T10:10:37.299 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T10:10:37.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:37.299 DEBUG:teuthology.orchestra.run.smithi099:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-04T10:10:37.327 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:10:37.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:37.328 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:10:37.382 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:10:37.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.523+0000 7fa7cf6c0640 1 Processor -- start 2024-03-04T10:10:37.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.523+0000 7fa7cf6c0640 1 -- start start 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cf6c0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7c80cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cf6c0640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c815f020 con 0x7fa7c8070de0 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cf6c0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c80cd010 con 0x7fa7c8154a80 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cf6c0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c80cfff0 con 0x7fa7c80cf650 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cdc5d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa7c8070de0 0x7fa7c8071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58874/0 (socket says 172.21.15.99:58874) 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cdc5d640 1 -- 172.21.15.99:0/980333121 learned_addr learned my addr 172.21.15.99:0/980333121 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764099935 0 0) 0x7fa7c0001490 con 0x7fa7c8070de0 2024-03-04T10:10:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7ccc5b640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7c80cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b4003520 con 0x7fa7c8070de0 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994877360 0 0) 0x7fa7b4003520 con 0x7fa7c8070de0 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 msgr2=0x7fa7c80cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7c80cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 >> v1:172.21.15.162:6789/0 conn(0x7fa7c8154a80 legacy=0x7fa7c815e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c80cdaa0 con 0x7fa7c8070de0 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fa7c0001bc0 con 0x7fa7c8070de0 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7ccc5b640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7c80cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 >> v1:172.21.15.99:6789/0 conn(0x7fa7c8070de0 legacy=0x7fa7c8071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7b4004800 0x7fa7b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b400ba90 con 0x7fa7b40039e0 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b400bc40 con 0x7fa7b4004800 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b400be40 con 0x7fa7c80cf650 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.524+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7ccc5b640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7b4004800 0x7fa7b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7cdc5d640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7cd45c640 1 -- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 msgr2=0x7fa7b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c80cf650 0x7fa7b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7cd45c640 1 -- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7b4004800 msgr2=0x7fa7b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7b4004800 0x7fa7b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.525+0000 7fa7cd45c640 1 -- 172.21.15.99:0/980333121 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b4003520 con 0x7fa7b40039e0 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7b4003e00 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4001750 tx=0x7fa7b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ab35d1b60b7fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4001370 con 0x7fa7b40039e0 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c40052d0 con 0x7fa7b40039e0 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7bffff640 1 -- 172.21.15.99:0/980333121 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4003830 con 0x7fa7b40039e0 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/980333121 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 msgr2=0x7fa7b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/980333121 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7b4003e00 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4001750 tx=0x7fa7b4005830 comp rx=0 tx=0).stop 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cdc5d640 1 -- 172.21.15.99:0/980333121 reap_dead start 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/980333121 shutdown_connections 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/980333121 >> 172.21.15.99:0/980333121 conn(0x7fa7c806f5f0 msgr2=0x7fa7c806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/980333121 shutdown_connections 2024-03-04T10:10:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.526+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/980333121 wait complete. 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.527+0000 7fa7cf6c0640 1 Processor -- start 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.527+0000 7fa7cf6c0640 1 -- start start 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cf6c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7c80c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cf6c0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c8153a50 0x7fa7c80b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cf6c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c80b6480 0x7fa7c80ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cf6c0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c80ce340 con 0x7fa7b40039e0 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cf6c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c80ce060 con 0x7fa7c80b6480 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cf6c0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c80ce620 con 0x7fa7c8153a50 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7c80c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7c80c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37798/0 (socket says 172.21.15.99:37798) 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 -- 172.21.15.99:0/2615711463 learned_addr learned my addr 172.21.15.99:0/2615711463 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 -- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c8153a50 msgr2=0x7fa7c80b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c8153a50 0x7fa7c80b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 -- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c80b6480 msgr2=0x7fa7c80ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c80b6480 0x7fa7c80ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.528+0000 7fa7cd45c640 1 -- 172.21.15.99:0/2615711463 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c80c55f0 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.529+0000 7fa7cd45c640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7c80c8940 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4005040 tx=0x7fa7c4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.529+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4041070 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.529+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c4005480 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.529+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c80c58d0 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.529+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4001370 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.529+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c80c5e90 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.530+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c8074f60 con 0x7fa7b40039e0 2024-03-04T10:10:37.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.530+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa7c4044070 con 0x7fa7b40039e0 2024-03-04T10:10:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.531+0000 7fa7bdffb640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.531+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6303+0+0 (secure 0 0 0) 0x7fa7c40abe30 con 0x7fa7b40039e0 2024-03-04T10:10:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.531+0000 7fa7ccc5b640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.532+0000 7fa7ccc5b640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8006a90 tx=0x7fa7b8006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.535+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa7c407aaf0 con 0x7fa7b40039e0 2024-03-04T10:10:37.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.744+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa7c80c8e80 con 0x7fa7b40039e0 2024-03-04T10:10:37.746 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:10:37.744+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:10:37.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.746+0000 7fa7bdffb640 1 -- 172.21.15.99:0/2615711463 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa7c407a3f0 con 0x7fa7b40039e0 2024-03-04T10:10:37.747 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:10:37.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa7ac06b4d0 msgr2=0x7fa7ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8006a90 tx=0x7fa7b8006a20 comp rx=0 tx=0).stop 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 msgr2=0x7fa7c80c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7c80c8940 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4005040 tx=0x7fa7c4001120 comp rx=0 tx=0).stop 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 shutdown_connections 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa7ac06b4d0 0x7fa7ac06d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa7c80b6480 0x7fa7c80ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa7c8153a50 0x7fa7c80b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 --2- 172.21.15.99:0/2615711463 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa7b40039e0 0x7fa7c80c8940 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 >> 172.21.15.99:0/2615711463 conn(0x7fa7c806f5f0 msgr2=0x7fa7c815e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 shutdown_connections 2024-03-04T10:10:37.751 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.749+0000 7fa7cf6c0640 1 -- 172.21.15.99:0/2615711463 wait complete. 2024-03-04T10:10:37.752 INFO:tasks.ceph.mds.c.smithi099.stderr: -621> 2024-03-04T10:10:31.742+0000 7f23292e4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T10:10:37.752 INFO:tasks.ceph.mds.c.smithi099.stderr: -186> 2024-03-04T10:10:35.641+0000 7f23252dc640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/subdir) 2024-03-04T10:10:37.762 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-03-04T10:10:37.776 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:10:37.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.862+0000 7f020b0ca640 1 Processor -- start 2024-03-04T10:10:37.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.862+0000 7f020b0ca640 1 -- start start 2024-03-04T10:10:37.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.863+0000 7f020b0ca640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f02040b5b80 0x7f02040b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.863+0000 7f020b0ca640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f020415e1e0 con 0x7f020415de00 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.863+0000 7f020b0ca640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02040d1c10 con 0x7f02040b2e60 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.863+0000 7f020b0ca640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02040b98c0 con 0x7f02040b5b80 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.863+0000 7f0208e66640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f020415de00 0x7f02040b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58888/0 (socket says 172.21.15.99:58888) 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.863+0000 7f0208e66640 1 -- 172.21.15.99:0/21928758 learned_addr learned my addr 172.21.15.99:0/21928758 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780475027 0 0) 0x7f020415e1e0 con 0x7f020415de00 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ec003520 con 0x7f020415de00 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 --2- 172.21.15.99:0/21928758 >> v2:172.21.15.162:3300/0 conn(0x7f02040b5b80 0x7f02040b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063322192 0 0) 0x7f01ec003520 con 0x7f020415de00 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 >> v2:172.21.15.162:3300/0 conn(0x7f02040b5b80 msgr2=0x7f02040b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 --2- 172.21.15.99:0/21928758 >> v2:172.21.15.162:3300/0 conn(0x7f02040b5b80 0x7f02040b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 >> v1:172.21.15.162:6789/0 conn(0x7f02040b2e60 legacy=0x7f02040b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040d1ef0 con 0x7f020415de00 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f01f4001670 con 0x7f020415de00 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 >> v1:172.21.15.99:6789/0 conn(0x7f020415de00 legacy=0x7f02040b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01ec0039e0 0x7f01ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec004800 0x7f01ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01ec0051c0 0x7f01ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02040b98c0 con 0x7f01ec0051c0 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec00c090 con 0x7f01ec0039e0 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec00c290 con 0x7f01ec004800 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01ec0051c0 0x7f01ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0208e66640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01ec0039e0 0x7f01ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0203fff640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec004800 0x7f01ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 -- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec004800 msgr2=0x7f01ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec004800 0x7f01ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 -- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01ec0039e0 msgr2=0x7f01ec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f01ec0039e0 0x7f01ec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.864+0000 7f0209667640 1 -- 172.21.15.99:0/21928758 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec003520 con 0x7f01ec0051c0 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.865+0000 7f0209667640 1 -- 172.21.15.99:0/21928758 reap_dead start 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.865+0000 7f0209667640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01ec0051c0 0x7f01ec00b9c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009f90 tx=0x7f01ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0d4e8e150ec1de server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.865+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc009c10 con 0x7f01ec0051c0 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.865+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01fc02cc20 con 0x7f01ec0051c0 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.865+0000 7f02037fe640 1 -- 172.21.15.99:0/21928758 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc003990 con 0x7f01ec0051c0 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 -- 172.21.15.99:0/21928758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01ec0051c0 msgr2=0x7f01ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01ec0051c0 0x7f01ec00b9c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f01fc009f90 tx=0x7f01ec005830 comp rx=0 tx=0).stop 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 -- 172.21.15.99:0/21928758 shutdown_connections 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 --2- 172.21.15.99:0/21928758 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f01ec0051c0 0x7f01ec00b9c0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.870 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 -- 172.21.15.99:0/21928758 >> 172.21.15.99:0/21928758 conn(0x7f02040d0470 msgr2=0x7f02040d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:37.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 -- 172.21.15.99:0/21928758 shutdown_connections 2024-03-04T10:10:37.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.866+0000 7f020b0ca640 1 -- 172.21.15.99:0/21928758 wait complete. 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.867+0000 7f020b0ca640 1 Processor -- start 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.867+0000 7f020b0ca640 1 -- start start 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f020b0ca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec0051c0 0x7f020414d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f020b0ca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 0x7f020415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f020b0ca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0204160c10 0x7f020414d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f020b0ca640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02040d2820 con 0x7f0204160660 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f020b0ca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02040d24b0 con 0x7f0204160c10 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f020b0ca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02040b9380 con 0x7f01ec0051c0 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 0x7f020415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 0x7f020415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37828/0 (socket says 172.21.15.99:37828) 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 -- 172.21.15.99:0/1594971046 learned_addr learned my addr 172.21.15.99:0/1594971046 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0209667640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0204160c10 0x7f020414d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0208e66640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec0051c0 0x7f020414d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 -- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec0051c0 msgr2=0x7f020414d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec0051c0 0x7f020414d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 -- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0204160c10 msgr2=0x7f020414d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0204160c10 0x7f020414d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0203fff640 1 -- 172.21.15.99:0/1594971046 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020415f5c0 con 0x7f0204160660 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0208e66640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec0051c0 0x7f020414d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.868+0000 7f0209667640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0204160c10 0x7f020414d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.869+0000 7f0203fff640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 0x7f020415ed90 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f01f800f5f0 tx=0x7f01f800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.869+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f80052e0 con 0x7f0204160660 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.869+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f8018070 con 0x7f0204160660 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.869+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f020415f8a0 con 0x7f0204160660 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.869+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8003810 con 0x7f0204160660 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.869+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f020415fe60 con 0x7f0204160660 2024-03-04T10:10:37.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.870+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0204074f60 con 0x7f0204160660 2024-03-04T10:10:37.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.870+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f01f80039b0 con 0x7f0204160660 2024-03-04T10:10:37.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.870+0000 7f02017fa640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01f006b4d0 0x7f01f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:37.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.871+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f01f808a240 con 0x7f0204160660 2024-03-04T10:10:37.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.871+0000 7f0208e66640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01f006b4d0 0x7f01f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:37.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.871+0000 7f0208e66640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01f006b4d0 0x7f01f006d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f01f4007c70 tx=0x7f01f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:37.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:37.875+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f01f8058ea0 con 0x7f0204160660 2024-03-04T10:10:38.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.115+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) v1 -- 0x7f02040b5620 con 0x7f0204160660 2024-03-04T10:10:38.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.747+0000 7f02017fa640 1 -- 172.21.15.99:0/1594971046 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 nothing to do: rank is not damaged v89) v1 ==== 107+0+0 (secure 0 0 0) 0x7f01f80587a0 con 0x7f0204160660 2024-03-04T10:10:38.749 INFO:teuthology.orchestra.run.smithi099.stderr:nothing to do: rank is not damaged 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01f006b4d0 msgr2=0x7f01f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01f006b4d0 0x7f01f006d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f01f4007c70 tx=0x7f01f4009040 comp rx=0 tx=0).stop 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 msgr2=0x7f020415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 0x7f020415ed90 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f01f800f5f0 tx=0x7f01f800fab0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 shutdown_connections 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f01f006b4d0 0x7f01f006d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0204160c10 0x7f020414d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f0204160660 0x7f020415ed90 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 --2- 172.21.15.99:0/1594971046 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f01ec0051c0 0x7f020414d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 >> 172.21.15.99:0/1594971046 conn(0x7f02040d0470 msgr2=0x7f02040c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 shutdown_connections 2024-03-04T10:10:38.752 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.750+0000 7f020b0ca640 1 -- 172.21.15.99:0/1594971046 wait complete. 2024-03-04T10:10:38.764 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-03-04T10:10:38.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.846+0000 7faacc43df40 1 Processor -- start 2024-03-04T10:10:38.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.846+0000 7faacc43df40 1 -- start start 2024-03-04T10:10:38.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faacc43df40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 0x555c50886110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faacc43df40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c508895a0 0x555c50887d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faacc43df40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c5088f5f0 0x555c50889bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faacc43df40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c50802ab0 con 0x555c507b7c90 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faacc43df40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c50760b30 con 0x555c5088f5f0 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faacc43df40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c50802d50 con 0x555c508895a0 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 0x555c50886110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 0x555c50886110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37830/0 (socket says 172.21.15.99:37830) 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 -- 172.21.15.99:0/309722536 learned_addr learned my addr 172.21.15.99:0/309722536 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9f24640 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c5088f5f0 0x555c50889bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac8f22640 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c508895a0 0x555c50887d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 -- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c508895a0 msgr2=0x555c50887d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c508895a0 0x555c50887d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 -- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c5088f5f0 msgr2=0x555c50889bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c5088f5f0 0x555c50889bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 -- 172.21.15.99:0/309722536 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c50888c30 con 0x555c507b7c90 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faac9723640 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 0x555c50886110 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ae698bc66f98f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faabb7fe640 1 -- 172.21.15.99:0/309722536 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac0057070 con 0x555c507b7c90 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faabb7fe640 1 -- 172.21.15.99:0/309722536 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac0061070 con 0x555c507b7c90 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.847+0000 7faabb7fe640 1 -- 172.21.15.99:0/309722536 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac0052960 con 0x555c507b7c90 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 -- 172.21.15.99:0/309722536 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 msgr2=0x555c50886110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 0x555c50886110 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac004c000 comp rx=0 tx=0).stop 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 -- 172.21.15.99:0/309722536 shutdown_connections 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c5088f5f0 0x555c50889bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c508895a0 0x555c50887d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 --2- 172.21.15.99:0/309722536 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c507b7c90 0x555c50886110 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 -- 172.21.15.99:0/309722536 >> 172.21.15.99:0/309722536 conn(0x555c507b9400 msgr2=0x555c507e4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:38.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 -- 172.21.15.99:0/309722536 shutdown_connections 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 -- 172.21.15.99:0/309722536 wait complete. 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.848+0000 7faacc43df40 1 Processor -- start 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 -- start start 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c507b7c90 0x555c5088bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 0x555c509212c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c5088f5f0 0x555c5087fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c5088f2d0 con 0x555c508895a0 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c507e77d0 con 0x555c507b7c90 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faacc43df40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c507e3c60 con 0x555c5088f5f0 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faac8f22640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 0x555c509212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faac8f22640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 0x555c509212c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37846/0 (socket says 172.21.15.99:37846) 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 learned_addr learned my addr 172.21.15.99:0/2253332164 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faac9f24640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c5088f5f0 0x555c5087fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.849+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c507b7c90 0x555c5088bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c5088f5f0 msgr2=0x555c5087fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x555c5088f5f0 0x555c5087fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c507b7c90 msgr2=0x555c5088bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c507b7c90 0x555c5088bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c50923180 con 0x555c508895a0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555c507b7c90 0x555c5088bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 0x555c509212c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7faabc001c10 tx=0x7faabc00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faaba7fc640 1 -- 172.21.15.99:0/2253332164 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc00f650 con 0x555c508895a0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faaba7fc640 1 -- 172.21.15.99:0/2253332164 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc00f7f0 con 0x555c508895a0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555c50886c30 con 0x555c508895a0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faaba7fc640 1 -- 172.21.15.99:0/2253332164 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc00fbe0 con 0x555c508895a0 2024-03-04T10:10:38.855 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.850+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c509233f0 con 0x555c508895a0 2024-03-04T10:10:38.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.851+0000 7faaba7fc640 1 -- 172.21.15.99:0/2253332164 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7faabc017400 con 0x555c508895a0 2024-03-04T10:10:38.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faaba7fc640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaac069eb0 0x7faaac06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faaba7fc640 1 -- 172.21.15.99:0/2253332164 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6389+0+0 (secure 0 0 0) 0x7faabc0898c0 con 0x555c508895a0 2024-03-04T10:10:38.856 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaac069eb0 0x7faaac06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.857 INFO:teuthology.orchestra.run.smithi099.stdout:Importing pool 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555c50887670 0x555c5092d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5088c4d0 con 0x555c50887670 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faac9f24640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555c50887670 0x555c5092d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.852+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaac069eb0 0x7faaac06c370 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faac0060000 tx=0x7faac004f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.853+0000 7faac9f24640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555c50887670 0x555c5092d660 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.854+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 601.00000000 [create] v97'43 uv16 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.854+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:2 13.10 13:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c507b8070 con 0x555c50887670 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.855+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== osd_op_reply(2 601.00000000 [delete] v97'44 uv44 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.855+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:3 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5092dba0 con 0x555c50887670 2024-03-04T10:10:38.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.857+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 3 ==== osd_op_reply(3 601.00000000 [create] v97'45 uv45 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.859 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-03-04T10:10:38.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.857+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:4 13.10 13:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5092df40 con 0x555c50887670 2024-03-04T10:10:38.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.860+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v97'46 uv46 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.860+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:5 13.10 13:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937940 con 0x555c50887670 2024-03-04T10:10:38.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.863+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v97'47 uv47 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.864 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.863+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:6 13.10 13:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50936930 con 0x555c50887670 2024-03-04T10:10:38.866 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.865+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v97'48 uv48 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.867 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.865+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:7 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50936d20 con 0x555c50887670 2024-03-04T10:10:38.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.868+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 7 ==== osd_op_reply(7 602.00000000 [create] v97'49 uv22 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.868+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:8 13.10 13:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937110 con 0x555c50887670 2024-03-04T10:10:38.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.871+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 8 ==== osd_op_reply(8 602.00000000 [delete] v97'50 uv50 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.871+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:9 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937110 con 0x555c50887670 2024-03-04T10:10:38.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.873+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 9 ==== osd_op_reply(9 602.00000000 [create] v97'51 uv51 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.875 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-03-04T10:10:38.875 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.873+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:10 13.10 13:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.876+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v97'52 uv52 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.876+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:11 13.10 13:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.879+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v97'53 uv53 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.879+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:12 13.10 13:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.884+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v97'54 uv54 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.884+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:13 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.885+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 13 ==== osd_op_reply(13 10000000000.00000000 [create] v97'55 uv55 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.887 INFO:teuthology.orchestra.run.smithi099.stdout:Write #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-03-04T10:10:38.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.885+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:14 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.888+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 14 ==== osd_op_reply(14 10000000000.00000000 [setxattr (30)] v97'56 uv56 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.889 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.888+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:15 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.891+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 15 ==== osd_op_reply(15 10000000000.00000000 [setxattr (62)] v97'57 uv57 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.892 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.891+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:16 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509370c0 con 0x555c50887670 2024-03-04T10:10:38.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.894+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 16 ==== osd_op_reply(16 10000000000.00000000 [omap-set-header] v97'58 uv58 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.895 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.894+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:17 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509374c0 con 0x555c50887670 2024-03-04T10:10:38.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.896+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 17 ==== osd_op_reply(17 10000000000.00000000 [omap-set-vals] v97'59 uv59 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.898 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.896+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:18 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.900 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.899+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 18 ==== osd_op_reply(18 600.00000000 [create] v97'60 uv35 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.901 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.899+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:19 13.10 13:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.902+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 19 ==== osd_op_reply(19 600.00000000 [delete] v97'61 uv61 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.902+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:20 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.903+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 20 ==== osd_op_reply(20 600.00000000 [create] v97'62 uv62 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.904 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-03-04T10:10:38.904 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.903+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:21 13.10 13:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.906+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v97'63 uv63 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.906+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:22 13.10 13:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.909+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v97'64 uv64 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.909+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:23 13.10 13:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.912+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v97'65 uv65 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.912+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:24 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.914+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 24 ==== osd_op_reply(24 603.00000000 [create] v97'66 uv41 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.914+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:25 13.10 13:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.918+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 25 ==== osd_op_reply(25 603.00000000 [delete] v97'67 uv67 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.918+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:26 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.920+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 26 ==== osd_op_reply(26 603.00000000 [create] v97'68 uv68 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.922 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-03-04T10:10:38.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.920+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:27 13.10 13:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.926+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v97'69 uv69 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.926+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:28 13.10 13:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.927+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v97'70 uv70 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.927+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:29 13.10 13:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c50887670 2024-03-04T10:10:38.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.929+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.1 v2:172.21.15.99:6824/3369025248 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v97'71 uv71 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c50887670 2024-03-04T10:10:38.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.929+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x555c509381f0 0x555c5093a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.930+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:30 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50937370 con 0x555c509381f0 2024-03-04T10:10:38.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.930+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x555c509381f0 0x555c5093a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.931 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.930+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x555c509381f0 0x555c5093a5e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.931+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(30 1.00000000.inode [create] v97'46 uv25 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.933 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.931+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:31 13.14 13:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093ab20 con 0x555c509381f0 2024-03-04T10:10:38.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.933+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== osd_op_reply(31 1.00000000.inode [delete] v97'47 uv47 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.934 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.933+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:32 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093af60 con 0x555c509381f0 2024-03-04T10:10:38.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.934+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 3 ==== osd_op_reply(32 1.00000000.inode [create] v97'48 uv48 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.935 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-03-04T10:10:38.935 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.934+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:33 13.14 13:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093b820 con 0x555c509381f0 2024-03-04T10:10:38.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.935+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v97'49 uv49 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.936 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.935+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:34 13.14 13:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093bbc0 con 0x555c509381f0 2024-03-04T10:10:38.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.936+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v97'50 uv50 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.937 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.936+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:35 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093bf60 con 0x555c509381f0 2024-03-04T10:10:38.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.937+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 6 ==== osd_op_reply(35 200.00000000 [create] v97'51 uv44 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.938 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.937+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:36 13.14 13:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c300 con 0x555c509381f0 2024-03-04T10:10:38.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.938+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 7 ==== osd_op_reply(36 200.00000000 [delete] v97'52 uv52 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.939 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.938+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:37 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.940 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.939+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 8 ==== osd_op_reply(37 200.00000000 [create] v97'53 uv53 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.940 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-03-04T10:10:38.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.939+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:38 13.14 13:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.941 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.940+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v97'54 uv54 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.940+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:39 13.14 13:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.941+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v97'55 uv55 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.943 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.941+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:40 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.943+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 11 ==== osd_op_reply(40 200.00000001 [create] v97'56 uv45 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.944 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.943+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:41 13.14 13:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.944+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 12 ==== osd_op_reply(41 200.00000001 [delete] v97'57 uv57 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.944+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:42 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.945+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 13 ==== osd_op_reply(42 200.00000001 [create] v97'58 uv58 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.946 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-03-04T10:10:38.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.945+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:43 13.14 13:2e2fa760:::200.00000001:head [write 0~18075 in=18075b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.946+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 14 ==== osd_op_reply(43 200.00000001 [write 0~18075] v97'59 uv59 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.947 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.946+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:44 13.14 13:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.947+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v97'60 uv60 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.948 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.947+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:45 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.948+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 16 ==== osd_op_reply(45 606.00000000 [create] v97'61 uv41 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.949 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.948+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:46 13.14 13:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.949+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 17 ==== osd_op_reply(46 606.00000000 [delete] v97'62 uv62 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.950 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.949+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:47 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.951 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.950+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 18 ==== osd_op_reply(47 606.00000000 [create] v97'63 uv63 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.951 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-03-04T10:10:38.952 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.950+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:48 13.14 13:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.951+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v97'64 uv64 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.951+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:49 13.14 13:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.952+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v97'65 uv65 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.953 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.952+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:50 13.14 13:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093c6a0 con 0x555c509381f0 2024-03-04T10:10:38.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.955+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.0 v2:172.21.15.99:6800/224245114 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v97'66 uv66 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509381f0 2024-03-04T10:10:38.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.955+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555c5093cfa0 0x555c5093f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.955+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:51 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093f8d0 con 0x555c5093cfa0 2024-03-04T10:10:38.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.956+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555c5093cfa0 0x555c5093f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.956+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555c5093cfa0 0x555c5093f390 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.957+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(51 607.00000000 [create] v97'9 uv8 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:38.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.957+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:52 13.1c 13:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5093fd50 con 0x555c5093cfa0 2024-03-04T10:10:38.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.959+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== osd_op_reply(52 607.00000000 [delete] v97'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:38.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.959+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:53 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50940230 con 0x555c5093cfa0 2024-03-04T10:10:38.961 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-03-04T10:10:38.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.960+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 3 ==== osd_op_reply(53 607.00000000 [create] v97'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:38.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.960+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:54 13.1c 13:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50941ae0 con 0x555c5093cfa0 2024-03-04T10:10:38.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.961+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 4 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v97'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:38.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.961+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:55 13.1c 13:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50941e80 con 0x555c5093cfa0 2024-03-04T10:10:38.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.962+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 5 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v97'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:38.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.962+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:56 13.1c 13:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50940ad0 con 0x555c5093cfa0 2024-03-04T10:10:38.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.963+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 6 ==== osd_op_reply(56 607.00000000 [omap-set-header] v97'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:38.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.963+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x555c509413d0 0x555c50944230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.963+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:57 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50944770 con 0x555c509413d0 2024-03-04T10:10:38.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.963+0000 7faac9f24640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x555c509413d0 0x555c50944230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.963+0000 7faac9f24640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x555c509413d0 0x555c50944230 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.965+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v97'10 uv9 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.965+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:58 13.6 13:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50944770 con 0x555c509413d0 2024-03-04T10:10:38.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.966+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v97'11 uv11 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.966+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:59 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50944790 con 0x555c509413d0 2024-03-04T10:10:38.969 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-03-04T10:10:38.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.967+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v97'12 uv12 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.967+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:60 13.6 13:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50944bd0 con 0x555c509413d0 2024-03-04T10:10:38.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.969+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v97'13 uv13 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.969+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555c509450e0 0x555c509474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:38.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.969+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:61 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947a60 con 0x555c509450e0 2024-03-04T10:10:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.969+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555c509450e0 0x555c509474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.969+0000 7faac9723640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555c509450e0 0x555c509474d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.971+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(61 608.00000000 [create] v97'9 uv8 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.971+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:62 13.e 13:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947b20 con 0x555c509450e0 2024-03-04T10:10:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.972+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== osd_op_reply(62 608.00000000 [delete] v97'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.972+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:63 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:38.974 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-03-04T10:10:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.973+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 3 ==== osd_op_reply(63 608.00000000 [create] v97'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.973+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:64 13.e 13:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50948f20 con 0x555c509450e0 2024-03-04T10:10:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.974+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 4 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v97'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.974+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:65 13.e 13:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50948f20 con 0x555c509450e0 2024-03-04T10:10:38.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.975+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 5 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v97'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:38.977 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.975+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:66 13.e 13:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:38.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.976+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 6 ==== osd_op_reply(66 608.00000000 [omap-set-header] v97'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:38.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.976+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:67 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.978+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 5 ==== osd_op_reply(67 604.00000000 [create] v97'15 uv9 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.978+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:68 13.15 13:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.979+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 6 ==== osd_op_reply(68 604.00000000 [delete] v97'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.979+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:69 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.981 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-03-04T10:10:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.980+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 7 ==== osd_op_reply(69 604.00000000 [create] v97'17 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.980+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:70 13.15 13:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50948f20 con 0x555c509413d0 2024-03-04T10:10:38.984 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.983+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 8 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v97'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.985 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.983+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:71 13.15 13:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50948f20 con 0x555c509413d0 2024-03-04T10:10:38.987 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.986+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 9 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v97'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.988 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.986+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:72 13.15 13:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.992+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 10 ==== osd_op_reply(72 604.00000000 [omap-set-header] v97'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.992+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:73 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.993+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 11 ==== osd_op_reply(73 500.00000000 [create] v97'21 uv14 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.993+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:74 13.15 13:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.996+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 12 ==== osd_op_reply(74 500.00000000 [delete] v97'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.996+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:75 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:38.999 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-03-04T10:10:38.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.998+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 13 ==== osd_op_reply(75 500.00000000 [create] v97'23 uv23 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:38.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:38.998+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:76 13.15 13:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.003+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 14 ==== osd_op_reply(76 500.00000000 [write 0~90] v97'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.005 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.003+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:77 13.15 13:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.004+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 15 ==== osd_op_reply(77 500.00000000 [omap-set-header] v97'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.006 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.004+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:78 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.006+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 16 ==== osd_op_reply(78 mds_snaptable [create] v97'7 uv6 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.007 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.006+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:79 13.d 13:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.007+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 17 ==== osd_op_reply(79 mds_snaptable [delete] v97'8 uv8 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.008 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.007+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:80 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.011 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-03-04T10:10:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.010+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 18 ==== osd_op_reply(80 mds_snaptable [create] v97'9 uv9 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.011 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.010+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:81 13.d 13:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.012+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 19 ==== osd_op_reply(81 mds_snaptable [write 0~70] v97'10 uv10 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.012+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:82 13.d 13:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509413d0 2024-03-04T10:10:39.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.015+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 <== osd.4 v2:172.21.15.162:6800/3079837853 20 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v97'11 uv11 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac400d940 con 0x555c509413d0 2024-03-04T10:10:39.016 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.015+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:83 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.016+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 7 ==== osd_op_reply(83 605.00000000 [create] v97'9 uv8 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.017 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.016+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:84 13.1d 13:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.018 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.017+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 8 ==== osd_op_reply(84 605.00000000 [delete] v97'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.018 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.017+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:85 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.021 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-03-04T10:10:39.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.020+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 9 ==== osd_op_reply(85 605.00000000 [create] v97'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.022 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.020+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:86 13.1d 13:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50948f20 con 0x555c509450e0 2024-03-04T10:10:39.024 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.022+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 10 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v97'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.024 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.023+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:87 13.1d 13:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50948f20 con 0x555c509450e0 2024-03-04T10:10:39.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.025+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 11 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v97'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.026 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.025+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:88 13.1d 13:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.029 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.028+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 12 ==== osd_op_reply(88 605.00000000 [omap-set-header] v97'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.029 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.028+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:89 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.029+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 13 ==== osd_op_reply(89 mds0_inotable [create] v97'14 uv8 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.029+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:90 13.13 13:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.032+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 14 ==== osd_op_reply(90 mds0_inotable [delete] v97'15 uv15 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.032+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:91 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c50947f10 con 0x555c509450e0 2024-03-04T10:10:39.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.035+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 15 ==== osd_op_reply(91 mds0_inotable [create] v97'16 uv16 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.037 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-03-04T10:10:39.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.035+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:92 13.13 13:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509406e0 con 0x555c509450e0 2024-03-04T10:10:39.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.038+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 16 ==== osd_op_reply(92 mds0_inotable [write 0~34] v97'17 uv17 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.038+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:93 13.13 13:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509406e0 con 0x555c509450e0 2024-03-04T10:10:39.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.041+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 17 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v97'18 uv18 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.041+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:94 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509406e0 con 0x555c509450e0 2024-03-04T10:10:39.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.044+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 18 ==== osd_op_reply(94 100.00000000 [create] v97'19 uv13 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.044+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:95 13.13 13:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509406e0 con 0x555c509450e0 2024-03-04T10:10:39.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.046+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 19 ==== osd_op_reply(95 100.00000000 [delete] v97'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.046+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:96 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509406e0 con 0x555c509450e0 2024-03-04T10:10:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.049+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 20 ==== osd_op_reply(96 100.00000000 [create] v97'21 uv21 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.052 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-03-04T10:10:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.049+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:97 13.13 13:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c509406e0 con 0x555c509450e0 2024-03-04T10:10:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.052+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 21 ==== osd_op_reply(97 100.00000000 [omap-set-header] v97'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.052+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:98 13.13 13:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094ad60 con 0x555c509450e0 2024-03-04T10:10:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.054+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 22 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v97'23 uv23 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.055+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x555c50948a40 0x555c5094ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.055+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:99 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b370 con 0x555c50948a40 2024-03-04T10:10:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.055+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x555c50948a40 0x555c5094ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.055+0000 7faac8f22640 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x555c50948a40 0x555c5094ae30 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.056+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(99 mds0_sessionmap [create] v97'9 uv7 ondisk = -17 ((17) File exists)) v8 ==== 159+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c50948a40 2024-03-04T10:10:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.056+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:100 13.b 13:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b370 con 0x555c50948a40 2024-03-04T10:10:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.057+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v97'10 uv10 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c50948a40 2024-03-04T10:10:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.057+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:101 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b370 con 0x555c50948a40 2024-03-04T10:10:39.061 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-03-04T10:10:39.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.060+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.5 v2:172.21.15.162:6808/1764948489 3 ==== osd_op_reply(101 mds0_sessionmap [create] v97'11 uv11 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c50948a40 2024-03-04T10:10:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.060+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:102 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c50948a40 2024-03-04T10:10:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.061+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.5 v2:172.21.15.162:6808/1764948489 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v97'12 uv12 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c50948a40 2024-03-04T10:10:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.061+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:103 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c509450e0 2024-03-04T10:10:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.062+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 23 ==== osd_op_reply(103 609.00000000 [create] v97'9 uv8 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.062+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:104 13.7 13:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c509450e0 2024-03-04T10:10:39.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.065+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 24 ==== osd_op_reply(104 609.00000000 [delete] v97'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.066 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.065+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:105 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c509450e0 2024-03-04T10:10:39.069 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-03-04T10:10:39.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.068+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 25 ==== osd_op_reply(105 609.00000000 [create] v97'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.068+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:106 13.7 13:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094c770 con 0x555c509450e0 2024-03-04T10:10:39.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.070+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 26 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v97'12 uv12 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.072 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.070+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:107 13.7 13:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094c770 con 0x555c509450e0 2024-03-04T10:10:39.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.072+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 27 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v97'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.072+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:108 13.7 13:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c509450e0 2024-03-04T10:10:39.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.074+0000 7faac9723640 1 -- 172.21.15.99:0/2253332164 <== osd.7 v2:172.21.15.162:6816/2710319333 28 ==== osd_op_reply(108 609.00000000 [omap-set-header] v97'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faac005a070 con 0x555c509450e0 2024-03-04T10:10:39.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.074+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:109 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.077+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 7 ==== osd_op_reply(109 400.00000000 [create] v97'7 uv6 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.077+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:110 13.f 13:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.079 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.078+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 8 ==== osd_op_reply(110 400.00000000 [delete] v97'8 uv8 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.080 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.078+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:111 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.083 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-03-04T10:10:39.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.082+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 9 ==== osd_op_reply(111 400.00000000 [create] v97'9 uv9 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.082+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:112 13.f 13:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.083+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 10 ==== osd_op_reply(112 400.00000000 [write 0~22] v97'10 uv10 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.083+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:113 13.f 13:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.084+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 11 ==== osd_op_reply(113 400.00000000 [omap-set-header] v97'11 uv11 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.084+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:114 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.086+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 12 ==== osd_op_reply(114 100.00000000.inode [create] v97'20 uv11 ondisk = -17 ((17) File exists)) v8 ==== 162+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.086+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:115 13.1f 13:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.088+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 13 ==== osd_op_reply(115 100.00000000.inode [delete] v97'21 uv21 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.088+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:116 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.092 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-03-04T10:10:39.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.091+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 14 ==== osd_op_reply(116 100.00000000.inode [create] v97'22 uv22 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.091+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:117 13.1f 13:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.094+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 15 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v97'23 uv23 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.094+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:118 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.094+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 16 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v97'24 uv24 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.094+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:119 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.098+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 17 ==== osd_op_reply(119 1.00000000 [create] v97'25 uv18 ondisk = -17 ((17) File exists)) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.098+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:120 13.1f 13:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.098+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 18 ==== osd_op_reply(120 1.00000000 [delete] v97'26 uv26 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.099+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:121 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.102 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-03-04T10:10:39.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.101+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 19 ==== osd_op_reply(121 1.00000000 [create] v97'27 uv27 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.101+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:122 13.1f 13:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094c770 con 0x555c5093cfa0 2024-03-04T10:10:39.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.104+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 20 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v97'28 uv28 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.104+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:123 13.1f 13:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094c770 con 0x555c5093cfa0 2024-03-04T10:10:39.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.105+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 21 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v97'29 uv29 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.105+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:124 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094b760 con 0x555c5093cfa0 2024-03-04T10:10:39.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.107+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 22 ==== osd_op_reply(124 1.00000000 [omap-set-header] v97'30 uv30 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.108+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:125 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97) v9 -- 0x555c5094bc10 con 0x555c5093cfa0 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faac8f22640 1 -- 172.21.15.99:0/2253332164 <== osd.6 v2:172.21.15.162:6824/1448511037 23 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v97'31 uv31 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7faabc054490 con 0x555c5093cfa0 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x555c509381f0 msgr2=0x555c5093a5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x555c509381f0 0x555c5093a5e0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555c50887670 msgr2=0x555c5092d660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x555c50887670 0x555c5092d660 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x555c509413d0 msgr2=0x555c50944230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x555c509413d0 0x555c50944230 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x555c50948a40 msgr2=0x555c5094ae30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x555c50948a40 0x555c5094ae30 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555c5093cfa0 msgr2=0x555c5093f390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x555c5093cfa0 0x555c5093f390 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555c509450e0 msgr2=0x555c509474d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x555c509450e0 0x555c509474d0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 reap_dead start 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 reap_dead start 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faac9f24640 1 -- 172.21.15.99:0/2253332164 reap_dead start 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaac069eb0 msgr2=0x7faaac06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaac069eb0 0x7faaac06c370 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faac0060000 tx=0x7faac004f000 comp rx=0 tx=0).stop 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 msgr2=0x555c509212c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 0x555c509212c0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7faabc001c10 tx=0x7faabc00b6c0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 shutdown_connections 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faaac069eb0 0x7faaac06c370 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.114 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 --2- 172.21.15.99:0/2253332164 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x555c508895a0 0x555c509212c0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 >> 172.21.15.99:0/2253332164 conn(0x555c507b9400 msgr2=0x555c507e4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:39.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 shutdown_connections 2024-03-04T10:10:39.115 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.110+0000 7faacc43df40 1 -- 172.21.15.99:0/2253332164 wait complete. 2024-03-04T10:10:39.116 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 600.00000000 2024-03-04T10:10:39.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 Processor -- start 2024-03-04T10:10:39.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 -- start start 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 0x55e6d91b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d91a5050 0x55e6d91b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6d9107110 0x55e6d91b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d9083b30 con 0x7f595c0030f0 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d9087a90 con 0x55e6d9107110 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.196+0000 7f5976a3bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6d9125ab0 con 0x55e6d91a5050 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 0x55e6d91b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 0x55e6d91b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37864/0 (socket says 172.21.15.99:37864) 2024-03-04T10:10:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 -- 172.21.15.99:0/1089973797 learned_addr learned my addr 172.21.15.99:0/1089973797 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 -- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d91a5050 msgr2=0x55e6d91b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d91a5050 0x55e6d91b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5974504640 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6d9107110 0x55e6d91b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 -- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6d9107110 msgr2=0x55e6d91b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6d9107110 0x55e6d91b0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 -- 172.21.15.99:0/1089973797 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6d910c850 con 0x7f595c0030f0 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5973d03640 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 0x55e6d91b3660 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f596400f770 tx=0x7f596400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729aea27e69be55d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5972500640 1 -- 172.21.15.99:0/1089973797 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964005070 con 0x7f595c0030f0 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5972500640 1 -- 172.21.15.99:0/1089973797 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596400db50 con 0x7f595c0030f0 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.197+0000 7f5972500640 1 -- 172.21.15.99:0/1089973797 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964019400 con 0x7f595c0030f0 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 -- 172.21.15.99:0/1089973797 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 msgr2=0x55e6d91b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 0x55e6d91b3660 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f596400f770 tx=0x7f596400fc30 comp rx=0 tx=0).stop 2024-03-04T10:10:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 -- 172.21.15.99:0/1089973797 shutdown_connections 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e6d9107110 0x55e6d91b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d91a5050 0x55e6d91b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/1089973797 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f595c0030f0 0x55e6d91b3660 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 -- 172.21.15.99:0/1089973797 >> 172.21.15.99:0/1089973797 conn(0x55e6d90dc400 msgr2=0x55e6d9122f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 -- 172.21.15.99:0/1089973797 shutdown_connections 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 -- 172.21.15.99:0/1089973797 wait complete. 2024-03-04T10:10:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 Processor -- start 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.198+0000 7f5976a3bf40 1 -- start start 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5976a3bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d9107110 0x55e6d91a9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5976a3bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6d91a5050 0x55e6d91a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5976a3bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f595c0030f0 0x55e6d9121d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5976a3bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d90db670 con 0x55e6d91a5050 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5976a3bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d91b1dc0 con 0x7f595c0030f0 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5976a3bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e6d91b20a0 con 0x55e6d9107110 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6d91a5050 0x55e6d91a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6d91a5050 0x55e6d91a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37870/0 (socket says 172.21.15.99:37870) 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 -- 172.21.15.99:0/3339590685 learned_addr learned my addr 172.21.15.99:0/3339590685 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973d03640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d9107110 0x55e6d91a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5974504640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f595c0030f0 0x55e6d9121d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 -- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d9107110 msgr2=0x55e6d91a9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55e6d9107110 0x55e6d91a9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 -- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f595c0030f0 msgr2=0x55e6d9121d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f595c0030f0 0x55e6d9121d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 -- 172.21.15.99:0/3339590685 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6d91a1e60 con 0x55e6d91a5050 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.199+0000 7f5973502640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6d91a5050 0x55e6d91a1920 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f5968000c00 tx=0x7f5968053900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.200+0000 7f59714fe640 1 -- 172.21.15.99:0/3339590685 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968060020 con 0x55e6d91a5050 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.200+0000 7f59714fe640 1 -- 172.21.15.99:0/3339590685 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596804ab30 con 0x55e6d91a5050 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.200+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e6d9121260 con 0x55e6d91a5050 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.200+0000 7f59714fe640 1 -- 172.21.15.99:0/3339590685 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968001c20 con 0x55e6d91a5050 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.200+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6d91246b0 con 0x55e6d91a5050 2024-03-04T10:10:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.201+0000 7f59714fe640 1 -- 172.21.15.99:0/3339590685 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5968063020 con 0x55e6d91a5050 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.201+0000 7f59714fe640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5960069ee0 0x7f596006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.201+0000 7f59714fe640 1 -- 172.21.15.99:0/3339590685 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f59680ca130 con 0x55e6d91a5050 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.201+0000 7f5973d03640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5960069ee0 0x7f596006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.202+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e6d91b14c0 0x55e6d91acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.202+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e97) v9 -- 0x55e6d91ad0d0 con 0x55e6d91b14c0 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.202+0000 7f5974504640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e6d91b14c0 0x55e6d91acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.202+0000 7f5973d03640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5960069ee0 0x7f596006c3a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f596400fc70 tx=0x7f5964004000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.202+0000 7f5974504640 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e6d91b14c0 0x55e6d91acb90 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5974504640 1 -- 172.21.15.99:0/3339590685 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 600.00000000 [delete] v97'72 uv72 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f596c011040 con 0x55e6d91b14c0 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e6d91b14c0 msgr2=0x55e6d91acb90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55e6d91b14c0 0x55e6d91acb90 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5960069ee0 msgr2=0x7f596006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5960069ee0 0x7f596006c3a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f596400fc70 tx=0x7f5964004000 comp rx=0 tx=0).stop 2024-03-04T10:10:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6d91a5050 msgr2=0x55e6d91a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 --2- 172.21.15.99:0/3339590685 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55e6d91a5050 0x55e6d91a1920 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f5968000c00 tx=0x7f5968053900 comp rx=0 tx=0).stop 2024-03-04T10:10:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5974504640 1 -- 172.21.15.99:0/3339590685 reap_dead start 2024-03-04T10:10:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 shutdown_connections 2024-03-04T10:10:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.203+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 >> 172.21.15.99:0/3339590685 conn(0x55e6d90dc400 msgr2=0x55e6d91261a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.204+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 shutdown_connections 2024-03-04T10:10:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.204+0000 7f5976a3bf40 1 -- 172.21.15.99:0/3339590685 wait complete. 2024-03-04T10:10:39.215 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:10:39.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.351+0000 7fdb6388c640 1 Processor -- start 2024-03-04T10:10:39.352 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.351+0000 7fdb6388c640 1 -- start start 2024-03-04T10:10:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb6388c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdb5c0cec40 0x7fdb5c0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb6388c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0d2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb6388c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0ca970 con 0x7fdb5c0cf020 2024-03-04T10:10:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb6388c640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c161d90 con 0x7fdb5c0cec40 2024-03-04T10:10:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb6388c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c162070 con 0x7fdb5c15de40 2024-03-04T10:10:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb61628640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdb5c0cec40 0x7fdb5c0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb60e27640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdb5c0cf020 0x7fdb5c15d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58900/0 (socket says 172.21.15.99:58900) 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb60e27640 1 -- 172.21.15.99:0/1548768116 learned_addr learned my addr 172.21.15.99:0/1548768116 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb61e29640 1 --2- 172.21.15.99:0/1548768116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0d2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb61628640 1 -- 172.21.15.99:0/1548768116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 msgr2=0x7fdb5c0d2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb61628640 1 --2- 172.21.15.99:0/1548768116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0d2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb61628640 1 -- 172.21.15.99:0/1548768116 >> v1:172.21.15.99:6789/0 conn(0x7fdb5c0cf020 legacy=0x7fdb5c15d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb61628640 1 -- 172.21.15.99:0/1548768116 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c162350 con 0x7fdb5c0cec40 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.352+0000 7fdb53fff640 1 -- 172.21.15.99:0/1548768116 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627424274 0 0) 0x7fdb5c0ca970 con 0x7fdb5c0cf020 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb61628640 1 --2- 172.21.15.99:0/1548768116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0b6ea0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fdb580097a0 tx=0x7fdb5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353f95a1edf3e0bf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb53fff640 1 -- 172.21.15.99:0/1548768116 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58035020 con 0x7fdb5c0cec40 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb53fff640 1 -- 172.21.15.99:0/1548768116 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb58031830 con 0x7fdb5c0cec40 2024-03-04T10:10:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb53fff640 1 -- 172.21.15.99:0/1548768116 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58031c00 con 0x7fdb5c0cec40 2024-03-04T10:10:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb6388c640 1 -- 172.21.15.99:0/1548768116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cec40 msgr2=0x7fdb5c0b6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1548768116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0b6ea0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fdb580097a0 tx=0x7fdb5802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb6388c640 1 -- 172.21.15.99:0/1548768116 shutdown_connections 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1548768116 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0d2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.353+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1548768116 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0b6ea0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.354+0000 7fdb6388c640 1 -- 172.21.15.99:0/1548768116 >> 172.21.15.99:0/1548768116 conn(0x7fdb5c0d0580 msgr2=0x7fdb5c0d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.354+0000 7fdb6388c640 1 -- 172.21.15.99:0/1548768116 shutdown_connections 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.354+0000 7fdb6388c640 1 -- 172.21.15.99:0/1548768116 wait complete. 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 Processor -- start 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 -- start start 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 0x7fdb5c151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d3300 con 0x7fdb5c0cf020 2024-03-04T10:10:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d1bf0 con 0x7fdb5c0cec40 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.355+0000 7fdb6388c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d3480 con 0x7fdb5c15de40 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 0x7fdb5c151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 0x7fdb5c151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37888/0 (socket says 172.21.15.99:37888) 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 -- 172.21.15.99:0/1684408768 learned_addr learned my addr 172.21.15.99:0/1684408768 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb61628640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb61e29640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 -- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 msgr2=0x7fdb5c0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 -- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5c0cec40 msgr2=0x7fdb5c0c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 -- 172.21.15.99:0/1684408768 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0c7740 con 0x7fdb5c0cf020 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb60e27640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 0x7fdb5c151ef0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c4a0 tx=0x7fdb4c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c017070 con 0x7fdb5c0cf020 2024-03-04T10:10:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c005ce0 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0c7a20 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0072b0 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.356+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c152430 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.357+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c074f60 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.358+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdb4c01a070 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.358+0000 7fdb51ffb640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb4006b500 0x7fdb4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.358+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6389+0+0 (secure 0 0 0) 0x7fdb4c089700 con 0x7fdb5c0cf020 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.358+0000 7fdb61628640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb4006b500 0x7fdb4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.359+0000 7fdb61628640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb4006b500 0x7fdb4006d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdb58036000 tx=0x7fdb5803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:39.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.362+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdb4c057d10 con 0x7fdb5c0cf020 2024-03-04T10:10:39.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.567+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fdb5c0d1930 con 0x7fdb5c0cf020 2024-03-04T10:10:39.976 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.974+0000 7fdb51ffb640 1 -- 172.21.15.99:0/1684408768 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v90) v1 ==== 162+0+0 (secure 0 0 0) 0x7fdb4c05bdb0 con 0x7fdb5c0cf020 2024-03-04T10:10:39.976 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:10:39.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb4006b500 msgr2=0x7fdb4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb4006b500 0x7fdb4006d9c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fdb58036000 tx=0x7fdb5803c040 comp rx=0 tx=0).stop 2024-03-04T10:10:39.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 msgr2=0x7fdb5c151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:39.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 0x7fdb5c151ef0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c4a0 tx=0x7fdb4c00c960 comp rx=0 tx=0).stop 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 shutdown_connections 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdb4006b500 0x7fdb4006d9c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdb5c15de40 0x7fdb5c0ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdb5c0cf020 0x7fdb5c151ef0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 --2- 172.21.15.99:0/1684408768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdb5c0cec40 0x7fdb5c0c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 >> 172.21.15.99:0/1684408768 conn(0x7fdb5c0d0580 msgr2=0x7fdb5c0d3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 shutdown_connections 2024-03-04T10:10:39.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:39.977+0000 7fdb6388c640 1 -- 172.21.15.99:0/1684408768 wait complete. 2024-03-04T10:10:39.991 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:40.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.090+0000 7f8a244ea640 1 Processor -- start 2024-03-04T10:10:40.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.090+0000 7f8a244ea640 1 -- start start 2024-03-04T10:10:40.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.090+0000 7f8a244ea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8a1c0b0700 0x7f8a1c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a244ea640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0ccad0 con 0x7f8a1c0b47c0 2024-03-04T10:10:40.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a244ea640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0cfab0 con 0x7f8a1c15b960 2024-03-04T10:10:40.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a244ea640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cf860 con 0x7f8a1c0b0700 2024-03-04T10:10:40.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a22a87640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8a1c0b47c0 0x7f8a1c0cbec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:58912/0 (socket says 172.21.15.99:58912) 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a22a87640 1 -- 172.21.15.99:0/2765697643 learned_addr learned my addr 172.21.15.99:0/2765697643 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699429781 0 0) 0x7f8a1c0ccad0 con 0x7f8a1c0b47c0 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21a85640 1 --2- 172.21.15.99:0/2765697643 >> v2:172.21.15.162:3300/0 conn(0x7f8a1c0b0700 0x7f8a1c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a08003520 con 0x7f8a1c0b47c0 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2741665689 0 0) 0x7f8a08003520 con 0x7f8a1c0b47c0 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 >> v2:172.21.15.162:3300/0 conn(0x7f8a1c0b0700 msgr2=0x7f8a1c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 --2- 172.21.15.99:0/2765697643 >> v2:172.21.15.162:3300/0 conn(0x7f8a1c0b0700 0x7f8a1c0b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 >> v1:172.21.15.162:6789/0 conn(0x7f8a1c15b960 legacy=0x7f8a1c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0cfd90 con 0x7f8a1c0b47c0 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8a180019f0 con 0x7f8a1c0b47c0 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 >> v1:172.21.15.99:6789/0 conn(0x7f8a1c0b47c0 legacy=0x7f8a1c0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a08004800 0x7f8a08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a080051c0 0x7f8a0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cf860 con 0x7f8a080039b0 2024-03-04T10:10:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0800c160 con 0x7f8a080051c0 2024-03-04T10:10:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0800c360 con 0x7f8a08004800 2024-03-04T10:10:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a22286640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a21a85640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a08004800 0x7f8a08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.091+0000 7f8a22a87640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a080051c0 0x7f8a0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22286640 1 -- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a08004800 msgr2=0x7f8a08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22286640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a08004800 0x7f8a08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22286640 1 -- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a080051c0 msgr2=0x7f8a0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22286640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a080051c0 0x7f8a0800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22286640 1 -- 172.21.15.99:0/2765697643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a08003520 con 0x7f8a080039b0 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22a87640 1 -- 172.21.15.99:0/2765697643 reap_dead start 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a22286640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a08003dd0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8a14001750 tx=0x7f8a08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ce5483c4e5b82a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14001370 con 0x7f8a080039b0 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a140052d0 con 0x7f8a080039b0 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.092+0000 7f8a21284640 1 -- 172.21.15.99:0/2765697643 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14003830 con 0x7f8a080039b0 2024-03-04T10:10:40.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 -- 172.21.15.99:0/2765697643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 msgr2=0x7f8a08003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a08003dd0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f8a14001750 tx=0x7f8a08005830 comp rx=0 tx=0).stop 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 -- 172.21.15.99:0/2765697643 shutdown_connections 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 --2- 172.21.15.99:0/2765697643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a08003dd0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 -- 172.21.15.99:0/2765697643 >> 172.21.15.99:0/2765697643 conn(0x7f8a1c06f5f0 msgr2=0x7f8a1c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 -- 172.21.15.99:0/2765697643 shutdown_connections 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.093+0000 7f8a244ea640 1 -- 172.21.15.99:0/2765697643 wait complete. 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.094+0000 7f8a244ea640 1 Processor -- start 2024-03-04T10:10:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.094+0000 7f8a244ea640 1 -- start start 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a244ea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a1c0ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a244ea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a1c0c9630 0x7f8a1c0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a244ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a1c14b7f0 0x7f8a1c14bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a244ea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0b0b90 con 0x7f8a080039b0 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a244ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0d0100 con 0x7f8a1c14b7f0 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a244ea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0b0d10 con 0x7f8a1c0c9630 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a1c0ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a1c0ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:37894/0 (socket says 172.21.15.99:37894) 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 -- 172.21.15.99:0/261667328 learned_addr learned my addr 172.21.15.99:0/261667328 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a21a85640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a1c0c9630 0x7f8a1c0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22a87640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a1c14b7f0 0x7f8a1c14bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 -- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a1c0c9630 msgr2=0x7f8a1c0c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a1c0c9630 0x7f8a1c0c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 -- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a1c14b7f0 msgr2=0x7f8a1c14bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a1c14b7f0 0x7f8a1c14bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 -- 172.21.15.99:0/261667328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0c76d0 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22a87640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a1c14b7f0 0x7f8a1c14bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.095+0000 7f8a22286640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a1c0ca6f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f8a14005040 tx=0x7f8a14001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.096+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14041020 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.096+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14005480 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.096+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1c0c79b0 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.096+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14007a30 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.096+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1c0c7f70 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.097+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1c074f60 con 0x7f8a080039b0 2024-03-04T10:10:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.097+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8a14044020 con 0x7f8a080039b0 2024-03-04T10:10:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.097+0000 7f8a12ffd640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a0006b4d0 0x7f8a0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.098+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6389+0+0 (secure 0 0 0) 0x7f8a140abea0 con 0x7f8a080039b0 2024-03-04T10:10:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.098+0000 7f8a21a85640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a0006b4d0 0x7f8a0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.098+0000 7f8a21a85640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a0006b4d0 0x7f8a0006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c15b0c0 tx=0x7f8a0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.102+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8a1404b030 con 0x7f8a080039b0 2024-03-04T10:10:40.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.308+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a1c0b1380 con 0x7f8a080039b0 2024-03-04T10:10:40.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.308+0000 7f8a12ffd640 1 -- 172.21.15.99:0/261667328 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f8a1407a9e0 con 0x7f8a080039b0 2024-03-04T10:10:40.310 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:40.311 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":90,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:39.974086+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":97,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8445":{"gid":8445,"name":"c","rank":0,"incarnation":90,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/2735900597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":2735900597},{"type":"v1","addr":"172.21.15.99:6835","nonce":2735900597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:40.311 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 90 2024-03-04T10:10:40.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.310+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a0006b4d0 msgr2=0x7f8a0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.310+0000 7f8a244ea640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a0006b4d0 0x7f8a0006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c15b0c0 tx=0x7f8a0c008040 comp rx=0 tx=0).stop 2024-03-04T10:10:40.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.310+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 msgr2=0x7f8a1c0ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:40.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.310+0000 7f8a244ea640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a1c0ca6f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f8a14005040 tx=0x7f8a14001120 comp rx=0 tx=0).stop 2024-03-04T10:10:40.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 shutdown_connections 2024-03-04T10:10:40.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8a0006b4d0 0x7f8a0006d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a1c14b7f0 0x7f8a1c14bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8a1c0c9630 0x7f8a1c0c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 --2- 172.21.15.99:0/261667328 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8a080039b0 0x7f8a1c0ca6f0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:40.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 >> 172.21.15.99:0/261667328 conn(0x7f8a1c06f5f0 msgr2=0x7f8a1c0b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:40.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 shutdown_connections 2024-03-04T10:10:40.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:40.311+0000 7f8a244ea640 1 -- 172.21.15.99:0/261667328 wait complete. 2024-03-04T10:10:40.323 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:10:40.539 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:40.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:10:42.995 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:42.994+0000 7f1ed5302640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-03-04T10:10:42.995 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:42.994+0000 7f1ed5302640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-03-04T10:10:42.996 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:10:42.994+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:10:42.996 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:10:42.994+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-03-04T10:10:43.002 INFO:tasks.ceph.mds.c.smithi099.stderr: -16> 2024-03-04T10:10:42.994+0000 7f1ed5302640 -1 log_channel(cluster) log [ERR] : dir 0x600 object missing on disk; some files may be lost (~mds0/stray0) 2024-03-04T10:10:43.002 INFO:tasks.ceph.mds.c.smithi099.stderr: -14> 2024-03-04T10:10:42.994+0000 7f1ed5302640 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x600 is fatal because it is a system directory for this rank 2024-03-04T10:10:43.027 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:10:45.323 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:45.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.432+0000 7f08f73f1640 1 Processor -- start 2024-03-04T10:10:45.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.433+0000 7f08f73f1640 1 -- start start 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.433+0000 7f08f73f1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f08f015d660 0x7f08f015e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.433+0000 7f08f73f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f08f0071a60 0x7f08f0150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.433+0000 7f08f73f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f015da40 con 0x7f08f00aeb20 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.433+0000 7f08f73f1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f015cbc0 con 0x7f08f015d660 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.433+0000 7f08f73f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f015ebb0 con 0x7f08f0071a60 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f08f015d660 0x7f08f015e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f08f015d660 0x7f08f015e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57690/0 (socket says 172.21.15.99:57690) 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 -- 172.21.15.99:0/2305861678 learned_addr learned my addr 172.21.15.99:0/2305861678 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f598e640 1 --2- 172.21.15.99:0/2305861678 >> v2:172.21.15.162:3300/0 conn(0x7f08f0071a60 0x7f08f0150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 -- 172.21.15.99:0/2305861678 >> v2:172.21.15.162:3300/0 conn(0x7f08f0071a60 msgr2=0x7f08f0150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 --2- 172.21.15.99:0/2305861678 >> v2:172.21.15.162:3300/0 conn(0x7f08f0071a60 0x7f08f0150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 -- 172.21.15.99:0/2305861678 >> v1:172.21.15.162:6789/0 conn(0x7f08f00aeb20 legacy=0x7f08f0071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f518d640 1 -- 172.21.15.99:0/2305861678 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00af090 con 0x7f08f015d660 2024-03-04T10:10:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.434+0000 7f08f598e640 1 --2- 172.21.15.99:0/2305861678 >> v2:172.21.15.162:3300/0 conn(0x7f08f0071a60 0x7f08f0150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08f518d640 1 --2- 172.21.15.99:0/2305861678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f015e670 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f08f00d0450 tx=0x7f08ec02deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4800c9c937a70aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08e7fff640 1 -- 172.21.15.99:0/2305861678 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec034070 con 0x7f08f015d660 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08e7fff640 1 -- 172.21.15.99:0/2305861678 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec02ec60 con 0x7f08f015d660 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08e7fff640 1 -- 172.21.15.99:0/2305861678 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec038620 con 0x7f08f015d660 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08f73f1640 1 -- 172.21.15.99:0/2305861678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 msgr2=0x7f08f015e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08f73f1640 1 --2- 172.21.15.99:0/2305861678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f015e670 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f08f00d0450 tx=0x7f08ec02deb0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08f73f1640 1 -- 172.21.15.99:0/2305861678 shutdown_connections 2024-03-04T10:10:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08f73f1640 1 --2- 172.21.15.99:0/2305861678 >> v2:172.21.15.162:3300/0 conn(0x7f08f0071a60 0x7f08f0150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.435+0000 7f08f73f1640 1 --2- 172.21.15.99:0/2305861678 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f015e670 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.436+0000 7f08f73f1640 1 -- 172.21.15.99:0/2305861678 >> 172.21.15.99:0/2305861678 conn(0x7f08f006acd0 msgr2=0x7f08f006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.436+0000 7f08f73f1640 1 -- 172.21.15.99:0/2305861678 shutdown_connections 2024-03-04T10:10:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.436+0000 7f08f73f1640 1 -- 172.21.15.99:0/2305861678 wait complete. 2024-03-04T10:10:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.437+0000 7f08f73f1640 1 Processor -- start 2024-03-04T10:10:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.437+0000 7f08f73f1640 1 -- start start 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.437+0000 7f08f73f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08f0071a60 0x7f08f01f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f73f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08f00aeb20 0x7f08f0153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f73f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f0153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f73f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f015afb0 con 0x7f08f015d660 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f73f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f015acd0 con 0x7f08f00aeb20 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f73f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f015a660 con 0x7f08f0071a60 2024-03-04T10:10:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f0153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f0153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57694/0 (socket says 172.21.15.99:57694) 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 -- 172.21.15.99:0/1880237890 learned_addr learned my addr 172.21.15.99:0/1880237890 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f518d640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08f0071a60 0x7f08f01f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f498c640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08f00aeb20 0x7f08f0153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 -- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08f0071a60 msgr2=0x7f08f01f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08f0071a60 0x7f08f01f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 -- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08f00aeb20 msgr2=0x7f08f0153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08f00aeb20 0x7f08f0153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 -- 172.21.15.99:0/1880237890 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f00a8600 con 0x7f08f015d660 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f498c640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08f00aeb20 0x7f08f0153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f598e640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f0153f60 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f08e8007e20 tx=0x7f08e800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e801a070 con 0x7f08f015d660 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e8003b90 con 0x7f08f015d660 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f00a88e0 con 0x7f08f015d660 2024-03-04T10:10:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.438+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e800de80 con 0x7f08f015d660 2024-03-04T10:10:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.439+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f00a8ea0 con 0x7f08f015d660 2024-03-04T10:10:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.440+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f0074f60 con 0x7f08f015d660 2024-03-04T10:10:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.440+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f08e801d070 con 0x7f08f015d660 2024-03-04T10:10:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.440+0000 7f08e5ffb640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08d406b4d0 0x7f08d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.440+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f08e808a800 con 0x7f08f015d660 2024-03-04T10:10:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.440+0000 7f08f518d640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08d406b4d0 0x7f08d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.441+0000 7f08f518d640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08d406b4d0 0x7f08d406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f08ec039000 tx=0x7f08ec034000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:45.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.445+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f08e8010070 con 0x7f08f015d660 2024-03-04T10:10:45.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.648+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08f00a9070 con 0x7f08f015d660 2024-03-04T10:10:45.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.648+0000 7f08e5ffb640 1 -- 172.21.15.99:0/1880237890 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f08e8059380 con 0x7f08f015d660 2024-03-04T10:10:45.651 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:45.651 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":95,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/334550293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":334550293},{"type":"v1","addr":"172.21.15.99:6835","nonce":334550293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:42.995218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:45.652 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 95 2024-03-04T10:10:45.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.650+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08d406b4d0 msgr2=0x7f08d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.650+0000 7f08f73f1640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08d406b4d0 0x7f08d406d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f08ec039000 tx=0x7f08ec034000 comp rx=0 tx=0).stop 2024-03-04T10:10:45.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.650+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 msgr2=0x7f08f0153f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.650+0000 7f08f73f1640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f0153f60 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f08e8007e20 tx=0x7f08e800d4a0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 shutdown_connections 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f08d406b4d0 0x7f08d406d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f08f015d660 0x7f08f0153f60 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08f00aeb20 0x7f08f0153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 --2- 172.21.15.99:0/1880237890 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f08f0071a60 0x7f08f01f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 >> 172.21.15.99:0/1880237890 conn(0x7f08f006acd0 msgr2=0x7f08f0071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 shutdown_connections 2024-03-04T10:10:45.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.651+0000 7f08f73f1640 1 -- 172.21.15.99:0/1880237890 wait complete. 2024-03-04T10:10:45.664 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-03-04T10:10:45.664 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:45.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.761+0000 7f32e5685640 1 Processor -- start 2024-03-04T10:10:45.764 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.761+0000 7f32e5685640 1 -- start start 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32e5685640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f32e015de00 0x7f32e00b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32e5685640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f32e00b9380 0x7f32e00b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32e5685640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e00b70d0 0x7f32e00b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32e5685640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e00d2820 con 0x7f32e015de00 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32e5685640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e015e1e0 con 0x7f32e00b9380 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32e5685640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00d1c10 con 0x7f32e00b70d0 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32deffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f32e015de00 0x7f32e00b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32deffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f32e015de00 0x7f32e00b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57708/0 (socket says 172.21.15.99:57708) 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32deffd640 1 -- 172.21.15.99:0/690520106 learned_addr learned my addr 172.21.15.99:0/690520106 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32df7fe640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e00b70d0 0x7f32e00b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.766 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32deffd640 1 -- 172.21.15.99:0/690520106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e00b70d0 msgr2=0x7f32e00b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32deffd640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e00b70d0 0x7f32e00b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.762+0000 7f32de7fc640 1 --2- 172.21.15.99:0/690520106 >> v2:172.21.15.162:3300/0 conn(0x7f32e00b9380 0x7f32e00b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32deffd640 1 -- 172.21.15.99:0/690520106 >> v2:172.21.15.162:3300/0 conn(0x7f32e00b9380 msgr2=0x7f32e00b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32deffd640 1 --2- 172.21.15.99:0/690520106 >> v2:172.21.15.162:3300/0 conn(0x7f32e00b9380 0x7f32e00b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32deffd640 1 -- 172.21.15.99:0/690520106 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00d1ef0 con 0x7f32e015de00 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32df7fe640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e00b70d0 0x7f32e00b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32de7fc640 1 --2- 172.21.15.99:0/690520106 >> v2:172.21.15.162:3300/0 conn(0x7f32e00b9380 0x7f32e00b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32deffd640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e015de00 0x7f32e00b2e60 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f32e0161830 tx=0x7f32d002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6222ffd04741454 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32ddffb640 1 -- 172.21.15.99:0/690520106 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d002ca70 con 0x7f32e015de00 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32ddffb640 1 -- 172.21.15.99:0/690520106 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d002cc10 con 0x7f32e015de00 2024-03-04T10:10:45.767 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.763+0000 7f32ddffb640 1 -- 172.21.15.99:0/690520106 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d00343f0 con 0x7f32e015de00 2024-03-04T10:10:45.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 -- 172.21.15.99:0/690520106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e015de00 msgr2=0x7f32e00b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e015de00 0x7f32e00b2e60 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f32e0161830 tx=0x7f32d002bed0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 -- 172.21.15.99:0/690520106 shutdown_connections 2024-03-04T10:10:45.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e00b70d0 0x7f32e00b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 --2- 172.21.15.99:0/690520106 >> v2:172.21.15.162:3300/0 conn(0x7f32e00b9380 0x7f32e00b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 --2- 172.21.15.99:0/690520106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e015de00 0x7f32e00b2e60 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 -- 172.21.15.99:0/690520106 >> 172.21.15.99:0/690520106 conn(0x7f32e00d0470 msgr2=0x7f32e00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 -- 172.21.15.99:0/690520106 shutdown_connections 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.764+0000 7f32e5685640 1 -- 172.21.15.99:0/690520106 wait complete. 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.765+0000 7f32e5685640 1 Processor -- start 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.765+0000 7f32e5685640 1 -- start start 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32e5685640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e00b70d0 0x7f32e00ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32e5685640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 0x7f32e00cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32e5685640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e015de00 0x7f32e014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32e5685640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b97e0 con 0x7f32e00b9380 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32e5685640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00d24b0 con 0x7f32e00b70d0 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32e5685640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b59b0 con 0x7f32e015de00 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 0x7f32e00cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 0x7f32e00cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57712/0 (socket says 172.21.15.99:57712) 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 -- 172.21.15.99:0/3527556637 learned_addr learned my addr 172.21.15.99:0/3527556637 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32deffd640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e00b70d0 0x7f32e00ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32df7fe640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e015de00 0x7f32e014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 -- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e015de00 msgr2=0x7f32e014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e015de00 0x7f32e014cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 -- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e00b70d0 msgr2=0x7f32e00ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e00b70d0 0x7f32e00ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32de7fc640 1 -- 172.21.15.99:0/3527556637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e014ac70 con 0x7f32e00b9380 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32deffd640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e00b70d0 0x7f32e00ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.766+0000 7f32df7fe640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e015de00 0x7f32e014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.767+0000 7f32de7fc640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 0x7f32e00cea40 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f32d400c7f0 tx=0x7f32d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.767+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4015070 con 0x7f32e00b9380 2024-03-04T10:10:45.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.767+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d4007a30 con 0x7f32e00b9380 2024-03-04T10:10:45.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.767+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4007e00 con 0x7f32e00b9380 2024-03-04T10:10:45.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.767+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e014d8b0 con 0x7f32e00b9380 2024-03-04T10:10:45.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.767+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e014b210 con 0x7f32e00b9380 2024-03-04T10:10:45.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.768+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e0074f60 con 0x7f32e00b9380 2024-03-04T10:10:45.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.768+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f32d4018070 con 0x7f32e00b9380 2024-03-04T10:10:45.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.768+0000 7f32c77fe640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32cc06b4d0 0x7f32cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:45.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.768+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f32d408a4b0 con 0x7f32e00b9380 2024-03-04T10:10:45.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.769+0000 7f32deffd640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32cc06b4d0 0x7f32cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:45.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.769+0000 7f32deffd640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32cc06b4d0 0x7f32cc06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f32d0035000 tx=0x7f32d0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:45.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.773+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f32d4059030 con 0x7f32e00b9380 2024-03-04T10:10:45.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.976+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32e00b7ed0 con 0x7f32e00b9380 2024-03-04T10:10:45.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.976+0000 7f32c77fe640 1 -- 172.21.15.99:0/3527556637 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4929 (secure 0 0 0) 0x7f32d4058930 con 0x7f32e00b9380 2024-03-04T10:10:45.979 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:45.979 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":95,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/334550293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":334550293},{"type":"v1","addr":"172.21.15.99:6835","nonce":334550293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:42.995218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:45.979 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 95 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32cc06b4d0 msgr2=0x7f32cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32cc06b4d0 0x7f32cc06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f32d0035000 tx=0x7f32d0030000 comp rx=0 tx=0).stop 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 msgr2=0x7f32e00cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 0x7f32e00cea40 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f32d400c7f0 tx=0x7f32d400cbf0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 shutdown_connections 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f32cc06b4d0 0x7f32cc06d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f32e015de00 0x7f32e014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f32e00b9380 0x7f32e00cea40 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 --2- 172.21.15.99:0/3527556637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f32e00b70d0 0x7f32e00ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 >> 172.21.15.99:0/3527556637 conn(0x7f32e00d0470 msgr2=0x7f32e00c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 shutdown_connections 2024-03-04T10:10:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:45.978+0000 7f32e5685640 1 -- 172.21.15.99:0/3527556637 wait complete. 2024-03-04T10:10:45.992 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-04T10:10:45.992 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 Processor -- start 2024-03-04T10:10:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 -- start start 2024-03-04T10:10:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe2440cfd60 0x7fe2440cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe2440cf0d0 0x7fe2440cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2440d0140 con 0x7fe24406d3b0 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24415cbb0 con 0x7fe2440cf0d0 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.091+0000 7fe24a2c0640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24415c280 con 0x7fe2440cfd60 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe2440cf0d0 0x7fe2440cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe2440cf0d0 0x7fe2440cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57714/0 (socket says 172.21.15.99:57714) 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 -- 172.21.15.99:0/1775248277 learned_addr learned my addr 172.21.15.99:0/1775248277 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 -- 172.21.15.99:0/1775248277 >> v2:172.21.15.162:3300/0 conn(0x7fe2440cfd60 msgr2=0x7fe2440cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 --2- 172.21.15.99:0/1775248277 >> v2:172.21.15.162:3300/0 conn(0x7fe2440cfd60 0x7fe2440cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 -- 172.21.15.99:0/1775248277 >> v1:172.21.15.162:6789/0 conn(0x7fe24406d3b0 legacy=0x7fe2440ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.092+0000 7fe24885d640 1 -- 172.21.15.99:0/1775248277 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24415c560 con 0x7fe2440cf0d0 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24885d640 1 --2- 172.21.15.99:0/1775248277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cf0d0 0x7fe2440cc330 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe24415cd30 tx=0x7fe23c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565437628cb0c91f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe242ffd640 1 -- 172.21.15.99:0/1775248277 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c02ca70 con 0x7fe2440cf0d0 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe242ffd640 1 -- 172.21.15.99:0/1775248277 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c02cc10 con 0x7fe2440cf0d0 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe242ffd640 1 -- 172.21.15.99:0/1775248277 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c0343f0 con 0x7fe2440cf0d0 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1775248277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cf0d0 msgr2=0x7fe2440cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1775248277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cf0d0 0x7fe2440cc330 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fe24415cd30 tx=0x7fe23c02bed0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1775248277 shutdown_connections 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1775248277 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cf0d0 0x7fe2440cc330 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1775248277 >> v2:172.21.15.162:3300/0 conn(0x7fe2440cfd60 0x7fe2440cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1775248277 >> 172.21.15.99:0/1775248277 conn(0x7fe24406acd0 msgr2=0x7fe24406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1775248277 shutdown_connections 2024-03-04T10:10:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.093+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1775248277 wait complete. 2024-03-04T10:10:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 Processor -- start 2024-03-04T10:10:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 -- start start 2024-03-04T10:10:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe24406d3b0 0x7fe2441544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2440cf0d0 0x7fe2440b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 0x7fe2440b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24415f520 con 0x7fe2440cfd60 2024-03-04T10:10:46.100 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24415bf30 con 0x7fe24406d3b0 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.095+0000 7fe24a2c0640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24415f6a0 con 0x7fe2440cf0d0 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 0x7fe2440b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 0x7fe2440b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57722/0 (socket says 172.21.15.99:57722) 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 -- 172.21.15.99:0/1589756561 learned_addr learned my addr 172.21.15.99:0/1589756561 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe243fff640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe24406d3b0 0x7fe2441544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 -- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2440cf0d0 msgr2=0x7fe2440b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2440cf0d0 0x7fe2440b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 -- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe24406d3b0 msgr2=0x7fe2441544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe24406d3b0 0x7fe2441544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 -- 172.21.15.99:0/1589756561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244158580 con 0x7fe2440cfd60 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24885d640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 0x7fe2440b1080 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009f10 tx=0x7fe23c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c043020 con 0x7fe2440cfd60 2024-03-04T10:10:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c03e450 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c03e820 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe244157a00 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.096+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe244157fc0 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.097+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe244074f60 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.097+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe23c046020 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.098+0000 7fe240ff9640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe23006b500 0x7fe23006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.098+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7fe23c0acb50 con 0x7fe2440cfd60 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.098+0000 7fe243fff640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe23006b500 0x7fe23006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.099+0000 7fe243fff640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe23006b500 0x7fe23006d9c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe2440cd230 tx=0x7fe234008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.102+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe23c04d070 con 0x7fe2440cfd60 2024-03-04T10:10:46.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.306+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe24406b300 con 0x7fe2440cfd60 2024-03-04T10:10:46.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.307+0000 7fe240ff9640 1 -- 172.21.15.99:0/1589756561 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4929 (secure 0 0 0) 0x7fe23c07b6d0 con 0x7fe2440cfd60 2024-03-04T10:10:46.309 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:46.309 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":95,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/334550293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":334550293},{"type":"v1","addr":"172.21.15.99:6835","nonce":334550293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:42.995218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:46.310 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 95 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe23006b500 msgr2=0x7fe23006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe23006b500 0x7fe23006d9c0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe2440cd230 tx=0x7fe234008040 comp rx=0 tx=0).stop 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 msgr2=0x7fe2440b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 0x7fe2440b1080 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009f10 tx=0x7fe23c004a40 comp rx=0 tx=0).stop 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 shutdown_connections 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe23006b500 0x7fe23006d9c0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe2440cfd60 0x7fe2440b1080 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe2440cf0d0 0x7fe2440b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 --2- 172.21.15.99:0/1589756561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe24406d3b0 0x7fe2441544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.308+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 >> 172.21.15.99:0/1589756561 conn(0x7fe24406acd0 msgr2=0x7fe24415ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.309+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 shutdown_connections 2024-03-04T10:10:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.309+0000 7fe24a2c0640 1 -- 172.21.15.99:0/1589756561 wait complete. 2024-03-04T10:10:46.322 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 600.00000000' led to DAMAGED state 2024-03-04T10:10:46.322 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 1.00000000.inode' 2024-03-04T10:10:46.322 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:46.322 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:46.338 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:10:46.338 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:10:46.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:46.338 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:10:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:10:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-04T10:10:46.393 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:10:46.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.531+0000 7fa6b662b640 1 Processor -- start 2024-03-04T10:10:46.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.531+0000 7fa6b662b640 1 -- start start 2024-03-04T10:10:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6b662b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa6b015c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6b662b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa6b006b860 0x7fa6b00cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6b662b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b015cfe0 con 0x7fa6b006acd0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6b662b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b00d0240 con 0x7fa6b006b860 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6b662b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00cf630 con 0x7fa6b01597f0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6af7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa6b006acd0 0x7fa6b006b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45638/0 (socket says 172.21.15.99:45638) 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.532+0000 7fa6af7fe640 1 -- 172.21.15.99:0/111720058 learned_addr learned my addr 172.21.15.99:0/111720058 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361588928 0 0) 0x7fa6b015cfe0 con 0x7fa6b006acd0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698003520 con 0x7fa6b006acd0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa6b015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6b4bc8640 1 --2- 172.21.15.99:0/111720058 >> v2:172.21.15.162:3300/0 conn(0x7fa6b006b860 0x7fa6b00cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1514866259 0 0) 0x7fa698003520 con 0x7fa6b006acd0 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 msgr2=0x7fa6b015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa6b015c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 >> v2:172.21.15.162:3300/0 conn(0x7fa6b006b860 msgr2=0x7fa6b00cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 --2- 172.21.15.99:0/111720058 >> v2:172.21.15.162:3300/0 conn(0x7fa6b006b860 0x7fa6b00cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00cf910 con 0x7fa6b006acd0 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fa6a4001670 con 0x7fa6b006acd0 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa6b015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 >> v1:172.21.15.99:6789/0 conn(0x7fa6b006acd0 legacy=0x7fa6b006b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6980039b0 0x7fa698003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa698004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa698004d40 0x7fa6980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00d0240 con 0x7fa6980039b0 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00cf630 con 0x7fa698004d40 2024-03-04T10:10:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800ba10 con 0x7fa6b01597f0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6980039b0 0x7fa698003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6b4bc8640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa698004d40 0x7fa6980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6af7fe640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa698004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 -- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 msgr2=0x7fa698004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa698004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 -- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa698004d40 msgr2=0x7fa6980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa698004d40 0x7fa6980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6affff640 1 -- 172.21.15.99:0/111720058 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698003520 con 0x7fa6980039b0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6af7fe640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6b01597f0 0x7fa698004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.533+0000 7fa6b4bc8640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa698004d40 0x7fa6980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.534+0000 7fa6affff640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6980039b0 0x7fa698003dd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00098a0 tx=0x7fa698005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85982b717c73ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.534+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a000f040 con 0x7fa6980039b0 2024-03-04T10:10:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.534+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a0004230 con 0x7fa6980039b0 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.534+0000 7fa6aeffd640 1 -- 172.21.15.99:0/111720058 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0003610 con 0x7fa6980039b0 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b662b640 1 -- 172.21.15.99:0/111720058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6980039b0 msgr2=0x7fa698003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b662b640 1 --2- 172.21.15.99:0/111720058 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6980039b0 0x7fa698003dd0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00098a0 tx=0x7fa698005830 comp rx=0 tx=0).stop 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b4bc8640 1 -- 172.21.15.99:0/111720058 reap_dead start 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b662b640 1 -- 172.21.15.99:0/111720058 shutdown_connections 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b662b640 1 -- 172.21.15.99:0/111720058 >> 172.21.15.99:0/111720058 conn(0x7fa6b00d0710 msgr2=0x7fa6b00d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b662b640 1 -- 172.21.15.99:0/111720058 shutdown_connections 2024-03-04T10:10:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.535+0000 7fa6b662b640 1 -- 172.21.15.99:0/111720058 wait complete. 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.536+0000 7fa6b662b640 1 Processor -- start 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.536+0000 7fa6b662b640 1 -- start start 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6980039b0 0x7fa6b00c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00c6d40 0x7fa6b00b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 0x7fa6b00b0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00d1350 con 0x7fa6b00b5960 2024-03-04T10:10:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00cfed0 con 0x7fa6b00c6d40 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b015c030 con 0x7fa6980039b0 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 0x7fa6b00b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 0x7fa6b00b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57752/0 (socket says 172.21.15.99:57752) 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 -- 172.21.15.99:0/2588675568 learned_addr learned my addr 172.21.15.99:0/2588675568 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 -- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6980039b0 msgr2=0x7fa6b00c6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6980039b0 0x7fa6b00c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 -- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00c6d40 msgr2=0x7fa6b00b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00c6d40 0x7fa6b00b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 -- 172.21.15.99:0/2588675568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00c5a40 con 0x7fa6b00b5960 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b4bc8640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 0x7fa6b00b0150 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00ce230 tx=0x7fa6a8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a801a030 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8010d80 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b00c5d20 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.537+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8003bd0 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.538+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b00c62e0 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.539+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b00afe00 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.539+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa6a801d070 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.539+0000 7fa6acff9640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa69c06b4d0 0x7fa69c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.539+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7fa6a808aa30 con 0x7fa6b00b5960 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.540+0000 7fa6affff640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa69c06b4d0 0x7fa69c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.540+0000 7fa6affff640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa69c06b4d0 0x7fa69c06d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0035000 tx=0x7fa6a0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.544+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa6b00afe00 con 0x7fa6b00b5960 2024-03-04T10:10:46.754 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:46.751+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa6b0074f60 con 0x7fa6b00b5960 2024-03-04T10:10:47.009 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.006+0000 7fa6acff9640 1 -- 172.21.15.99:0/2588675568 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v96) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa6a8059640 con 0x7fa6b00b5960 2024-03-04T10:10:47.009 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:10:47.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa69c06b4d0 msgr2=0x7fa69c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.012 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa69c06b4d0 0x7fa69c06d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0035000 tx=0x7fa6a0030000 comp rx=0 tx=0).stop 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 msgr2=0x7fa6b00b0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 0x7fa6b00b0150 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00ce230 tx=0x7fa6a8010590 comp rx=0 tx=0).stop 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 shutdown_connections 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa69c06b4d0 0x7fa69c06d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6b00b5960 0x7fa6b00b0150 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6b00c6d40 0x7fa6b00b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 --2- 172.21.15.99:0/2588675568 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6980039b0 0x7fa6b00c6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 >> 172.21.15.99:0/2588675568 conn(0x7fa6b00d0710 msgr2=0x7fa6b00cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:47.013 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 shutdown_connections 2024-03-04T10:10:47.014 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.009+0000 7fa6b662b640 1 -- 172.21.15.99:0/2588675568 wait complete. 2024-03-04T10:10:47.024 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-03-04T10:10:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.123+0000 7f5b14cb2640 1 Processor -- start 2024-03-04T10:10:47.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.123+0000 7f5b14cb2640 1 -- start start 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b14cb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 0x7f5b1015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b14cb2640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1006d8b0 con 0x7f5b1015b030 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b14cb2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1006db90 con 0x7f5b1015b410 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b14cb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100b2720 con 0x7f5b1006d3b0 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0e59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5b1015b030 0x7f5b1015dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45654/0 (socket says 172.21.15.99:45654) 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0e59c640 1 -- 172.21.15.99:0/2632293728 learned_addr learned my addr 172.21.15.99:0/2632293728 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:47.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b07fff640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 0x7f5b1015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608794266 0 0) 0x7f5b1006d8b0 con 0x7f5b1015b030 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc003520 con 0x7f5b1015b030 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 461288943 0 0) 0x7f5afc003520 con 0x7f5b1015b030 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 msgr2=0x7f5b1015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 0x7f5b1015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 >> v1:172.21.15.162:6789/0 conn(0x7f5b1015b410 legacy=0x7f5b1015b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1006de70 con 0x7f5b1015b030 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5b00001670 con 0x7f5b1015b030 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 >> v1:172.21.15.99:6789/0 conn(0x7f5b1015b030 legacy=0x7f5b1015dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 0x7f5afc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5afc003f60 0x7f5afc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc004d40 0x7f5afc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100b2720 con 0x7f5afc003f60 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc00bb50 con 0x7f5afc004d40 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc00bd50 con 0x7f5b1006d3b0 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.124+0000 7f5b07fff640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5afc003f60 0x7f5afc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b0e59c640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 0x7f5afc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b0ed9d640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc004d40 0x7f5afc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b07fff640 1 -- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 msgr2=0x7f5afc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b07fff640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b1006d3b0 0x7f5afc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b07fff640 1 -- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc004d40 msgr2=0x7f5afc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b07fff640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc004d40 0x7f5afc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b07fff640 1 -- 172.21.15.99:0/2632293728 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc003520 con 0x7f5afc003f60 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b0ed9d640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc004d40 0x7f5afc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.125+0000 7f5b07fff640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5afc003f60 0x7f5afc004800 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009ac0 tx=0x7f5afc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb5ee1625702b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af8039070 con 0x7f5afc003f60 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af802c6b0 con 0x7f5afc003f60 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b0dd9b640 1 -- 172.21.15.99:0/2632293728 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af8009690 con 0x7f5afc003f60 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b14cb2640 1 -- 172.21.15.99:0/2632293728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5afc003f60 msgr2=0x7f5afc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/2632293728 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5afc003f60 0x7f5afc004800 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009ac0 tx=0x7f5afc005830 comp rx=0 tx=0).stop 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b0ed9d640 1 -- 172.21.15.99:0/2632293728 reap_dead start 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b14cb2640 1 -- 172.21.15.99:0/2632293728 shutdown_connections 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b14cb2640 1 -- 172.21.15.99:0/2632293728 >> 172.21.15.99:0/2632293728 conn(0x7f5b1006acd0 msgr2=0x7f5b1006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b14cb2640 1 -- 172.21.15.99:0/2632293728 shutdown_connections 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.126+0000 7f5b14cb2640 1 -- 172.21.15.99:0/2632293728 wait complete. 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.127+0000 7f5b14cb2640 1 Processor -- start 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.127+0000 7f5b14cb2640 1 -- start start 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b14cb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc003f60 0x7f5b1015c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b14cb2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 0x7f5b10141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b14cb2640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b10142470 0x7f5b101effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b14cb2640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006e810 con 0x7f5b1015ca40 2024-03-04T10:10:47.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b14cb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006e690 con 0x7f5afc003f60 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b14cb2640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1015c120 con 0x7f5b10142470 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b07fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 0x7f5b10141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b07fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 0x7f5b10141f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57764/0 (socket says 172.21.15.99:57764) 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b07fff640 1 -- 172.21.15.99:0/1983787893 learned_addr learned my addr 172.21.15.99:0/1983787893 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b0ed9d640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b10142470 0x7f5b101effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.128+0000 7f5b0e59c640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc003f60 0x7f5b1015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b07fff640 1 -- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b10142470 msgr2=0x7f5b101effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b07fff640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b10142470 0x7f5b101effe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b07fff640 1 -- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc003f60 msgr2=0x7f5b1015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b07fff640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc003f60 0x7f5b1015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:47.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b07fff640 1 -- 172.21.15.99:0/1983787893 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100cdc80 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b07fff640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 0x7f5b10141f30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009be0 tx=0x7f5af8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af80044c0 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af8003610 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b100cdf60 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af80039e0 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.129+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b100ce520 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.130+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b10074f60 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.130+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5af8003b80 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.130+0000 7f5b06ffd640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5af006b4d0 0x7f5af006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.131+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f5af80ae240 con 0x7f5b1015ca40 2024-03-04T10:10:47.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.131+0000 7f5b0e59c640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5af006b4d0 0x7f5af006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:47.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.131+0000 7f5b0e59c640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5af006b4d0 0x7f5af006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5b00007cb0 tx=0x7f5b00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:47.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.135+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5af807ce50 con 0x7f5b1015ca40 2024-03-04T10:10:47.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:47.375+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) v1 -- 0x7f5b10154a40 con 0x7f5b1015ca40 2024-03-04T10:10:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.015+0000 7f5b06ffd640 1 -- 172.21.15.99:0/1983787893 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v97) v1 ==== 101+0+0 (secure 0 0 0) 0x7f5af807c750 con 0x7f5b1015ca40 2024-03-04T10:10:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:repaired: restoring rank 6:0 2024-03-04T10:10:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.017+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5af006b4d0 msgr2=0x7f5af006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.017+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5af006b4d0 0x7f5af006d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5b00007cb0 tx=0x7f5b00009040 comp rx=0 tx=0).stop 2024-03-04T10:10:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.017+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 msgr2=0x7f5b10141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.017+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 0x7f5b10141f30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009be0 tx=0x7f5af8004730 comp rx=0 tx=0).stop 2024-03-04T10:10:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.017+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 shutdown_connections 2024-03-04T10:10:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5af006b4d0 0x7f5af006d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5b10142470 0x7f5b101effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5b1015ca40 0x7f5b10141f30 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 --2- 172.21.15.99:0/1983787893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5afc003f60 0x7f5b1015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 >> 172.21.15.99:0/1983787893 conn(0x7f5b1006acd0 msgr2=0x7f5b100affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 shutdown_connections 2024-03-04T10:10:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.018+0000 7f5b14cb2640 1 -- 172.21.15.99:0/1983787893 wait complete. 2024-03-04T10:10:48.032 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.071+0000 7f436e974f40 1 Processor -- start 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.071+0000 7f436e974f40 1 -- start start 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436e974f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43540030f0 0x55dbf3cc4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436e974f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55dbf3cf2530 0x55dbf3d8dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436e974f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3cc6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436e974f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf3c6ab30 con 0x55dbf3cc67a0 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436e974f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf3c6ea90 con 0x7f43540030f0 2024-03-04T10:10:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436e974f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dbf3d0dd40 con 0x55dbf3cf2530 2024-03-04T10:10:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3cc6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3cc6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57784/0 (socket says 172.21.15.99:57784) 2024-03-04T10:10:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 -- 172.21.15.99:0/3289100572 learned_addr learned my addr 172.21.15.99:0/3289100572 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436c472640 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43540030f0 0x55dbf3cc4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436bc71640 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55dbf3cf2530 0x55dbf3d8dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 -- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55dbf3cf2530 msgr2=0x55dbf3d8dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55dbf3cf2530 0x55dbf3d8dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 -- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43540030f0 msgr2=0x55dbf3cc4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43540030f0 0x55dbf3cc4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 -- 172.21.15.99:0/3289100572 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbf3d9b330 con 0x55dbf3cc67a0 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436cc73640 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3cc6c30 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x55dbf3cc8240 tx=0x7f43640029f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da5d6136e7d42cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436ac6f640 1 -- 172.21.15.99:0/3289100572 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364006e40 con 0x55dbf3cc67a0 2024-03-04T10:10:48.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436ac6f640 1 -- 172.21.15.99:0/3289100572 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364017070 con 0x55dbf3cc67a0 2024-03-04T10:10:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.072+0000 7f436ac6f640 1 -- 172.21.15.99:0/3289100572 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640117d0 con 0x55dbf3cc67a0 2024-03-04T10:10:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 -- 172.21.15.99:0/3289100572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 msgr2=0x55dbf3cc6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3cc6c30 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x55dbf3cc8240 tx=0x7f43640029f0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 -- 172.21.15.99:0/3289100572 shutdown_connections 2024-03-04T10:10:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3cc6c30 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x55dbf3cf2530 0x55dbf3d8dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 --2- 172.21.15.99:0/3289100572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f43540030f0 0x55dbf3cc4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 -- 172.21.15.99:0/3289100572 >> 172.21.15.99:0/3289100572 conn(0x55dbf3cc3400 msgr2=0x55dbf3cc3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 -- 172.21.15.99:0/3289100572 shutdown_connections 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 -- 172.21.15.99:0/3289100572 wait complete. 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 Processor -- start 2024-03-04T10:10:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.073+0000 7f436e974f40 1 -- start start 2024-03-04T10:10:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436e974f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3d8c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436e974f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf3cf2530 0x55dbf3d90130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436e974f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43540030f0 0x55dbf3d91940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436e974f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf3cf2140 con 0x55dbf3cc67a0 2024-03-04T10:10:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436e974f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dbf3cc27c0 con 0x55dbf3cf2530 2024-03-04T10:10:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436e974f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dbf3cc5160 con 0x7f43540030f0 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3d8c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3d8c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57794/0 (socket says 172.21.15.99:57794) 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 learned_addr learned my addr 172.21.15.99:0/2514944130 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf3cf2530 0x55dbf3d90130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436cc73640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43540030f0 0x55dbf3d91940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43540030f0 msgr2=0x55dbf3d91940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f43540030f0 0x55dbf3d91940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf3cf2530 msgr2=0x55dbf3d90130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf3cf2530 0x55dbf3d90130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.074+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dbf3d9aa20 con 0x55dbf3cc67a0 2024-03-04T10:10:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55dbf3cf2530 0x55dbf3d90130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3d8c840 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f435c009d40 tx=0x7f435c0107d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f4369c6d640 1 -- 172.21.15.99:0/2514944130 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c005070 con 0x55dbf3cc67a0 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f4369c6d640 1 -- 172.21.15.99:0/2514944130 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435c00e7f0 con 0x55dbf3cc67a0 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dbf3d8f930 con 0x55dbf3cc67a0 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f4369c6d640 1 -- 172.21.15.99:0/2514944130 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435c00ebc0 con 0x55dbf3cc67a0 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.075+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dbf3d9a360 con 0x55dbf3cc67a0 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.076+0000 7f4369c6d640 1 -- 172.21.15.99:0/2514944130 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f435c01a3a0 con 0x55dbf3cc67a0 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f4369c6d640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4358069eb0 0x7f435806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f4369c6d640 1 -- 172.21.15.99:0/2514944130 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f435c08c1c0 con 0x55dbf3cc67a0 2024-03-04T10:10:48.090 INFO:teuthology.orchestra.run.smithi099.stdout:Importing pool 2024-03-04T10:10:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4358069eb0 0x7f435806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55dbf3e2df40 0x55dbf3e2e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e2e860 con 0x55dbf3e2df40 2024-03-04T10:10:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f436cc73640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55dbf3e2df40 0x55dbf3e2e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f436cc73640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55dbf3e2df40 0x55dbf3e2e320 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.077+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4358069eb0 0x7f435806c370 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4360056000 tx=0x7f4360051000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.078+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 601.00000000 [create] v98'73 uv48 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.078+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:2 13.10 13:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e2ec40 con 0x55dbf3e2df40 2024-03-04T10:10:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.079+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== osd_op_reply(2 601.00000000 [delete] v98'74 uv74 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.079+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:3 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3a500 con 0x55dbf3e2df40 2024-03-04T10:10:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.082+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 3 ==== osd_op_reply(3 601.00000000 [create] v98'75 uv75 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.095 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-03-04T10:10:48.097 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.082+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:4 13.10 13:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b8b0 con 0x55dbf3e2df40 2024-03-04T10:10:48.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.085+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v98'76 uv76 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.085+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:5 13.10 13:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3bc50 con 0x55dbf3e2df40 2024-03-04T10:10:48.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.088+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v98'77 uv77 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.088+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:6 13.10 13:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3a8a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.091+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v98'78 uv78 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.098 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.091+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:7 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3ac40 con 0x55dbf3e2df40 2024-03-04T10:10:48.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.092+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 7 ==== osd_op_reply(7 602.00000000 [create] v98'79 uv54 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.092+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:8 13.10 13:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b030 con 0x55dbf3e2df40 2024-03-04T10:10:48.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.093+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 8 ==== osd_op_reply(8 602.00000000 [delete] v98'80 uv80 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.093+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:9 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b420 con 0x55dbf3e2df40 2024-03-04T10:10:48.099 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.093+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 9 ==== osd_op_reply(9 602.00000000 [create] v98'81 uv81 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.100 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-03-04T10:10:48.102 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.093+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:10 13.10 13:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b420 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.094+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v98'82 uv82 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.094+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:11 13.10 13:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.095+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v98'83 uv83 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.095+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:12 13.10 13:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.096+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v98'84 uv84 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.096+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:13 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.103 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.097+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 13 ==== osd_op_reply(13 10000000000.00000000 [create] v98'85 uv59 ondisk = -17 ((17) File exists)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.097+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:14 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.098+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v98'86 uv86 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.098+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:15 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.099+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 15 ==== osd_op_reply(15 10000000000.00000000 [create] v98'87 uv87 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.105 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-03-04T10:10:48.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.099+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:16 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.100+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v98'88 uv88 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.100+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:17 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.101+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v98'89 uv89 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.101+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:18 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3b3d0 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.102+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v98'90 uv90 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.102+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:19 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.103+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v98'91 uv91 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.103+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:20 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.104+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 20 ==== osd_op_reply(20 600.00000000 [create] v98'92 uv92 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.109 INFO:teuthology.orchestra.run.smithi099.stdout:Write #-9223372036854775808:00000000:::600.00000000:head# 2024-03-04T10:10:48.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.104+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:21 13.10 13:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3d4b0 con 0x55dbf3e2df40 2024-03-04T10:10:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.105+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 21 ==== osd_op_reply(21 600.00000000 [setxattr (30)] v98'93 uv93 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.105+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:22 13.10 13:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3d4b0 con 0x55dbf3e2df40 2024-03-04T10:10:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.106+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 22 ==== osd_op_reply(22 600.00000000 [setxattr (62)] v98'94 uv94 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.106+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:23 13.10 13:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.108+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 23 ==== osd_op_reply(23 600.00000000 [omap-set-header] v98'95 uv95 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.108+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:24 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.111+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 24 ==== osd_op_reply(24 603.00000000 [create] v98'96 uv71 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.113 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.111+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:25 13.10 13:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.113+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 25 ==== osd_op_reply(25 603.00000000 [delete] v98'97 uv97 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.113+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:26 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.116+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 26 ==== osd_op_reply(26 603.00000000 [create] v98'98 uv98 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.119 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-03-04T10:10:48.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.116+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:27 13.10 13:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3d4b0 con 0x55dbf3e2df40 2024-03-04T10:10:48.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.119+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 27 ==== osd_op_reply(27 603.00000000 [setxattr (30)] v98'99 uv99 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.119+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:28 13.10 13:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3d4b0 con 0x55dbf3e2df40 2024-03-04T10:10:48.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.121+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 28 ==== osd_op_reply(28 603.00000000 [setxattr (62)] v98'100 uv100 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.121+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:29 13.10 13:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3c4a0 con 0x55dbf3e2df40 2024-03-04T10:10:48.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.124+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.1 v2:172.21.15.99:6824/3369025248 29 ==== osd_op_reply(29 603.00000000 [omap-set-header] v98'101 uv101 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e2df40 2024-03-04T10:10:48.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.124+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55dbf3e3c9b0 0x55dbf3e3eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.124+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:30 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3f2e0 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.124+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55dbf3e3c9b0 0x55dbf3e3eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.125+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55dbf3e3c9b0 0x55dbf3e3eda0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.126+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(30 1.00000000.inode [create] v98'68 uv50 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.126+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:31 13.14 13:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3f680 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.127+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== osd_op_reply(31 1.00000000.inode [delete] v98'69 uv69 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.130 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.127+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:32 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e3fa20 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.128+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 3 ==== osd_op_reply(32 1.00000000.inode [create] v98'70 uv70 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.131 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-03-04T10:10:48.131 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.128+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:33 13.14 13:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40270 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.131+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 4 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v98'71 uv71 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.131+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:34 13.14 13:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40610 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.134+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 5 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v98'72 uv72 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.134+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:35 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e409b0 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.135+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 6 ==== osd_op_reply(35 200.00000000 [create] v98'73 uv55 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.135+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:36 13.14 13:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.136+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 7 ==== osd_op_reply(36 200.00000000 [delete] v98'74 uv74 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.138 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.136+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:37 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.137+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 8 ==== osd_op_reply(37 200.00000000 [create] v98'75 uv75 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.139 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-03-04T10:10:48.139 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.137+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:38 13.14 13:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.138+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 9 ==== osd_op_reply(38 200.00000000 [write 0~90] v98'76 uv76 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.140 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.138+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:39 13.14 13:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.139+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 10 ==== osd_op_reply(39 200.00000000 [omap-set-header] v98'77 uv77 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.141 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.139+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:40 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.140+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 11 ==== osd_op_reply(40 200.00000001 [create] v98'78 uv60 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.140+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:41 13.14 13:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.141+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 12 ==== osd_op_reply(41 200.00000001 [delete] v98'79 uv79 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.141+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:42 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.144 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.142+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 13 ==== osd_op_reply(42 200.00000001 [create] v98'80 uv80 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.145 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-03-04T10:10:48.145 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.142+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:43 13.14 13:2e2fa760:::200.00000001:head [write 0~18075 in=18075b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e45410 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.144+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 14 ==== osd_op_reply(43 200.00000001 [write 0~18075] v98'81 uv81 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.144+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:44 13.14 13:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e45410 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.145+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 15 ==== osd_op_reply(44 200.00000001 [omap-set-header] v98'82 uv82 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.145+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:45 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.146+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 16 ==== osd_op_reply(45 606.00000000 [create] v98'83 uv66 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.146+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:46 13.14 13:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.147+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 17 ==== osd_op_reply(46 606.00000000 [delete] v98'84 uv84 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.147+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:47 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.150 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-03-04T10:10:48.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.148+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 18 ==== osd_op_reply(47 606.00000000 [create] v98'85 uv85 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.150 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.148+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:48 13.14 13:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e41d60 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.149+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 19 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v98'86 uv86 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.151 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.149+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:49 13.14 13:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e41d60 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.150+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 20 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v98'87 uv87 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.150+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:50 13.14 13:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e40d50 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.151+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.0 v2:172.21.15.99:6800/224245114 21 ==== osd_op_reply(50 606.00000000 [omap-set-header] v98'88 uv88 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e3c9b0 2024-03-04T10:10:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.151+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55dbf3e41260 0x55dbf3e43650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.151+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:51 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e43b90 con 0x55dbf3e41260 2024-03-04T10:10:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.151+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55dbf3e41260 0x55dbf3e43650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.151+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55dbf3e41260 0x55dbf3e43650 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.152+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(51 607.00000000 [create] v98'15 uv14 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.152+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:52 13.1c 13:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e43f70 con 0x55dbf3e41260 2024-03-04T10:10:48.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.153+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== osd_op_reply(52 607.00000000 [delete] v98'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.153+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:53 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e44450 con 0x55dbf3e41260 2024-03-04T10:10:48.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.154+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 3 ==== osd_op_reply(53 607.00000000 [create] v98'17 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.156 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-03-04T10:10:48.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.154+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:54 13.1c 13:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e45d00 con 0x55dbf3e41260 2024-03-04T10:10:48.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.155+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 4 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v98'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.157 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.155+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:55 13.1c 13:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e460a0 con 0x55dbf3e41260 2024-03-04T10:10:48.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.156+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 5 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v98'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.156+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:56 13.1c 13:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e44cf0 con 0x55dbf3e41260 2024-03-04T10:10:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.157+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 6 ==== osd_op_reply(56 607.00000000 [omap-set-header] v98'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.157+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55dbf3e455f0 0x55dbf3e48450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.157+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:57 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e48990 con 0x55dbf3e455f0 2024-03-04T10:10:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.157+0000 7f436cc73640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55dbf3e455f0 0x55dbf3e48450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.158+0000 7f436cc73640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55dbf3e455f0 0x55dbf3e48450 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.159+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v98'14 uv13 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.159+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:58 13.6 13:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e48990 con 0x55dbf3e455f0 2024-03-04T10:10:48.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.161+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v98'15 uv15 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.161+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:59 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e48990 con 0x55dbf3e455f0 2024-03-04T10:10:48.164 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-03-04T10:10:48.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.162+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v98'16 uv16 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.162+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:60 13.6 13:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e48d80 con 0x55dbf3e455f0 2024-03-04T10:10:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.163+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v98'17 uv17 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.163+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55dbf3e49290 0x55dbf3e4b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.163+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:61 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bbc0 con 0x55dbf3e49290 2024-03-04T10:10:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.163+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55dbf3e49290 0x55dbf3e4b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.163+0000 7f436c472640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55dbf3e49290 0x55dbf3e4b680 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.164+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(61 608.00000000 [create] v98'15 uv14 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.164+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:62 13.e 13:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.166+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== osd_op_reply(62 608.00000000 [delete] v98'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.166+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:63 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.169 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-03-04T10:10:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.167+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 3 ==== osd_op_reply(63 608.00000000 [create] v98'17 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.167+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:64 13.e 13:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4cfa0 con 0x55dbf3e49290 2024-03-04T10:10:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.168+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 4 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v98'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.168+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:65 13.e 13:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4cfa0 con 0x55dbf3e49290 2024-03-04T10:10:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.170+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 5 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v98'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.170+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:66 13.e 13:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.170+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 6 ==== osd_op_reply(66 608.00000000 [omap-set-header] v98'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.170+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:67 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.172+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 5 ==== osd_op_reply(67 604.00000000 [create] v98'26 uv20 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.172+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:68 13.15 13:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.173+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 6 ==== osd_op_reply(68 604.00000000 [delete] v98'27 uv27 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.173+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:69 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.174+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 7 ==== osd_op_reply(69 604.00000000 [create] v98'28 uv28 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.176 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-03-04T10:10:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.174+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:70 13.15 13:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.175+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 8 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v98'29 uv29 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.175+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:71 13.15 13:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.176+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 9 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v98'30 uv30 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.176+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:72 13.15 13:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.177+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 10 ==== osd_op_reply(72 604.00000000 [omap-set-header] v98'31 uv31 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.177+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:73 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.178+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 11 ==== osd_op_reply(73 500.00000000 [create] v98'32 uv25 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.178+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:74 13.15 13:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.179+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 12 ==== osd_op_reply(74 500.00000000 [delete] v98'33 uv33 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.179+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:75 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.183 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-03-04T10:10:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.181+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 13 ==== osd_op_reply(75 500.00000000 [create] v98'34 uv34 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.181+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:76 13.15 13:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.182+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 14 ==== osd_op_reply(76 500.00000000 [write 0~90] v98'35 uv35 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.182+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:77 13.15 13:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.183+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 15 ==== osd_op_reply(77 500.00000000 [omap-set-header] v98'36 uv36 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.183+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:78 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.184+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 16 ==== osd_op_reply(78 mds_snaptable [create] v98'12 uv11 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.184+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:79 13.d 13:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.185+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 17 ==== osd_op_reply(79 mds_snaptable [delete] v98'13 uv13 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.185+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:80 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.186+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 18 ==== osd_op_reply(80 mds_snaptable [create] v98'14 uv14 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.189 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-03-04T10:10:48.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.186+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:81 13.d 13:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.187+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 19 ==== osd_op_reply(81 mds_snaptable [write 0~70] v98'15 uv15 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.187+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:82 13.d 13:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e455f0 2024-03-04T10:10:48.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.188+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 <== osd.4 v2:172.21.15.162:6800/3079837853 20 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v98'16 uv16 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4364011a70 con 0x55dbf3e455f0 2024-03-04T10:10:48.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.188+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:83 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.190+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 7 ==== osd_op_reply(83 605.00000000 [create] v98'15 uv14 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.190+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:84 13.1d 13:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.191+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 8 ==== osd_op_reply(84 605.00000000 [delete] v98'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.191+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:85 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.194 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-03-04T10:10:48.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.192+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 9 ==== osd_op_reply(85 605.00000000 [create] v98'17 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.192+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:86 13.1d 13:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.193+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 10 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v98'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.195 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.193+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:87 13.1d 13:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.194+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 11 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v98'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.194+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:88 13.1d 13:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.195+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 12 ==== osd_op_reply(88 605.00000000 [omap-set-header] v98'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.195+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:89 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.196+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 13 ==== osd_op_reply(89 mds0_inotable [create] v98'24 uv18 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.196+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:90 13.13 13:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.197+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 14 ==== osd_op_reply(90 mds0_inotable [delete] v98'25 uv25 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.197+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:91 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4bf90 con 0x55dbf3e49290 2024-03-04T10:10:48.201 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-03-04T10:10:48.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.198+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 15 ==== osd_op_reply(91 mds0_inotable [create] v98'26 uv26 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.198+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:92 13.13 13:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4c3a0 con 0x55dbf3e49290 2024-03-04T10:10:48.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.200+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 16 ==== osd_op_reply(92 mds0_inotable [write 0~34] v98'27 uv27 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.200+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:93 13.13 13:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4c3a0 con 0x55dbf3e49290 2024-03-04T10:10:48.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.201+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 17 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v98'28 uv28 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.201+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:94 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4c350 con 0x55dbf3e49290 2024-03-04T10:10:48.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.202+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 18 ==== osd_op_reply(94 100.00000000 [create] v98'29 uv23 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.202+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:95 13.13 13:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4c350 con 0x55dbf3e49290 2024-03-04T10:10:48.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.204+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 19 ==== osd_op_reply(95 100.00000000 [delete] v98'30 uv30 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.204+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:96 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4c350 con 0x55dbf3e49290 2024-03-04T10:10:48.207 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-03-04T10:10:48.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.205+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 20 ==== osd_op_reply(96 100.00000000 [create] v98'31 uv31 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.205+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:97 13.13 13:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4c350 con 0x55dbf3e49290 2024-03-04T10:10:48.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.206+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 21 ==== osd_op_reply(97 100.00000000 [omap-set-header] v98'32 uv32 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.206+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:98 13.13 13:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f160 con 0x55dbf3e49290 2024-03-04T10:10:48.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.208+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 22 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v98'33 uv33 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.208+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55dbf3e4cd10 0x55dbf3e4f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.208+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:99 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e4cd10 2024-03-04T10:10:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.208+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55dbf3e4cd10 0x55dbf3e4f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.209+0000 7f436bc71640 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55dbf3e4cd10 0x55dbf3e4f550 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.210+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(99 mds0_sessionmap [create] v98'14 uv12 ondisk = -17 ((17) File exists)) v8 ==== 159+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e4cd10 2024-03-04T10:10:48.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.210+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:100 13.b 13:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e4cd10 2024-03-04T10:10:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.211+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v98'15 uv15 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e4cd10 2024-03-04T10:10:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.211+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:101 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e4cd10 2024-03-04T10:10:48.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.212+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.5 v2:172.21.15.162:6808/1764948489 3 ==== osd_op_reply(101 mds0_sessionmap [create] v98'16 uv16 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e4cd10 2024-03-04T10:10:48.214 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-03-04T10:10:48.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.212+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:102 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e4cd10 2024-03-04T10:10:48.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.213+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.5 v2:172.21.15.162:6808/1764948489 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v98'17 uv17 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e4cd10 2024-03-04T10:10:48.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.213+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:103 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e49290 2024-03-04T10:10:48.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.214+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 23 ==== osd_op_reply(103 609.00000000 [create] v98'15 uv14 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.214+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:104 13.7 13:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e49290 2024-03-04T10:10:48.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.215+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 24 ==== osd_op_reply(104 609.00000000 [delete] v98'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.215+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:105 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e49290 2024-03-04T10:10:48.218 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-03-04T10:10:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.216+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 25 ==== osd_op_reply(105 609.00000000 [create] v98'17 uv17 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.216+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:106 13.7 13:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e49290 2024-03-04T10:10:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.217+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 26 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v98'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.217+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:107 13.7 13:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e49290 2024-03-04T10:10:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.218+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 27 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v98'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.218+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:108 13.7 13:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e49290 2024-03-04T10:10:48.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.219+0000 7f436c472640 1 -- 172.21.15.99:0/2514944130 <== osd.7 v2:172.21.15.162:6816/2710319333 28 ==== osd_op_reply(108 609.00000000 [omap-set-header] v98'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f435c021020 con 0x55dbf3e49290 2024-03-04T10:10:48.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.220+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:109 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.221+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 7 ==== osd_op_reply(109 400.00000000 [create] v98'12 uv11 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.221+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:110 13.f 13:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.222+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 8 ==== osd_op_reply(110 400.00000000 [delete] v98'13 uv13 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.222+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:111 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.225 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-03-04T10:10:48.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.223+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 9 ==== osd_op_reply(111 400.00000000 [create] v98'14 uv14 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.223+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:112 13.f 13:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.224+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 10 ==== osd_op_reply(112 400.00000000 [write 0~22] v98'15 uv15 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.224+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:113 13.f 13:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.225+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 11 ==== osd_op_reply(113 400.00000000 [omap-set-header] v98'16 uv16 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.225+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:114 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.228+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 12 ==== osd_op_reply(114 100.00000000.inode [create] v98'33 uv24 ondisk = -17 ((17) File exists)) v8 ==== 162+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.228+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:115 13.1f 13:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.229+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 13 ==== osd_op_reply(115 100.00000000.inode [delete] v98'34 uv34 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.229+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:116 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.234 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-03-04T10:10:48.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.232+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 14 ==== osd_op_reply(116 100.00000000.inode [create] v98'35 uv35 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.232+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:117 13.1f 13:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.234+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 15 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v98'36 uv36 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.234+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:118 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f120 con 0x55dbf3e41260 2024-03-04T10:10:48.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.236+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 16 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v98'37 uv37 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.236+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:119 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f120 con 0x55dbf3e41260 2024-03-04T10:10:48.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.238+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 17 ==== osd_op_reply(119 1.00000000 [create] v98'38 uv31 ondisk = -17 ((17) File exists)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.238+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:120 13.1f 13:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f120 con 0x55dbf3e41260 2024-03-04T10:10:48.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.241+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 18 ==== osd_op_reply(120 1.00000000 [delete] v98'39 uv39 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.243 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.241+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:121 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f120 con 0x55dbf3e41260 2024-03-04T10:10:48.244 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-03-04T10:10:48.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.242+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 19 ==== osd_op_reply(121 1.00000000 [create] v98'40 uv40 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.242+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:122 13.1f 13:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f120 con 0x55dbf3e41260 2024-03-04T10:10:48.245 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.243+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 20 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v98'41 uv41 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.243+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:123 13.1f 13:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.246 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.244+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 21 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v98'42 uv42 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.244+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:124 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4f100 con 0x55dbf3e41260 2024-03-04T10:10:48.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.245+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 22 ==== osd_op_reply(124 1.00000000 [omap-set-header] v98'43 uv43 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.245+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:125 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) v9 -- 0x55dbf3e4fa90 con 0x55dbf3e41260 2024-03-04T10:10:48.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436bc71640 1 -- 172.21.15.99:0/2514944130 <== osd.6 v2:172.21.15.162:6824/1448511037 23 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v98'44 uv44 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f436005f550 con 0x55dbf3e41260 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55dbf3e3c9b0 msgr2=0x55dbf3e3eda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x55dbf3e3c9b0 0x55dbf3e3eda0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55dbf3e2df40 msgr2=0x55dbf3e2e320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x55dbf3e2df40 0x55dbf3e2e320 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55dbf3e455f0 msgr2=0x55dbf3e48450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x55dbf3e455f0 0x55dbf3e48450 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55dbf3e4cd10 msgr2=0x55dbf3e4f550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x55dbf3e4cd10 0x55dbf3e4f550 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55dbf3e41260 msgr2=0x55dbf3e43650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x55dbf3e41260 0x55dbf3e43650 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55dbf3e49290 msgr2=0x55dbf3e4b680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x55dbf3e49290 0x55dbf3e4b680 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 reap_dead start 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436cc73640 1 -- 172.21.15.99:0/2514944130 reap_dead start 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4358069eb0 msgr2=0x7f435806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4358069eb0 0x7f435806c370 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4360056000 tx=0x7f4360051000 comp rx=0 tx=0).stop 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 msgr2=0x55dbf3d8c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3d8c840 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f435c009d40 tx=0x7f435c0107d0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 shutdown_connections 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4358069eb0 0x7f435806c370 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 --2- 172.21.15.99:0/2514944130 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x55dbf3cc67a0 0x55dbf3d8c840 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.246+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 >> 172.21.15.99:0/2514944130 conn(0x55dbf3cc3400 msgr2=0x55dbf3d997c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.247+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 shutdown_connections 2024-03-04T10:10:48.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.247+0000 7f436e974f40 1 -- 172.21.15.99:0/2514944130 wait complete. 2024-03-04T10:10:48.255 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 1.00000000.inode 2024-03-04T10:10:48.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.330+0000 7f4a793d6f40 1 Processor -- start 2024-03-04T10:10:48.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.330+0000 7f4a793d6f40 1 -- start start 2024-03-04T10:10:48.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a793d6f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 0x5650b51c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a793d6f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b52655a0 0x5650b526c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a793d6f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51dd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a793d6f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b513cb30 con 0x5650b5193c90 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a793d6f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b5140a90 con 0x5650b526b5f0 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a793d6f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650b51deab0 con 0x5650b52655a0 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 0x5650b51c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 0x5650b51c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57810/0 (socket says 172.21.15.99:57810) 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 -- 172.21.15.99:0/2940027997 learned_addr learned my addr 172.21.15.99:0/2940027997 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a766d3640 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b52655a0 0x5650b526c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a776d5640 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51dd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 -- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b52655a0 msgr2=0x5650b526c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b52655a0 0x5650b526c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 -- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 msgr2=0x5650b51dd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51dd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 -- 172.21.15.99:0/2940027997 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650b526df30 con 0x5650b5193c90 2024-03-04T10:10:48.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a776d5640 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51dd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.331+0000 7f4a76ed4640 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 0x5650b51c54c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000b370 tx=0x7f4a7000b830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae93f9c62fc45a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a756d1640 1 -- 172.21.15.99:0/2940027997 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a70016070 con 0x5650b5193c90 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a756d1640 1 -- 172.21.15.99:0/2940027997 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7000d6b0 con 0x5650b5193c90 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a756d1640 1 -- 172.21.15.99:0/2940027997 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000daa0 con 0x5650b5193c90 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2940027997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 msgr2=0x5650b51c54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 0x5650b51c54c0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000b370 tx=0x7f4a7000b830 comp rx=0 tx=0).stop 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2940027997 shutdown_connections 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51dd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b52655a0 0x5650b526c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2940027997 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b5193c90 0x5650b51c54c0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2940027997 >> 172.21.15.99:0/2940027997 conn(0x5650b5195400 msgr2=0x5650b525fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2940027997 shutdown_connections 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.332+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2940027997 wait complete. 2024-03-04T10:10:48.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 Processor -- start 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 -- start start 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b5193c90 0x5650b5265b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b52655a0 0x5650b5252250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b526aef0 con 0x5650b52655a0 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b526e7a0 con 0x5650b526b5f0 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.333+0000 7f4a793d6f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x5650b51bfbf0 con 0x5650b5193c90 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b52655a0 0x5650b5252250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b52655a0 0x5650b5252250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57826/0 (socket says 172.21.15.99:57826) 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 -- 172.21.15.99:0/2962560986 learned_addr learned my addr 172.21.15.99:0/2962560986 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a76ed4640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b5193c90 0x5650b5265b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a776d5640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 -- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b5193c90 msgr2=0x5650b5265b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x5650b5193c90 0x5650b5265b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 -- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 msgr2=0x5650b51c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 -- 172.21.15.99:0/2962560986 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650b5266290 con 0x5650b52655a0 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a776d5640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5650b526b5f0 0x5650b51c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a766d3640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b52655a0 0x5650b5252250 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000c660 tx=0x7f4a6000cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a6ffff640 1 -- 172.21.15.99:0/2962560986 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6000ce50 con 0x5650b52655a0 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a6ffff640 1 -- 172.21.15.99:0/2962560986 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60013070 con 0x5650b52655a0 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5650b5266570 con 0x5650b52655a0 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a6ffff640 1 -- 172.21.15.99:0/2962560986 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6000bad0 con 0x5650b52655a0 2024-03-04T10:10:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.334+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650b5266b30 con 0x5650b52655a0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.335+0000 7f4a6ffff640 1 -- 172.21.15.99:0/2962560986 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4a6000bc70 con 0x5650b52655a0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a6ffff640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4a5806a050 0x7f4a5806c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a6ffff640 1 -- 172.21.15.99:0/2962560986 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7f4a60089c70 con 0x5650b52655a0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a76ed4640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4a5806a050 0x7f4a5806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5650b5307410 0x5650b5307820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e98) v9 -- 0x5650b5307d60 con 0x5650b5307410 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a776d5640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5650b5307410 0x5650b5307820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a76ed4640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4a5806a050 0x7f4a5806c510 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000b370 tx=0x7f4a70002000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.336+0000 7f4a776d5640 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5650b5307410 0x5650b5307820 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a776d5640 1 -- 172.21.15.99:0/2962560986 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 1.00000000.inode [delete] v98'89 uv89 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4a68052430 con 0x5650b5307410 2024-03-04T10:10:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5650b5307410 msgr2=0x5650b5307820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x5650b5307410 0x5650b5307820 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4a5806a050 msgr2=0x7f4a5806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4a5806a050 0x7f4a5806c510 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000b370 tx=0x7f4a70002000 comp rx=0 tx=0).stop 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b52655a0 msgr2=0x5650b5252250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 --2- 172.21.15.99:0/2962560986 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x5650b52655a0 0x5650b5252250 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000c660 tx=0x7f4a6000cb20 comp rx=0 tx=0).stop 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a776d5640 1 -- 172.21.15.99:0/2962560986 reap_dead start 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 shutdown_connections 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 >> 172.21.15.99:0/2962560986 conn(0x5650b5195400 msgr2=0x5650b52691a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 shutdown_connections 2024-03-04T10:10:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.338+0000 7f4a793d6f40 1 -- 172.21.15.99:0/2962560986 wait complete. 2024-03-04T10:10:48.345 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:10:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.485+0000 7fafa6a44640 1 Processor -- start 2024-03-04T10:10:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.485+0000 7fafa6a44640 1 -- start start 2024-03-04T10:10:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.485+0000 7fafa6a44640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fafa015bb70 0x7fafa00b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.485+0000 7fafa6a44640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafa015f660 0x7fafa00d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7fafa6a44640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa015bf50 con 0x7fafa00d1cc0 2024-03-04T10:10:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7fafa6a44640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa00cf890 con 0x7fafa015f660 2024-03-04T10:10:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7fafa6a44640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa015fb60 con 0x7fafa015bb70 2024-03-04T10:10:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafa015f660 0x7fafa00d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fafa015f660 0x7fafa00d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57830/0 (socket says 172.21.15.99:57830) 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1648916939 learned_addr learned my addr 172.21.15.99:0/1648916939 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9ffff640 1 --2- 172.21.15.99:0/1648916939 >> v2:172.21.15.162:3300/0 conn(0x7fafa015bb70 0x7fafa00b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1648916939 >> v2:172.21.15.162:3300/0 conn(0x7fafa015bb70 msgr2=0x7fafa00b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 --2- 172.21.15.99:0/1648916939 >> v2:172.21.15.162:3300/0 conn(0x7fafa015bb70 0x7fafa00b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1648916939 >> v1:172.21.15.162:6789/0 conn(0x7fafa00d1cc0 legacy=0x7fafa00ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.486+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1648916939 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa015fe40 con 0x7fafa015f660 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.487+0000 7faf9f7fe640 1 --2- 172.21.15.99:0/1648916939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015f660 0x7fafa00d1780 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fafa00cf310 tx=0x7faf9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a728d5c23b1cf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.487+0000 7faf9effd640 1 -- 172.21.15.99:0/1648916939 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94035020 con 0x7fafa015f660 2024-03-04T10:10:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.487+0000 7faf9effd640 1 -- 172.21.15.99:0/1648916939 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf94031870 con 0x7fafa015f660 2024-03-04T10:10:48.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.487+0000 7faf9effd640 1 -- 172.21.15.99:0/1648916939 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94031c40 con 0x7fafa015f660 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.487+0000 7fafa6a44640 1 -- 172.21.15.99:0/1648916939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015f660 msgr2=0x7fafa00d1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.487+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1648916939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015f660 0x7fafa00d1780 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fafa00cf310 tx=0x7faf9402b910 comp rx=0 tx=0).stop 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.488+0000 7fafa6a44640 1 -- 172.21.15.99:0/1648916939 shutdown_connections 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.488+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1648916939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015f660 0x7fafa00d1780 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.488+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1648916939 >> v2:172.21.15.162:3300/0 conn(0x7fafa015bb70 0x7fafa00b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.488+0000 7fafa6a44640 1 -- 172.21.15.99:0/1648916939 >> 172.21.15.99:0/1648916939 conn(0x7fafa00b33a0 msgr2=0x7fafa00b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.488+0000 7fafa6a44640 1 -- 172.21.15.99:0/1648916939 shutdown_connections 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.488+0000 7fafa6a44640 1 -- 172.21.15.99:0/1648916939 wait complete. 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.489+0000 7fafa6a44640 1 Processor -- start 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.489+0000 7fafa6a44640 1 -- start start 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa00d1cc0 0x7fafa015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 0x7fafa00aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa015f660 0x7fafa0159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00b0f80 con 0x7fafa015bb70 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0160400 con 0x7fafa00d1cc0 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00b1100 con 0x7fafa015f660 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 0x7fafa00aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 0x7fafa00aaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57846/0 (socket says 172.21.15.99:57846) 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1405065065 learned_addr learned my addr 172.21.15.99:0/1405065065 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa4fe1640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa015f660 0x7fafa0159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9ffff640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa00d1cc0 0x7fafa015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa015f660 msgr2=0x7fafa0159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa015f660 0x7fafa0159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa00d1cc0 msgr2=0x7fafa015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa00d1cc0 0x7fafa015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 -- 172.21.15.99:0/1405065065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa015d170 con 0x7fafa015bb70 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa4fe1640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa015f660 0x7fafa0159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9ffff640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa00d1cc0 0x7fafa015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9f7fe640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 0x7fafa00aaf30 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7faf9402b950 tx=0x7faf940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94040070 con 0x7fafa015bb70 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf94030c80 con 0x7fafa015bb70 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf94035040 con 0x7fafa015bb70 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa00a9380 con 0x7fafa015bb70 2024-03-04T10:10:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.490+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa00a9940 con 0x7fafa015bb70 2024-03-04T10:10:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.491+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa015d340 con 0x7fafa015bb70 2024-03-04T10:10:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.492+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7faf9404b060 con 0x7fafa015bb70 2024-03-04T10:10:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.492+0000 7faf9cff9640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf8c06b4d0 0x7faf8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.492+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6475+0+0 (secure 0 0 0) 0x7faf940ac160 con 0x7fafa015bb70 2024-03-04T10:10:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.492+0000 7faf9ffff640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf8c06b4d0 0x7faf8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.493+0000 7faf9ffff640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf8c06b4d0 0x7faf8c06d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7faf90009ac0 tx=0x7faf90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:48.498 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.496+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7faf9402f320 con 0x7fafa015bb70 2024-03-04T10:10:48.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:48.702+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fafa00b2360 con 0x7fafa015bb70 2024-03-04T10:10:49.024 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.022+0000 7faf9cff9640 1 -- 172.21.15.99:0/1405065065 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v98) v1 ==== 162+0+0 (secure 0 0 0) 0x7faf9407ad70 con 0x7fafa015bb70 2024-03-04T10:10:49.025 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:10:49.029 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf8c06b4d0 msgr2=0x7faf8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.029 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf8c06b4d0 0x7faf8c06d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7faf90009ac0 tx=0x7faf90008040 comp rx=0 tx=0).stop 2024-03-04T10:10:49.029 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 msgr2=0x7fafa00aaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 0x7fafa00aaf30 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7faf9402b950 tx=0x7faf940046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 shutdown_connections 2024-03-04T10:10:49.030 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7faf8c06b4d0 0x7faf8c06d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fafa015f660 0x7fafa0159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fafa015bb70 0x7fafa00aaf30 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.031 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 --2- 172.21.15.99:0/1405065065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fafa00d1cc0 0x7fafa015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 >> 172.21.15.99:0/1405065065 conn(0x7fafa00b33a0 msgr2=0x7fafa00b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:49.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 shutdown_connections 2024-03-04T10:10:49.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.025+0000 7fafa6a44640 1 -- 172.21.15.99:0/1405065065 wait complete. 2024-03-04T10:10:49.034 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:49.032+0000 7f4cee41b640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-03-04T10:10:49.035 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:49.032+0000 7f4cee41b640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-03-04T10:10:49.035 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:10:49.032+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:10:49.036 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:10:49.032+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE) 2024-03-04T10:10:49.041 INFO:tasks.ceph.mds.c.smithi099.stderr: -16> 2024-03-04T10:10:49.032+0000 7f4cee41b640 -1 mds.0.cache.ino(0x1) No data while reading inode 0x1 2024-03-04T10:10:49.041 INFO:tasks.ceph.mds.c.smithi099.stderr: -14> 2024-03-04T10:10:49.032+0000 7f4cee41b640 -1 log_channel(cluster) log [ERR] : Error loading MDS rank 0: (2) No such file or directory 2024-03-04T10:10:49.043 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:49.064 INFO:tasks.ceph.mds.c.smithi099.stdout:starting mds.c at 2024-03-04T10:10:49.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.143+0000 7f15d20d1640 1 Processor -- start 2024-03-04T10:10:49.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.143+0000 7f15d20d1640 1 -- start start 2024-03-04T10:10:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15d20d1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f15cc154a80 0x7f15cc15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15d20d1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f15cc0cf650 0x7f15cc0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15d20d1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15cc0cdaa0 con 0x7f15cc1616c0 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15d20d1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15cc071df0 con 0x7f15cc0cf650 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15d20d1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15cc0cd010 con 0x7f15cc154a80 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f15cc0cf650 0x7f15cc0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f15cc0cf650 0x7f15cc0cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57870/0 (socket says 172.21.15.99:57870) 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 -- 172.21.15.99:0/3498742657 learned_addr learned my addr 172.21.15.99:0/3498742657 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15cb7fe640 1 --2- 172.21.15.99:0/3498742657 >> v2:172.21.15.162:3300/0 conn(0x7f15cc154a80 0x7f15cc15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 -- 172.21.15.99:0/3498742657 >> v2:172.21.15.162:3300/0 conn(0x7f15cc154a80 msgr2=0x7f15cc15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 --2- 172.21.15.99:0/3498742657 >> v2:172.21.15.162:3300/0 conn(0x7f15cc154a80 0x7f15cc15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 -- 172.21.15.99:0/3498742657 >> v1:172.21.15.162:6789/0 conn(0x7f15cc1616c0 legacy=0x7f15cc161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15caffd640 1 -- 172.21.15.99:0/3498742657 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc0cdd80 con 0x7f15cc0cf650 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.144+0000 7f15cb7fe640 1 --2- 172.21.15.99:0/3498742657 >> v2:172.21.15.162:3300/0 conn(0x7f15cc154a80 0x7f15cc15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.145+0000 7f15caffd640 1 --2- 172.21.15.99:0/3498742657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc0cfae0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f15cc06e970 tx=0x7f15c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc77ab8ef77edbd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.145+0000 7f15ca7fc640 1 -- 172.21.15.99:0/3498742657 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0035020 con 0x7f15cc0cf650 2024-03-04T10:10:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.145+0000 7f15ca7fc640 1 -- 172.21.15.99:0/3498742657 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c0031870 con 0x7f15cc0cf650 2024-03-04T10:10:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.145+0000 7f15ca7fc640 1 -- 172.21.15.99:0/3498742657 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0031c40 con 0x7f15cc0cf650 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 -- 172.21.15.99:0/3498742657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 msgr2=0x7f15cc0cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 --2- 172.21.15.99:0/3498742657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc0cfae0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f15cc06e970 tx=0x7f15c002b910 comp rx=0 tx=0).stop 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 -- 172.21.15.99:0/3498742657 shutdown_connections 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 --2- 172.21.15.99:0/3498742657 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc0cfae0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 --2- 172.21.15.99:0/3498742657 >> v2:172.21.15.162:3300/0 conn(0x7f15cc154a80 0x7f15cc15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 -- 172.21.15.99:0/3498742657 >> 172.21.15.99:0/3498742657 conn(0x7f15cc06f5f0 msgr2=0x7f15cc06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 -- 172.21.15.99:0/3498742657 shutdown_connections 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.146+0000 7f15d20d1640 1 -- 172.21.15.99:0/3498742657 wait complete. 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.147+0000 7f15d20d1640 1 Processor -- start 2024-03-04T10:10:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.147+0000 7f15d20d1640 1 -- start start 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15cc154a80 0x7f15cc156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15cc1616c0 0x7f15cc1f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc0ce620 con 0x7f15cc0cf650 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc0ce340 con 0x7f15cc154a80 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15cc154e60 con 0x7f15cc1616c0 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57880/0 (socket says 172.21.15.99:57880) 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 -- 172.21.15.99:0/662140264 learned_addr learned my addr 172.21.15.99:0/662140264 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15caffd640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15cc154a80 0x7f15cc156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cbfff640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15cc1616c0 0x7f15cc1f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 -- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15cc1616c0 msgr2=0x7f15cc1f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15cc1616c0 0x7f15cc1f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 -- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15cc154a80 msgr2=0x7f15cc156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15cc154a80 0x7f15cc156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 -- 172.21.15.99:0/662140264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15cc0cb600 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15caffd640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15cc154a80 0x7f15cc156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15cb7fe640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc156940 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00c820 tx=0x7f15bc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc015070 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc007a30 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15cc1575d0 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.148+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc007e00 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.149+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15cc157b90 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.149+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15cc157d90 con 0x7f15cc0cf650 2024-03-04T10:10:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.150+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f15bc018070 con 0x7f15cc0cf650 2024-03-04T10:10:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.150+0000 7f15b3fff640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15b806b4d0 0x7f15b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.150+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f15bc08a460 con 0x7f15cc0cf650 2024-03-04T10:10:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.150+0000 7f15caffd640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15b806b4d0 0x7f15b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.151+0000 7f15caffd640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15b806b4d0 0x7f15b806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f15c0036000 tx=0x7f15c003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.154+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f15cc157d90 con 0x7f15cc0cf650 2024-03-04T10:10:49.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.358+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15cc1573c0 con 0x7f15cc0cf650 2024-03-04T10:10:49.361 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.359+0000 7f15b3fff640 1 -- 172.21.15.99:0/662140264 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4143 (secure 0 0 0) 0x7f15bc059010 con 0x7f15cc0cf650 2024-03-04T10:10:49.362 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:49.362 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":100,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:49.034050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:49.362 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 100 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15b806b4d0 msgr2=0x7f15b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15b806b4d0 0x7f15b806d990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f15c0036000 tx=0x7f15c003c040 comp rx=0 tx=0).stop 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 msgr2=0x7f15cc156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc156940 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f15bc00c820 tx=0x7f15bc00cc20 comp rx=0 tx=0).stop 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 shutdown_connections 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f15b806b4d0 0x7f15b806d990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f15cc1616c0 0x7f15cc1f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f15cc154a80 0x7f15cc156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 --2- 172.21.15.99:0/662140264 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f15cc0cf650 0x7f15cc156940 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 >> 172.21.15.99:0/662140264 conn(0x7f15cc06f5f0 msgr2=0x7f15cc0d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:49.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 shutdown_connections 2024-03-04T10:10:49.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.361+0000 7f15d20d1640 1 -- 172.21.15.99:0/662140264 wait complete. 2024-03-04T10:10:49.375 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-04T10:10:49.375 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:49.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.513+0000 7f615c945640 1 Processor -- start 2024-03-04T10:10:49.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.513+0000 7f615c945640 1 -- start start 2024-03-04T10:10:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f615c945640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f61580cfae0 0x7f61580cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f615c945640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580cb300 con 0x7f61580d1290 2024-03-04T10:10:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f615c945640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f615806ae50 con 0x7f61580cc8e0 2024-03-04T10:10:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f615c945640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615806b130 con 0x7f61580cfae0 2024-03-04T10:10:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f615659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f61580cfae0 0x7f61580cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f6156d9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f61580d1290 0x7f61580cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45696/0 (socket says 172.21.15.99:45696) 2024-03-04T10:10:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.514+0000 7f6156d9d640 1 -- 172.21.15.99:0/3524609611 learned_addr learned my addr 172.21.15.99:0/3524609611 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.515+0000 7f615659c640 1 -- 172.21.15.99:0/3524609611 >> v1:172.21.15.162:6789/0 conn(0x7f61580cc8e0 legacy=0x7f61580cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.515+0000 7f615659c640 1 -- 172.21.15.99:0/3524609611 >> v1:172.21.15.99:6789/0 conn(0x7f61580d1290 legacy=0x7f61580cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.515+0000 7f615659c640 1 -- 172.21.15.99:0/3524609611 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615806b410 con 0x7f61580cfae0 2024-03-04T10:10:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.515+0000 7f615559a640 1 -- 172.21.15.99:0/3524609611 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424654639 0 0) 0x7f61580cb300 con 0x7f61580d1290 2024-03-04T10:10:49.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.515+0000 7f615659c640 1 --2- 172.21.15.99:0/3524609611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580cfae0 0x7f61580cadc0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6148009d90 tx=0x7f614802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31889e22ba6a553a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615559a640 1 -- 172.21.15.99:0/3524609611 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148039070 con 0x7f61580cfae0 2024-03-04T10:10:49.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615559a640 1 -- 172.21.15.99:0/3524609611 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614802cc60 con 0x7f61580cfae0 2024-03-04T10:10:49.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615559a640 1 -- 172.21.15.99:0/3524609611 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148034660 con 0x7f61580cfae0 2024-03-04T10:10:49.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 -- 172.21.15.99:0/3524609611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580cfae0 msgr2=0x7f61580cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.519 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 --2- 172.21.15.99:0/3524609611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580cfae0 0x7f61580cadc0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f6148009d90 tx=0x7f614802beb0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 -- 172.21.15.99:0/3524609611 shutdown_connections 2024-03-04T10:10:49.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 --2- 172.21.15.99:0/3524609611 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580cfae0 0x7f61580cadc0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 -- 172.21.15.99:0/3524609611 >> 172.21.15.99:0/3524609611 conn(0x7f61580d0770 msgr2=0x7f61580d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:49.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 -- 172.21.15.99:0/3524609611 shutdown_connections 2024-03-04T10:10:49.520 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.516+0000 7f615c945640 1 -- 172.21.15.99:0/3524609611 wait complete. 2024-03-04T10:10:49.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.517+0000 7f615c945640 1 Processor -- start 2024-03-04T10:10:49.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 -- start start 2024-03-04T10:10:49.522 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61580cc8e0 0x7f61581f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61580cfae0 0x7f615815fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 0x7f61580b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615806b9d0 con 0x7f61580d1290 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61580ce440 con 0x7f61580cfae0 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f615c945640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61580cdb50 con 0x7f61580cc8e0 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f6156d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 0x7f61580b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f6156d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 0x7f61580b3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57898/0 (socket says 172.21.15.99:57898) 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.518+0000 7f6156d9d640 1 -- 172.21.15.99:0/3573631552 learned_addr learned my addr 172.21.15.99:0/3573631552 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f615659c640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61580cc8e0 0x7f61581f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6156d9d640 1 -- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61580cc8e0 msgr2=0x7f61581f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6155d9b640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61580cfae0 0x7f615815fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6156d9d640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61580cc8e0 0x7f61581f3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6156d9d640 1 -- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61580cfae0 msgr2=0x7f615815fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6156d9d640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61580cfae0 0x7f615815fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6156d9d640 1 -- 172.21.15.99:0/3573631552 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580c8330 con 0x7f61580d1290 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6156d9d640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 0x7f61580b3370 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f615000c440 tx=0x7f615000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61500074e0 con 0x7f61580d1290 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61500047e0 con 0x7f61580d1290 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61580c8610 con 0x7f61580d1290 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6150004bd0 con 0x7f61580d1290 2024-03-04T10:10:49.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.519+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61580c8bd0 con 0x7f61580d1290 2024-03-04T10:10:49.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.520+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615815b1f0 con 0x7f61580d1290 2024-03-04T10:10:49.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.520+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6150011070 con 0x7f61580d1290 2024-03-04T10:10:49.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.521+0000 7f6146ffd640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f613806b4d0 0x7f613806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.525 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.521+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f615008a110 con 0x7f61580d1290 2024-03-04T10:10:49.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.521+0000 7f615659c640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f613806b4d0 0x7f613806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.521+0000 7f615659c640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f613806b4d0 0x7f613806d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6148036000 tx=0x7f6148030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.525+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f615815b1f0 con 0x7f61580d1290 2024-03-04T10:10:49.736 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.734+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f615806cd70 con 0x7f61580d1290 2024-03-04T10:10:49.737 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.734+0000 7f6146ffd640 1 -- 172.21.15.99:0/3573631552 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+4143 (secure 0 0 0) 0x7f6150058c30 con 0x7f61580d1290 2024-03-04T10:10:49.737 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:49.737 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":100,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:49.034050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:49.737 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 100 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f613806b4d0 msgr2=0x7f613806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f613806b4d0 0x7f613806d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6148036000 tx=0x7f6148030000 comp rx=0 tx=0).stop 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 msgr2=0x7f61580b3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 0x7f61580b3370 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f615000c440 tx=0x7f615000c900 comp rx=0 tx=0).stop 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 shutdown_connections 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f613806b4d0 0x7f613806d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.739 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f61580d1290 0x7f61580b3370 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61580cfae0 0x7f615815fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 --2- 172.21.15.99:0/3573631552 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f61580cc8e0 0x7f61581f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 >> 172.21.15.99:0/3573631552 conn(0x7f61580d0770 msgr2=0x7f61580d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 shutdown_connections 2024-03-04T10:10:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.736+0000 7f615c945640 1 -- 172.21.15.99:0/3573631552 wait complete. 2024-03-04T10:10:49.750 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-04T10:10:49.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:49.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.848+0000 7fd0a2cfc640 1 Processor -- start 2024-03-04T10:10:49.851 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.848+0000 7fd0a2cfc640 1 -- start start 2024-03-04T10:10:49.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a2cfc640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd09c15dea0 0x7fd09c0cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a2cfc640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd09c0cc8d0 0x7fd09c06acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a2cfc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c06b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a2cfc640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c06fad0 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a2cfc640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c06fd20 con 0x7fd09c15dea0 2024-03-04T10:10:49.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a2cfc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c070000 con 0x7fd09c0d1200 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd09c0cc8d0 0x7fd09c06acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fd09c0cc8d0 0x7fd09c06acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57904/0 (socket says 172.21.15.99:57904) 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 -- 172.21.15.99:0/3781219793 learned_addr learned my addr 172.21.15.99:0/3781219793 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a0a98640 1 --2- 172.21.15.99:0/3781219793 >> v2:172.21.15.162:3300/0 conn(0x7fd09c15dea0 0x7fd09c0cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a1299640 1 --2- 172.21.15.99:0/3781219793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 -- 172.21.15.99:0/3781219793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 msgr2=0x7fd09c06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 --2- 172.21.15.99:0/3781219793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c06b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 -- 172.21.15.99:0/3781219793 >> v2:172.21.15.162:3300/0 conn(0x7fd09c15dea0 msgr2=0x7fd09c0cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 --2- 172.21.15.99:0/3781219793 >> v2:172.21.15.162:3300/0 conn(0x7fd09c15dea0 0x7fd09c0cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd09bfff640 1 -- 172.21.15.99:0/3781219793 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0702e0 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.849+0000 7fd0a0a98640 1 --2- 172.21.15.99:0/3781219793 >> v2:172.21.15.162:3300/0 conn(0x7fd09c15dea0 0x7fd09c0cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.850+0000 7fd09bfff640 1 --2- 172.21.15.99:0/3781219793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c06acd0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0ce200 tx=0x7fd08802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f0877040d674b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.850+0000 7fd09b7fe640 1 -- 172.21.15.99:0/3781219793 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08802e070 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.850+0000 7fd09b7fe640 1 -- 172.21.15.99:0/3781219793 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd088037070 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.850+0000 7fd09b7fe640 1 -- 172.21.15.99:0/3781219793 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088031a00 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/3781219793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 msgr2=0x7fd09c06acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/3781219793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c06acd0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0ce200 tx=0x7fd08802b880 comp rx=0 tx=0).stop 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/3781219793 shutdown_connections 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/3781219793 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c06b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/3781219793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c06acd0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/3781219793 >> v2:172.21.15.162:3300/0 conn(0x7fd09c15dea0 0x7fd09c0cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/3781219793 >> 172.21.15.99:0/3781219793 conn(0x7fd09c0d0710 msgr2=0x7fd09c0d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/3781219793 shutdown_connections 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.851+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/3781219793 wait complete. 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.852+0000 7fd0a2cfc640 1 Processor -- start 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.852+0000 7fd0a2cfc640 1 -- start start 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c15b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c1e9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd09c15dea0 0x7fd09c0af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0cd480 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0cd300 con 0x7fd09c15dea0 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0cd600 con 0x7fd09c0d1200 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c15b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c15b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57914/0 (socket says 172.21.15.99:57914) 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 -- 172.21.15.99:0/4155722896 learned_addr learned my addr 172.21.15.99:0/4155722896 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd09bfff640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c1e9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a1299640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd09c15dea0 0x7fd09c0af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 -- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 msgr2=0x7fd09c1e9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c1e9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 -- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd09c15dea0 msgr2=0x7fd09c0af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd09c15dea0 0x7fd09c0af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 -- 172.21.15.99:0/4155722896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c0c9530 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a0a98640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c15b2a0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd09000d7d0 tx=0x7fd09000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090004030 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0900041d0 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.853+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c0c9810 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.854+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0900096a0 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.854+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c0c9dd0 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.854+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c074f60 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.855+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd090009840 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.855+0000 7fd0997fa640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd08006b4d0 0x7fd08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.855+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fd09008a8e0 con 0x7fd09c0cc8d0 2024-03-04T10:10:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.855+0000 7fd09bfff640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd08006b4d0 0x7fd08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.856+0000 7fd09bfff640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd08006b4d0 0x7fd08006d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd088036000 tx=0x7fd08803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:49.859+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd090059370 con 0x7fd09c0cc8d0 2024-03-04T10:10:50.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.068+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd09c0caa40 con 0x7fd09c0cc8d0 2024-03-04T10:10:50.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.069+0000 7fd0997fa640 1 -- 172.21.15.99:0/4155722896 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+4932 (secure 0 0 0) 0x7fd090058c70 con 0x7fd09c0cc8d0 2024-03-04T10:10:50.071 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:50.071 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":101,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74},{"gid":8550,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6835/301817793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":301817793},{"type":"v1","addr":"172.21.15.99:6835","nonce":301817793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:49.034050+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[],"damaged":[0],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:50.071 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 101 2024-03-04T10:10:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd08006b4d0 msgr2=0x7fd08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd08006b4d0 0x7fd08006d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd088036000 tx=0x7fd08803d040 comp rx=0 tx=0).stop 2024-03-04T10:10:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 msgr2=0x7fd09c15b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c15b2a0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd09000d7d0 tx=0x7fd09000dc90 comp rx=0 tx=0).stop 2024-03-04T10:10:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 shutdown_connections 2024-03-04T10:10:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd08006b4d0 0x7fd08006d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd09c15dea0 0x7fd09c0af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd09c0d1200 0x7fd09c1e9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 --2- 172.21.15.99:0/4155722896 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd09c0cc8d0 0x7fd09c15b2a0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 >> 172.21.15.99:0/4155722896 conn(0x7fd09c0d0710 msgr2=0x7fd09c06e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 shutdown_connections 2024-03-04T10:10:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.071+0000 7fd0a2cfc640 1 -- 172.21.15.99:0/4155722896 wait complete. 2024-03-04T10:10:50.085 INFO:tasks.cephfs.test_damage:Result: Mutation 'Delete 1.00000000.inode' led to DAMAGED state 2024-03-04T10:10:50.085 INFO:tasks.cephfs.test_damage:Applying mutation 'Delete 200.00000000' 2024-03-04T10:10:50.085 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:10:50.085 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:10:50.101 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:10:50.102 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:10:50.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:10:50.102 DEBUG:teuthology.orchestra.run.smithi099:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T10:10:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:10:50.157 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-04T10:10:50.157 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:10:50.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.296+0000 7fe00a681640 1 Processor -- start 2024-03-04T10:10:50.298 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.296+0000 7fe00a681640 1 -- start start 2024-03-04T10:10:50.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe00a681640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe00415dea0 0x7fe0040cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:50.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe00a681640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe0040cc8d0 0x7fe00406acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:50.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe00a681640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00406fd20 con 0x7fe0040d1200 2024-03-04T10:10:50.303 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe00a681640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00406fad0 con 0x7fe00415dea0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe00a681640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe004070000 con 0x7fe0040cc8d0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe00415dea0 0x7fe0040cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe00415dea0 0x7fe0040cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57916/0 (socket says 172.21.15.99:57916) 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 -- 172.21.15.99:0/1173336057 learned_addr learned my addr 172.21.15.99:0/1173336057 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe0037fe640 1 --2- 172.21.15.99:0/1173336057 >> v2:172.21.15.162:3300/0 conn(0x7fe0040cc8d0 0x7fe00406acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 -- 172.21.15.99:0/1173336057 >> v2:172.21.15.162:3300/0 conn(0x7fe0040cc8d0 msgr2=0x7fe00406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 --2- 172.21.15.99:0/1173336057 >> v2:172.21.15.162:3300/0 conn(0x7fe0040cc8d0 0x7fe00406acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 -- 172.21.15.99:0/1173336057 >> v1:172.21.15.162:6789/0 conn(0x7fe0040d1200 legacy=0x7fe0040cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.297+0000 7fe003fff640 1 -- 172.21.15.99:0/1173336057 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0040702e0 con 0x7fe00415dea0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.298+0000 7fe003fff640 1 --2- 172.21.15.99:0/1173336057 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe00415dea0 0x7fe0040cad60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fe00406e310 tx=0x7fdff402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5decc7bc50a898ee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.298+0000 7fe002ffd640 1 -- 172.21.15.99:0/1173336057 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4035020 con 0x7fe00415dea0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.298+0000 7fe002ffd640 1 -- 172.21.15.99:0/1173336057 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff4031870 con 0x7fe00415dea0 2024-03-04T10:10:50.304 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.298+0000 7fe002ffd640 1 -- 172.21.15.99:0/1173336057 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff4031c40 con 0x7fe00415dea0 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.298+0000 7fe00a681640 1 -- 172.21.15.99:0/1173336057 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe00415dea0 msgr2=0x7fe0040cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.298+0000 7fe00a681640 1 --2- 172.21.15.99:0/1173336057 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe00415dea0 0x7fe0040cad60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fe00406e310 tx=0x7fdff402b910 comp rx=0 tx=0).stop 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.299+0000 7fe00a681640 1 -- 172.21.15.99:0/1173336057 shutdown_connections 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.299+0000 7fe00a681640 1 --2- 172.21.15.99:0/1173336057 >> v2:172.21.15.162:3300/0 conn(0x7fe0040cc8d0 0x7fe00406acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.299+0000 7fe00a681640 1 --2- 172.21.15.99:0/1173336057 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe00415dea0 0x7fe0040cad60 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.299+0000 7fe00a681640 1 -- 172.21.15.99:0/1173336057 >> 172.21.15.99:0/1173336057 conn(0x7fe0040d0710 msgr2=0x7fe0040d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.299+0000 7fe00a681640 1 -- 172.21.15.99:0/1173336057 shutdown_connections 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.299+0000 7fe00a681640 1 -- 172.21.15.99:0/1173336057 wait complete. 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.300+0000 7fe00a681640 1 Processor -- start 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.300+0000 7fe00a681640 1 -- start start 2024-03-04T10:10:50.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040cc8d0 0x7fe0040c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 0x7fe00414fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe00415dea0 0x7fe0040c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00406b390 con 0x7fe0040d1200 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00406b210 con 0x7fe00415dea0 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00406b510 con 0x7fe0040cc8d0 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 0x7fe00414fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 0x7fe00414fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57932/0 (socket says 172.21.15.99:57932) 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 -- 172.21.15.99:0/3689384289 learned_addr learned my addr 172.21.15.99:0/3689384289 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe003fff640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040cc8d0 0x7fe0040c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe008c1e640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe00415dea0 0x7fe0040c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 -- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040cc8d0 msgr2=0x7fe0040c8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040cc8d0 0x7fe0040c8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 -- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe00415dea0 msgr2=0x7fe0040c77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:50.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe00415dea0 0x7fe0040c77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 -- 172.21.15.99:0/3689384289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0040c7f50 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe0037fe640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 0x7fe00414fd30 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fdff800d5f0 tx=0x7fdff800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff80125f0 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8019070 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0040c8230 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff80129d0 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.301+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0040c87f0 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.302+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe004074f60 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.303+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdff8012b70 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.303+0000 7fe000ff9640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdff006b4d0 0x7fdff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.303+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fdff808a870 con 0x7fe0040d1200 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.303+0000 7fe003fff640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdff006b4d0 0x7fdff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:50.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.304+0000 7fe003fff640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdff006b4d0 0x7fdff006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdff4036000 tx=0x7fdff403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:50.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.307+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdff8058d40 con 0x7fe0040d1200 2024-03-04T10:10:50.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:50.514+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe0040cbbd0 con 0x7fe0040d1200 2024-03-04T10:10:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.051+0000 7fe000ff9640 1 -- 172.21.15.99:0/3689384289 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v102) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdff805cde0 con 0x7fe0040d1200 2024-03-04T10:10:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdff006b4d0 msgr2=0x7fdff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdff006b4d0 0x7fdff006d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fdff4036000 tx=0x7fdff403c040 comp rx=0 tx=0).stop 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 msgr2=0x7fe00414fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 0x7fe00414fd30 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fdff800d5f0 tx=0x7fdff800dab0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 shutdown_connections 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdff006b4d0 0x7fdff006d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe00415dea0 0x7fe0040c77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe0040d1200 0x7fe00414fd30 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 --2- 172.21.15.99:0/3689384289 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe0040cc8d0 0x7fe0040c8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 >> 172.21.15.99:0/3689384289 conn(0x7fe0040d0710 msgr2=0x7fe0040b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 shutdown_connections 2024-03-04T10:10:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.054+0000 7fe00a681640 1 -- 172.21.15.99:0/3689384289 wait complete. 2024-03-04T10:10:51.069 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds repaired 0 2024-03-04T10:10:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.171+0000 7f7fcdcfe640 1 Processor -- start 2024-03-04T10:10:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.171+0000 7f7fcdcfe640 1 -- start start 2024-03-04T10:10:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fcdcfe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7fc806c090 0x7f7fc806f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fcdcfe640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7fc806c470 0x7f7fc80af220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fcdcfe640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc80ad6c0 con 0x7f7fc8066de0 2024-03-04T10:10:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fcdcfe640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc80ad960 con 0x7f7fc806c090 2024-03-04T10:10:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fcdcfe640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fc806ebc0 con 0x7f7fc806c470 2024-03-04T10:10:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fc77fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7fc806c090 0x7f7fc806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fc77fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f7fc806c090 0x7f7fc806f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57942/0 (socket says 172.21.15.99:57942) 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fc77fe640 1 -- 172.21.15.99:0/1636835957 learned_addr learned my addr 172.21.15.99:0/1636835957 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.172+0000 7f7fc77fe640 1 -- 172.21.15.99:0/1636835957 >> v2:172.21.15.162:3300/0 conn(0x7f7fc806c470 msgr2=0x7f7fc80af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc6ffd640 1 --2- 172.21.15.99:0/1636835957 >> v2:172.21.15.162:3300/0 conn(0x7f7fc806c470 0x7f7fc80af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc77fe640 1 --2- 172.21.15.99:0/1636835957 >> v2:172.21.15.162:3300/0 conn(0x7f7fc806c470 0x7f7fc80af220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc77fe640 1 -- 172.21.15.99:0/1636835957 >> v1:172.21.15.162:6789/0 conn(0x7f7fc8066de0 legacy=0x7f7fc815f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc77fe640 1 -- 172.21.15.99:0/1636835957 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc806eea0 con 0x7f7fc806c090 2024-03-04T10:10:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc6ffd640 1 --2- 172.21.15.99:0/1636835957 >> v2:172.21.15.162:3300/0 conn(0x7f7fc806c470 0x7f7fc80af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc77fe640 1 --2- 172.21.15.99:0/1636835957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc806c090 0x7f7fc806f680 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f7fc806d540 tx=0x7f7fb802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371c254b553af1c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc67fc640 1 -- 172.21.15.99:0/1636835957 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb802e070 con 0x7f7fc806c090 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc67fc640 1 -- 172.21.15.99:0/1636835957 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb8037070 con 0x7f7fc806c090 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.173+0000 7f7fc67fc640 1 -- 172.21.15.99:0/1636835957 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8031a00 con 0x7f7fc806c090 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/1636835957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc806c090 msgr2=0x7f7fc806f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/1636835957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc806c090 0x7f7fc806f680 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f7fc806d540 tx=0x7f7fb802b880 comp rx=0 tx=0).stop 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/1636835957 shutdown_connections 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/1636835957 >> v2:172.21.15.162:3300/0 conn(0x7f7fc806c470 0x7f7fc80af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/1636835957 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc806c090 0x7f7fc806f680 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/1636835957 >> 172.21.15.99:0/1636835957 conn(0x7f7fc80cfef0 msgr2=0x7f7fc80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/1636835957 shutdown_connections 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.174+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/1636835957 wait complete. 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.175+0000 7f7fcdcfe640 1 Processor -- start 2024-03-04T10:10:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.175+0000 7f7fcdcfe640 1 -- start start 2024-03-04T10:10:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fcdcfe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 0x7f7fc80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fcdcfe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fc80b33a0 0x7f7fc80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fcdcfe640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7fc81f7550 0x7f7fc81f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fcdcfe640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc80adc40 con 0x7f7fc8066de0 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fcdcfe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc80af760 con 0x7f7fc80b33a0 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fcdcfe640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc80addc0 con 0x7f7fc81f7550 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 0x7f7fc80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 0x7f7fc80b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57954/0 (socket says 172.21.15.99:57954) 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 -- 172.21.15.99:0/4061694615 learned_addr learned my addr 172.21.15.99:0/4061694615 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc6ffd640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fc80b33a0 0x7f7fc80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc7fff640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7fc81f7550 0x7f7fc81f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 -- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7fc81f7550 msgr2=0x7f7fc81f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7fc81f7550 0x7f7fc81f79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 -- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fc80b33a0 msgr2=0x7f7fc80b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fc80b33a0 0x7f7fc80b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc77fe640 1 -- 172.21.15.99:0/4061694615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8155920 con 0x7f7fc8066de0 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc7fff640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7fc81f7550 0x7f7fc81f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.176+0000 7f7fc6ffd640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fc80b33a0 0x7f7fc80b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:10:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.177+0000 7f7fc77fe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 0x7f7fc80b2e60 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000c00 tx=0x7f7fb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.177+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8032880 con 0x7f7fc8066de0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.177+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb8032a40 con 0x7f7fc8066de0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.177+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8032e30 con 0x7f7fc8066de0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.177+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc814f4f0 con 0x7f7fc8066de0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.177+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc8152970 con 0x7f7fc8066de0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.178+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc8074f60 con 0x7f7fc8066de0 2024-03-04T10:10:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.178+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7fb804c060 con 0x7f7fc8066de0 2024-03-04T10:10:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.179+0000 7f7faffff640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7fb406b500 0x7f7fb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.179+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f7fb80abcd0 con 0x7f7fc8066de0 2024-03-04T10:10:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.179+0000 7f7fc6ffd640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7fb406b500 0x7f7fb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.179+0000 7f7fc6ffd640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7fb406b500 0x7f7fb406d9c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7fc809dfc0 tx=0x7f7fbc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:51.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.183+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7fb8040070 con 0x7f7fc8066de0 2024-03-04T10:10:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:51.424+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "mds repaired", "role": "0"} v 0) v1 -- 0x7f7fc806def0 con 0x7f7fc8066de0 2024-03-04T10:10:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.057+0000 7f7faffff640 1 -- 172.21.15.99:0/4061694615 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "mds repaired", "role": "0"}]=0 repaired: restoring rank 6:0 v103) v1 ==== 101+0+0 (secure 0 0 0) 0x7f7fb807a1a0 con 0x7f7fc8066de0 2024-03-04T10:10:52.059 INFO:teuthology.orchestra.run.smithi099.stderr:repaired: restoring rank 6:0 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7fb406b500 msgr2=0x7f7fb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7fb406b500 0x7f7fb406d9c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7fc809dfc0 tx=0x7f7fbc008040 comp rx=0 tx=0).stop 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 msgr2=0x7f7fc80b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 0x7f7fc80b2e60 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8000c00 tx=0x7f7fb80046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 shutdown_connections 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7fb406b500 0x7f7fb406d9c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7fc81f7550 0x7f7fc81f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fc80b33a0 0x7f7fc80b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 --2- 172.21.15.99:0/4061694615 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7fc8066de0 0x7f7fc80b2e60 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 >> 172.21.15.99:0/4061694615 conn(0x7f7fc80cfef0 msgr2=0x7f7fc815fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 shutdown_connections 2024-03-04T10:10:52.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.059+0000 7f7fcdcfe640 1 -- 172.21.15.99:0/4061694615 wait complete. 2024-03-04T10:10:52.073 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata import - 2024-03-04T10:10:52.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.113+0000 7f4ade436f40 1 Processor -- start 2024-03-04T10:10:52.116 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.113+0000 7f4ade436f40 1 -- start start 2024-03-04T10:10:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ade436f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 0x557ebf4862d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ade436f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557ebf3e4160 0x557ebf3b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ade436f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e45e0 0x557ebf48c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ade436f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ebf35fb30 con 0x7f4ac4003ab0 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ade436f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ebf363a90 con 0x557ebf3e45e0 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ade436f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ebf401ab0 con 0x557ebf3e4160 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 0x557ebf4862d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 0x557ebf4862d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57976/0 (socket says 172.21.15.99:57976) 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 -- 172.21.15.99:0/4220381732 learned_addr learned my addr 172.21.15.99:0/4220381732 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e45e0 0x557ebf48c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 -- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557ebf3e4160 msgr2=0x557ebf3b99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adafca640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557ebf3e4160 0x557ebf3b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557ebf3e4160 0x557ebf3b99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 -- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e45e0 msgr2=0x557ebf48c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e45e0 0x557ebf48c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 -- 172.21.15.99:0/4220381732 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ebf48dd50 con 0x7f4ac4003ab0 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e45e0 0x557ebf48c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adafca640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557ebf3e4160 0x557ebf3b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 0x557ebf4862d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0097b0 tx=0x7f4acc005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fb1ed3915f87ff server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ad9fc8640 1 -- 172.21.15.99:0/4220381732 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc007830 con 0x7f4ac4003ab0 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ad9fc8640 1 -- 172.21.15.99:0/4220381732 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc0079d0 con 0x7f4ac4003ab0 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.114+0000 7f4ad9fc8640 1 -- 172.21.15.99:0/4220381732 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc007d80 con 0x7f4ac4003ab0 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 -- 172.21.15.99:0/4220381732 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 msgr2=0x557ebf4862d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 0x557ebf4862d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4acc0097b0 tx=0x7f4acc005a00 comp rx=0 tx=0).stop 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 -- 172.21.15.99:0/4220381732 shutdown_connections 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e45e0 0x557ebf48c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x557ebf3e4160 0x557ebf3b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 --2- 172.21.15.99:0/4220381732 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ac4003ab0 0x557ebf4862d0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 -- 172.21.15.99:0/4220381732 >> 172.21.15.99:0/4220381732 conn(0x557ebf3b8400 msgr2=0x557ebf3e8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 -- 172.21.15.99:0/4220381732 shutdown_connections 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.115+0000 7f4ade436f40 1 -- 172.21.15.99:0/4220381732 wait complete. 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 Processor -- start 2024-03-04T10:10:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 -- start start 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e4160 0x557ebf480bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 0x557ebf48c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ac4003ab0 0x557ebf46fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ebf3ffaa0 con 0x557ebf3e45e0 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ebf487790 con 0x557ebf3e4160 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4ade436f40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ebf48e2a0 con 0x7f4ac4003ab0 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4adafca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 0x557ebf48c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4adafca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 0x557ebf48c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57980/0 (socket says 172.21.15.99:57980) 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 learned_addr learned my addr 172.21.15.99:0/797728092 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.116+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ac4003ab0 0x557ebf46fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e4160 0x557ebf480bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ac4003ab0 msgr2=0x557ebf46fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4ac4003ab0 0x557ebf46fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e4160 msgr2=0x557ebf480bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x557ebf3e4160 0x557ebf480bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ebf488c30 con 0x557ebf3e45e0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 0x557ebf48c380 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0000c00 tx=0x7f4ad000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4ad8fc6640 1 -- 172.21.15.99:0/797728092 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0019430 con 0x557ebf3e45e0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4ad8fc6640 1 -- 172.21.15.99:0/797728092 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad0013030 con 0x557ebf3e45e0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557ebf488ed0 con 0x557ebf3e45e0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4ad8fc6640 1 -- 172.21.15.99:0/797728092 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad0019810 con 0x557ebf3e45e0 2024-03-04T10:10:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.117+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ebf489490 con 0x557ebf3e45e0 2024-03-04T10:10:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.118+0000 7f4ad8fc6640 1 -- 172.21.15.99:0/797728092 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4ad000f3f0 con 0x557ebf3e45e0 2024-03-04T10:10:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4ad8fc6640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4ac8069eb0 0x7f4ac806c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4ad8fc6640 1 -- 172.21.15.99:0/797728092 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f4ad00894f0 con 0x557ebf3e45e0 2024-03-04T10:10:52.125 INFO:teuthology.orchestra.run.smithi099.stdout:Importing pool 2024-03-04T10:10:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4ac8069eb0 0x7f4ac806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x557ebf4810f0 0x557ebf3e7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:1 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf527090 con 0x557ebf4810f0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x557ebf4810f0 0x557ebf3e7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x557ebf4810f0 0x557ebf3e7770 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.119+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4ac8069eb0 0x7f4ac806c370 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4acc006010 tx=0x7f4acc007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.121+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== osd_op_reply(1 601.00000000 [create] v99'102 uv78 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.121+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:2 13.10 13:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf3baed0 con 0x557ebf4810f0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.122+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== osd_op_reply(2 601.00000000 [delete] v99'103 uv103 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.122+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:3 13.10 13:09401a81:::601.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf48c8c0 con 0x557ebf4810f0 2024-03-04T10:10:52.127 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::601.00000000:head# 2024-03-04T10:10:52.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.124+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 3 ==== osd_op_reply(3 601.00000000 [create] v99'104 uv104 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.128 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.124+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:4 13.10 13:09401a81:::601.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf3e7cb0 con 0x557ebf4810f0 2024-03-04T10:10:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.127+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 4 ==== osd_op_reply(4 601.00000000 [setxattr (30)] v99'105 uv105 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.127+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:5 13.10 13:09401a81:::601.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534820 con 0x557ebf4810f0 2024-03-04T10:10:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.130+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 5 ==== osd_op_reply(5 601.00000000 [setxattr (62)] v99'106 uv106 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.130+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:6 13.10 13:09401a81:::601.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533810 con 0x557ebf4810f0 2024-03-04T10:10:52.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.132+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 6 ==== osd_op_reply(6 601.00000000 [omap-set-header] v99'107 uv107 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.135 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.133+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:7 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533bd0 con 0x557ebf4810f0 2024-03-04T10:10:52.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.135+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 7 ==== osd_op_reply(7 602.00000000 [create] v99'108 uv84 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.135+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:8 13.10 13:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf3e8050 con 0x557ebf4810f0 2024-03-04T10:10:52.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.140+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 8 ==== osd_op_reply(8 602.00000000 [delete] v99'109 uv109 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.142 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.140+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:9 13.10 13:0bd6d154:::602.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533fc0 con 0x557ebf4810f0 2024-03-04T10:10:52.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.141+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 9 ==== osd_op_reply(9 602.00000000 [create] v99'110 uv110 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.143 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::602.00000000:head# 2024-03-04T10:10:52.143 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.141+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:10 13.10 13:0bd6d154:::602.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.144+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 10 ==== osd_op_reply(10 602.00000000 [setxattr (30)] v99'111 uv111 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.146 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.144+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:11 13.10 13:0bd6d154:::602.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.144+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 11 ==== osd_op_reply(11 602.00000000 [setxattr (62)] v99'112 uv112 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.147 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.145+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:12 13.10 13:0bd6d154:::602.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.149+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 12 ==== osd_op_reply(12 602.00000000 [omap-set-header] v99'113 uv113 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.152 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.150+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:13 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.150+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 13 ==== osd_op_reply(13 10000000000.00000000 [create] v99'114 uv91 ondisk = -17 ((17) File exists)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.153 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.150+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:14 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.153+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 14 ==== osd_op_reply(14 10000000000.00000000 [delete] v99'115 uv115 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.155 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.153+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:15 13.10 13:0cf6ad0f:::10000000000.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.156+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 15 ==== osd_op_reply(15 10000000000.00000000 [create] v99'116 uv116 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.158 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::10000000000.00000000:head# 2024-03-04T10:10:52.158 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.156+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:16 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.159+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 16 ==== osd_op_reply(16 10000000000.00000000 [setxattr (30)] v99'117 uv117 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.159+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:17 13.10 13:0cf6ad0f:::10000000000.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.159+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 17 ==== osd_op_reply(17 10000000000.00000000 [setxattr (62)] v99'118 uv118 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.159+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:18 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf533f70 con 0x557ebf4810f0 2024-03-04T10:10:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.165+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 18 ==== osd_op_reply(18 10000000000.00000000 [omap-set-header] v99'119 uv119 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.165+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:19 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=585b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.166+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 19 ==== osd_op_reply(19 10000000000.00000000 [omap-set-vals] v99'120 uv120 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.166+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:20 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.169+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 20 ==== osd_op_reply(20 600.00000000 [create] v99'121 uv95 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.169+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:21 13.10 13:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.171+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 21 ==== osd_op_reply(21 600.00000000 [delete] v99'122 uv122 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.171+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:22 13.10 13:0d82a743:::600.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.174+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 22 ==== osd_op_reply(22 600.00000000 [create] v99'123 uv123 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.177 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::600.00000000:head# 2024-03-04T10:10:52.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.175+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:23 13.10 13:0d82a743:::600.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf535bd0 con 0x557ebf4810f0 2024-03-04T10:10:52.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.177+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 23 ==== osd_op_reply(23 600.00000000 [setxattr (30)] v99'124 uv124 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.177+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:24 13.10 13:0d82a743:::600.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf535bd0 con 0x557ebf4810f0 2024-03-04T10:10:52.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.180+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 24 ==== osd_op_reply(24 600.00000000 [setxattr (62)] v99'125 uv125 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.180+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:25 13.10 13:0d82a743:::600.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.183+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 25 ==== osd_op_reply(25 600.00000000 [omap-set-header] v99'126 uv126 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.183+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:26 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.186+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 26 ==== osd_op_reply(26 603.00000000 [create] v99'127 uv101 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.186+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:27 13.10 13:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.187+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 27 ==== osd_op_reply(27 603.00000000 [delete] v99'128 uv128 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.189 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.187+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:28 13.10 13:0d89b25e:::603.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.188+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 28 ==== osd_op_reply(28 603.00000000 [create] v99'129 uv129 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.190 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::603.00000000:head# 2024-03-04T10:10:52.190 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.188+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:29 13.10 13:0d89b25e:::603.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf535bd0 con 0x557ebf4810f0 2024-03-04T10:10:52.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.189+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 29 ==== osd_op_reply(29 603.00000000 [setxattr (30)] v99'130 uv130 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.191 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.189+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:30 13.10 13:0d89b25e:::603.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf535bd0 con 0x557ebf4810f0 2024-03-04T10:10:52.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.190+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 30 ==== osd_op_reply(30 603.00000000 [setxattr (62)] v99'131 uv131 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.192 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.190+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- osd_op(unknown.0.0:31 13.10 13:0d89b25e:::603.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf534bc0 con 0x557ebf4810f0 2024-03-04T10:10:52.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.191+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.1 v2:172.21.15.99:6824/3369025248 31 ==== osd_op_reply(31 603.00000000 [omap-set-header] v99'132 uv132 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf4810f0 2024-03-04T10:10:52.193 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.191+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x557ebf5350d0 0x557ebf5374c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.191+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:32 13.14 13:287034c4:::1.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf537a00 con 0x557ebf5350d0 2024-03-04T10:10:52.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.191+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x557ebf5350d0 0x557ebf5374c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.194 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.191+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x557ebf5350d0 0x557ebf5374c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.195 INFO:teuthology.orchestra.run.smithi099.stdout:Write #-9223372036854775808:00000000:::1.00000000.inode:head# 2024-03-04T10:10:52.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.193+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(32 1.00000000.inode [create] v99'90 uv90 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.196 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.193+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:33 13.14 13:287034c4:::1.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5382d0 con 0x557ebf5350d0 2024-03-04T10:10:52.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.194+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== osd_op_reply(33 1.00000000.inode [write 0~582] v99'91 uv91 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.197 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.194+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:34 13.14 13:287034c4:::1.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf537ea0 con 0x557ebf5350d0 2024-03-04T10:10:52.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.196+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 3 ==== osd_op_reply(34 1.00000000.inode [omap-set-header] v99'92 uv92 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.198 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.196+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:35 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf538ba0 con 0x557ebf5350d0 2024-03-04T10:10:52.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.197+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 4 ==== osd_op_reply(35 200.00000000 [create] v99'93 uv77 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.199 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.197+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:36 13.14 13:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf538f90 con 0x557ebf5350d0 2024-03-04T10:10:52.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.198+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 5 ==== osd_op_reply(36 200.00000000 [delete] v99'94 uv94 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.200 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.198+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:37 13.14 13:292cf221:::200.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539330 con 0x557ebf5350d0 2024-03-04T10:10:52.201 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000000:head# 2024-03-04T10:10:52.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.199+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 6 ==== osd_op_reply(37 200.00000000 [create] v99'95 uv95 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.201 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.199+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:38 13.14 13:292cf221:::200.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5396d0 con 0x557ebf5350d0 2024-03-04T10:10:52.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.200+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 7 ==== osd_op_reply(38 200.00000000 [write 0~90] v99'96 uv96 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.202 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.200+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:39 13.14 13:292cf221:::200.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539ac0 con 0x557ebf5350d0 2024-03-04T10:10:52.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.201+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 8 ==== osd_op_reply(39 200.00000000 [omap-set-header] v99'97 uv97 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.203 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.201+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:40 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539ac0 con 0x557ebf5350d0 2024-03-04T10:10:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.202+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 9 ==== osd_op_reply(40 200.00000001 [create] v99'98 uv82 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.202+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:41 13.14 13:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.203+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 10 ==== osd_op_reply(41 200.00000001 [delete] v99'99 uv99 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.206 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.203+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:42 13.14 13:2e2fa760:::200.00000001:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.207 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::200.00000001:head# 2024-03-04T10:10:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.205+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 11 ==== osd_op_reply(42 200.00000001 [create] v99'100 uv100 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.205+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:43 13.14 13:2e2fa760:::200.00000001:head [write 0~18075 in=18075b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.206+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 12 ==== osd_op_reply(43 200.00000001 [write 0~18075] v99'101 uv101 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.206+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:44 13.14 13:2e2fa760:::200.00000001:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.207+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 13 ==== osd_op_reply(44 200.00000001 [omap-set-header] v99'102 uv102 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.207+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:45 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.208+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 14 ==== osd_op_reply(45 606.00000000 [create] v99'103 uv88 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.208+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:46 13.14 13:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.210+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 15 ==== osd_op_reply(46 606.00000000 [delete] v99'104 uv104 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.210+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:47 13.14 13:2f55791f:::606.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.213 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::606.00000000:head# 2024-03-04T10:10:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.211+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 16 ==== osd_op_reply(47 606.00000000 [create] v99'105 uv105 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.211+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:48 13.14 13:2f55791f:::606.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.212+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 17 ==== osd_op_reply(48 606.00000000 [setxattr (30)] v99'106 uv106 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.212+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:49 13.14 13:2f55791f:::606.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.213+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 18 ==== osd_op_reply(49 606.00000000 [setxattr (62)] v99'107 uv107 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.213+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:50 13.14 13:2f55791f:::606.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf539a70 con 0x557ebf5350d0 2024-03-04T10:10:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.214+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.0 v2:172.21.15.99:6800/224245114 19 ==== osd_op_reply(50 606.00000000 [omap-set-header] v99'108 uv108 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf5350d0 2024-03-04T10:10:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.214+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x557ebf53a370 0x557ebf53a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.214+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:51 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53cef0 con 0x557ebf53a370 2024-03-04T10:10:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.214+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x557ebf53a370 0x557ebf53a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.215+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x557ebf53a370 0x557ebf53a750 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.216+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== osd_op_reply(51 607.00000000 [create] v99'21 uv20 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.216+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:52 13.1c 13:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53d290 con 0x557ebf53a370 2024-03-04T10:10:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.218+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== osd_op_reply(52 607.00000000 [delete] v99'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.218+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:53 13.1c 13:3ed09add:::607.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53d630 con 0x557ebf53a370 2024-03-04T10:10:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.219+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 3 ==== osd_op_reply(53 607.00000000 [create] v99'23 uv23 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.221 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::607.00000000:head# 2024-03-04T10:10:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.219+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:54 13.1c 13:3ed09add:::607.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53eee0 con 0x557ebf53a370 2024-03-04T10:10:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.220+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 4 ==== osd_op_reply(54 607.00000000 [setxattr (30)] v99'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.220+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:55 13.1c 13:3ed09add:::607.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53f280 con 0x557ebf53a370 2024-03-04T10:10:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.221+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 5 ==== osd_op_reply(55 607.00000000 [setxattr (62)] v99'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.221+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:56 13.1c 13:3ed09add:::607.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53ded0 con 0x557ebf53a370 2024-03-04T10:10:52.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.222+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 6 ==== osd_op_reply(56 607.00000000 [omap-set-header] v99'26 uv26 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.222+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x557ebf53e7f0 0x557ebf541630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.222+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:57 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf541b70 con 0x557ebf53e7f0 2024-03-04T10:10:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.222+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x557ebf53e7f0 0x557ebf541630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.223+0000 7f4adbfcc640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x557ebf53e7f0 0x557ebf541630 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.224+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== osd_op_reply(57 mds0_openfiles.0 [create] v99'18 uv17 ondisk = -17 ((17) File exists)) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.224+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:58 13.6 13:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf541b70 con 0x557ebf53e7f0 2024-03-04T10:10:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.225+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== osd_op_reply(58 mds0_openfiles.0 [delete] v99'19 uv19 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.225+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:59 13.6 13:654134d2:::mds0_openfiles.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf541f60 con 0x557ebf53e7f0 2024-03-04T10:10:52.230 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_openfiles.0:head# 2024-03-04T10:10:52.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.229+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 3 ==== osd_op_reply(59 mds0_openfiles.0 [create] v99'20 uv20 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.229+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:60 13.6 13:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf541f60 con 0x557ebf53e7f0 2024-03-04T10:10:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.231+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 4 ==== osd_op_reply(60 mds0_openfiles.0 [omap-set-header] v99'21 uv21 ondisk = 0) v8 ==== 160+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.231+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x557ebf542470 0x557ebf544860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.231+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:61 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf544de0 con 0x557ebf542470 2024-03-04T10:10:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.231+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x557ebf542470 0x557ebf544860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.231+0000 7f4adb7cb640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x557ebf542470 0x557ebf544860 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.233+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== osd_op_reply(61 608.00000000 [create] v99'21 uv20 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.233+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:62 13.e 13:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf544ea0 con 0x557ebf542470 2024-03-04T10:10:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.234+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== osd_op_reply(62 608.00000000 [delete] v99'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.234+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:63 13.e 13:717a0223:::608.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.239 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::608.00000000:head# 2024-03-04T10:10:52.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.236+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 3 ==== osd_op_reply(63 608.00000000 [create] v99'23 uv23 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.237+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:64 13.e 13:717a0223:::608.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5462d0 con 0x557ebf542470 2024-03-04T10:10:52.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.239+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 4 ==== osd_op_reply(64 608.00000000 [setxattr (30)] v99'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.239+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:65 13.e 13:717a0223:::608.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5462d0 con 0x557ebf542470 2024-03-04T10:10:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.242+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 5 ==== osd_op_reply(65 608.00000000 [setxattr (62)] v99'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.244 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.242+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:66 13.e 13:717a0223:::608.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.245+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 6 ==== osd_op_reply(66 608.00000000 [omap-set-header] v99'26 uv26 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.245+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:67 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.246+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 5 ==== osd_op_reply(67 604.00000000 [create] v99'37 uv31 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.246+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:68 13.15 13:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.247+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 6 ==== osd_op_reply(68 604.00000000 [delete] v99'38 uv38 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.247+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:69 13.15 13:a93a17c2:::604.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.255 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::604.00000000:head# 2024-03-04T10:10:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.253+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 7 ==== osd_op_reply(69 604.00000000 [create] v99'39 uv39 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.253+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:70 13.15 13:a93a17c2:::604.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5462d0 con 0x557ebf53e7f0 2024-03-04T10:10:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.254+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 8 ==== osd_op_reply(70 604.00000000 [setxattr (30)] v99'40 uv40 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.254+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:71 13.15 13:a93a17c2:::604.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5462d0 con 0x557ebf53e7f0 2024-03-04T10:10:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.257+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 9 ==== osd_op_reply(71 604.00000000 [setxattr (62)] v99'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.257+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:72 13.15 13:a93a17c2:::604.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.260+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 10 ==== osd_op_reply(72 604.00000000 [omap-set-header] v99'42 uv42 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.262 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.260+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:73 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.263+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 11 ==== osd_op_reply(73 500.00000000 [create] v99'43 uv36 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.265 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.263+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:74 13.15 13:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.265+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 12 ==== osd_op_reply(74 500.00000000 [delete] v99'44 uv44 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.265+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:75 13.15 13:aa448500:::500.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.270 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::500.00000000:head# 2024-03-04T10:10:52.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.268+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 13 ==== osd_op_reply(75 500.00000000 [create] v99'45 uv45 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.270 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.268+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:76 13.15 13:aa448500:::500.00000000:head [write 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.271+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 14 ==== osd_op_reply(76 500.00000000 [write 0~90] v99'46 uv46 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.273 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.271+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:77 13.15 13:aa448500:::500.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.274+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 15 ==== osd_op_reply(77 500.00000000 [omap-set-header] v99'47 uv47 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.274+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:78 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.275+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 16 ==== osd_op_reply(78 mds_snaptable [create] v99'17 uv16 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.275+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:79 13.d 13:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.279+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 17 ==== osd_op_reply(79 mds_snaptable [delete] v99'18 uv18 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.280+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:80 13.d 13:b50e409b:::mds_snaptable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.283 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds_snaptable:head# 2024-03-04T10:10:52.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.281+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 18 ==== osd_op_reply(80 mds_snaptable [create] v99'19 uv19 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.281+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:81 13.d 13:b50e409b:::mds_snaptable:head [write 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.284+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 19 ==== osd_op_reply(81 mds_snaptable [write 0~70] v99'20 uv20 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.284+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- osd_op(unknown.0.0:82 13.d 13:b50e409b:::mds_snaptable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf53e7f0 2024-03-04T10:10:52.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.285+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 <== osd.4 v2:172.21.15.162:6800/3079837853 20 ==== osd_op_reply(82 mds_snaptable [omap-set-header] v99'21 uv21 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4ad4003030 con 0x557ebf53e7f0 2024-03-04T10:10:52.287 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.285+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:83 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.286+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 7 ==== osd_op_reply(83 605.00000000 [create] v99'21 uv20 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.288 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.286+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:84 13.1d 13:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.289 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.287+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 8 ==== osd_op_reply(84 605.00000000 [delete] v99'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.287+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:85 13.1d 13:b871830b:::605.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.290 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::605.00000000:head# 2024-03-04T10:10:52.290 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.288+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 9 ==== osd_op_reply(85 605.00000000 [create] v99'23 uv23 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.288+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:86 13.1d 13:b871830b:::605.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5462d0 con 0x557ebf542470 2024-03-04T10:10:52.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.289+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 10 ==== osd_op_reply(86 605.00000000 [setxattr (30)] v99'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.291 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.289+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:87 13.1d 13:b871830b:::605.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5462d0 con 0x557ebf542470 2024-03-04T10:10:52.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.290+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 11 ==== osd_op_reply(87 605.00000000 [setxattr (62)] v99'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.292 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.290+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:88 13.1d 13:b871830b:::605.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.291+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 12 ==== osd_op_reply(88 605.00000000 [omap-set-header] v99'26 uv26 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.293 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.291+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:89 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.292+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 13 ==== osd_op_reply(89 mds0_inotable [create] v99'34 uv28 ondisk = -17 ((17) File exists)) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.294 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.292+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:90 13.13 13:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.295 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.293+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 14 ==== osd_op_reply(90 mds0_inotable [delete] v99'35 uv35 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.296 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.293+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:91 13.13 13:c91d4a1d:::mds0_inotable:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5452c0 con 0x557ebf542470 2024-03-04T10:10:52.298 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_inotable:head# 2024-03-04T10:10:52.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.296+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 15 ==== osd_op_reply(91 mds0_inotable [create] v99'36 uv36 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.296+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:92 13.13 13:c91d4a1d:::mds0_inotable:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53dac0 con 0x557ebf542470 2024-03-04T10:10:52.299 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.297+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 16 ==== osd_op_reply(92 mds0_inotable [write 0~34] v99'37 uv37 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.300 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.297+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:93 13.13 13:c91d4a1d:::mds0_inotable:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53dac0 con 0x557ebf542470 2024-03-04T10:10:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.300+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 17 ==== osd_op_reply(93 mds0_inotable [omap-set-header] v99'38 uv38 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.300+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:94 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53dac0 con 0x557ebf542470 2024-03-04T10:10:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.303+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 18 ==== osd_op_reply(94 100.00000000 [create] v99'39 uv33 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.303+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:95 13.13 13:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53dac0 con 0x557ebf542470 2024-03-04T10:10:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.308+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 19 ==== osd_op_reply(95 100.00000000 [delete] v99'40 uv40 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.308+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:96 13.13 13:cd5a64a3:::100.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53dac0 con 0x557ebf542470 2024-03-04T10:10:52.311 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000:head# 2024-03-04T10:10:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.309+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 20 ==== osd_op_reply(96 100.00000000 [create] v99'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.309+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:97 13.13 13:cd5a64a3:::100.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf53dac0 con 0x557ebf542470 2024-03-04T10:10:52.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.310+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 21 ==== osd_op_reply(97 100.00000000 [omap-set-header] v99'42 uv42 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.310+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:98 13.13 13:cd5a64a3:::100.00000000:head [omap-set-vals in=5194b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548130 con 0x557ebf542470 2024-03-04T10:10:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.312+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 22 ==== osd_op_reply(98 100.00000000 [omap-set-vals] v99'43 uv43 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.312+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x557ebf5457d0 0x557ebf5487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.312+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:99 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf547bc0 con 0x557ebf5457d0 2024-03-04T10:10:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.312+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x557ebf5457d0 0x557ebf5487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.312+0000 7f4adafca640 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x557ebf5457d0 0x557ebf5487f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.313+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== osd_op_reply(99 mds0_sessionmap [create] v99'18 uv17 ondisk = -17 ((17) File exists)) v8 ==== 159+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf5457d0 2024-03-04T10:10:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.313+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:100 13.b 13:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf5457d0 2024-03-04T10:10:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.314+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== osd_op_reply(100 mds0_sessionmap [delete] v99'19 uv19 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf5457d0 2024-03-04T10:10:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.314+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:101 13.b 13:d0630e4c:::mds0_sessionmap:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf5457d0 2024-03-04T10:10:52.319 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.317+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.5 v2:172.21.15.162:6808/1764948489 3 ==== osd_op_reply(101 mds0_sessionmap [create] v99'20 uv20 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf5457d0 2024-03-04T10:10:52.319 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::mds0_sessionmap:head# 2024-03-04T10:10:52.320 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.317+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- osd_op(unknown.0.0:102 13.b 13:d0630e4c:::mds0_sessionmap:head [omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf5457d0 2024-03-04T10:10:52.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.320+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.5 v2:172.21.15.162:6808/1764948489 4 ==== osd_op_reply(102 mds0_sessionmap [omap-set-header] v99'21 uv21 ondisk = 0) v8 ==== 159+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf5457d0 2024-03-04T10:10:52.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.320+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:103 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf542470 2024-03-04T10:10:52.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.321+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 23 ==== osd_op_reply(103 609.00000000 [create] v99'21 uv20 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.321+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:104 13.7 13:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf542470 2024-03-04T10:10:52.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.322+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 24 ==== osd_op_reply(104 609.00000000 [delete] v99'22 uv22 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.322+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:105 13.7 13:e0b41d06:::609.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf542470 2024-03-04T10:10:52.325 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::609.00000000:head# 2024-03-04T10:10:52.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.323+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 25 ==== osd_op_reply(105 609.00000000 [create] v99'23 uv23 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.323+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:106 13.7 13:e0b41d06:::609.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf542470 2024-03-04T10:10:52.326 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.324+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 26 ==== osd_op_reply(106 609.00000000 [setxattr (30)] v99'24 uv24 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.327 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.324+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:107 13.7 13:e0b41d06:::609.00000000:head [setxattr parent (62) in=68b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548d30 con 0x557ebf542470 2024-03-04T10:10:52.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.325+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 27 ==== osd_op_reply(107 609.00000000 [setxattr (62)] v99'25 uv25 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.325+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- osd_op(unknown.0.0:108 13.7 13:e0b41d06:::609.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf549120 con 0x557ebf542470 2024-03-04T10:10:52.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.327+0000 7f4adb7cb640 1 -- 172.21.15.99:0/797728092 <== osd.7 v2:172.21.15.162:6816/2710319333 28 ==== osd_op_reply(108 609.00000000 [omap-set-header] v99'26 uv26 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4acc0154b0 con 0x557ebf542470 2024-03-04T10:10:52.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.327+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:109 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf549120 con 0x557ebf53a370 2024-03-04T10:10:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.329+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 7 ==== osd_op_reply(109 400.00000000 [create] v99'17 uv16 ondisk = -17 ((17) File exists)) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.329+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:110 13.f 13:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf549120 con 0x557ebf53a370 2024-03-04T10:10:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.330+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 8 ==== osd_op_reply(110 400.00000000 [delete] v99'18 uv18 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.330+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:111 13.f 13:f1f69726:::400.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548e00 con 0x557ebf53a370 2024-03-04T10:10:52.333 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::400.00000000:head# 2024-03-04T10:10:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.331+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 9 ==== osd_op_reply(111 400.00000000 [create] v99'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.331+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:112 13.f 13:f1f69726:::400.00000000:head [write 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548e00 con 0x557ebf53a370 2024-03-04T10:10:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.332+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 10 ==== osd_op_reply(112 400.00000000 [write 0~22] v99'20 uv20 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.332+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:113 13.f 13:f1f69726:::400.00000000:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548db0 con 0x557ebf53a370 2024-03-04T10:10:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.334+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 11 ==== osd_op_reply(113 400.00000000 [omap-set-header] v99'21 uv21 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.334+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:114 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5491a0 con 0x557ebf53a370 2024-03-04T10:10:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.335+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 12 ==== osd_op_reply(114 100.00000000.inode [create] v99'45 uv37 ondisk = -17 ((17) File exists)) v8 ==== 162+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.335+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:115 13.1f 13:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf5491a0 con 0x557ebf53a370 2024-03-04T10:10:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.336+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 13 ==== osd_op_reply(115 100.00000000.inode [delete] v99'46 uv46 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.336+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:116 13.1f 13:fe07bba1:::100.00000000.inode:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548e90 con 0x557ebf53a370 2024-03-04T10:10:52.339 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::100.00000000.inode:head# 2024-03-04T10:10:52.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.337+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 14 ==== osd_op_reply(116 100.00000000.inode [create] v99'47 uv47 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.337+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:117 13.1f 13:fe07bba1:::100.00000000.inode:head [write 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf549280 con 0x557ebf53a370 2024-03-04T10:10:52.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.338+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 15 ==== osd_op_reply(117 100.00000000.inode [write 0~582] v99'48 uv48 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.338+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:118 13.1f 13:fe07bba1:::100.00000000.inode:head [omap-set-header] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf549280 con 0x557ebf53a370 2024-03-04T10:10:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.339+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 16 ==== osd_op_reply(118 100.00000000.inode [omap-set-header] v99'49 uv49 ondisk = 0) v8 ==== 162+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.339+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:119 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548f40 con 0x557ebf53a370 2024-03-04T10:10:52.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.341+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 17 ==== osd_op_reply(119 1.00000000 [create] v99'50 uv44 ondisk = -17 ((17) File exists)) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.341+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:120 13.1f 13:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548e90 con 0x557ebf53a370 2024-03-04T10:10:52.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.341+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 18 ==== osd_op_reply(120 1.00000000 [delete] v99'51 uv51 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.341+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:121 13.1f 13:ff5b34d6:::1.00000000:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548e90 con 0x557ebf53a370 2024-03-04T10:10:52.347 INFO:teuthology.orchestra.run.smithi099.stdout:***Overwrite*** #-9223372036854775808:00000000:::1.00000000:head# 2024-03-04T10:10:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.345+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 19 ==== osd_op_reply(121 1.00000000 [create] v99'52 uv52 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.345+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:122 13.1f 13:ff5b34d6:::1.00000000:head [setxattr layout (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548e10 con 0x557ebf53a370 2024-03-04T10:10:52.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.347+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 20 ==== osd_op_reply(122 1.00000000 [setxattr (30)] v99'53 uv53 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.349 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.347+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:123 13.1f 13:ff5b34d6:::1.00000000:head [setxattr parent (30) in=36b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548db0 con 0x557ebf53a370 2024-03-04T10:10:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.348+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 21 ==== osd_op_reply(123 1.00000000 [setxattr (30)] v99'54 uv54 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.348+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:124 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf548db0 con 0x557ebf53a370 2024-03-04T10:10:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.351+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 22 ==== osd_op_reply(124 1.00000000 [omap-set-header] v99'55 uv55 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.351+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- osd_op(unknown.0.0:125 13.1f 13:ff5b34d6:::1.00000000:head [omap-set-vals in=584b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v9 -- 0x557ebf549210 con 0x557ebf53a370 2024-03-04T10:10:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4adafca640 1 -- 172.21.15.99:0/797728092 <== osd.6 v2:172.21.15.162:6824/1448511037 23 ==== osd_op_reply(125 1.00000000 [omap-set-vals] v99'56 uv56 ondisk = 0) v8 ==== 154+0+0 (crc 0 0 0) 0x7f4ad0020070 con 0x557ebf53a370 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x557ebf5350d0 msgr2=0x557ebf5374c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x557ebf5350d0 0x557ebf5374c0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x557ebf4810f0 msgr2=0x557ebf3e7770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x557ebf4810f0 0x557ebf3e7770 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x557ebf53e7f0 msgr2=0x557ebf541630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x557ebf53e7f0 0x557ebf541630 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x557ebf5457d0 msgr2=0x557ebf5487f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x557ebf5457d0 0x557ebf5487f0 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x557ebf53a370 msgr2=0x557ebf53a750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x557ebf53a370 0x557ebf53a750 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 reap_dead start 2024-03-04T10:10:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x557ebf542470 msgr2=0x557ebf544860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x557ebf542470 0x557ebf544860 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 reap_dead start 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4adbfcc640 1 -- 172.21.15.99:0/797728092 reap_dead start 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4ac8069eb0 msgr2=0x7f4ac806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4ac8069eb0 0x7f4ac806c370 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4acc006010 tx=0x7f4acc007000 comp rx=0 tx=0).stop 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 msgr2=0x557ebf48c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 0x557ebf48c380 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4ad0000c00 tx=0x7f4ad000ef70 comp rx=0 tx=0).stop 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 shutdown_connections 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f4ac8069eb0 0x7f4ac806c370 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 --2- 172.21.15.99:0/797728092 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x557ebf3e45e0 0x557ebf48c380 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 >> 172.21.15.99:0/797728092 conn(0x557ebf3b8400 msgr2=0x557ebf3ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 shutdown_connections 2024-03-04T10:10:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.355+0000 7f4ade436f40 1 -- 172.21.15.99:0/797728092 wait complete. 2024-03-04T10:10:52.360 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 200.00000000 2024-03-04T10:10:52.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.439+0000 7f124be4bf40 1 Processor -- start 2024-03-04T10:10:52.442 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.439+0000 7f124be4bf40 1 -- start start 2024-03-04T10:10:52.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f124be4bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f139dab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f124be4bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139bb1c0 0x561f139d3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.446 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f124be4bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139d8670 0x561f139d9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f124be4bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f139d9610 con 0x561f138fec90 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f124be4bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f138a7b30 con 0x561f139d8670 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f124be4bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f13949b50 con 0x561f139bb1c0 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f139dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f139dab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57992/0 (socket says 172.21.15.99:57992) 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 -- 172.21.15.99:0/3940086226 learned_addr learned my addr 172.21.15.99:0/3940086226 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 -- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139bb1c0 msgr2=0x561f139d3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139bb1c0 0x561f139d3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 -- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139d8670 msgr2=0x561f139d9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249912640 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139d8670 0x561f139d9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139d8670 0x561f139d9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249111640 1 -- 172.21.15.99:0/3940086226 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f139db0b0 con 0x561f138fec90 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.440+0000 7f1249912640 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139d8670 0x561f139d9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f1249111640 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f139dab70 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f123c055040 tx=0x7f123c04c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d75e248d6f3664 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f123b7fe640 1 -- 172.21.15.99:0/3940086226 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c057070 con 0x561f138fec90 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f123b7fe640 1 -- 172.21.15.99:0/3940086226 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c061070 con 0x561f138fec90 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f123b7fe640 1 -- 172.21.15.99:0/3940086226 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0529a0 con 0x561f138fec90 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 -- 172.21.15.99:0/3940086226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 msgr2=0x561f139dab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f139dab70 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f123c055040 tx=0x7f123c04c000 comp rx=0 tx=0).stop 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 -- 172.21.15.99:0/3940086226 shutdown_connections 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139d8670 0x561f139d9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139bb1c0 0x561f139d3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 --2- 172.21.15.99:0/3940086226 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f139dab70 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 -- 172.21.15.99:0/3940086226 >> 172.21.15.99:0/3940086226 conn(0x561f13900400 msgr2=0x561f139d78f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 -- 172.21.15.99:0/3940086226 shutdown_connections 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.441+0000 7f124be4bf40 1 -- 172.21.15.99:0/3940086226 wait complete. 2024-03-04T10:10:52.448 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.442+0000 7f124be4bf40 1 Processor -- start 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.442+0000 7f124be4bf40 1 -- start start 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f13947400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139bb1c0 0x561f139d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139d8670 0x561f139b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f139d3700 con 0x561f138fec90 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f139d3140 con 0x561f139bb1c0 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x561f139b95a0 con 0x561f139d8670 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f13947400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f13947400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:57998/0 (socket says 172.21.15.99:57998) 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 -- 172.21.15.99:0/537990218 learned_addr learned my addr 172.21.15.99:0/537990218 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249912640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139d8670 0x561f139b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1248910640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139bb1c0 0x561f139d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 -- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139d8670 msgr2=0x561f139b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x561f139d8670 0x561f139b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 -- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139bb1c0 msgr2=0x561f139d55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f139bb1c0 0x561f139d55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 -- 172.21.15.99:0/537990218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f139d6f60 con 0x561f138fec90 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f1249111640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f13947400 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f123c05e040 tx=0x7f123c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f123a7fc640 1 -- 172.21.15.99:0/537990218 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c06d420 con 0x561f138fec90 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f123a7fc640 1 -- 172.21.15.99:0/537990218 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c058040 con 0x561f138fec90 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561f139d7200 con 0x561f138fec90 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f123a7fc640 1 -- 172.21.15.99:0/537990218 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c06d820 con 0x561f138fec90 2024-03-04T10:10:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.443+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f139d4df0 con 0x561f138fec90 2024-03-04T10:10:52.452 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f123a7fc640 1 -- 172.21.15.99:0/537990218 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f123c076430 con 0x561f138fec90 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f123a7fc640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f122c069eb0 0x7f122c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f123a7fc640 1 -- 172.21.15.99:0/537990218 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f123c0cc0f0 con 0x561f138fec90 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f124be4bf40 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561f13a71210 0x561f13a715f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- osd_op(unknown.0.0:1 13.14 13:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e99) v9 -- 0x561f139d7ec0 con 0x561f13a71210 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f1248910640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f122c069eb0 0x7f122c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.445+0000 7f1249912640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561f13a71210 0x561f13a715f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.446+0000 7f1249912640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561f13a71210 0x561f13a715f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.446+0000 7f1248910640 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f122c069eb0 0x7f122c06c370 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x561f139be9e0 tx=0x7f1240005950 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f1249912640 1 -- 172.21.15.99:0/537990218 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== osd_op_reply(1 200.00000000 [delete] v99'109 uv109 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x7f1244003450 con 0x561f13a71210 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561f13a71210 msgr2=0x561f13a715f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x561f13a71210 0x561f13a715f0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f122c069eb0 msgr2=0x7f122c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f122c069eb0 0x7f122c06c370 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x561f139be9e0 tx=0x7f1240005950 comp rx=0 tx=0).stop 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 msgr2=0x561f13947400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 --2- 172.21.15.99:0/537990218 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x561f138fec90 0x561f13947400 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f123c05e040 tx=0x7f123c002fb0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f1249912640 1 -- 172.21.15.99:0/537990218 reap_dead start 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 shutdown_connections 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 >> 172.21.15.99:0/537990218 conn(0x561f13900400 msgr2=0x561f1392e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 shutdown_connections 2024-03-04T10:10:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.447+0000 7f124be4bf40 1 -- 172.21.15.99:0/537990218 wait complete. 2024-03-04T10:10:52.454 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-03-04T10:10:52.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.593+0000 7fc530eca640 1 Processor -- start 2024-03-04T10:10:52.596 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.593+0000 7fc530eca640 1 -- start start 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc530eca640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc52c15da20 0x7fc52c0ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc530eca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c161df0 0x7fc52c0cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc530eca640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52c162450 con 0x7fc52c0cf660 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc530eca640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c0d0000 con 0x7fc52c15da20 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc530eca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0cc920 con 0x7fc52c161df0 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52a59c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc52c15da20 0x7fc52c0ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc529d9b640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc52c0cf660 0x7fc52c1616e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:45722/0 (socket says 172.21.15.99:45722) 2024-03-04T10:10:52.597 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc529d9b640 1 -- 172.21.15.99:0/125065669 learned_addr learned my addr 172.21.15.99:0/125065669 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:10:52.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52ad9d640 1 --2- 172.21.15.99:0/125065669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c161df0 0x7fc52c0cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52a59c640 1 -- 172.21.15.99:0/125065669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c161df0 msgr2=0x7fc52c0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:52.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52a59c640 1 --2- 172.21.15.99:0/125065669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c161df0 0x7fc52c0cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52a59c640 1 -- 172.21.15.99:0/125065669 >> v1:172.21.15.99:6789/0 conn(0x7fc52c0cf660 legacy=0x7fc52c1616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52a59c640 1 -- 172.21.15.99:0/125065669 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c0cc2d0 con 0x7fc52c15da20 2024-03-04T10:10:52.598 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.594+0000 7fc52959a640 1 -- 172.21.15.99:0/125065669 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724293562 0 0) 0x7fc52c162450 con 0x7fc52c0cf660 2024-03-04T10:10:52.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.595+0000 7fc52a59c640 1 --2- 172.21.15.99:0/125065669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c15da20 0x7fc52c0ccdb0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0097a0 tx=0x7fc51c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc198dbb72edbf6d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.595+0000 7fc52959a640 1 -- 172.21.15.99:0/125065669 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c035020 con 0x7fc52c15da20 2024-03-04T10:10:52.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.595+0000 7fc52959a640 1 -- 172.21.15.99:0/125065669 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c031830 con 0x7fc52c15da20 2024-03-04T10:10:52.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.595+0000 7fc52959a640 1 -- 172.21.15.99:0/125065669 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c031c00 con 0x7fc52c15da20 2024-03-04T10:10:52.599 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 -- 172.21.15.99:0/125065669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c15da20 msgr2=0x7fc52c0ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 --2- 172.21.15.99:0/125065669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c15da20 0x7fc52c0ccdb0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0097a0 tx=0x7fc51c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 -- 172.21.15.99:0/125065669 shutdown_connections 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 --2- 172.21.15.99:0/125065669 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c161df0 0x7fc52c0cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 --2- 172.21.15.99:0/125065669 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c15da20 0x7fc52c0ccdb0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 -- 172.21.15.99:0/125065669 >> 172.21.15.99:0/125065669 conn(0x7fc52c0b33a0 msgr2=0x7fc52c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 -- 172.21.15.99:0/125065669 shutdown_connections 2024-03-04T10:10:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.596+0000 7fc530eca640 1 -- 172.21.15.99:0/125065669 wait complete. 2024-03-04T10:10:52.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.597+0000 7fc530eca640 1 Processor -- start 2024-03-04T10:10:52.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.597+0000 7fc530eca640 1 -- start start 2024-03-04T10:10:52.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc530eca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 0x7fc52c1f3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.602 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc530eca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c15da20 0x7fc52c1f6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc530eca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c161df0 0x7fc52c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc530eca640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0cbf80 con 0x7fc52c0cf660 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc530eca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0cd2f0 con 0x7fc52c161df0 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc530eca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c15de00 con 0x7fc52c15da20 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 0x7fc52c1f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 0x7fc52c1f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58018/0 (socket says 172.21.15.99:58018) 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 -- 172.21.15.99:0/3455455252 learned_addr learned my addr 172.21.15.99:0/3455455252 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc529d9b640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c15da20 0x7fc52c1f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52ad9d640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c161df0 0x7fc52c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 -- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c15da20 msgr2=0x7fc52c1f6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c15da20 0x7fc52c1f6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 -- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c161df0 msgr2=0x7fc52c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c161df0 0x7fc52c0c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.598+0000 7fc52a59c640 1 -- 172.21.15.99:0/3455455252 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c0c5850 con 0x7fc52c0cf660 2024-03-04T10:10:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.599+0000 7fc52a59c640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 0x7fc52c1f3320 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc51c02bee0 tx=0x7fc51c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.599+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c035040 con 0x7fc52c0cf660 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.599+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c03b920 con 0x7fc52c0cf660 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.599+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c0c5b30 con 0x7fc52c0cf660 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.599+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c03bd10 con 0x7fc52c0cf660 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.599+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c0c63d0 con 0x7fc52c0cf660 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.600+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c074f60 con 0x7fc52c0cf660 2024-03-04T10:10:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.600+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc51c038020 con 0x7fc52c0cf660 2024-03-04T10:10:52.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.600+0000 7fc51affd640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc50c06b4d0 0x7fc50c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:52.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.601+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fc51c0ac5c0 con 0x7fc52c0cf660 2024-03-04T10:10:52.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.601+0000 7fc529d9b640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc50c06b4d0 0x7fc50c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:52.605 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.601+0000 7fc529d9b640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc50c06b4d0 0x7fc50c06d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc52c15b460 tx=0x7fc520007e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:52.607 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.605+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc51c031800 con 0x7fc52c0cf660 2024-03-04T10:10:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:52.810+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fc52c0c5ec0 con 0x7fc52c0cf660 2024-03-04T10:10:53.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.058+0000 7fc51affd640 1 -- 172.21.15.99:0/3455455252 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v104) v1 ==== 162+0+0 (secure 0 0 0) 0x7fc51c07aaa0 con 0x7fc52c0cf660 2024-03-04T10:10:53.061 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked joinable; MDS may join as newly active. 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc50c06b4d0 msgr2=0x7fc50c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc50c06b4d0 0x7fc50c06d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc52c15b460 tx=0x7fc520007e40 comp rx=0 tx=0).stop 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 msgr2=0x7fc52c1f3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 0x7fc52c1f3320 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc51c02bee0 tx=0x7fc51c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 shutdown_connections 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc50c06b4d0 0x7fc50c06d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c161df0 0x7fc52c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc52c15da20 0x7fc52c1f6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 --2- 172.21.15.99:0/3455455252 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc52c0cf660 0x7fc52c1f3320 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 >> 172.21.15.99:0/3455455252 conn(0x7fc52c0b33a0 msgr2=0x7fc52c0b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 shutdown_connections 2024-03-04T10:10:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.060+0000 7fc530eca640 1 -- 172.21.15.99:0/3455455252 wait complete. 2024-03-04T10:10:53.071 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:53.069+0000 7fed969d4640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:10:53.071 INFO:tasks.ceph.mds.c.smithi099.stderr:*** Caught signal (Floating point exception) ** 2024-03-04T10:10:53.071 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7fed969d4640 thread_name:md_recov_open 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7feda2c54db0] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x5643a81f7783] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x5643a7f9476f] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x5643a7f58532] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x5643a7f5862f] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x5643a8205917] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x5643a81f8175] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7feda2c9f802] 2024-03-04T10:10:53.072 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7feda2c3f450] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr:2024-03-04T10:10:53.070+0000 7fed969d4640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7fed969d4640 thread_name:md_recov_open 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7feda2c54db0] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x5643a81f7783] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x5643a7f9476f] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x5643a7f58532] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x5643a7f5862f] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x5643a8205917] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x5643a81f8175] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7feda2c9f802] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7feda2c3f450] 2024-03-04T10:10:53.073 INFO:tasks.ceph.mds.c.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:10:53.074 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:10:53.074 INFO:tasks.ceph.mds.c.smithi099.stderr: -13> 2024-03-04T10:10:53.069+0000 7fed969d4640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:10:53.074 INFO:tasks.ceph.mds.c.smithi099.stderr: 0> 2024-03-04T10:10:53.070+0000 7fed969d4640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:10:53.074 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7fed969d4640 thread_name:md_recov_open 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7feda2c54db0] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x5643a81f7783] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x5643a7f9476f] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x5643a7f58532] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x5643a7f5862f] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x5643a8205917] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x5643a81f8175] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7feda2c9f802] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7feda2c3f450] 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:10:53.075 INFO:tasks.ceph.mds.c.smithi099.stderr: -17> 2024-03-04T10:10:53.069+0000 7fed969d4640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: -4> 2024-03-04T10:10:53.070+0000 7fed969d4640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: in thread 7fed969d4640 thread_name:md_recov_open 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7feda2c54db0] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x5643a81f7783] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x5643a7f9476f] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x5643a7f58532] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x5643a7f5862f] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x5643a8205917] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x5643a81f8175] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7feda2c9f802] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7feda2c3f450] 2024-03-04T10:10:53.076 INFO:tasks.ceph.mds.c.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:10:53.077 INFO:tasks.ceph.mds.c.smithi099.stderr: 2024-03-04T10:10:53.077 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:53.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.175+0000 7fea6369a640 1 Processor -- start 2024-03-04T10:10:53.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.175+0000 7fea6369a640 1 -- start start 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea6369a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea5c15d660 0x7fea5c15e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea6369a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c0aeb20 0x7fea5c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea6369a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fea5c071890 0x7fea5c150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea6369a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c15da40 con 0x7fea5c15d660 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea6369a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5c15ebb0 con 0x7fea5c071890 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea6369a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c0af090 con 0x7fea5c0aeb20 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea5c15d660 0x7fea5c15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fea5c15d660 0x7fea5c15e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58022/0 (socket says 172.21.15.99:58022) 2024-03-04T10:10:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 -- 172.21.15.99:0/3800463075 learned_addr learned my addr 172.21.15.99:0/3800463075 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 -- 172.21.15.99:0/3800463075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c0aeb20 msgr2=0x7fea5c071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea60c35640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c0aeb20 0x7fea5c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61c37640 1 --2- 172.21.15.99:0/3800463075 >> v2:172.21.15.162:3300/0 conn(0x7fea5c071890 0x7fea5c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c0aeb20 0x7fea5c071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 -- 172.21.15.99:0/3800463075 >> v2:172.21.15.162:3300/0 conn(0x7fea5c071890 msgr2=0x7fea5c150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 --2- 172.21.15.99:0/3800463075 >> v2:172.21.15.162:3300/0 conn(0x7fea5c071890 0x7fea5c150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61436640 1 -- 172.21.15.99:0/3800463075 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c15a9f0 con 0x7fea5c15d660 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea61c37640 1 --2- 172.21.15.99:0/3800463075 >> v2:172.21.15.162:3300/0 conn(0x7fea5c071890 0x7fea5c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.176+0000 7fea60c35640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c0aeb20 0x7fea5c071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:10:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.177+0000 7fea61436640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c15d660 0x7fea5c15e670 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fea5c06f5a0 tx=0x7fea5802dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9e34b784b790c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.177+0000 7fea53fff640 1 -- 172.21.15.99:0/3800463075 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea58034070 con 0x7fea5c15d660 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.177+0000 7fea53fff640 1 -- 172.21.15.99:0/3800463075 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5802eb30 con 0x7fea5c15d660 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.177+0000 7fea53fff640 1 -- 172.21.15.99:0/3800463075 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea58038400 con 0x7fea5c15d660 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 -- 172.21.15.99:0/3800463075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c15d660 msgr2=0x7fea5c15e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c15d660 0x7fea5c15e670 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fea5c06f5a0 tx=0x7fea5802dd80 comp rx=0 tx=0).stop 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 -- 172.21.15.99:0/3800463075 shutdown_connections 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 --2- 172.21.15.99:0/3800463075 >> v2:172.21.15.162:3300/0 conn(0x7fea5c071890 0x7fea5c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c0aeb20 0x7fea5c071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 --2- 172.21.15.99:0/3800463075 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c15d660 0x7fea5c15e670 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 -- 172.21.15.99:0/3800463075 >> 172.21.15.99:0/3800463075 conn(0x7fea5c06acd0 msgr2=0x7fea5c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 -- 172.21.15.99:0/3800463075 shutdown_connections 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.178+0000 7fea6369a640 1 -- 172.21.15.99:0/3800463075 wait complete. 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.179+0000 7fea6369a640 1 Processor -- start 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.179+0000 7fea6369a640 1 -- start start 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.179+0000 7fea6369a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c071890 0x7fea5c146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 0x7fea5c15bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea5c15d660 0x7fea5c154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c15a660 con 0x7fea5c0aeb20 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c15afb0 con 0x7fea5c15d660 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5c15b4d0 con 0x7fea5c071890 2024-03-04T10:10:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 0x7fea5c15bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 0x7fea5c15bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:58034/0 (socket says 172.21.15.99:58034) 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 -- 172.21.15.99:0/1783338892 learned_addr learned my addr 172.21.15.99:0/1783338892 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea61436640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c071890 0x7fea5c146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 -- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c071890 msgr2=0x7fea5c146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c071890 0x7fea5c146110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea61c37640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea5c15d660 0x7fea5c154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 -- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea5c15d660 msgr2=0x7fea5c154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea5c15d660 0x7fea5c154090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 -- 172.21.15.99:0/1783338892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c15c590 con 0x7fea5c0aeb20 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea60c35640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 0x7fea5c15bbb0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00c820 tx=0x7fea4c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c015070 con 0x7fea5c0aeb20 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c007a30 con 0x7fea5c0aeb20 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea5c15c870 con 0x7fea5c0aeb20 2024-03-04T10:10:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c007de0 con 0x7fea5c0aeb20 2024-03-04T10:10:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.180+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea5c155280 con 0x7fea5c0aeb20 2024-03-04T10:10:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.182+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea5c15ca40 con 0x7fea5c0aeb20 2024-03-04T10:10:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.182+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fea4c018070 con 0x7fea5c0aeb20 2024-03-04T10:10:53.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.182+0000 7fea51ffb640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea4006b4d0 0x7fea4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:53.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.182+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fea4c08a400 con 0x7fea5c0aeb20 2024-03-04T10:10:53.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.182+0000 7fea61436640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea4006b4d0 0x7fea4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:53.187 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.183+0000 7fea61436640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea4006b4d0 0x7fea4006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fea58039000 tx=0x7fea58034000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.186+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fea4c058fb0 con 0x7fea5c0aeb20 2024-03-04T10:10:53.264 INFO:tasks.ceph.mds.c.smithi099.stderr:daemon-helper: command crashed with signal 8 2024-03-04T10:10:53.401 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.399+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea5c15e370 con 0x7fea5c0aeb20 2024-03-04T10:10:53.402 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.399+0000 7fea51ffb640 1 -- 172.21.15.99:0/1783338892 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4941 (secure 0 0 0) 0x7fea5c15e370 con 0x7fea5c0aeb20 2024-03-04T10:10:53.402 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:53.402 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":105,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:53.062239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8550":{"gid":8550,"name":"c","rank":0,"incarnation":105,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/301817793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":301817793},{"type":"v1","addr":"172.21.15.99:6835","nonce":301817793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:53.402 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 105 2024-03-04T10:10:53.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea4006b4d0 msgr2=0x7fea4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea4006b4d0 0x7fea4006d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fea58039000 tx=0x7fea58034000 comp rx=0 tx=0).stop 2024-03-04T10:10:53.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 msgr2=0x7fea5c15bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:53.404 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 0x7fea5c15bbb0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00c820 tx=0x7fea4c00cce0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 shutdown_connections 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea4006b4d0 0x7fea4006d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea5c15d660 0x7fea5c154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea5c0aeb20 0x7fea5c15bbb0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 --2- 172.21.15.99:0/1783338892 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea5c071890 0x7fea5c146110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 >> 172.21.15.99:0/1783338892 conn(0x7fea5c06acd0 msgr2=0x7fea5c1529f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.401+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 shutdown_connections 2024-03-04T10:10:53.405 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:53.402+0000 7fea6369a640 1 -- 172.21.15.99:0/1783338892 wait complete. 2024-03-04T10:10:53.415 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:10:53.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-04T10:10:58.416 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:10:58.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.525+0000 7fb2ff905640 1 Processor -- start 2024-03-04T10:10:58.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.526+0000 7fb2ff905640 1 -- start start 2024-03-04T10:10:58.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.526+0000 7fb2ff905640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fb2f815de00 0x7fb2f80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:58.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.526+0000 7fb2ff905640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb2f80b9380 0x7fb2f80b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.526+0000 7fb2ff905640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f815e1e0 con 0x7fb2f80b70d0 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.526+0000 7fb2ff905640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f80d2820 con 0x7fb2f80b9380 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.526+0000 7fb2ff905640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f80d1c10 con 0x7fb2f815de00 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb2f80b9380 0x7fb2f80b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fb2f80b9380 0x7fb2f80b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47026/0 (socket says 172.21.15.99:47026) 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 -- 172.21.15.99:0/3340909958 learned_addr learned my addr 172.21.15.99:0/3340909958 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fd6a1640 1 --2- 172.21.15.99:0/3340909958 >> v2:172.21.15.162:3300/0 conn(0x7fb2f815de00 0x7fb2f80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 -- 172.21.15.99:0/3340909958 >> v2:172.21.15.162:3300/0 conn(0x7fb2f815de00 msgr2=0x7fb2f80b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 --2- 172.21.15.99:0/3340909958 >> v2:172.21.15.162:3300/0 conn(0x7fb2f815de00 0x7fb2f80b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 -- 172.21.15.99:0/3340909958 >> v1:172.21.15.162:6789/0 conn(0x7fb2f80b70d0 legacy=0x7fb2f80b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:10:58.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.527+0000 7fb2fcea0640 1 -- 172.21.15.99:0/3340909958 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f80d1ef0 con 0x7fb2f80b9380 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2fcea0640 1 --2- 172.21.15.99:0/3340909958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f80b9380 0x7fb2f80b6b90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80ce240 tx=0x7fb2e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7dc6f3372cee80 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2effff640 1 -- 172.21.15.99:0/3340909958 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8035020 con 0x7fb2f80b9380 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2effff640 1 -- 172.21.15.99:0/3340909958 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e8031870 con 0x7fb2f80b9380 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2effff640 1 -- 172.21.15.99:0/3340909958 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8031c40 con 0x7fb2f80b9380 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 -- 172.21.15.99:0/3340909958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f80b9380 msgr2=0x7fb2f80b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 --2- 172.21.15.99:0/3340909958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f80b9380 0x7fb2f80b6b90 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80ce240 tx=0x7fb2e802b910 comp rx=0 tx=0).stop 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 -- 172.21.15.99:0/3340909958 shutdown_connections 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 --2- 172.21.15.99:0/3340909958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f80b9380 0x7fb2f80b6b90 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 --2- 172.21.15.99:0/3340909958 >> v2:172.21.15.162:3300/0 conn(0x7fb2f815de00 0x7fb2f80b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 -- 172.21.15.99:0/3340909958 >> 172.21.15.99:0/3340909958 conn(0x7fb2f80d0470 msgr2=0x7fb2f80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 -- 172.21.15.99:0/3340909958 shutdown_connections 2024-03-04T10:10:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.528+0000 7fb2ff905640 1 -- 172.21.15.99:0/3340909958 wait complete. 2024-03-04T10:10:58.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 Processor -- start 2024-03-04T10:10:58.533 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 -- start start 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb2f80b70d0 0x7fb2f814ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2f80b9380 0x7fb2f815ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 0x7fb2f80cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80b97e0 con 0x7fb2f815de00 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80d24b0 con 0x7fb2f80b9380 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2ff905640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f80b5b80 con 0x7fb2f80b70d0 2024-03-04T10:10:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2fdea2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 0x7fb2f80cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2fdea2640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 0x7fb2f80cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47042/0 (socket says 172.21.15.99:47042) 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.530+0000 7fb2fdea2640 1 -- 172.21.15.99:0/1467014179 learned_addr learned my addr 172.21.15.99:0/1467014179 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fd6a1640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb2f80b70d0 0x7fb2f814ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fdea2640 1 -- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb2f80b70d0 msgr2=0x7fb2f814ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fdea2640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb2f80b70d0 0x7fb2f814ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fcea0640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2f80b9380 0x7fb2f815ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fdea2640 1 -- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2f80b9380 msgr2=0x7fb2f815ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fdea2640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2f80b9380 0x7fb2f815ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fdea2640 1 -- 172.21.15.99:0/1467014179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f815f320 con 0x7fb2f815de00 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2fdea2640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 0x7fb2f80cc5c0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000c390 tx=0x7fb2f000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0017070 con 0x7fb2f815de00 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f00035f0 con 0x7fb2f815de00 2024-03-04T10:10:58.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f80c9f40 con 0x7fb2f815de00 2024-03-04T10:10:58.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f00039e0 con 0x7fb2f815de00 2024-03-04T10:10:58.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.531+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f80c80d0 con 0x7fb2f815de00 2024-03-04T10:10:58.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.532+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f80b66a0 con 0x7fb2f815de00 2024-03-04T10:10:58.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.533+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fb2f001a070 con 0x7fb2f815de00 2024-03-04T10:10:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.533+0000 7fb2edffb640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2dc06b4d0 0x7fb2dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:10:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.533+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7fb2f0089950 con 0x7fb2f815de00 2024-03-04T10:10:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.533+0000 7fb2fd6a1640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2dc06b4d0 0x7fb2dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:10:58.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.534+0000 7fb2fd6a1640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2dc06b4d0 0x7fb2dc06d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80b8bf0 tx=0x7fb2f40083c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:10:58.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.537+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fb2f80b66a0 con 0x7fb2f815de00 2024-03-04T10:10:58.744 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.742+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2f80c7ec0 con 0x7fb2f815de00 2024-03-04T10:10:58.745 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.742+0000 7fb2edffb640 1 -- 172.21.15.99:0/1467014179 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4941 (secure 0 0 0) 0x7fb2f815f320 con 0x7fb2f815de00 2024-03-04T10:10:58.745 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:10:58.745 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":105,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:53.062239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8550":{"gid":8550,"name":"c","rank":0,"incarnation":105,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/301817793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":301817793},{"type":"v1","addr":"172.21.15.99:6835","nonce":301817793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:10:58.745 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 105 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2dc06b4d0 msgr2=0x7fb2dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2dc06b4d0 0x7fb2dc06d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80b8bf0 tx=0x7fb2f40083c0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 msgr2=0x7fb2f80cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 0x7fb2f80cc5c0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fb2f000c390 tx=0x7fb2f000c850 comp rx=0 tx=0).stop 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 shutdown_connections 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fb2dc06b4d0 0x7fb2dc06d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fb2f815de00 0x7fb2f80cc5c0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb2f80b9380 0x7fb2f815ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 --2- 172.21.15.99:0/1467014179 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fb2f80b70d0 0x7fb2f814ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:10:58.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 >> 172.21.15.99:0/1467014179 conn(0x7fb2f80d0470 msgr2=0x7fb2f80ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:10:58.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 shutdown_connections 2024-03-04T10:10:58.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:10:58.744+0000 7fb2ff905640 1 -- 172.21.15.99:0/1467014179 wait complete. 2024-03-04T10:10:58.758 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:00.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~7s 2024-03-04T10:11:03.759 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:03.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.871+0000 7f83341fa640 1 Processor -- start 2024-03-04T10:11:03.873 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.871+0000 7f83341fa640 1 -- start start 2024-03-04T10:11:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f83341fa640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f832c154a80 0x7f832c15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f83341fa640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c0cf650 0x7f832c0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f83341fa640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0cdaa0 con 0x7f832c1616c0 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f83341fa640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c0cfff0 con 0x7f832c154a80 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f83341fa640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c071df0 con 0x7f832c0cf650 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f8331f96640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f832c154a80 0x7f832c15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f8331f96640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f832c154a80 0x7f832c15e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45252/0 (socket says 172.21.15.99:45252) 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f8331f96640 1 -- 172.21.15.99:0/1850606516 learned_addr learned my addr 172.21.15.99:0/1850606516 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.872+0000 7f8331795640 1 --2- 172.21.15.99:0/1850606516 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c0cf650 0x7f832c0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:03.877 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.873+0000 7f8331f96640 1 -- 172.21.15.99:0/1850606516 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c0cf650 msgr2=0x7f832c0cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.873+0000 7f8331f96640 1 --2- 172.21.15.99:0/1850606516 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c0cf650 0x7f832c0cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.873+0000 7f8331f96640 1 -- 172.21.15.99:0/1850606516 >> v1:172.21.15.162:6789/0 conn(0x7f832c1616c0 legacy=0x7f832c161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.873+0000 7f8331f96640 1 -- 172.21.15.99:0/1850606516 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0cd010 con 0x7f832c154a80 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.874+0000 7f8331f96640 1 --2- 172.21.15.99:0/1850606516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c154a80 0x7f832c15e910 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f83280098a0 tx=0x7f832802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92738a38f523813e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.874+0000 7f8330f94640 1 -- 172.21.15.99:0/1850606516 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328034070 con 0x7f832c154a80 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.874+0000 7f8330f94640 1 -- 172.21.15.99:0/1850606516 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328030840 con 0x7f832c154a80 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.874+0000 7f8330f94640 1 -- 172.21.15.99:0/1850606516 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328030c10 con 0x7f832c154a80 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.874+0000 7f83341fa640 1 -- 172.21.15.99:0/1850606516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c154a80 msgr2=0x7f832c15e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.874+0000 7f83341fa640 1 --2- 172.21.15.99:0/1850606516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c154a80 0x7f832c15e910 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f83280098a0 tx=0x7f832802b980 comp rx=0 tx=0).stop 2024-03-04T10:11:03.878 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.875+0000 7f83341fa640 1 -- 172.21.15.99:0/1850606516 shutdown_connections 2024-03-04T10:11:03.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.875+0000 7f83341fa640 1 --2- 172.21.15.99:0/1850606516 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c0cf650 0x7f832c0cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:03.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.875+0000 7f83341fa640 1 --2- 172.21.15.99:0/1850606516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c154a80 0x7f832c15e910 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:03.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.875+0000 7f83341fa640 1 -- 172.21.15.99:0/1850606516 >> 172.21.15.99:0/1850606516 conn(0x7f832c06f5f0 msgr2=0x7f832c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:03.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.875+0000 7f83341fa640 1 -- 172.21.15.99:0/1850606516 shutdown_connections 2024-03-04T10:11:03.879 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.875+0000 7f83341fa640 1 -- 172.21.15.99:0/1850606516 wait complete. 2024-03-04T10:11:03.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.876+0000 7f83341fa640 1 Processor -- start 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.876+0000 7f83341fa640 1 -- start start 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 0x7f832c153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f832c154a80 0x7f832c1f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c1616c0 0x7f832c15fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0ce340 con 0x7f832c0cf650 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0ce060 con 0x7f832c154a80 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0ce620 con 0x7f832c1616c0 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 0x7f832c153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 0x7f832c153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45268/0 (socket says 172.21.15.99:45268) 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 -- 172.21.15.99:0/2673136071 learned_addr learned my addr 172.21.15.99:0/2673136071 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:03.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331795640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f832c154a80 0x7f832c1f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8332797640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c1616c0 0x7f832c15fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 -- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c1616c0 msgr2=0x7f832c15fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c1616c0 0x7f832c15fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 -- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f832c154a80 msgr2=0x7f832c1f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f832c154a80 0x7f832c1f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 -- 172.21.15.99:0/2673136071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0c71c0 con 0x7f832c0cf650 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f8331f96640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 0x7f832c153f50 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f8328000c00 tx=0x7f8328004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328048070 con 0x7f832c0cf650 2024-03-04T10:11:03.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8328030640 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c0c74a0 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328034040 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.877+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c1f4f00 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.878+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c074f60 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.879+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f832804b070 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.879+0000 7f83227fc640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f831006b4d0 0x7f831006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.879+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f83280ac320 con 0x7f832c0cf650 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.880+0000 7f8331795640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f831006b4d0 0x7f831006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.880+0000 7f8331795640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f831006b4d0 0x7f831006d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f832c154490 tx=0x7f831c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:03.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:03.884+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f832807a910 con 0x7f832c0cf650 2024-03-04T10:11:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.091+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f832c0b1c50 con 0x7f832c0cf650 2024-03-04T10:11:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.091+0000 7f83227fc640 1 -- 172.21.15.99:0/2673136071 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f832807e9b0 con 0x7f832c0cf650 2024-03-04T10:11:04.093 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:04.093 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":105,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:53.062239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8550":{"gid":8550,"name":"c","rank":0,"incarnation":105,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/301817793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":301817793},{"type":"v1","addr":"172.21.15.99:6835","nonce":301817793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 105 2024-03-04T10:11:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f831006b4d0 msgr2=0x7f831006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f831006b4d0 0x7f831006d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f832c154490 tx=0x7f831c009190 comp rx=0 tx=0).stop 2024-03-04T10:11:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 msgr2=0x7f832c153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 0x7f832c153f50 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f8328000c00 tx=0x7f8328004730 comp rx=0 tx=0).stop 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 shutdown_connections 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f831006b4d0 0x7f831006d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f832c1616c0 0x7f832c15fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f832c154a80 0x7f832c1f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.093+0000 7f83341fa640 1 --2- 172.21.15.99:0/2673136071 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f832c0cf650 0x7f832c153f50 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.094+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 >> 172.21.15.99:0/2673136071 conn(0x7f832c06f5f0 msgr2=0x7f832c0d1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.094+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 shutdown_connections 2024-03-04T10:11:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:04.094+0000 7f83341fa640 1 -- 172.21.15.99:0/2673136071 wait complete. 2024-03-04T10:11:04.106 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:06.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~13s 2024-03-04T10:11:09.108 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.217+0000 7f3a12419640 1 Processor -- start 2024-03-04T10:11:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.217+0000 7f3a12419640 1 -- start start 2024-03-04T10:11:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a12419640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a0c15b960 0x7f3a0c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a12419640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3a0c0b0e10 0x7f3a0c0b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a12419640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0ccad0 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a12419640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0c0cf860 con 0x7f3a0c15b960 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a12419640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a0c0d0100 con 0x7f3a0c0b0e10 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a0bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f3a0c15b960 0x7f3a0c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a0b7fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3a0c0d0ad0 0x7f3a0c0b0700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:44192/0 (socket says 172.21.15.99:44192) 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.218+0000 7f3a0b7fe640 1 -- 172.21.15.99:0/857133835 learned_addr learned my addr 172.21.15.99:0/857133835 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.219+0000 7f3a0bfff640 1 -- 172.21.15.99:0/857133835 >> v2:172.21.15.162:3300/0 conn(0x7f3a0c0b0e10 msgr2=0x7f3a0c0b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.219+0000 7f3a0bfff640 1 --2- 172.21.15.99:0/857133835 >> v2:172.21.15.162:3300/0 conn(0x7f3a0c0b0e10 0x7f3a0c0b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.219+0000 7f3a0bfff640 1 -- 172.21.15.99:0/857133835 >> v1:172.21.15.99:6789/0 conn(0x7f3a0c0d0ad0 legacy=0x7f3a0c0b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.219+0000 7f3a0bfff640 1 -- 172.21.15.99:0/857133835 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0cfab0 con 0x7f3a0c15b960 2024-03-04T10:11:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.219+0000 7f3a0affd640 1 -- 172.21.15.99:0/857133835 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139306885 0 0) 0x7f3a0c0ccad0 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.219+0000 7f3a0bfff640 1 --2- 172.21.15.99:0/857133835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0d0590 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0ce1b0 tx=0x7f39fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da20088d71a651a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a0affd640 1 -- 172.21.15.99:0/857133835 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc0315a0 con 0x7f3a0c15b960 2024-03-04T10:11:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a0affd640 1 -- 172.21.15.99:0/857133835 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc037070 con 0x7f3a0c15b960 2024-03-04T10:11:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a0affd640 1 -- 172.21.15.99:0/857133835 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc031960 con 0x7f3a0c15b960 2024-03-04T10:11:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 -- 172.21.15.99:0/857133835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c15b960 msgr2=0x7f3a0c0d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 --2- 172.21.15.99:0/857133835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0d0590 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0ce1b0 tx=0x7f39fc02b840 comp rx=0 tx=0).stop 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 -- 172.21.15.99:0/857133835 shutdown_connections 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 --2- 172.21.15.99:0/857133835 >> v2:172.21.15.162:3300/0 conn(0x7f3a0c0b0e10 0x7f3a0c0b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 --2- 172.21.15.99:0/857133835 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0d0590 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 -- 172.21.15.99:0/857133835 >> 172.21.15.99:0/857133835 conn(0x7f3a0c06f5f0 msgr2=0x7f3a0c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 -- 172.21.15.99:0/857133835 shutdown_connections 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.220+0000 7f3a12419640 1 -- 172.21.15.99:0/857133835 wait complete. 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.221+0000 7f3a12419640 1 Processor -- start 2024-03-04T10:11:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 -- start start 2024-03-04T10:11:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a0c0b0e10 0x7f3a0c158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 0x7f3a0c0b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c15bec0 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c15bd40 con 0x7f3a0c15b960 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a12419640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0b4d00 con 0x7f3a0c0b0e10 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 0x7f3a0c0b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 0x7f3a0c0b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:45288/0 (socket says 172.21.15.99:45288) 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.222+0000 7f3a0b7fe640 1 -- 172.21.15.99:0/1726220641 learned_addr learned my addr 172.21.15.99:0/1726220641 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0b7fe640 1 -- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a0c0b0e10 msgr2=0x7f3a0c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0bfff640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a0c0b0e10 0x7f3a0c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a109b6640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0b7fe640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a0c0b0e10 0x7f3a0c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0b7fe640 1 -- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a0c15b960 msgr2=0x7f3a0c0b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0b7fe640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0b2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0b7fe640 1 -- 172.21.15.99:0/1726220641 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0cbe30 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a0b7fe640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 0x7f3a0c0b1ec0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000c500 tx=0x7f3a0000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00018070 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00007580 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c0cc110 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00004a20 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.223+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c158650 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.224+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c074f60 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.225+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3a0001b070 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.225+0000 7f3a08ff9640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f39f806b4d0 0x7f39f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.225+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6561+0+0 (secure 0 0 0) 0x7f3a000897e0 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.225+0000 7f3a0bfff640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f39f806b4d0 0x7f39f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:09.230 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.226+0000 7f3a0bfff640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f39f806b4d0 0x7f39f806d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f39fc036000 tx=0x7f39fc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:09.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.230+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3a00058390 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.435+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a0c0cd5e0 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.437 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.435+0000 7f3a08ff9640 1 -- 172.21.15.99:0/1726220641 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f3a00057c90 con 0x7f3a0c0d0ad0 2024-03-04T10:11:09.437 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:09.437 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":105,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7641,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:10:53.062239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8550":{"gid":8550,"name":"c","rank":0,"incarnation":105,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6835/301817793","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6834","nonce":301817793},{"type":"v1","addr":"172.21.15.99:6835","nonce":301817793}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:09.437 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 105 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f39f806b4d0 msgr2=0x7f39f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f39f806b4d0 0x7f39f806d990 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f39fc036000 tx=0x7f39fc008030 comp rx=0 tx=0).stop 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 msgr2=0x7f3a0c0b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 0x7f3a0c0b1ec0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000c500 tx=0x7f3a0000c9c0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 shutdown_connections 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f39f806b4d0 0x7f39f806d990 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.439 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3a0c15b960 0x7f3a0c0b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3a0c0d0ad0 0x7f3a0c0b1ec0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 --2- 172.21.15.99:0/1726220641 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3a0c0b0e10 0x7f3a0c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 >> 172.21.15.99:0/1726220641 conn(0x7f3a0c06f5f0 msgr2=0x7f3a0c0ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 shutdown_connections 2024-03-04T10:11:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:09.437+0000 7f3a12419640 1 -- 172.21.15.99:0/1726220641 wait complete. 2024-03-04T10:11:09.450 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:09.993 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:11:09.992+0000 7fceef499640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr:*** Caught signal (Floating point exception) ** 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7fceef499640 thread_name:md_recov_open 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcefb654db0] 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x557aec94d783] 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x557aec6ea76f] 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x557aec6ae532] 2024-03-04T10:11:09.994 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x557aec6ae62f] 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x557aec95b917] 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x557aec94e175] 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fcefb69f802] 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fcefb63f450] 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr:2024-03-04T10:11:09.992+0000 7fceef499640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7fceef499640 thread_name:md_recov_open 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcefb654db0] 2024-03-04T10:11:09.995 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x557aec94d783] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x557aec6ea76f] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x557aec6ae532] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x557aec6ae62f] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x557aec95b917] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x557aec94e175] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fcefb69f802] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fcefb63f450] 2024-03-04T10:11:09.996 INFO:tasks.ceph.mds.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:09.997 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:11:09.997 INFO:tasks.ceph.mds.a.smithi099.stderr: -6> 2024-03-04T10:11:09.992+0000 7fceef499640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:09.998 INFO:tasks.ceph.mds.a.smithi099.stderr: 0> 2024-03-04T10:11:09.992+0000 7fceef499640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:09.998 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7fceef499640 thread_name:md_recov_open 2024-03-04T10:11:09.998 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:11:09.998 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:09.998 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcefb654db0] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x557aec94d783] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x557aec6ea76f] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x557aec6ae532] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x557aec6ae62f] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x557aec95b917] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x557aec94e175] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fcefb69f802] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fcefb63f450] 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:09.999 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: -6> 2024-03-04T10:11:09.992+0000 7fceef499640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 0> 2024-03-04T10:11:09.992+0000 7fceef499640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: in thread 7fceef499640 thread_name:md_recov_open 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fcefb654db0] 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x557aec94d783] 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x557aec6ea76f] 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 4: (MDSRank::damaged()+0x52) [0x557aec6ae532] 2024-03-04T10:11:10.000 INFO:tasks.ceph.mds.a.smithi099.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x557aec6ae62f] 2024-03-04T10:11:10.001 INFO:tasks.ceph.mds.a.smithi099.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x557aec95b917] 2024-03-04T10:11:10.001 INFO:tasks.ceph.mds.a.smithi099.stderr: 7: ceph-mds(+0x3fc175) [0x557aec94e175] 2024-03-04T10:11:10.001 INFO:tasks.ceph.mds.a.smithi099.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7fcefb69f802] 2024-03-04T10:11:10.001 INFO:tasks.ceph.mds.a.smithi099.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7fcefb63f450] 2024-03-04T10:11:10.001 INFO:tasks.ceph.mds.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:10.001 INFO:tasks.ceph.mds.a.smithi099.stderr: 2024-03-04T10:11:10.087 INFO:tasks.ceph.mds.a.smithi099.stderr:daemon-helper: command crashed with signal 8 2024-03-04T10:11:10.617 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:10.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:13.415 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-04T10:11:13.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~20s 2024-03-04T10:11:14.451 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:14.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.565+0000 7fdf8bfa9640 1 Processor -- start 2024-03-04T10:11:14.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.565+0000 7fdf8bfa9640 1 -- start start 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf8bfa9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdf8415b030 0x7fdf8415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf8bfa9640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8406d8b0 con 0x7fdf8415b410 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf8bfa9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8406db90 con 0x7fdf8406d3b0 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf8bfa9640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840b2720 con 0x7fdf8415b030 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf89d45640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fdf8415b030 0x7fdf8415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf8a546640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fdf8415b410 0x7fdf8415b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57386/0 (socket says 172.21.15.99:57386) 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf8a546640 1 -- 172.21.15.99:0/3304286296 learned_addr learned my addr 172.21.15.99:0/3304286296 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf89d45640 1 -- 172.21.15.99:0/3304286296 >> v1:172.21.15.162:6789/0 conn(0x7fdf8406d3b0 legacy=0x7fdf8415dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf89d45640 1 -- 172.21.15.99:0/3304286296 >> v1:172.21.15.99:6789/0 conn(0x7fdf8415b410 legacy=0x7fdf8415b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf88d43640 1 -- 172.21.15.99:0/3304286296 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031830416 0 0) 0x7fdf8406d8b0 con 0x7fdf8415b410 2024-03-04T10:11:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.566+0000 7fdf89d45640 1 -- 172.21.15.99:0/3304286296 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8406de70 con 0x7fdf8415b030 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.567+0000 7fdf89d45640 1 --2- 172.21.15.99:0/3304286296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf8415c120 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fdf800097a0 tx=0x7fdf8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe68ab431a0e53d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.567+0000 7fdf88d43640 1 -- 172.21.15.99:0/3304286296 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80035020 con 0x7fdf8415b030 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.567+0000 7fdf88d43640 1 -- 172.21.15.99:0/3304286296 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80031830 con 0x7fdf8415b030 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.567+0000 7fdf88d43640 1 -- 172.21.15.99:0/3304286296 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80031c00 con 0x7fdf8415b030 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/3304286296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 msgr2=0x7fdf8415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/3304286296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf8415c120 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fdf800097a0 tx=0x7fdf8002b8d0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/3304286296 shutdown_connections 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/3304286296 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf8415c120 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/3304286296 >> 172.21.15.99:0/3304286296 conn(0x7fdf8406acd0 msgr2=0x7fdf8406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:14.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/3304286296 shutdown_connections 2024-03-04T10:11:14.571 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.568+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/3304286296 wait complete. 2024-03-04T10:11:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.569+0000 7fdf8bfa9640 1 Processor -- start 2024-03-04T10:11:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.569+0000 7fdf8bfa9640 1 -- start start 2024-03-04T10:11:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8bfa9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8406d3b0 0x7fdf84147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8bfa9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf84155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8bfa9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8415b410 0x7fdf840b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8bfa9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8406e810 con 0x7fdf8415b030 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8bfa9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8406e690 con 0x7fdf8406d3b0 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8bfa9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840b2bb0 con 0x7fdf8415b410 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf84155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf84155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39780/0 (socket says 172.21.15.99:39780) 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 -- 172.21.15.99:0/481095989 learned_addr learned my addr 172.21.15.99:0/481095989 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8a546640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8415b410 0x7fdf840b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89d45640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8406d3b0 0x7fdf84147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 -- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8415b410 msgr2=0x7fdf840b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8415b410 0x7fdf840b2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 -- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8406d3b0 msgr2=0x7fdf84147800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8406d3b0 0x7fdf84147800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 -- 172.21.15.99:0/481095989 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf841f4a70 con 0x7fdf8415b030 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf8a546640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8415b410 0x7fdf840b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89d45640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8406d3b0 0x7fdf84147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.570+0000 7fdf89544640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf84155210 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c400 tx=0x7fdf7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.571+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74017070 con 0x7fdf8415b030 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.571+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf740052d0 con 0x7fdf8415b030 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.571+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf841f4d50 con 0x7fdf8415b030 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.571+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003830 con 0x7fdf8415b030 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.571+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf841f5310 con 0x7fdf8415b030 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.572+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf84074f60 con 0x7fdf8415b030 2024-03-04T10:11:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.572+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fdf7401a070 con 0x7fdf8415b030 2024-03-04T10:11:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.572+0000 7fdf7a7fc640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf6806b4d0 0x7fdf6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.572+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6647+0+0 (secure 0 0 0) 0x7fdf740899b0 con 0x7fdf8415b030 2024-03-04T10:11:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.573+0000 7fdf89d45640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf6806b4d0 0x7fdf6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.573+0000 7fdf89d45640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf6806b4d0 0x7fdf6806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fdf80036000 tx=0x7fdf80008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:14.578 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.577+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fdf74058480 con 0x7fdf8415b030 2024-03-04T10:11:14.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.784+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf8415d4c0 con 0x7fdf8415b030 2024-03-04T10:11:14.787 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.784+0000 7fdf7a7fc640 1 -- 172.21.15.99:0/481095989 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4154 (secure 0 0 0) 0x7fdf74057d80 con 0x7fdf8415b030 2024-03-04T10:11:14.787 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:14.787 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":106,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:09.979254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7641":{"gid":7641,"name":"a","rank":0,"incarnation":106,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 106 2024-03-04T10:11:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf6806b4d0 msgr2=0x7fdf6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:14.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf6806b4d0 0x7fdf6806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fdf80036000 tx=0x7fdf80008030 comp rx=0 tx=0).stop 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 msgr2=0x7fdf84155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf84155210 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c400 tx=0x7fdf7400c8c0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 shutdown_connections 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fdf6806b4d0 0x7fdf6806d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fdf8415b410 0x7fdf840b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fdf8415b030 0x7fdf84155210 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 --2- 172.21.15.99:0/481095989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf8406d3b0 0x7fdf84147800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 >> 172.21.15.99:0/481095989 conn(0x7fdf8406acd0 msgr2=0x7fdf840b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 shutdown_connections 2024-03-04T10:11:14.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:14.786+0000 7fdf8bfa9640 1 -- 172.21.15.99:0/481095989 wait complete. 2024-03-04T10:11:14.799 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:19.801 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:19.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 Processor -- start 2024-03-04T10:11:19.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 -- start start 2024-03-04T10:11:19.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff70406d3b0 0x7ff7040ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:19.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7ff7040cc500 0x7ff7040cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7040d0140 con 0x7ff7040cfd60 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7040cf440 con 0x7ff70406d3b0 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.915+0000 7ff70a19e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70415cbb0 con 0x7ff7040cc500 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff70406d3b0 0x7ff7040ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7ff70406d3b0 0x7ff7040ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39792/0 (socket says 172.21.15.99:39792) 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff7037fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7ff7040cfd60 0x7ff7040cbdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57388/0 (socket says 172.21.15.99:57388) 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 -- 172.21.15.99:0/3429203260 learned_addr learned my addr 172.21.15.99:0/3429203260 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 -- 172.21.15.99:0/3429203260 >> v2:172.21.15.162:3300/0 conn(0x7ff7040cc500 msgr2=0x7ff7040cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 --2- 172.21.15.99:0/3429203260 >> v2:172.21.15.162:3300/0 conn(0x7ff7040cc500 0x7ff7040cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 -- 172.21.15.99:0/3429203260 >> v1:172.21.15.99:6789/0 conn(0x7ff7040cfd60 legacy=0x7ff7040cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:19.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.916+0000 7ff702ffd640 1 -- 172.21.15.99:0/3429203260 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70415c280 con 0x7ff70406d3b0 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff702ffd640 1 --2- 172.21.15.99:0/3429203260 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff70406d3b0 0x7ff7040ce9c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff70415cd30 tx=0x7ff6f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f25cb4397f151bb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff7027fc640 1 -- 172.21.15.99:0/3429203260 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8034070 con 0x7ff70406d3b0 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff7027fc640 1 -- 172.21.15.99:0/3429203260 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f8030860 con 0x7ff70406d3b0 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff7027fc640 1 -- 172.21.15.99:0/3429203260 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8030c30 con 0x7ff70406d3b0 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 -- 172.21.15.99:0/3429203260 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff70406d3b0 msgr2=0x7ff7040ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3429203260 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff70406d3b0 0x7ff7040ce9c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7ff70415cd30 tx=0x7ff6f802b9a0 comp rx=0 tx=0).stop 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 -- 172.21.15.99:0/3429203260 shutdown_connections 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3429203260 >> v2:172.21.15.162:3300/0 conn(0x7ff7040cc500 0x7ff7040cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3429203260 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff70406d3b0 0x7ff7040ce9c0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 -- 172.21.15.99:0/3429203260 >> 172.21.15.99:0/3429203260 conn(0x7ff70406acd0 msgr2=0x7ff70406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 -- 172.21.15.99:0/3429203260 shutdown_connections 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.917+0000 7ff70a19e640 1 -- 172.21.15.99:0/3429203260 wait complete. 2024-03-04T10:11:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 Processor -- start 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 -- start start 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff70406d3b0 0x7ff704158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 0x7ff7041448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7040cfd60 0x7ff7041f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70415bf30 con 0x7ff7040cc500 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70415c840 con 0x7ff7040cfd60 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.919+0000 7ff70a19e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70415f520 con 0x7ff70406d3b0 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 0x7ff7041448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 0x7ff7041448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39804/0 (socket says 172.21.15.99:39804) 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 -- 172.21.15.99:0/3459438042 learned_addr learned my addr 172.21.15.99:0/3459438042 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff703fff640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7040cfd60 0x7ff7041f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:19.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff7037fe640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff70406d3b0 0x7ff704158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 -- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff70406d3b0 msgr2=0x7ff704158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff70406d3b0 0x7ff704158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 -- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7040cfd60 msgr2=0x7ff7041f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7040cfd60 0x7ff7041f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 -- 172.21.15.99:0/3459438042 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff704158950 con 0x7ff7040cc500 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff7037fe640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff70406d3b0 0x7ff704158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff702ffd640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 0x7ff7041448a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8048070 con 0x7ff7040cc500 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f8039d90 con 0x7ff7040cc500 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff704158c30 con 0x7ff7040cc500 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8034040 con 0x7ff7040cc500 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.920+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7040ad4b0 con 0x7ff7040cc500 2024-03-04T10:11:19.924 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.921+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff704074f60 con 0x7ff7040cc500 2024-03-04T10:11:19.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.922+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7ff6f804b070 con 0x7ff7040cc500 2024-03-04T10:11:19.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.922+0000 7ff6ebfff640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6f006b4d0 0x7ff6f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:19.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.922+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6647+0+0 (secure 0 0 0) 0x7ff6f80ac6d0 con 0x7ff7040cc500 2024-03-04T10:11:19.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.922+0000 7ff7037fe640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6f006b4d0 0x7ff6f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:19.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.923+0000 7ff7037fe640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6f006b4d0 0x7ff6f006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4000f60 tx=0x7ff6f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:19.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~7s 2024-03-04T10:11:19.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~26s 2024-03-04T10:11:19.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:19.926+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7ff6f807b1a0 con 0x7ff7040cc500 2024-03-04T10:11:20.133 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.132+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff70406e8d0 con 0x7ff7040cc500 2024-03-04T10:11:20.134 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.132+0000 7ff6ebfff640 1 -- 172.21.15.99:0/3459438042 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4154 (secure 0 0 0) 0x7ff6f807aaa0 con 0x7ff7040cc500 2024-03-04T10:11:20.134 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:20.134 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":106,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":6517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:09.979254+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":100,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7641":{"gid":7641,"name":"a","rank":0,"incarnation":106,"state":"up:replay","state_seq":1,"addr":"172.21.15.99:6833/584503338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6832","nonce":584503338},{"type":"v1","addr":"172.21.15.99:6833","nonce":584503338}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:20.134 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 106 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6f006b4d0 msgr2=0x7ff6f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6f006b4d0 0x7ff6f006d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4000f60 tx=0x7ff6f4009040 comp rx=0 tx=0).stop 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 msgr2=0x7ff7041448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 0x7ff7041448a0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f8004730 comp rx=0 tx=0).stop 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 shutdown_connections 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7ff6f006b4d0 0x7ff6f006d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff7040cfd60 0x7ff7041f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:20.136 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7ff7040cc500 0x7ff7041448a0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:20.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 --2- 172.21.15.99:0/3459438042 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7ff70406d3b0 0x7ff704158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:20.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 >> 172.21.15.99:0/3459438042 conn(0x7ff70406acd0 msgr2=0x7ff704155e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:20.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 shutdown_connections 2024-03-04T10:11:20.137 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:20.134+0000 7ff70a19e640 1 -- 172.21.15.99:0/3459438042 wait complete. 2024-03-04T10:11:20.147 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:25.000 INFO:tasks.ceph.mds.d.smithi162.stderr:2024-03-04T10:11:24.998+0000 7f89d695f640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:25.000 INFO:tasks.ceph.mds.d.smithi162.stderr:*** Caught signal (Floating point exception) ** 2024-03-04T10:11:25.000 INFO:tasks.ceph.mds.d.smithi162.stderr: in thread 7f89d695f640 thread_name:md_recov_open 2024-03-04T10:11:25.000 INFO:tasks.ceph.mds.d.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:25.000 INFO:tasks.ceph.mds.d.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f89e2c54db0] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x55eee6896783] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x55eee663376f] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x55eee65f7532] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x55eee65f762f] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x55eee68a4917] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x55eee6897175] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f89e2c9f802] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f89e2c3f450] 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr:2024-03-04T10:11:24.998+0000 7f89d695f640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:25.001 INFO:tasks.ceph.mds.d.smithi162.stderr: in thread 7f89d695f640 thread_name:md_recov_open 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f89e2c54db0] 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x55eee6896783] 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x55eee663376f] 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x55eee65f7532] 2024-03-04T10:11:25.002 INFO:tasks.ceph.mds.d.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x55eee65f762f] 2024-03-04T10:11:25.003 INFO:tasks.ceph.mds.d.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x55eee68a4917] 2024-03-04T10:11:25.003 INFO:tasks.ceph.mds.d.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x55eee6897175] 2024-03-04T10:11:25.003 INFO:tasks.ceph.mds.d.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f89e2c9f802] 2024-03-04T10:11:25.003 INFO:tasks.ceph.mds.d.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f89e2c3f450] 2024-03-04T10:11:25.003 INFO:tasks.ceph.mds.d.smithi162.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:25.003 INFO:tasks.ceph.mds.d.smithi162.stderr: 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: -6> 2024-03-04T10:11:24.998+0000 7f89d695f640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 0> 2024-03-04T10:11:24.998+0000 7f89d695f640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: in thread 7f89d695f640 thread_name:md_recov_open 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f89e2c54db0] 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x55eee6896783] 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x55eee663376f] 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x55eee65f7532] 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x55eee65f762f] 2024-03-04T10:11:25.004 INFO:tasks.ceph.mds.d.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x55eee68a4917] 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x55eee6897175] 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f89e2c9f802] 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f89e2c3f450] 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: -6> 2024-03-04T10:11:24.998+0000 7f89d695f640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 0> 2024-03-04T10:11:24.998+0000 7f89d695f640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: in thread 7f89d695f640 thread_name:md_recov_open 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f89e2c54db0] 2024-03-04T10:11:25.005 INFO:tasks.ceph.mds.d.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x55eee6896783] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x55eee663376f] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x55eee65f7532] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x55eee65f762f] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x55eee68a4917] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x55eee6897175] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f89e2c9f802] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f89e2c3f450] 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:25.006 INFO:tasks.ceph.mds.d.smithi162.stderr: 2024-03-04T10:11:25.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.259+0000 7f7ea7bc4640 1 Processor -- start 2024-03-04T10:11:25.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.259+0000 7f7ea7bc4640 1 -- start start 2024-03-04T10:11:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.260+0000 7f7ea7bc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7ea00b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.260+0000 7f7ea7bc4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea015e1e0 con 0x7f7ea00b5470 2024-03-04T10:11:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.260+0000 7f7ea7bc4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea00d1c10 con 0x7f7ea015de00 2024-03-04T10:11:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.260+0000 7f7ea7bc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea00b98c0 con 0x7f7ea00b2e60 2024-03-04T10:11:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.260+0000 7f7ea6161640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7ea00b5470 0x7f7ea00d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:52930/0 (socket says 172.21.15.99:52930) 2024-03-04T10:11:25.274 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.260+0000 7f7ea6161640 1 -- 172.21.15.99:0/3858466883 learned_addr learned my addr 172.21.15.99:0/3858466883 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:25.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7ea00b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315400641 0 0) 0x7f7ea015e1e0 con 0x7f7ea00b5470 2024-03-04T10:11:25.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e8c003520 con 0x7f7ea00b5470 2024-03-04T10:11:25.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68744811 0 0) 0x7f7ea00d1c10 con 0x7f7ea015de00 2024-03-04T10:11:25.275 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea015e1e0 con 0x7f7ea015de00 2024-03-04T10:11:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371335828 0 0) 0x7f7e8c003520 con 0x7f7ea00b5470 2024-03-04T10:11:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 msgr2=0x7f7ea00b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7ea00b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 >> v1:172.21.15.162:6789/0 conn(0x7f7ea015de00 legacy=0x7f7ea00b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea00d1ef0 con 0x7f7ea00b5470 2024-03-04T10:11:25.276 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7e9c0019f0 con 0x7f7ea00b5470 2024-03-04T10:11:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7ea00b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:11:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 >> v1:172.21.15.99:6789/0 conn(0x7f7ea00b5470 legacy=0x7f7ea00d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e8c003b70 0x7f7e8c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7e8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7e8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.277 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea00b98c0 con 0x7f7e8c004800 2024-03-04T10:11:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c00ba70 con 0x7f7e8c003b70 2024-03-04T10:11:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c00bc40 con 0x7f7ea00b2e60 2024-03-04T10:11:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7e8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea5960640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e8c003b70 0x7f7e8c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea6161640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7e8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.278 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 -- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 msgr2=0x7f7e8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7e8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 -- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e8c003b70 msgr2=0x7f7e8c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e8c003b70 0x7f7e8c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea515f640 1 -- 172.21.15.99:0/3858466883 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c003520 con 0x7f7e8c004800 2024-03-04T10:11:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea5960640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e8c003b70 0x7f7e8c003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:25.279 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.261+0000 7f7ea6161640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea00b2e60 0x7f7e8c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.262+0000 7f7ea515f640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7e8c004c80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009c60 tx=0x7f7e8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db7b6c299a0fd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.262+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90039070 con 0x7f7e8c004800 2024-03-04T10:11:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.262+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9002c8a0 con 0x7f7e8c004800 2024-03-04T10:11:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.262+0000 7f7ea495e640 1 -- 172.21.15.99:0/3858466883 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90009860 con 0x7f7e8c004800 2024-03-04T10:11:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/3858466883 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 msgr2=0x7f7e8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.280 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/3858466883 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7e8c004c80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f7e90009c60 tx=0x7f7e8c005830 comp rx=0 tx=0).stop 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea6161640 1 -- 172.21.15.99:0/3858466883 reap_dead start 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/3858466883 shutdown_connections 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/3858466883 >> 172.21.15.99:0/3858466883 conn(0x7f7ea00d0470 msgr2=0x7f7ea00d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/3858466883 shutdown_connections 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.263+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/3858466883 wait complete. 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.264+0000 7f7ea7bc4640 1 Processor -- start 2024-03-04T10:11:25.281 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.264+0000 7f7ea7bc4640 1 -- start start 2024-03-04T10:11:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea7bc4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7ea00c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea7bc4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea01e6950 0x7f7ea01e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea7bc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ea00d2bb0 0x7f7ea00d3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea7bc4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea00d2820 con 0x7f7e8c004800 2024-03-04T10:11:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea7bc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea00d24b0 con 0x7f7ea00d2bb0 2024-03-04T10:11:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea7bc4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea00b72a0 con 0x7f7ea01e6950 2024-03-04T10:11:25.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7ea00c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7ea00c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41442/0 (socket says 172.21.15.99:41442) 2024-03-04T10:11:25.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 -- 172.21.15.99:0/2616484246 learned_addr learned my addr 172.21.15.99:0/2616484246 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:25.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea515f640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea01e6950 0x7f7ea01e6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea6161640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ea00d2bb0 0x7f7ea00d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.283 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 -- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea01e6950 msgr2=0x7f7ea01e6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea01e6950 0x7f7ea01e6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 -- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ea00d2bb0 msgr2=0x7f7ea00d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ea00d2bb0 0x7f7ea00d3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 -- 172.21.15.99:0/2616484246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea00c9cb0 con 0x7f7e8c004800 2024-03-04T10:11:25.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea6161640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ea00d2bb0 0x7f7ea00d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:25.284 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.265+0000 7f7ea5960640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7ea00c9770 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800e7b0 tx=0x7f7e9800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:25.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.266+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800ee60 con 0x7f7e8c004800 2024-03-04T10:11:25.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.266+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e980052e0 con 0x7f7e8c004800 2024-03-04T10:11:25.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.266+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea01483d0 con 0x7f7e8c004800 2024-03-04T10:11:25.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.266+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98007020 con 0x7f7e8c004800 2024-03-04T10:11:25.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.266+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea0148990 con 0x7f7e8c004800 2024-03-04T10:11:25.285 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.267+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea00d3570 con 0x7f7e8c004800 2024-03-04T10:11:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.267+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7e9801a020 con 0x7f7e8c004800 2024-03-04T10:11:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.268+0000 7f7e967fc640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e8406b500 0x7f7e8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.268+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6733+0+0 (secure 0 0 0) 0x7f7e98089d80 con 0x7f7e8c004800 2024-03-04T10:11:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.268+0000 7f7ea515f640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e8406b500 0x7f7e8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.268+0000 7f7ea515f640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e8406b500 0x7f7e8406d9c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7e90036000 tx=0x7f7e90030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:25.286 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.272+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7e98058770 con 0x7f7e8c004800 2024-03-04T10:11:25.327 INFO:tasks.ceph.mds.d.smithi162.stderr:daemon-helper: command crashed with signal 8 2024-03-04T10:11:25.478 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.477+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ea0074f60 con 0x7f7e8c004800 2024-03-04T10:11:25.479 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.477+0000 7f7e967fc640 1 -- 172.21.15.99:0/2616484246 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3366 (secure 0 0 0) 0x7f7e98058070 con 0x7f7e8c004800 2024-03-04T10:11:25.479 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:25.479 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":107,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:24.982776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6517":{"gid":6517,"name":"d","rank":0,"incarnation":107,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:25.480 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 107 2024-03-04T10:11:25.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e8406b500 msgr2=0x7f7e8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e8406b500 0x7f7e8406d9c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7e90036000 tx=0x7f7e90030000 comp rx=0 tx=0).stop 2024-03-04T10:11:25.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 msgr2=0x7f7ea00c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:25.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7ea00c9770 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800e7b0 tx=0x7f7e9800ec70 comp rx=0 tx=0).stop 2024-03-04T10:11:25.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 shutdown_connections 2024-03-04T10:11:25.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7e8406b500 0x7f7e8406d9c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7ea00d2bb0 0x7f7ea00d3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7ea01e6950 0x7f7ea01e6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 --2- 172.21.15.99:0/2616484246 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7e8c004800 0x7f7ea00c9770 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:25.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 >> 172.21.15.99:0/2616484246 conn(0x7f7ea00d0470 msgr2=0x7f7ea00b8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:25.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 shutdown_connections 2024-03-04T10:11:25.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:25.479+0000 7f7ea7bc4640 1 -- 172.21.15.99:0/2616484246 wait complete. 2024-03-04T10:11:25.492 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:26.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~13s 2024-03-04T10:11:26.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~33s 2024-03-04T10:11:26.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-04T10:11:30.493 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:30.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.603+0000 7f09acc8c640 1 Processor -- start 2024-03-04T10:11:30.606 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.603+0000 7f09acc8c640 1 -- start start 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.604+0000 7f09acc8c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a80b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.604+0000 7f09acc8c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f09a80b5470 0x7f09a80b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.604+0000 7f09acc8c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a815e1e0 con 0x7f09a815de00 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.604+0000 7f09acc8c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a80b98c0 con 0x7f09a80b5470 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.604+0000 7f09acc8c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80b32c0 con 0x7f09a80b2e60 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a6d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f09a80b5470 0x7f09a80b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a5d9b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a80b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a6d9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f09a80b5470 0x7f09a80b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:34776/0 (socket says 172.21.15.99:34776) 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a659c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f09a815de00 0x7f09a80b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:47216/0 (socket says 172.21.15.99:47216) 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a6d9d640 1 -- 172.21.15.99:0/3764978034 learned_addr learned my addr 172.21.15.99:0/3764978034 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a5d9b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a80b9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:35916/0 (socket says 172.21.15.99:35916) 2024-03-04T10:11:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a5d9b640 1 -- 172.21.15.99:0/3764978034 >> v2:172.21.15.162:3300/0 conn(0x7f09a80b5470 msgr2=0x7f09a80b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a5d9b640 1 --2- 172.21.15.99:0/3764978034 >> v2:172.21.15.162:3300/0 conn(0x7f09a80b5470 0x7f09a80b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a5d9b640 1 -- 172.21.15.99:0/3764978034 >> v1:172.21.15.162:6789/0 conn(0x7f09a815de00 legacy=0x7f09a80b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.605+0000 7f09a5d9b640 1 -- 172.21.15.99:0/3764978034 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80d1c10 con 0x7f09a80b2e60 2024-03-04T10:11:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.606+0000 7f09a5d9b640 1 --2- 172.21.15.99:0/3764978034 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a80b9380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f099c0097a0 tx=0x7f099c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63e2f1a601437ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.606+0000 7f09a559a640 1 -- 172.21.15.99:0/3764978034 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c035020 con 0x7f09a80b2e60 2024-03-04T10:11:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.606+0000 7f09a559a640 1 -- 172.21.15.99:0/3764978034 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c031830 con 0x7f09a80b2e60 2024-03-04T10:11:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.606+0000 7f09a559a640 1 -- 172.21.15.99:0/3764978034 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c031c00 con 0x7f09a80b2e60 2024-03-04T10:11:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 -- 172.21.15.99:0/3764978034 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 msgr2=0x7f09a80b9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 --2- 172.21.15.99:0/3764978034 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a80b9380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f099c0097a0 tx=0x7f099c02b8d0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 -- 172.21.15.99:0/3764978034 shutdown_connections 2024-03-04T10:11:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 --2- 172.21.15.99:0/3764978034 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a80b9380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 --2- 172.21.15.99:0/3764978034 >> v2:172.21.15.162:3300/0 conn(0x7f09a80b5470 0x7f09a80b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 -- 172.21.15.99:0/3764978034 >> 172.21.15.99:0/3764978034 conn(0x7f09a80d0470 msgr2=0x7f09a80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 -- 172.21.15.99:0/3764978034 shutdown_connections 2024-03-04T10:11:30.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.607+0000 7f09acc8c640 1 -- 172.21.15.99:0/3764978034 wait complete. 2024-03-04T10:11:30.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.608+0000 7f09acc8c640 1 Processor -- start 2024-03-04T10:11:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.608+0000 7f09acc8c640 1 -- start start 2024-03-04T10:11:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09acc8c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a815f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09acc8c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 0x7f09a815ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09acc8c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a815de00 0x7f09a815f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09acc8c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80d21d0 con 0x7f09a80b5470 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09acc8c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80d2820 con 0x7f09a815de00 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09acc8c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80d24b0 con 0x7f09a80b2e60 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 0x7f09a815ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 0x7f09a815ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:41458/0 (socket says 172.21.15.99:41458) 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 -- 172.21.15.99:0/843854106 learned_addr learned my addr 172.21.15.99:0/843854106 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a659c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a815f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a6d9d640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a815de00 0x7f09a815f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 -- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 msgr2=0x7f09a815f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a815f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 -- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a815de00 msgr2=0x7f09a815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a815de00 0x7f09a815f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.609+0000 7f09a5d9b640 1 -- 172.21.15.99:0/843854106 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a8160250 con 0x7f09a80b5470 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.610+0000 7f09a5d9b640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 0x7f09a815ba60 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f099c02bee0 tx=0x7f099c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.610+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c035040 con 0x7f09a80b5470 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.610+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c03b920 con 0x7f09a80b5470 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.610+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a815e470 con 0x7f09a80b5470 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.610+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c03bd10 con 0x7f09a80b5470 2024-03-04T10:11:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.610+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a815ea30 con 0x7f09a80b5470 2024-03-04T10:11:30.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.611+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a815ef20 con 0x7f09a80b5470 2024-03-04T10:11:30.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.611+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f099c038020 con 0x7f09a80b5470 2024-03-04T10:11:30.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.612+0000 7f0996ffd640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f098806b4d0 0x7f098806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:30.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.612+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6733+0+0 (secure 0 0 0) 0x7f099c0ac660 con 0x7f09a80b5470 2024-03-04T10:11:30.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.612+0000 7f09a659c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f098806b4d0 0x7f098806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:30.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.612+0000 7f09a659c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f098806b4d0 0x7f098806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0998007fd0 tx=0x7f0998001630 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:30.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.616+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f099c031800 con 0x7f09a80b5470 2024-03-04T10:11:30.823 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.821+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09a8074f60 con 0x7f09a80b5470 2024-03-04T10:11:30.824 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.822+0000 7f0996ffd640 1 -- 172.21.15.99:0/843854106 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3366 (secure 0 0 0) 0x7f099c07aaa0 con 0x7f09a80b5470 2024-03-04T10:11:30.824 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:30.824 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":107,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:24.982776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6517":{"gid":6517,"name":"d","rank":0,"incarnation":107,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:30.825 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 107 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f098806b4d0 msgr2=0x7f098806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f098806b4d0 0x7f098806d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0998007fd0 tx=0x7f0998001630 comp rx=0 tx=0).stop 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 msgr2=0x7f09a815ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 0x7f09a815ba60 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f099c02bee0 tx=0x7f099c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 shutdown_connections 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f098806b4d0 0x7f098806d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f09a815de00 0x7f09a815f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f09a80b5470 0x7f09a815ba60 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 --2- 172.21.15.99:0/843854106 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f09a80b2e60 0x7f09a815f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:30.827 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 >> 172.21.15.99:0/843854106 conn(0x7f09a80d0470 msgr2=0x7f09a80b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:30.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 shutdown_connections 2024-03-04T10:11:30.828 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:30.824+0000 7f09acc8c640 1 -- 172.21.15.99:0/843854106 wait complete. 2024-03-04T10:11:30.838 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:32.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~19s 2024-03-04T10:11:32.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~39s 2024-03-04T10:11:32.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~6s 2024-03-04T10:11:35.839 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:35.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.958+0000 7fcc20116640 1 Processor -- start 2024-03-04T10:11:35.963 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.959+0000 7fcc20116640 1 -- start start 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.959+0000 7fcc20116640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcc1815b030 0x7fcc1815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.959+0000 7fcc20116640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fcc1815b410 0x7fcc1815b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.959+0000 7fcc20116640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1806d8b0 con 0x7fcc1806d3b0 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.959+0000 7fcc20116640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180b2720 con 0x7fcc1815b030 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.959+0000 7fcc20116640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1806db90 con 0x7fcc1815b410 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcc1815b030 0x7fcc1815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fcc1815b030 0x7fcc1815c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43804/0 (socket says 172.21.15.99:43804) 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 -- 172.21.15.99:0/3539289626 learned_addr learned my addr 172.21.15.99:0/3539289626 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 -- 172.21.15.99:0/3539289626 >> v2:172.21.15.162:3300/0 conn(0x7fcc1815b410 msgr2=0x7fcc1815b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1e6b3640 1 --2- 172.21.15.99:0/3539289626 >> v2:172.21.15.162:3300/0 conn(0x7fcc1815b410 0x7fcc1815b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 --2- 172.21.15.99:0/3539289626 >> v2:172.21.15.162:3300/0 conn(0x7fcc1815b410 0x7fcc1815b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 -- 172.21.15.99:0/3539289626 >> v1:172.21.15.162:6789/0 conn(0x7fcc1806d3b0 legacy=0x7fcc1815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1deb2640 1 -- 172.21.15.99:0/3539289626 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1806de70 con 0x7fcc1815b030 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.960+0000 7fcc1e6b3640 1 --2- 172.21.15.99:0/3539289626 >> v2:172.21.15.162:3300/0 conn(0x7fcc1815b410 0x7fcc1815b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc1deb2640 1 --2- 172.21.15.99:0/3539289626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1815c120 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fcc180b28a0 tx=0x7fcc1002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89c097b75f85ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc1ceb0640 1 -- 172.21.15.99:0/3539289626 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1003b070 con 0x7fcc1815b030 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc1ceb0640 1 -- 172.21.15.99:0/3539289626 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1002ee00 con 0x7fcc1815b030 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc1ceb0640 1 -- 172.21.15.99:0/3539289626 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10036660 con 0x7fcc1815b030 2024-03-04T10:11:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc20116640 1 -- 172.21.15.99:0/3539289626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 msgr2=0x7fcc1815c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc20116640 1 --2- 172.21.15.99:0/3539289626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1815c120 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fcc180b28a0 tx=0x7fcc1002e050 comp rx=0 tx=0).stop 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc20116640 1 -- 172.21.15.99:0/3539289626 shutdown_connections 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc20116640 1 --2- 172.21.15.99:0/3539289626 >> v2:172.21.15.162:3300/0 conn(0x7fcc1815b410 0x7fcc1815b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.961+0000 7fcc20116640 1 --2- 172.21.15.99:0/3539289626 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1815c120 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.962+0000 7fcc20116640 1 -- 172.21.15.99:0/3539289626 >> 172.21.15.99:0/3539289626 conn(0x7fcc1806acd0 msgr2=0x7fcc1806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.962+0000 7fcc20116640 1 -- 172.21.15.99:0/3539289626 shutdown_connections 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.962+0000 7fcc20116640 1 -- 172.21.15.99:0/3539289626 wait complete. 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 Processor -- start 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 -- start start 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc1806d3b0 0x7fcc1813ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1813e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc1815b410 0x7fcc181406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1806e690 con 0x7fcc1815b030 2024-03-04T10:11:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1815bdc0 con 0x7fcc1815b410 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.963+0000 7fcc20116640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1806e810 con 0x7fcc1806d3b0 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1813e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1813e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43816/0 (socket says 172.21.15.99:43816) 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 -- 172.21.15.99:0/2422371832 learned_addr learned my addr 172.21.15.99:0/2422371832 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 -- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc1806d3b0 msgr2=0x7fcc1813ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc1806d3b0 0x7fcc1813ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 -- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc1815b410 msgr2=0x7fcc181406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1e6b3640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc1815b410 0x7fcc181406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc1815b410 0x7fcc181406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 -- 172.21.15.99:0/2422371832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1813e330 con 0x7fcc1815b030 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1e6b3640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc1815b410 0x7fcc181406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc1d6b1640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1813e640 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c400 tx=0x7fcc0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08017070 con 0x7fcc1815b030 2024-03-04T10:11:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc080035f0 con 0x7fcc1815b030 2024-03-04T10:11:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08003a20 con 0x7fcc1815b030 2024-03-04T10:11:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1813f100 con 0x7fcc1815b030 2024-03-04T10:11:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.964+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1813f6c0 con 0x7fcc1815b030 2024-03-04T10:11:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.965+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc18074f60 con 0x7fcc1815b030 2024-03-04T10:11:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.966+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fcc0801a070 con 0x7fcc1815b030 2024-03-04T10:11:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.966+0000 7fcc0e7fc640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbfc06b4d0 0x7fcbfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.966+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6733+0+0 (secure 0 0 0) 0x7fcc08089930 con 0x7fcc1815b030 2024-03-04T10:11:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.966+0000 7fcc1deb2640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbfc06b4d0 0x7fcbfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.967+0000 7fcc1deb2640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbfc06b4d0 0x7fcbfc06d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcc10038000 tx=0x7fcc10030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:35.973 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:35.970+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fcc08057d60 con 0x7fcc1815b030 2024-03-04T10:11:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.175+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc180b3870 con 0x7fcc1815b030 2024-03-04T10:11:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.176+0000 7fcc0e7fc640 1 -- 172.21.15.99:0/2422371832 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3366 (secure 0 0 0) 0x7fcc0805be00 con 0x7fcc1815b030 2024-03-04T10:11:36.179 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:36.179 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":107,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6514,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:24.982776+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6517":{"gid":6517,"name":"d","rank":0,"incarnation":107,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6837/526865753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":526865753},{"type":"v1","addr":"172.21.15.162:6837","nonce":526865753}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 107 2024-03-04T10:11:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbfc06b4d0 msgr2=0x7fcbfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbfc06b4d0 0x7fcbfc06d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcc10038000 tx=0x7fcc10030000 comp rx=0 tx=0).stop 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 msgr2=0x7fcc1813e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1813e640 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c400 tx=0x7fcc0800c8c0 comp rx=0 tx=0).stop 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 shutdown_connections 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fcbfc06b4d0 0x7fcbfc06d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcc1815b410 0x7fcc181406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fcc1815b030 0x7fcc1813e640 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 --2- 172.21.15.99:0/2422371832 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fcc1806d3b0 0x7fcc1813ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 >> 172.21.15.99:0/2422371832 conn(0x7fcc1806acd0 msgr2=0x7fcc180b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 shutdown_connections 2024-03-04T10:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:36.178+0000 7fcc20116640 1 -- 172.21.15.99:0/2422371832 wait complete. 2024-03-04T10:11:36.192 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:39.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~26s 2024-03-04T10:11:39.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~46s 2024-03-04T10:11:39.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~13s 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr:2024-03-04T10:11:40.000+0000 7f0c188e8640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr:*** Caught signal (Floating point exception) ** 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: in thread 7f0c188e8640 thread_name:md_recov_open 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0c24c54db0] 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x561edb63a783] 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x561edb3d776f] 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x561edb39b532] 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x561edb39b62f] 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x561edb648917] 2024-03-04T10:11:40.003 INFO:tasks.ceph.mds.b.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x561edb63b175] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f0c24c9f802] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f0c24c3f450] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr:2024-03-04T10:11:40.001+0000 7f0c188e8640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: in thread 7f0c188e8640 thread_name:md_recov_open 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0c24c54db0] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x561edb63a783] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x561edb3d776f] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x561edb39b532] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x561edb39b62f] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x561edb648917] 2024-03-04T10:11:40.004 INFO:tasks.ceph.mds.b.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x561edb63b175] 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f0c24c9f802] 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f0c24c3f450] 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: -6> 2024-03-04T10:11:40.000+0000 7f0c188e8640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: 0> 2024-03-04T10:11:40.001+0000 7f0c188e8640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: in thread 7f0c188e8640 thread_name:md_recov_open 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:40.005 INFO:tasks.ceph.mds.b.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0c24c54db0] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x561edb63a783] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x561edb3d776f] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x561edb39b532] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x561edb39b62f] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x561edb648917] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x561edb63b175] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f0c24c9f802] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f0c24c3f450] 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: 2024-03-04T10:11:40.006 INFO:tasks.ceph.mds.b.smithi162.stderr: -6> 2024-03-04T10:11:40.000+0000 7f0c188e8640 -1 log_channel(cluster) log [ERR] : Error recovering journal 0x200: (2) No such file or directory 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 0> 2024-03-04T10:11:40.001+0000 7f0c188e8640 -1 *** Caught signal (Floating point exception) ** 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: in thread 7f0c188e8640 thread_name:md_recov_open 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f0c24c54db0] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 2: (MDLog::get_estimated_replay_finish_time()+0x73) [0x561edb63a783] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 3: (Beacon::notify_health(MDSRank const*)+0x26bf) [0x561edb3d776f] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 4: (MDSRank::damaged()+0x52) [0x561edb39b532] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 5: (MDSRank::damaged_unlocked()+0x7f) [0x561edb39b62f] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 6: (MDLog::_recovery_thread(MDSContext*)+0x1337) [0x561edb648917] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 7: ceph-mds(+0x3fc175) [0x561edb63b175] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 8: /lib64/libc.so.6(+0x9f802) [0x7f0c24c9f802] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: 9: /lib64/libc.so.6(+0x3f450) [0x7f0c24c3f450] 2024-03-04T10:11:40.007 INFO:tasks.ceph.mds.b.smithi162.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T10:11:40.008 INFO:tasks.ceph.mds.b.smithi162.stderr: 2024-03-04T10:11:40.344 INFO:tasks.ceph.mds.b.smithi162.stderr:daemon-helper: command crashed with signal 8 2024-03-04T10:11:40.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:40.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:11:41.192 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:41.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.303+0000 7fd8c3212640 1 Processor -- start 2024-03-04T10:11:41.306 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.303+0000 7fd8c3212640 1 -- start start 2024-03-04T10:11:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8c3212640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd8bc158410 0x7fd8bc0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8c3212640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc06b500 con 0x7fd8bc06acd0 2024-03-04T10:11:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8c3212640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc06b7e0 con 0x7fd8bc161e70 2024-03-04T10:11:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8c3212640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b3f00 con 0x7fd8bc158410 2024-03-04T10:11:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8bbfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fd8bc06acd0 0x7fd8bc161760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:39898/0 (socket says 172.21.15.99:39898) 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8bbfff640 1 -- 172.21.15.99:0/699821741 learned_addr learned my addr 172.21.15.99:0/699821741 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8c0fae640 1 --2- 172.21.15.99:0/699821741 >> v2:172.21.15.162:3300/0 conn(0x7fd8bc158410 0x7fd8bc0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8bb7fe640 1 -- 172.21.15.99:0/699821741 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443920328 0 0) 0x7fd8bc06b500 con 0x7fd8bc06acd0 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.304+0000 7fd8bb7fe640 1 -- 172.21.15.99:0/699821741 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4003520 con 0x7fd8bc06acd0 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.305+0000 7fd8c0fae640 1 -- 172.21.15.99:0/699821741 >> v1:172.21.15.162:6789/0 conn(0x7fd8bc161e70 legacy=0x7fd8bc06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.305+0000 7fd8c0fae640 1 -- 172.21.15.99:0/699821741 >> v1:172.21.15.99:6789/0 conn(0x7fd8bc06acd0 legacy=0x7fd8bc161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.305+0000 7fd8c0fae640 1 -- 172.21.15.99:0/699821741 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc06bac0 con 0x7fd8bc158410 2024-03-04T10:11:41.309 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.305+0000 7fd8c0fae640 1 --2- 172.21.15.99:0/699821741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc158410 0x7fd8bc0b39c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0097a0 tx=0x7fd8ac02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a679b0a9e4968572 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8bb7fe640 1 -- 172.21.15.99:0/699821741 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac02f810 con 0x7fd8bc158410 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8bb7fe640 1 -- 172.21.15.99:0/699821741 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac02f9b0 con 0x7fd8bc158410 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8bb7fe640 1 -- 172.21.15.99:0/699821741 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac02fd60 con 0x7fd8bc158410 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 -- 172.21.15.99:0/699821741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc158410 msgr2=0x7fd8bc0b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 --2- 172.21.15.99:0/699821741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc158410 0x7fd8bc0b39c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac0097a0 tx=0x7fd8ac02b9f0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 -- 172.21.15.99:0/699821741 shutdown_connections 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 --2- 172.21.15.99:0/699821741 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc158410 0x7fd8bc0b39c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 -- 172.21.15.99:0/699821741 >> 172.21.15.99:0/699821741 conn(0x7fd8bc0cfef0 msgr2=0x7fd8bc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 -- 172.21.15.99:0/699821741 shutdown_connections 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.306+0000 7fd8c3212640 1 -- 172.21.15.99:0/699821741 wait complete. 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.307+0000 7fd8c3212640 1 Processor -- start 2024-03-04T10:11:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.307+0000 7fd8c3212640 1 -- start start 2024-03-04T10:11:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c3212640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc06acd0 0x7fd8bc158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c3212640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8bc158410 0x7fd8bc14bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c3212640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 0x7fd8bc14c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c3212640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc06c5a0 con 0x7fd8bc161e70 2024-03-04T10:11:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c3212640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc06c080 con 0x7fd8bc06acd0 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c3212640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc06b160 con 0x7fd8bc158410 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c17af640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 0x7fd8bc14c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c17af640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 0x7fd8bc14c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:43822/0 (socket says 172.21.15.99:43822) 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.308+0000 7fd8c17af640 1 -- 172.21.15.99:0/2763695922 learned_addr learned my addr 172.21.15.99:0/2763695922 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c0fae640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc06acd0 0x7fd8bc158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8bbfff640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8bc158410 0x7fd8bc14bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c17af640 1 -- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8bc158410 msgr2=0x7fd8bc14bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c17af640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8bc158410 0x7fd8bc14bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c17af640 1 -- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc06acd0 msgr2=0x7fd8bc158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c17af640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc06acd0 0x7fd8bc158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c17af640 1 -- 172.21.15.99:0/2763695922 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0cebb0 con 0x7fd8bc161e70 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c17af640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 0x7fd8bc14c330 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c680 tx=0x7fd8b400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4003b70 con 0x7fd8bc161e70 2024-03-04T10:11:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b4003d10 con 0x7fd8bc161e70 2024-03-04T10:11:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc1464e0 con 0x7fd8bc161e70 2024-03-04T10:11:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4005310 con 0x7fd8bc161e70 2024-03-04T10:11:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.309+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc146aa0 con 0x7fd8bc161e70 2024-03-04T10:11:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.310+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc146ca0 con 0x7fd8bc161e70 2024-03-04T10:11:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.311+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd8b40054b0 con 0x7fd8bc161e70 2024-03-04T10:11:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.311+0000 7fd8b97fa640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd8a806b500 0x7fd8a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.311+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fd8b408a430 con 0x7fd8bc161e70 2024-03-04T10:11:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.312+0000 7fd8c0fae640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd8a806b500 0x7fd8a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.312+0000 7fd8c0fae640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd8a806b500 0x7fd8a806d9c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac036000 tx=0x7fd8ac008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.316+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd8bc146ca0 con 0x7fd8bc161e70 2024-03-04T10:11:41.523 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.521+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8bc074f60 con 0x7fd8bc161e70 2024-03-04T10:11:41.524 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.521+0000 7fd8b97fa640 1 -- 172.21.15.99:0/2763695922 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2576 (secure 0 0 0) 0x7fd8b40586f0 con 0x7fd8bc161e70 2024-03-04T10:11:41.524 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:41.524 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":108,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:39.985556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:41.525 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 108 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd8a806b500 msgr2=0x7fd8a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd8a806b500 0x7fd8a806d9c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac036000 tx=0x7fd8ac008030 comp rx=0 tx=0).stop 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 msgr2=0x7fd8bc14c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 0x7fd8bc14c330 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c680 tx=0x7fd8b400cb40 comp rx=0 tx=0).stop 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 shutdown_connections 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd8a806b500 0x7fd8a806d9c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.526 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd8bc161e70 0x7fd8bc14c330 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd8bc158410 0x7fd8bc14bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 --2- 172.21.15.99:0/2763695922 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd8bc06acd0 0x7fd8bc158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:41.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 >> 172.21.15.99:0/2763695922 conn(0x7fd8bc0cfef0 msgr2=0x7fd8bc0b0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:41.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 shutdown_connections 2024-03-04T10:11:41.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:41.523+0000 7fd8c3212640 1 -- 172.21.15.99:0/2763695922 wait complete. 2024-03-04T10:11:41.536 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:45.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~32s 2024-03-04T10:11:45.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~52s 2024-03-04T10:11:45.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-04T10:11:45.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~19s 2024-03-04T10:11:46.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.648+0000 7f1051b9d640 1 Processor -- start 2024-03-04T10:11:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.648+0000 7f1051b9d640 1 -- start start 2024-03-04T10:11:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1051b9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cce90 0x7f104c0b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1051b9d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f104c15ba10 0x7f104c15e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1051b9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f104c0cd270 0x7f104c15e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1051b9d640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104c158800 con 0x7f104c15ba10 2024-03-04T10:11:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1051b9d640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f104c0ce0b0 con 0x7f104c0cd270 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1051b9d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104c0ce350 con 0x7f104c0cce90 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f104c15ba10 0x7f104c15e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f104c15ba10 0x7f104c15e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39880/0 (socket says 172.21.15.99:39880) 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 -- 172.21.15.99:0/1639388152 learned_addr learned my addr 172.21.15.99:0/1639388152 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f105139c640 1 --2- 172.21.15.99:0/1639388152 >> v2:172.21.15.162:3300/0 conn(0x7f104c0cd270 0x7f104c15e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f1050b9b640 1 --2- 172.21.15.99:0/1639388152 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cce90 0x7f104c0b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 -- 172.21.15.99:0/1639388152 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cce90 msgr2=0x7f104c0b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 --2- 172.21.15.99:0/1639388152 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cce90 0x7f104c0b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 -- 172.21.15.99:0/1639388152 >> v2:172.21.15.162:3300/0 conn(0x7f104c0cd270 msgr2=0x7f104c15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 --2- 172.21.15.99:0/1639388152 >> v2:172.21.15.162:3300/0 conn(0x7f104c0cd270 0x7f104c15e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.649+0000 7f104bfff640 1 -- 172.21.15.99:0/1639388152 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c0ce630 con 0x7f104c15ba10 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.650+0000 7f104bfff640 1 --2- 172.21.15.99:0/1639388152 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c15e2c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f104c070ad0 tx=0x7f103802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2650746c3f13a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.650+0000 7f104b7fe640 1 -- 172.21.15.99:0/1639388152 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038035020 con 0x7f104c15ba10 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.650+0000 7f104b7fe640 1 -- 172.21.15.99:0/1639388152 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1038031870 con 0x7f104c15ba10 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.650+0000 7f104b7fe640 1 -- 172.21.15.99:0/1639388152 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038031c40 con 0x7f104c15ba10 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.650+0000 7f1051b9d640 1 -- 172.21.15.99:0/1639388152 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 msgr2=0x7f104c15e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.650+0000 7f1051b9d640 1 --2- 172.21.15.99:0/1639388152 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c15e2c0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f104c070ad0 tx=0x7f103802b910 comp rx=0 tx=0).stop 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 -- 172.21.15.99:0/1639388152 shutdown_connections 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 --2- 172.21.15.99:0/1639388152 >> v2:172.21.15.162:3300/0 conn(0x7f104c0cd270 0x7f104c15e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 --2- 172.21.15.99:0/1639388152 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c15e2c0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 --2- 172.21.15.99:0/1639388152 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cce90 0x7f104c0b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 -- 172.21.15.99:0/1639388152 >> 172.21.15.99:0/1639388152 conn(0x7f104c099a00 msgr2=0x7f104c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 -- 172.21.15.99:0/1639388152 shutdown_connections 2024-03-04T10:11:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.651+0000 7f1051b9d640 1 -- 172.21.15.99:0/1639388152 wait complete. 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.652+0000 7f1051b9d640 1 Processor -- start 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.652+0000 7f1051b9d640 1 -- start start 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1051b9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f104c0cce90 0x7f104c0b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1051b9d640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cd270 0x7f104c1f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1051b9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1051b9d640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c0b2cd0 con 0x7f104c15ba10 2024-03-04T10:11:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1051b9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c0cebf0 con 0x7f104c0cce90 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1051b9d640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104c0b2e50 con 0x7f104c0cd270 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39888/0 (socket says 172.21.15.99:39888) 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 -- 172.21.15.99:0/3697423285 learned_addr learned my addr 172.21.15.99:0/3697423285 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f1050b9b640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f104c0cce90 0x7f104c0b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f104bfff640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cd270 0x7f104c1f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 -- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cd270 msgr2=0x7f104c1f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cd270 0x7f104c1f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 -- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f104c0cce90 msgr2=0x7f104c0b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f104c0cce90 0x7f104c0b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 -- 172.21.15.99:0/3697423285 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c0c7850 con 0x7f104c15ba10 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.653+0000 7f105139c640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c155330 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1044007e90 tx=0x7f104400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.654+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044014420 con 0x7f104c15ba10 2024-03-04T10:11:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.654+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044019070 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.654+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c0c7b10 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.654+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044014800 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.654+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c0c80d0 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.655+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c04f100 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.655+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f10440149a0 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.655+0000 7f10497fa640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.656+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f104408ad30 con 0x7f104c15ba10 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.656+0000 7f1050b9b640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.656+0000 7f1050b9b640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1040003610 tx=0x7f1040008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:46.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.660+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1044059100 con 0x7f104c15ba10 2024-03-04T10:11:46.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.867+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f104c0cdc70 con 0x7f104c15ba10 2024-03-04T10:11:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.868+0000 7f10497fa640 1 -- 172.21.15.99:0/3697423285 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2576 (secure 0 0 0) 0x7f104405d1a0 con 0x7f104c15ba10 2024-03-04T10:11:46.869 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:46.869 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":108,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:39.985556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:46.870 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 108 2024-03-04T10:11:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 msgr2=0x7f103006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f1040003610 tx=0x7f1040008040 comp rx=0 tx=0).stop 2024-03-04T10:11:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 msgr2=0x7f104c155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c155330 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f1044007e90 tx=0x7f104400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 shutdown_connections 2024-03-04T10:11:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f103006b4d0 0x7f103006d990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f104c15ba10 0x7f104c155330 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f104c0cd270 0x7f104c1f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.869+0000 7f1051b9d640 1 --2- 172.21.15.99:0/3697423285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f104c0cce90 0x7f104c0b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:46.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.870+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 >> 172.21.15.99:0/3697423285 conn(0x7f104c099a00 msgr2=0x7f104c06e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:46.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.870+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 shutdown_connections 2024-03-04T10:11:46.872 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:46.870+0000 7f1051b9d640 1 -- 172.21.15.99:0/3697423285 wait complete. 2024-03-04T10:11:46.882 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:51.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~39s 2024-03-04T10:11:51.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~58s 2024-03-04T10:11:51.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~6s 2024-03-04T10:11:51.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~26s 2024-03-04T10:11:51.884 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:51.992 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.991+0000 7f9a6ce9a640 1 Processor -- start 2024-03-04T10:11:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.991+0000 7f9a6ce9a640 1 -- start start 2024-03-04T10:11:51.994 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6ce9a640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a6815d660 0x7f9a6815e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6ce9a640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f9a680aeb20 0x7f9a68071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6ce9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a68071890 0x7f9a680cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6ce9a640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6815da40 con 0x7f9a6815d660 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6ce9a640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6815ebb0 con 0x7f9a680aeb20 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6ce9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a680af090 con 0x7f9a68071890 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a6815d660 0x7f9a6815e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a6815d660 0x7f9a6815e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39896/0 (socket says 172.21.15.99:39896) 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 -- 172.21.15.99:0/1438379771 learned_addr learned my addr 172.21.15.99:0/1438379771 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a66d9d640 1 --2- 172.21.15.99:0/1438379771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a68071890 0x7f9a680cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a65d9b640 1 --2- 172.21.15.99:0/1438379771 >> v2:172.21.15.162:3300/0 conn(0x7f9a680aeb20 0x7f9a68071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 -- 172.21.15.99:0/1438379771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a68071890 msgr2=0x7f9a680cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 --2- 172.21.15.99:0/1438379771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a68071890 0x7f9a680cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 -- 172.21.15.99:0/1438379771 >> v2:172.21.15.162:3300/0 conn(0x7f9a680aeb20 msgr2=0x7f9a68071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 --2- 172.21.15.99:0/1438379771 >> v2:172.21.15.162:3300/0 conn(0x7f9a680aeb20 0x7f9a68071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.992+0000 7f9a6659c640 1 -- 172.21.15.99:0/1438379771 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6815a9f0 con 0x7f9a6815d660 2024-03-04T10:11:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.993+0000 7f9a6659c640 1 --2- 172.21.15.99:0/1438379771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a6815d660 0x7f9a6815e670 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f9a6806d200 tx=0x7f9a5802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a2c39237223eb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.993+0000 7f9a6559a640 1 -- 172.21.15.99:0/1438379771 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5802c950 con 0x7f9a6815d660 2024-03-04T10:11:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.993+0000 7f9a6559a640 1 -- 172.21.15.99:0/1438379771 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5802caf0 con 0x7f9a6815d660 2024-03-04T10:11:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.993+0000 7f9a6559a640 1 -- 172.21.15.99:0/1438379771 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580343f0 con 0x7f9a6815d660 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1438379771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a6815d660 msgr2=0x7f9a6815e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1438379771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a6815d660 0x7f9a6815e670 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f9a6806d200 tx=0x7f9a5802bdb0 comp rx=0 tx=0).stop 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1438379771 shutdown_connections 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1438379771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a68071890 0x7f9a680cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1438379771 >> v2:172.21.15.162:3300/0 conn(0x7f9a680aeb20 0x7f9a68071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1438379771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a6815d660 0x7f9a6815e670 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1438379771 >> 172.21.15.99:0/1438379771 conn(0x7f9a6806acd0 msgr2=0x7f9a6806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:51.999 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1438379771 shutdown_connections 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.994+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1438379771 wait complete. 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.995+0000 7f9a6ce9a640 1 Processor -- start 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.995+0000 7f9a6ce9a640 1 -- start start 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6ce9a640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 0x7f9a68159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6ce9a640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a680aeb20 0x7f9a680b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6ce9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a6815d660 0x7f9a681520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6ce9a640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6815a660 con 0x7f9a68071890 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6ce9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6815afb0 con 0x7f9a6815d660 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6ce9a640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6815b4d0 con 0x7f9a680aeb20 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 0x7f9a68159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 0x7f9a68159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39906/0 (socket says 172.21.15.99:39906) 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 -- 172.21.15.99:0/1084103887 learned_addr learned my addr 172.21.15.99:0/1084103887 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:52.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a66d9d640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a6815d660 0x7f9a681520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a65d9b640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a680aeb20 0x7f9a680b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 -- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a680aeb20 msgr2=0x7f9a680b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a680aeb20 0x7f9a680b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 -- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a6815d660 msgr2=0x7f9a681520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a6815d660 0x7f9a681520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a6659c640 1 -- 172.21.15.99:0/1084103887 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68152950 con 0x7f9a68071890 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.996+0000 7f9a66d9d640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a6815d660 0x7f9a681520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.997+0000 7f9a6659c640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 0x7f9a68159ce0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f9a58000f80 tx=0x7f9a58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.997+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58043020 con 0x7f9a68071890 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.997+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5803e450 con 0x7f9a68071890 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.997+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a680b3c80 con 0x7f9a68071890 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.997+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58009510 con 0x7f9a68071890 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.997+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a680b4240 con 0x7f9a68071890 2024-03-04T10:11:52.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.998+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a68074f60 con 0x7f9a68071890 2024-03-04T10:11:52.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.998+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9a58046020 con 0x7f9a68071890 2024-03-04T10:11:52.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.998+0000 7f9a56ffd640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4806b4d0 0x7f9a4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:52.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.999+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f9a580ace50 con 0x7f9a68071890 2024-03-04T10:11:52.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.999+0000 7f9a65d9b640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4806b4d0 0x7f9a4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:52.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:51.999+0000 7f9a65d9b640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4806b4d0 0x7f9a4806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006fd0 tx=0x7f9a5c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:52.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.003+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f9a5804d030 con 0x7f9a68071890 2024-03-04T10:11:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.207+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a68072460 con 0x7f9a68071890 2024-03-04T10:11:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.207+0000 7f9a56ffd640 1 -- 172.21.15.99:0/1084103887 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+2576 (secure 0 0 0) 0x7f9a5807b900 con 0x7f9a68071890 2024-03-04T10:11:52.210 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:52.210 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":108,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:39.985556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:52.210 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 108 2024-03-04T10:11:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4806b4d0 msgr2=0x7f9a4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4806b4d0 0x7f9a4806d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c006fd0 tx=0x7f9a5c008040 comp rx=0 tx=0).stop 2024-03-04T10:11:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 msgr2=0x7f9a68159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 0x7f9a68159ce0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f9a58000f80 tx=0x7f9a58004a40 comp rx=0 tx=0).stop 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 shutdown_connections 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4806b4d0 0x7f9a4806d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a6815d660 0x7f9a681520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a680aeb20 0x7f9a680b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 --2- 172.21.15.99:0/1084103887 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a68071890 0x7f9a68159ce0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.209+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 >> 172.21.15.99:0/1084103887 conn(0x7f9a6806acd0 msgr2=0x7f9a68154200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.210+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 shutdown_connections 2024-03-04T10:11:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:52.210+0000 7f9a6ce9a640 1 -- 172.21.15.99:0/1084103887 wait complete. 2024-03-04T10:11:52.222 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-03-04T10:11:57.223 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.333+0000 7f973ba44640 1 Processor -- start 2024-03-04T10:11:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 -- start start 2024-03-04T10:11:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97340b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f97340b9380 0x7f97340b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f97340b70d0 0x7f97340b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97340d2820 con 0x7f97340b70d0 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f973415e1e0 con 0x7f97340b9380 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.334+0000 7f973ba44640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97340d1c10 con 0x7f973415de00 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f97340b70d0 0x7f97340b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f97340b70d0 0x7f97340b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47004/0 (socket says 172.21.15.99:47004) 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 -- 172.21.15.99:0/3996158342 learned_addr learned my addr 172.21.15.99:0/3996158342 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 -- 172.21.15.99:0/3996158342 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 msgr2=0x7f97340b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 --2- 172.21.15.99:0/3996158342 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97340b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 -- 172.21.15.99:0/3996158342 >> v2:172.21.15.162:3300/0 conn(0x7f97340b9380 msgr2=0x7f97340b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 --2- 172.21.15.99:0/3996158342 >> v2:172.21.15.162:3300/0 conn(0x7f97340b9380 0x7f97340b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.335+0000 7f9739fe1640 1 -- 172.21.15.99:0/3996158342 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340d1ef0 con 0x7f97340b70d0 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f9739fe1640 1 --2- 172.21.15.99:0/3996158342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97340b5470 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f97340d4a50 tx=0x7f972402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a3e2adca604f29 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f9733fff640 1 -- 172.21.15.99:0/3996158342 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724035020 con 0x7f97340b70d0 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f9733fff640 1 -- 172.21.15.99:0/3996158342 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9724031870 con 0x7f97340b70d0 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f9733fff640 1 -- 172.21.15.99:0/3996158342 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9724031c40 con 0x7f97340b70d0 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 -- 172.21.15.99:0/3996158342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 msgr2=0x7f97340b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 --2- 172.21.15.99:0/3996158342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97340b5470 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f97340d4a50 tx=0x7f972402b910 comp rx=0 tx=0).stop 2024-03-04T10:11:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 -- 172.21.15.99:0/3996158342 shutdown_connections 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 --2- 172.21.15.99:0/3996158342 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97340b5470 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 --2- 172.21.15.99:0/3996158342 >> v2:172.21.15.162:3300/0 conn(0x7f97340b9380 0x7f97340b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 --2- 172.21.15.99:0/3996158342 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97340b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 -- 172.21.15.99:0/3996158342 >> 172.21.15.99:0/3996158342 conn(0x7f97340d0470 msgr2=0x7f97340d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.336+0000 7f973ba44640 1 -- 172.21.15.99:0/3996158342 shutdown_connections 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.337+0000 7f973ba44640 1 -- 172.21.15.99:0/3996158342 wait complete. 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 Processor -- start 2024-03-04T10:11:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 -- start start 2024-03-04T10:11:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97341574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97340b9380 0x7f97341f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97341f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97340b97e0 con 0x7f97340b70d0 2024-03-04T10:11:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97340d24b0 con 0x7f97340b9380 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f973ba44640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97340b59b0 con 0x7f973415de00 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f97397e0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97341574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f97397e0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97341574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47012/0 (socket says 172.21.15.99:47012) 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.338+0000 7f97397e0640 1 -- 172.21.15.99:0/510287723 learned_addr learned my addr 172.21.15.99:0/510287723 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f9739fe1640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97341f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f97397e0640 1 -- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 msgr2=0x7f97341f6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f97397e0640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97341f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f9738fdf640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97340b9380 0x7f97341f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f97397e0640 1 -- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97340b9380 msgr2=0x7f97341f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f97397e0640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97340b9380 0x7f97341f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f97397e0640 1 -- 172.21.15.99:0/510287723 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97341592a0 con 0x7f97340b70d0 2024-03-04T10:11:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f9738fdf640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97340b9380 0x7f97341f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f97397e0640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97341574b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f972c00ba80 tx=0x7f972c00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c019070 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c00cc20 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9734159580 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c014440 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.339+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9734159b40 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.340+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9734074f60 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.341+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f972c01c070 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.341+0000 7f9731ffb640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f972006b4d0 0x7f972006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.341+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f972c08a270 con 0x7f97340b70d0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.341+0000 7f9738fdf640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f972006b4d0 0x7f972006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.342+0000 7f9738fdf640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f972006b4d0 0x7f972006d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9728006fd0 tx=0x7f9728008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:57.347 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.346+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f972c0586d0 con 0x7f97340b70d0 2024-03-04T10:11:57.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.551+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9734159f10 con 0x7f97340b70d0 2024-03-04T10:11:57.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.551+0000 7f9731ffb640 1 -- 172.21.15.99:0/510287723 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+2624 (secure 0 0 0) 0x7f972c05c770 con 0x7f97340b70d0 2024-03-04T10:11:57.553 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:57.553 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":109,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:54.988200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:11:54.988200+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:57.553 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 109 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f972006b4d0 msgr2=0x7f972006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f972006b4d0 0x7f972006d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9728006fd0 tx=0x7f9728008040 comp rx=0 tx=0).stop 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 msgr2=0x7f97341574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97341574b0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f972c00ba80 tx=0x7f972c00bf40 comp rx=0 tx=0).stop 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 shutdown_connections 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f972006b4d0 0x7f972006d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f973415de00 0x7f97341f6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97340b9380 0x7f97341f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 --2- 172.21.15.99:0/510287723 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f97340b70d0 0x7f97341574b0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 >> 172.21.15.99:0/510287723 conn(0x7f97340d0470 msgr2=0x7f973415ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 shutdown_connections 2024-03-04T10:11:57.556 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.553+0000 7f973ba44640 1 -- 172.21.15.99:0/510287723 wait complete. 2024-03-04T10:11:57.566 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c2c20640 1 Processor -- start 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c2c20640 1 -- start start 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c2c20640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 0x7f07bc0b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c2c20640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07bc15e1e0 con 0x7f07bc15de00 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c2c20640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07bc0d1c10 con 0x7f07bc0b5470 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c2c20640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0b98c0 con 0x7f07bc0b2e60 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c09bc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f07bc15de00 0x7f07bc0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42572/0 (socket says 172.21.15.99:42572) 2024-03-04T10:11:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.665+0000 7f07c09bc640 1 -- 172.21.15.99:0/2072790507 learned_addr learned my addr 172.21.15.99:0/2072790507 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027407728 0 0) 0x7f07bc15e1e0 con 0x7f07bc15de00 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ac003520 con 0x7f07bc15de00 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bbfff640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 0x7f07bc0b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2806936934 0 0) 0x7f07ac003520 con 0x7f07bc15de00 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 msgr2=0x7f07bc0b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 0x7f07bc0b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 >> v1:172.21.15.162:6789/0 conn(0x7f07bc0b5470 legacy=0x7f07bc0d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc0d1ef0 con 0x7f07bc15de00 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f07b0001670 con 0x7f07bc15de00 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 >> v1:172.21.15.99:6789/0 conn(0x7f07bc15de00 legacy=0x7f07bc0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07ac0039e0 0x7f07ac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 0x7f07ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac004d40 0x7f07ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0b98c0 con 0x7f07ac0039e0 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac00bb50 con 0x7f07ac004d40 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac00bd20 con 0x7f07bc0b2e60 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07ac0039e0 0x7f07ac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07bbfff640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 0x7f07ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c11bd640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac004d40 0x7f07ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c09bc640 1 -- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 msgr2=0x7f07ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b2e60 0x7f07ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c09bc640 1 -- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac004d40 msgr2=0x7f07ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac004d40 0x7f07ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.666+0000 7f07c09bc640 1 -- 172.21.15.99:0/2072790507 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac003520 con 0x7f07ac0039e0 2024-03-04T10:11:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.667+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07ac0039e0 0x7f07ac003e00 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f07b000c270 tx=0x7f07ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c052b677eb4b3ace server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.667+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0007cf0 con 0x7f07ac0039e0 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.667+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0030070 con 0x7f07ac0039e0 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.667+0000 7f07bb7fe640 1 -- 172.21.15.99:0/2072790507 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0005d30 con 0x7f07ac0039e0 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c2c20640 1 -- 172.21.15.99:0/2072790507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07ac0039e0 msgr2=0x7f07ac003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2072790507 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07ac0039e0 0x7f07ac003e00 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f07b000c270 tx=0x7f07ac005830 comp rx=0 tx=0).stop 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c11bd640 1 -- 172.21.15.99:0/2072790507 reap_dead start 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c2c20640 1 -- 172.21.15.99:0/2072790507 shutdown_connections 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c2c20640 1 -- 172.21.15.99:0/2072790507 >> 172.21.15.99:0/2072790507 conn(0x7f07bc0d0470 msgr2=0x7f07bc0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c2c20640 1 -- 172.21.15.99:0/2072790507 shutdown_connections 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.668+0000 7f07c2c20640 1 -- 172.21.15.99:0/2072790507 wait complete. 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.669+0000 7f07c2c20640 1 Processor -- start 2024-03-04T10:11:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.669+0000 7f07c2c20640 1 -- start start 2024-03-04T10:11:57.677 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c2c20640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac0039e0 0x7f07bc159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c2c20640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 0x7f07bc14dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c2c20640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b3a10 0x7f07bc0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c2c20640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0d2820 con 0x7f07bc158510 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c2c20640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0d24b0 con 0x7f07ac0039e0 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c2c20640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc0b72a0 con 0x7f07bc0b3a10 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 0x7f07bc14dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 0x7f07bc14dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47034/0 (socket says 172.21.15.99:47034) 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 -- 172.21.15.99:0/2325778771 learned_addr learned my addr 172.21.15.99:0/2325778771 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac0039e0 0x7f07bc159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07c11bd640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b3a10 0x7f07bc0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 -- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b3a10 msgr2=0x7f07bc0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b3a10 0x7f07bc0b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 -- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac0039e0 msgr2=0x7f07bc159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac0039e0 0x7f07bc159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 -- 172.21.15.99:0/2325778771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc15a430 con 0x7f07bc158510 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07bbfff640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 0x7f07bc14dfc0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b7f0 tx=0x7f07a800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.670+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8018070 con 0x7f07bc158510 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.671+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a8010830 con 0x7f07bc158510 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.671+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc15a6d0 con 0x7f07bc158510 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.671+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8010c00 con 0x7f07bc158510 2024-03-04T10:11:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.671+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc151f20 con 0x7f07bc158510 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.671+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc074f60 con 0x7f07bc158510 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.672+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f07a801b070 con 0x7f07bc158510 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.672+0000 7f07b97fa640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f07a006b500 0x7f07a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.672+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f07a808a8b0 con 0x7f07bc158510 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.673+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f07a006b500 0x7f07a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.673+0000 7f07c09bc640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f07a006b500 0x7f07a006d9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f07b0034000 tx=0x7f07b0007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.677+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f07a8059360 con 0x7f07bc158510 2024-03-04T10:11:57.884 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.883+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07bc0d09f0 con 0x7f07bc158510 2024-03-04T10:11:57.885 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.883+0000 7f07b97fa640 1 -- 172.21.15.99:0/2325778771 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+2624 (secure 0 0 0) 0x7f07a8058c60 con 0x7f07bc158510 2024-03-04T10:11:57.885 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:57.885 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":109,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:54.988200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:11:54.988200+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:57.886 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 109 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f07a006b500 msgr2=0x7f07a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f07a006b500 0x7f07a006d9c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f07b0034000 tx=0x7f07b0007000 comp rx=0 tx=0).stop 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 msgr2=0x7f07bc14dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 0x7f07bc14dfc0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b7f0 tx=0x7f07a800bcb0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 shutdown_connections 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f07a006b500 0x7f07a006d9c0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f07bc0b3a10 0x7f07bc0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f07bc158510 0x7f07bc14dfc0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.887 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 --2- 172.21.15.99:0/2325778771 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f07ac0039e0 0x7f07bc159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:57.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 >> 172.21.15.99:0/2325778771 conn(0x7f07bc0d0470 msgr2=0x7f07bc0b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:57.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 shutdown_connections 2024-03-04T10:11:57.888 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:57.885+0000 7f07c2c20640 1 -- 172.21.15.99:0/2325778771 wait complete. 2024-03-04T10:11:57.898 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-04T10:11:57.898 DEBUG:teuthology.orchestra.run.smithi099:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:11:57.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-04T10:11:57.999 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-04T10:11:57.999 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:11:57.999 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:11:58.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:11:58.019 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:11:58.019 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:11:58.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:11:58.038 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:11:58.038 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:11:58.074 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:11:58.074 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:11:58.074 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:11:58.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~45s 2024-03-04T10:11:58.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~65s 2024-03-04T10:11:58.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~13s 2024-03-04T10:11:58.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~32s 2024-03-04T10:11:58.094 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:11:58.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:58.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.214+0000 7f6b49f80640 1 Processor -- start 2024-03-04T10:11:58.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.214+0000 7f6b49f80640 1 -- start start 2024-03-04T10:11:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.214+0000 7f6b49f80640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6b440cbf80 0x7f6b440cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.214+0000 7f6b49f80640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b440cfb40 con 0x7f6b440d1ca0 2024-03-04T10:11:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.214+0000 7f6b49f80640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b440cfe20 con 0x7f6b4415bbb0 2024-03-04T10:11:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.214+0000 7f6b49f80640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b440d2880 con 0x7f6b440cbf80 2024-03-04T10:11:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b42ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6b440cbf80 0x7f6b440cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b43fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f6b440d1ca0 0x7f6b440b4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42588/0 (socket says 172.21.15.99:42588) 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b42ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6b440cbf80 0x7f6b440cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47044/0 (socket says 172.21.15.99:47044) 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b43fff640 1 -- 172.21.15.99:0/1926555654 learned_addr learned my addr 172.21.15.99:0/1926555654 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b42ffd640 1 -- 172.21.15.99:0/1926555654 >> v1:172.21.15.162:6789/0 conn(0x7f6b4415bbb0 legacy=0x7f6b440d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b42ffd640 1 -- 172.21.15.99:0/1926555654 >> v1:172.21.15.99:6789/0 conn(0x7f6b440d1ca0 legacy=0x7f6b440b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b427fc640 1 -- 172.21.15.99:0/1926555654 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626569821 0 0) 0x7f6b440cfb40 con 0x7f6b440d1ca0 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.215+0000 7f6b42ffd640 1 -- 172.21.15.99:0/1926555654 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440d0100 con 0x7f6b440cbf80 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b42ffd640 1 --2- 172.21.15.99:0/1926555654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b440cbf80 0x7f6b440cc3c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f6b380097a0 tx=0x7f6b3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562cc70bd2ce0c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b427fc640 1 -- 172.21.15.99:0/1926555654 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b38035020 con 0x7f6b440cbf80 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b427fc640 1 -- 172.21.15.99:0/1926555654 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b38031830 con 0x7f6b440cbf80 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b427fc640 1 -- 172.21.15.99:0/1926555654 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b38031c00 con 0x7f6b440cbf80 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 -- 172.21.15.99:0/1926555654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b440cbf80 msgr2=0x7f6b440cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 --2- 172.21.15.99:0/1926555654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b440cbf80 0x7f6b440cc3c0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f6b380097a0 tx=0x7f6b3802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 -- 172.21.15.99:0/1926555654 shutdown_connections 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 --2- 172.21.15.99:0/1926555654 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b440cbf80 0x7f6b440cc3c0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 -- 172.21.15.99:0/1926555654 >> 172.21.15.99:0/1926555654 conn(0x7f6b440b3480 msgr2=0x7f6b440b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 -- 172.21.15.99:0/1926555654 shutdown_connections 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.216+0000 7f6b49f80640 1 -- 172.21.15.99:0/1926555654 wait complete. 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 Processor -- start 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 -- start start 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b440cbf80 0x7f6b44154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b440d1ca0 0x7f6b4415df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 0x7f6b441f1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440d1010 con 0x7f6b4415bbb0 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440cc900 con 0x7f6b440d1ca0 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b49f80640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4415bf90 con 0x7f6b440cbf80 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.218+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 0x7f6b441f1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 0x7f6b441f1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47048/0 (socket says 172.21.15.99:47048) 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 -- 172.21.15.99:0/4229700337 learned_addr learned my addr 172.21.15.99:0/4229700337 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b437fe640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b440cbf80 0x7f6b44154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 -- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b440cbf80 msgr2=0x7f6b44154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b440cbf80 0x7f6b44154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 -- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b440d1ca0 msgr2=0x7f6b4415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b440d1ca0 0x7f6b4415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 -- 172.21.15.99:0/4229700337 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b441574d0 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b43fff640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 0x7f6b441f1740 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00c7a0 tx=0x7f6b3c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c018070 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c00a640 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b44156150 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00aab0 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.219+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b44156710 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.220+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b44074f60 con 0x7f6b4415bbb0 2024-03-04T10:11:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.221+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6b3c01b070 con 0x7f6b4415bbb0 2024-03-04T10:11:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.221+0000 7f6b2bfff640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6b3006b4d0 0x7f6b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.221+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f6b3c089bf0 con 0x7f6b4415bbb0 2024-03-04T10:11:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.221+0000 7f6b437fe640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6b3006b4d0 0x7f6b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.222+0000 7f6b437fe640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6b3006b4d0 0x7f6b3006d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f6b4415e780 tx=0x7f6b34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.225+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f6b3c0586a0 con 0x7f6b4415bbb0 2024-03-04T10:11:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.431+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b440b2a60 con 0x7f6b4415bbb0 2024-03-04T10:11:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.431+0000 7f6b2bfff640 1 -- 172.21.15.99:0/4229700337 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+2624 (secure 0 0 0) 0x7f6b3c057fa0 con 0x7f6b4415bbb0 2024-03-04T10:11:58.433 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:58.433 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":109,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:54.988200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:11:54.988200+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 109 2024-03-04T10:11:58.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6b3006b4d0 msgr2=0x7f6b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6b3006b4d0 0x7f6b3006d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f6b4415e780 tx=0x7f6b34009040 comp rx=0 tx=0).stop 2024-03-04T10:11:58.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 msgr2=0x7f6b441f1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.435 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 0x7f6b441f1740 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00c7a0 tx=0x7f6b3c00cc60 comp rx=0 tx=0).stop 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 shutdown_connections 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f6b3006b4d0 0x7f6b3006d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6b4415bbb0 0x7f6b441f1740 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b440d1ca0 0x7f6b4415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 --2- 172.21.15.99:0/4229700337 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f6b440cbf80 0x7f6b44154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.433+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 >> 172.21.15.99:0/4229700337 conn(0x7f6b440b3480 msgr2=0x7f6b440b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.434+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 shutdown_connections 2024-03-04T10:11:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.434+0000 7f6b49f80640 1 -- 172.21.15.99:0/4229700337 wait complete. 2024-03-04T10:11:58.447 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:11:58.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.545+0000 7f38b1eea640 1 Processor -- start 2024-03-04T10:11:58.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.545+0000 7f38b1eea640 1 -- start start 2024-03-04T10:11:58.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38b1eea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac15de00 0x7f38ac0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38b1eea640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f38ac0b5470 0x7f38ac0b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38b1eea640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ac15e1e0 con 0x7f38ac0b9380 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38b1eea640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ac0d2820 con 0x7f38ac0b5470 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38b1eea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0d1c10 con 0x7f38ac15de00 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38aaffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f38ac0b9380 0x7f38ac0b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42604/0 (socket says 172.21.15.99:42604) 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38aaffd640 1 -- 172.21.15.99:0/2582995826 learned_addr learned my addr 172.21.15.99:0/2582995826 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38abfff640 1 --2- 172.21.15.99:0/2582995826 >> v2:172.21.15.162:3300/0 conn(0x7f38ac0b5470 0x7f38ac0b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.546+0000 7f38ab7fe640 1 --2- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac15de00 0x7f38ac0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.547+0000 7f38aa7fc640 1 -- 172.21.15.99:0/2582995826 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730796017 0 0) 0x7f38ac15e1e0 con 0x7f38ac0b9380 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.547+0000 7f38aa7fc640 1 -- 172.21.15.99:0/2582995826 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3894003520 con 0x7f38ac0b9380 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.547+0000 7f38abfff640 1 -- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac15de00 msgr2=0x7f38ac0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.547+0000 7f38abfff640 1 --2- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac15de00 0x7f38ac0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.547+0000 7f38abfff640 1 -- 172.21.15.99:0/2582995826 >> v1:172.21.15.99:6789/0 conn(0x7f38ac0b9380 legacy=0x7f38ac0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.547+0000 7f38abfff640 1 -- 172.21.15.99:0/2582995826 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac0d1ef0 con 0x7f38ac0b5470 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38abfff640 1 --2- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b5470 0x7f38ac0b5900 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f38a40097a0 tx=0x7f38a402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b7cf693244e880d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38aa7fc640 1 -- 172.21.15.99:0/2582995826 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4035020 con 0x7f38ac0b5470 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38aa7fc640 1 -- 172.21.15.99:0/2582995826 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a4031830 con 0x7f38ac0b5470 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38aa7fc640 1 -- 172.21.15.99:0/2582995826 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4031c00 con 0x7f38ac0b5470 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 -- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b5470 msgr2=0x7f38ac0b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b5470 0x7f38ac0b5900 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f38a40097a0 tx=0x7f38a402b8d0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 -- 172.21.15.99:0/2582995826 shutdown_connections 2024-03-04T10:11:58.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b5470 0x7f38ac0b5900 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2582995826 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac15de00 0x7f38ac0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 -- 172.21.15.99:0/2582995826 >> 172.21.15.99:0/2582995826 conn(0x7f38ac0d0470 msgr2=0x7f38ac0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:58.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 -- 172.21.15.99:0/2582995826 shutdown_connections 2024-03-04T10:11:58.553 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.548+0000 7f38b1eea640 1 -- 172.21.15.99:0/2582995826 wait complete. 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 Processor -- start 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 -- start start 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac0b5470 0x7f38ac1f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b9380 0x7f38ac159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 0x7f38ac160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0b97e0 con 0x7f38ac15de00 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0d24b0 con 0x7f38ac0b9380 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.550+0000 7f38b1eea640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ac0b72a0 con 0x7f38ac0b5470 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 0x7f38ac160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 0x7f38ac160900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47062/0 (socket says 172.21.15.99:47062) 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 -- 172.21.15.99:0/2241798401 learned_addr learned my addr 172.21.15.99:0/2241798401 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38aaffd640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b9380 0x7f38ac159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38ab7fe640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac0b5470 0x7f38ac1f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 -- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac0b5470 msgr2=0x7f38ac1f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac0b5470 0x7f38ac1f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 -- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b9380 msgr2=0x7f38ac159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b9380 0x7f38ac159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 -- 172.21.15.99:0/2241798401 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac1588f0 con 0x7f38ac15de00 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38ab7fe640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac0b5470 0x7f38ac1f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38aaffd640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b9380 0x7f38ac159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38abfff640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 0x7f38ac160900 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f38a40097a0 tx=0x7f38a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4035040 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a403b840 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ac158bd0 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a403bc30 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.551+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ac159190 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.552+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ac159960 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.553+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f38a4038020 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.553+0000 7f3893fff640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f389806b500 0x7f389806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.553+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f38a40ac6b0 con 0x7f38ac15de00 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.553+0000 7f38ab7fe640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f389806b500 0x7f389806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.559 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.554+0000 7f38ab7fe640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f389806b500 0x7f389806d9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f389c007440 tx=0x7f389c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.557+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f38a407b0d0 con 0x7f38ac15de00 2024-03-04T10:11:58.730 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.729+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38ac074f60 con 0x7f38ac15de00 2024-03-04T10:11:58.731 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.730+0000 7f3893fff640 1 -- 172.21.15.99:0/2241798401 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+17945 (secure 0 0 0) 0x7f38a4039e60 con 0x7f38ac15de00 2024-03-04T10:11:58.732 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:58.732 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":102,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:11:39.985592+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.31","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6837/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6835/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6832/584503338":"2024-03-05T10:11:24.982726+0000","172.21.15.162:6836/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6835/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6834/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6835/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6834/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6835/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6834/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6835/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6833/584503338":"2024-03-05T10:11:24.982726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f389806b500 msgr2=0x7f389806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f389806b500 0x7f389806d9c0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f389c007440 tx=0x7f389c00b040 comp rx=0 tx=0).stop 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 msgr2=0x7f38ac160900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 0x7f38ac160900 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f38a40097a0 tx=0x7f38a40046c0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 shutdown_connections 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f389806b500 0x7f389806d9c0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f38ac15de00 0x7f38ac160900 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38ac0b9380 0x7f38ac159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 --2- 172.21.15.99:0/2241798401 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f38ac0b5470 0x7f38ac1f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 >> 172.21.15.99:0/2241798401 conn(0x7f38ac0d0470 msgr2=0x7f38ac06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:58.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 shutdown_connections 2024-03-04T10:11:58.735 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.732+0000 7f38b1eea640 1 -- 172.21.15.99:0/2241798401 wait complete. 2024-03-04T10:11:58.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T10:03:27.478194+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T10:10:05.411440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T10:10:06.166240+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T10:11:58.746 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T10:11:58.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.847+0000 7f8f0b769640 1 Processor -- start 2024-03-04T10:11:58.849 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.847+0000 7f8f0b769640 1 -- start start 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f0b769640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f04071320 0x7f8f04071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f0b769640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8f0406d3c0 0x7f8f041508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f0b769640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8f0406d7e0 0x7f8f04150df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f0b769640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f04071c40 con 0x7f8f0406d7e0 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f0b769640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0406dd10 con 0x7f8f0406d3c0 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f0b769640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0406dff0 con 0x7f8f04071320 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8f0406d7e0 0x7f8f04150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8f0406d7e0 0x7f8f04150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47072/0 (socket says 172.21.15.99:47072) 2024-03-04T10:11:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 -- 172.21.15.99:0/1231611828 learned_addr learned my addr 172.21.15.99:0/1231611828 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f08d04640 1 --2- 172.21.15.99:0/1231611828 >> v2:172.21.15.162:3300/0 conn(0x7f8f0406d3c0 0x7f8f041508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 -- 172.21.15.99:0/1231611828 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f04071320 msgr2=0x7f8f04071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09505640 1 --2- 172.21.15.99:0/1231611828 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f04071320 0x7f8f04071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 --2- 172.21.15.99:0/1231611828 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f04071320 0x7f8f04071700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 -- 172.21.15.99:0/1231611828 >> v2:172.21.15.162:3300/0 conn(0x7f8f0406d3c0 msgr2=0x7f8f041508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.848+0000 7f8f09d06640 1 --2- 172.21.15.99:0/1231611828 >> v2:172.21.15.162:3300/0 conn(0x7f8f0406d3c0 0x7f8f041508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.849+0000 7f8f09d06640 1 -- 172.21.15.99:0/1231611828 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0406e2d0 con 0x7f8f0406d7e0 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.849+0000 7f8f09d06640 1 --2- 172.21.15.99:0/1231611828 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f0406d7e0 0x7f8f04150df0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f8f04071e20 tx=0x7f8efc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b2ef941852a3ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.849+0000 7f8efbfff640 1 -- 172.21.15.99:0/1231611828 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc035020 con 0x7f8f0406d7e0 2024-03-04T10:11:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.849+0000 7f8efbfff640 1 -- 172.21.15.99:0/1231611828 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc031640 con 0x7f8f0406d7e0 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.849+0000 7f8efbfff640 1 -- 172.21.15.99:0/1231611828 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc02d040 con 0x7f8f0406d7e0 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 -- 172.21.15.99:0/1231611828 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f0406d7e0 msgr2=0x7f8f04150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1231611828 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f0406d7e0 0x7f8f04150df0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f8f04071e20 tx=0x7f8efc02b8f0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 -- 172.21.15.99:0/1231611828 shutdown_connections 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1231611828 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f0406d7e0 0x7f8f04150df0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1231611828 >> v2:172.21.15.162:3300/0 conn(0x7f8f0406d3c0 0x7f8f041508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1231611828 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f04071320 0x7f8f04071700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 -- 172.21.15.99:0/1231611828 >> 172.21.15.99:0/1231611828 conn(0x7f8f0406acd0 msgr2=0x7f8f0406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 -- 172.21.15.99:0/1231611828 shutdown_connections 2024-03-04T10:11:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.850+0000 7f8f0b769640 1 -- 172.21.15.99:0/1231611828 wait complete. 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.851+0000 7f8f0b769640 1 Processor -- start 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.851+0000 7f8f0b769640 1 -- start start 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f0b769640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f0406d3c0 0x7f8f04153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f0b769640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f0406d7e0 0x7f8f0415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f0b769640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 0x7f8f04154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f0b769640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040b3e50 con 0x7f8f04071320 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f0b769640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04151330 con 0x7f8f0406d7e0 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f0b769640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040b3fd0 con 0x7f8f0406d3c0 2024-03-04T10:11:58.858 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 0x7f8f04154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 0x7f8f04154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47082/0 (socket says 172.21.15.99:47082) 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 -- 172.21.15.99:0/1725493030 learned_addr learned my addr 172.21.15.99:0/1725493030 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f08d04640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f0406d7e0 0x7f8f0415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09505640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f0406d3c0 0x7f8f04153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 -- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f0406d3c0 msgr2=0x7f8f04153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f0406d3c0 0x7f8f04153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 -- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f0406d7e0 msgr2=0x7f8f0415a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f0406d7e0 0x7f8f0415a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 -- 172.21.15.99:0/1725493030 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0415a840 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f08d04640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f0406d7e0 0x7f8f0415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.852+0000 7f8f09d06640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 0x7f8f04154bc0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f8efc02b930 tx=0x7f8efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.853+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc040070 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.853+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc039c10 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.853+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f04155610 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.853+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc035040 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.853+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f041565d0 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.854+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f04155b30 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.854+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8efc038020 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.855+0000 7f8ef9ffb640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ee806b4d0 0x7f8ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.855+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f8efc0ac1c0 con 0x7f8f04071320 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.855+0000 7f8f09505640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ee806b4d0 0x7f8ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.855+0000 7f8f09505640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ee806b4d0 0x7f8ee806d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8f00007130 tx=0x7f8f00007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:58.861 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:58.859+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8efc07abe0 con 0x7f8f04071320 2024-03-04T10:11:59.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.032+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f04074f60 con 0x7f8f04071320 2024-03-04T10:11:59.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.032+0000 7f8ef9ffb640 1 -- 172.21.15.99:0/1725493030 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8efc07a4e0 con 0x7f8f04071320 2024-03-04T10:11:59.034 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 1 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ee806b4d0 msgr2=0x7f8ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ee806b4d0 0x7f8ee806d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f8f00007130 tx=0x7f8f00007040 comp rx=0 tx=0).stop 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 msgr2=0x7f8f04154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 0x7f8f04154bc0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f8efc02b930 tx=0x7f8efc0046c0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 shutdown_connections 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8ee806b4d0 0x7f8ee806d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8f04071320 0x7f8f04154bc0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8f0406d7e0 0x7f8f0415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 --2- 172.21.15.99:0/1725493030 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8f0406d3c0 0x7f8f04153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 >> 172.21.15.99:0/1725493030 conn(0x7f8f0406acd0 msgr2=0x7f8f04150790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 shutdown_connections 2024-03-04T10:11:59.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.034+0000 7f8f0b769640 1 -- 172.21.15.99:0/1725493030 wait complete. 2024-03-04T10:11:59.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T10:11:59.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.148+0000 7f8d121e6640 1 Processor -- start 2024-03-04T10:11:59.159 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.148+0000 7f8d121e6640 1 -- start start 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d121e6640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8d0c0cdd70 0x7f8d0c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d121e6640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c06ae00 con 0x7f8d0c0ccfe0 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d121e6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c06b0e0 con 0x7f8d0c0cc900 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d121e6640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d0c0ccde0 con 0x7f8d0c0cdd70 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d0bfff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f8d0c0ccfe0 0x7f8d0c15db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42606/0 (socket says 172.21.15.99:42606) 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d0bfff640 1 -- 172.21.15.99:0/2773981939 learned_addr learned my addr 172.21.15.99:0/2773981939 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d0b7fe640 1 --2- 172.21.15.99:0/2773981939 >> v2:172.21.15.162:3300/0 conn(0x7f8d0c0cdd70 0x7f8d0c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249917829 0 0) 0x7f8d0c06ae00 con 0x7f8d0c0ccfe0 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.149+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf4003520 con 0x7f8d0c0ccfe0 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452776289 0 0) 0x7f8cf4003520 con 0x7f8d0c0ccfe0 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 >> v2:172.21.15.162:3300/0 conn(0x7f8d0c0cdd70 msgr2=0x7f8d0c0cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 --2- 172.21.15.99:0/2773981939 >> v2:172.21.15.162:3300/0 conn(0x7f8d0c0cdd70 0x7f8d0c0cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 >> v1:172.21.15.162:6789/0 conn(0x7f8d0c0cc900 legacy=0x7f8d0c15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c06b3c0 con 0x7f8d0c0ccfe0 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f8d040019f0 con 0x7f8d0c0ccfe0 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 >> v1:172.21.15.99:6789/0 conn(0x7f8d0c0ccfe0 legacy=0x7f8d0c15db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf40039b0 0x7f8cf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf4004800 0x7f8cf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8cf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0ccde0 con 0x7f8cf40051c0 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf400c160 con 0x7f8cf40039b0 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf400c360 con 0x7f8cf4004800 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0bfff640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8cf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0b7fe640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf40039b0 0x7f8cf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0affd640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf4004800 0x7f8cf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0bfff640 1 -- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf4004800 msgr2=0x7f8cf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0bfff640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8cf4004800 0x7f8cf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0bfff640 1 -- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf40039b0 msgr2=0x7f8cf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0bfff640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8cf40039b0 0x7f8cf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.150+0000 7f8d0bfff640 1 -- 172.21.15.99:0/2773981939 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf4003520 con 0x7f8cf40051c0 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d0bfff640 1 -- 172.21.15.99:0/2773981939 reap_dead start 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d0bfff640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8cf400b9c0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400a850 tx=0x7f8cf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c8a89df04a511d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0400a4a0 con 0x7f8cf40051c0 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04034650 con 0x7f8cf40051c0 2024-03-04T10:11:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d0a7fc640 1 -- 172.21.15.99:0/2773981939 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04034a40 con 0x7f8cf40051c0 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d121e6640 1 -- 172.21.15.99:0/2773981939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 msgr2=0x7f8cf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.151+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8cf400b9c0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400a850 tx=0x7f8cf4005830 comp rx=0 tx=0).stop 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.152+0000 7f8d121e6640 1 -- 172.21.15.99:0/2773981939 shutdown_connections 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.152+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2773981939 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8cf400b9c0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.152+0000 7f8d121e6640 1 -- 172.21.15.99:0/2773981939 >> 172.21.15.99:0/2773981939 conn(0x7f8d0c0b22e0 msgr2=0x7f8d0c0b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.152+0000 7f8d121e6640 1 -- 172.21.15.99:0/2773981939 shutdown_connections 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.152+0000 7f8d121e6640 1 -- 172.21.15.99:0/2773981939 wait complete. 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.153+0000 7f8d121e6640 1 Processor -- start 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.153+0000 7f8d121e6640 1 -- start start 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8d0c159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c0c68a0 0x7f8d0c0c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d0c0c5ae0 0x7f8d0c0c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0b3010 con 0x7f8cf40051c0 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0b2e90 con 0x7f8d0c0c68a0 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c0af950 con 0x7f8d0c0c5ae0 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8d0c159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8d0c159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47106/0 (socket says 172.21.15.99:47106) 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 -- 172.21.15.99:0/2591471693 learned_addr learned my addr 172.21.15.99:0/2591471693 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0bfff640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d0c0c5ae0 0x7f8d0c0c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0affd640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c0c68a0 0x7f8d0c0c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 -- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d0c0c5ae0 msgr2=0x7f8d0c0c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d0c0c5ae0 0x7f8d0c0c5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 -- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c0c68a0 msgr2=0x7f8d0c0c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c0c68a0 0x7f8d0c0c5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 -- 172.21.15.99:0/2591471693 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0c6e60 con 0x7f8cf40051c0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d0b7fe640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8d0c159cc0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0118e0 tx=0x7f8cfc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc00faf0 con 0x7f8cf40051c0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc00fc90 con 0x7f8cf40051c0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c0c7140 con 0x7f8cf40051c0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc014440 con 0x7f8cf40051c0 2024-03-04T10:11:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.154+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c0b47a0 con 0x7f8cf40051c0 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.155+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c074f60 con 0x7f8cf40051c0 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.156+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8cfc012040 con 0x7f8cf40051c0 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.156+0000 7f8cf3fff640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cf806b4d0 0x7f8cf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.156+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f8cfc08ada0 con 0x7f8cf40051c0 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.156+0000 7f8d0affd640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cf806b4d0 0x7f8cf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.157+0000 7f8d0affd640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cf806b4d0 0x7f8cf806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c15fae0 tx=0x7f8d00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.160+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8cfc019030 con 0x7f8cf40051c0 2024-03-04T10:11:59.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.329+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d0c0c7310 con 0x7f8cf40051c0 2024-03-04T10:11:59.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.330+0000 7f8cf3fff640 1 -- 172.21.15.99:0/2591471693 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8cfc059200 con 0x7f8cf40051c0 2024-03-04T10:11:59.331 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 32 2024-03-04T10:11:59.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cf806b4d0 msgr2=0x7f8cf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cf806b4d0 0x7f8cf806d990 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c15fae0 tx=0x7f8d00009040 comp rx=0 tx=0).stop 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 msgr2=0x7f8d0c159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8d0c159cc0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc0118e0 tx=0x7f8cfc011da0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 shutdown_connections 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8cf806b4d0 0x7f8cf806d990 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8d0c0c5ae0 0x7f8d0c0c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.334 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8d0c0c68a0 0x7f8d0c0c5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 --2- 172.21.15.99:0/2591471693 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8cf40051c0 0x7f8d0c159cc0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.331+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 >> 172.21.15.99:0/2591471693 conn(0x7f8d0c0b22e0 msgr2=0x7f8d0c06f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.332+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 shutdown_connections 2024-03-04T10:11:59.335 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.332+0000 7f8d121e6640 1 -- 172.21.15.99:0/2591471693 wait complete. 2024-03-04T10:11:59.344 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T10:11:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.444+0000 7fbf18ce9640 1 Processor -- start 2024-03-04T10:11:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.444+0000 7fbf18ce9640 1 -- start start 2024-03-04T10:11:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf18ce9640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbf140d34a0 0x7fbf14161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf18ce9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf14162120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf18ce9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf140d2400 con 0x7fbf140b5700 2024-03-04T10:11:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf18ce9640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf140d26e0 con 0x7fbf140d34a0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf18ce9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf140d29c0 con 0x7fbf140b9410 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf09d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbf140d34a0 0x7fbf14161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf09d9b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbf140d34a0 0x7fbf14161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47120/0 (socket says 172.21.15.99:47120) 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf09d9b640 1 -- 172.21.15.99:0/4127092633 learned_addr learned my addr 172.21.15.99:0/4127092633 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.445+0000 7fbf12d9d640 1 --2- 172.21.15.99:0/4127092633 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf14162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf09d9b640 1 -- 172.21.15.99:0/4127092633 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 msgr2=0x7fbf14162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf09d9b640 1 --2- 172.21.15.99:0/4127092633 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf14162120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf09d9b640 1 -- 172.21.15.99:0/4127092633 >> v1:172.21.15.162:6789/0 conn(0x7fbf140b5700 legacy=0x7fbf140d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf09d9b640 1 -- 172.21.15.99:0/4127092633 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf140d2ca0 con 0x7fbf140d34a0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf09d9b640 1 --2- 172.21.15.99:0/4127092633 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf14161be0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009870 tx=0x7fbefc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80bb197af45ee82 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf11d9b640 1 -- 172.21.15.99:0/4127092633 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc035020 con 0x7fbf140d34a0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf11d9b640 1 -- 172.21.15.99:0/4127092633 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbefc031640 con 0x7fbf140d34a0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.446+0000 7fbf11d9b640 1 -- 172.21.15.99:0/4127092633 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbefc02d040 con 0x7fbf140d34a0 2024-03-04T10:11:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 -- 172.21.15.99:0/4127092633 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 msgr2=0x7fbf14161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/4127092633 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf14161be0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fbefc009870 tx=0x7fbefc02b8f0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 -- 172.21.15.99:0/4127092633 shutdown_connections 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/4127092633 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf14162120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/4127092633 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf14161be0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 -- 172.21.15.99:0/4127092633 >> 172.21.15.99:0/4127092633 conn(0x7fbf140d0490 msgr2=0x7fbf140d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 -- 172.21.15.99:0/4127092633 shutdown_connections 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.447+0000 7fbf18ce9640 1 -- 172.21.15.99:0/4127092633 wait complete. 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.448+0000 7fbf18ce9640 1 Processor -- start 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 -- start start 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf140b5700 0x7fbf141f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf140c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf1415ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.456 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf14162660 con 0x7fbf140d34a0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf140b5d30 con 0x7fbf140b5700 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf18ce9640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf141627e0 con 0x7fbf140b9410 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf12d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf1415ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf12d9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf1415ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47136/0 (socket says 172.21.15.99:47136) 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.449+0000 7fbf12d9d640 1 -- 172.21.15.99:0/1426555643 learned_addr learned my addr 172.21.15.99:0/1426555643 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf09d9b640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf140c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf12d9d640 1 -- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 msgr2=0x7fbf140c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf1259c640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf140b5700 0x7fbf141f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf12d9d640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf140c8ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf12d9d640 1 -- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf140b5700 msgr2=0x7fbf141f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf12d9d640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf140b5700 0x7fbf141f67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf12d9d640 1 -- 172.21.15.99:0/1426555643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf140c9510 con 0x7fbf140d34a0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf12d9d640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf1415ffc0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c007e70 tx=0x7fbf0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c014620 con 0x7fbf140d34a0 2024-03-04T10:11:59.457 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c019070 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf140c97f0 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c014a20 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.450+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf140c9db0 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.451+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf14074f60 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.451+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbf0c014bc0 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.452+0000 7fbf0b7fe640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbef406b4d0 0x7fbef406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.452+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fbf0c08aed0 con 0x7fbf140d34a0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.452+0000 7fbf1259c640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbef406b4d0 0x7fbef406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.452+0000 7fbf1259c640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbef406b4d0 0x7fbef406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbf040015a0 tx=0x7fbf04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.456+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbf0c059980 con 0x7fbf140d34a0 2024-03-04T10:11:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.623+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf14161140 con 0x7fbf140d34a0 2024-03-04T10:11:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.624+0000 7fbf0b7fe640 1 -- 172.21.15.99:0/1426555643 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf0c059280 con 0x7fbf140d34a0 2024-03-04T10:11:59.625 INFO:teuthology.orchestra.run.smithi099.stdout:pg_num: 64 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbef406b4d0 msgr2=0x7fbef406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbef406b4d0 0x7fbef406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fbf040015a0 tx=0x7fbf04009040 comp rx=0 tx=0).stop 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 msgr2=0x7fbf1415ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf1415ffc0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c007e70 tx=0x7fbf0c00e4b0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 shutdown_connections 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbef406b4d0 0x7fbef406d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbf140d34a0 0x7fbf1415ffc0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbf140b9410 0x7fbf140c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 --2- 172.21.15.99:0/1426555643 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbf140b5700 0x7fbf141f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 >> 172.21.15.99:0/1426555643 conn(0x7fbf140d0490 msgr2=0x7fbf140d1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 shutdown_connections 2024-03-04T10:11:59.629 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.626+0000 7fbf18ce9640 1 -- 172.21.15.99:0/1426555643 wait complete. 2024-03-04T10:11:59.639 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:11:59.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.737+0000 7fe52d3ab640 1 Processor -- start 2024-03-04T10:11:59.738 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.737+0000 7fe52d3ab640 1 -- start start 2024-03-04T10:11:59.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.737+0000 7fe52d3ab640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe52815de00 0x7fe5280b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.737+0000 7fe52d3ab640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52815e1e0 con 0x7fe5280b9380 2024-03-04T10:11:59.741 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.737+0000 7fe52d3ab640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5280d1c10 con 0x7fe5280b5470 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.737+0000 7fe52d3ab640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5280d2820 con 0x7fe52815de00 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe5267fc640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fe5280b9380 0x7fe5280b6b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42612/0 (socket says 172.21.15.99:42612) 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe5267fc640 1 -- 172.21.15.99:0/557956413 learned_addr learned my addr 172.21.15.99:0/557956413 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe526ffd640 1 --2- 172.21.15.99:0/557956413 >> v2:172.21.15.162:3300/0 conn(0x7fe52815de00 0x7fe5280b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe525ffb640 1 -- 172.21.15.99:0/557956413 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260970281 0 0) 0x7fe52815e1e0 con 0x7fe5280b9380 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe525ffb640 1 -- 172.21.15.99:0/557956413 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510003520 con 0x7fe5280b9380 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe526ffd640 1 -- 172.21.15.99:0/557956413 >> v1:172.21.15.162:6789/0 conn(0x7fe5280b5470 legacy=0x7fe5280d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe525ffb640 1 -- 172.21.15.99:0/557956413 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542480370 0 0) 0x7fe510003520 con 0x7fe5280b9380 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe526ffd640 1 -- 172.21.15.99:0/557956413 >> v1:172.21.15.99:6789/0 conn(0x7fe5280b9380 legacy=0x7fe5280b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.738+0000 7fe526ffd640 1 -- 172.21.15.99:0/557956413 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280d1ef0 con 0x7fe52815de00 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.739+0000 7fe526ffd640 1 --2- 172.21.15.99:0/557956413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52815de00 0x7fe5280b2e60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe518009d90 tx=0x7fe51802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c181c471d70c3449 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.739+0000 7fe525ffb640 1 -- 172.21.15.99:0/557956413 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518039070 con 0x7fe52815de00 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.739+0000 7fe525ffb640 1 -- 172.21.15.99:0/557956413 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51802c940 con 0x7fe52815de00 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.739+0000 7fe525ffb640 1 -- 172.21.15.99:0/557956413 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c001670 con 0x7fe52815de00 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.739+0000 7fe52d3ab640 1 -- 172.21.15.99:0/557956413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52815de00 msgr2=0x7fe5280b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.739+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/557956413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52815de00 0x7fe5280b2e60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe518009d90 tx=0x7fe51802be10 comp rx=0 tx=0).stop 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.740+0000 7fe52d3ab640 1 -- 172.21.15.99:0/557956413 shutdown_connections 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.740+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/557956413 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe52815de00 0x7fe5280b2e60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.740+0000 7fe52d3ab640 1 -- 172.21.15.99:0/557956413 >> 172.21.15.99:0/557956413 conn(0x7fe5280d0470 msgr2=0x7fe5280d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.740+0000 7fe52d3ab640 1 -- 172.21.15.99:0/557956413 shutdown_connections 2024-03-04T10:11:59.743 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.740+0000 7fe52d3ab640 1 -- 172.21.15.99:0/557956413 wait complete. 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.741+0000 7fe52d3ab640 1 Processor -- start 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.741+0000 7fe52d3ab640 1 -- start start 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280b5470 0x7fe5280b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280b9380 0x7fe5280cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 0x7fe52814d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b97e0 con 0x7fe52815de00 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280d24b0 con 0x7fe5280b5470 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b72a0 con 0x7fe5280b9380 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 0x7fe52814d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.747 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 0x7fe52814d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47152/0 (socket says 172.21.15.99:47152) 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 -- 172.21.15.99:0/3515807652 learned_addr learned my addr 172.21.15.99:0/3515807652 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5267fc640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280b9380 0x7fe5280cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 -- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280b9380 msgr2=0x7fe5280cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe526ffd640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280b5470 0x7fe5280b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280b9380 0x7fe5280cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 -- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280b5470 msgr2=0x7fe5280b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.748 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280b5470 0x7fe5280b80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 -- 172.21.15.99:0/3515807652 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52814db60 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5267fc640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280b9380 0x7fe5280cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe526ffd640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280b5470 0x7fe5280b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe5277fe640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 0x7fe52814d620 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe52000c400 tx=0x7fe52000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520017070 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5200052d0 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe520003810 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52814acb0 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.742+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52814b230 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.743+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5280cf710 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.744+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe52001a070 con 0x7fe52815de00 2024-03-04T10:11:59.749 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.744+0000 7fe50f7fe640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:11:59.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.744+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fe520089930 con 0x7fe52815de00 2024-03-04T10:11:59.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.744+0000 7fe526ffd640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:11:59.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.745+0000 7fe526ffd640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe518036000 tx=0x7fe518030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:11:59.750 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.748+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe5280cf710 con 0x7fe52815de00 2024-03-04T10:11:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.956+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe528074f60 con 0x7fe52815de00 2024-03-04T10:11:59.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.957+0000 7fe50f7fe640 1 -- 172.21.15.99:0/3515807652 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+2624 (secure 0 0 0) 0x7fe520087e20 con 0x7fe52815de00 2024-03-04T10:11:59.959 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:11:59.959 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":109,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:54.988200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:11:54.988200+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:11:59.960 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 109 2024-03-04T10:11:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 msgr2=0x7fe51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe518036000 tx=0x7fe518030000 comp rx=0 tx=0).stop 2024-03-04T10:11:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 msgr2=0x7fe52814d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:11:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 0x7fe52814d620 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fe52000c400 tx=0x7fe52000c8c0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 shutdown_connections 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe51406b4d0 0x7fe51406d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe52815de00 0x7fe52814d620 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe5280b9380 0x7fe5280cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 --2- 172.21.15.99:0/3515807652 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe5280b5470 0x7fe5280b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 >> 172.21.15.99:0/3515807652 conn(0x7fe5280d0470 msgr2=0x7fe52815ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 shutdown_connections 2024-03-04T10:11:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:11:59.959+0000 7fe52d3ab640 1 -- 172.21.15.99:0/3515807652 wait complete. 2024-03-04T10:11:59.972 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:12:00.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.072+0000 7f613e116640 1 Processor -- start 2024-03-04T10:12:00.074 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.072+0000 7f613e116640 1 -- start start 2024-03-04T10:12:00.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f613e116640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f613806d3b0 0x7f613815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f613e116640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f613815b410 0x7f613815b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f613e116640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f613806d8b0 con 0x7f613815b030 2024-03-04T10:12:00.075 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f613e116640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61380b2720 con 0x7f613815b410 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f613e116640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f613806db90 con 0x7f613806d3b0 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f613815b410 0x7f613815b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f613815b410 0x7f613815b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47168/0 (socket says 172.21.15.99:47168) 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 -- 172.21.15.99:0/2009205793 learned_addr learned my addr 172.21.15.99:0/2009205793 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 -- 172.21.15.99:0/2009205793 >> v2:172.21.15.162:3300/0 conn(0x7f613806d3b0 msgr2=0x7f613815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 --2- 172.21.15.99:0/2009205793 >> v2:172.21.15.162:3300/0 conn(0x7f613806d3b0 0x7f613815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 -- 172.21.15.99:0/2009205793 >> v1:172.21.15.162:6789/0 conn(0x7f613815b030 legacy=0x7f613815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.073+0000 7f6137fff640 1 -- 172.21.15.99:0/2009205793 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613806de70 con 0x7f613815b410 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.074+0000 7f6137fff640 1 --2- 172.21.15.99:0/2009205793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f613815b880 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f61380b28a0 tx=0x7f613002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f97794445107bc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.074+0000 7f61367fc640 1 -- 172.21.15.99:0/2009205793 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613002c9e0 con 0x7f613815b410 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.074+0000 7f61367fc640 1 -- 172.21.15.99:0/2009205793 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613002cb80 con 0x7f613815b410 2024-03-04T10:12:00.076 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.074+0000 7f61367fc640 1 -- 172.21.15.99:0/2009205793 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61300343f0 con 0x7f613815b410 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 -- 172.21.15.99:0/2009205793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 msgr2=0x7f613815b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 --2- 172.21.15.99:0/2009205793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f613815b880 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f61380b28a0 tx=0x7f613002be40 comp rx=0 tx=0).stop 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 -- 172.21.15.99:0/2009205793 shutdown_connections 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 --2- 172.21.15.99:0/2009205793 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f613815b880 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 --2- 172.21.15.99:0/2009205793 >> v2:172.21.15.162:3300/0 conn(0x7f613806d3b0 0x7f613815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 -- 172.21.15.99:0/2009205793 >> 172.21.15.99:0/2009205793 conn(0x7f613806acd0 msgr2=0x7f613806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 -- 172.21.15.99:0/2009205793 shutdown_connections 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.075+0000 7f613e116640 1 -- 172.21.15.99:0/2009205793 wait complete. 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.076+0000 7f613e116640 1 Processor -- start 2024-03-04T10:12:00.078 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.076+0000 7f613e116640 1 -- start start 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f613e116640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f613806d3b0 0x7f61381510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f613e116640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f613815b030 0x7f6138154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f613e116640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f6138155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f613e116640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613806e690 con 0x7f613815b410 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f613e116640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613815bdc0 con 0x7f613815b030 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f613e116640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613806e810 con 0x7f613806d3b0 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f6138155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f6138155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47184/0 (socket says 172.21.15.99:47184) 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 -- 172.21.15.99:0/1665254045 learned_addr learned my addr 172.21.15.99:0/1665254045 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6136ffd640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f613815b030 0x7f6138154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f61377fe640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f613806d3b0 0x7f61381510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.081 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 -- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f613806d3b0 msgr2=0x7f61381510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f613806d3b0 0x7f61381510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 -- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f613815b030 msgr2=0x7f6138154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f613815b030 0x7f6138154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 -- 172.21.15.99:0/1665254045 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138158a60 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f61377fe640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f613806d3b0 0x7f61381510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f6137fff640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f6138155070 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6130036040 tx=0x7f6130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130043020 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f613003e450 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.077+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130009510 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.078+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6138159ec0 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.078+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6138158d10 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.079+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6138074f60 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.079+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6130046020 con 0x7f613815b410 2024-03-04T10:12:00.082 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.079+0000 7f611ffff640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f612406b4d0 0x7f612406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.079+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f61300acf70 con 0x7f613815b410 2024-03-04T10:12:00.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.079+0000 7f61377fe640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f612406b4d0 0x7f612406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.083 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.080+0000 7f61377fe640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f612406b4d0 0x7f612406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6128001750 tx=0x7f6128009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.085 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.083+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f613004d030 con 0x7f613815b410 2024-03-04T10:12:00.257 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.256+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61380b3430 con 0x7f613815b410 2024-03-04T10:12:00.258 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.257+0000 7f611ffff640 1 -- 172.21.15.99:0/1665254045 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+17945 (secure 0 0 0) 0x7f613007ba20 con 0x7f613815b410 2024-03-04T10:12:00.258 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:00.259 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":102,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:11:39.985592+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.31","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6837/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6835/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6832/584503338":"2024-03-05T10:11:24.982726+0000","172.21.15.162:6836/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6835/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6834/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6835/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6834/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6835/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6834/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6835/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6833/584503338":"2024-03-05T10:11:24.982726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:12:00.260 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f612406b4d0 msgr2=0x7f612406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f612406b4d0 0x7f612406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f6128001750 tx=0x7f6128009040 comp rx=0 tx=0).stop 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 msgr2=0x7f6138155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f6138155070 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6130036040 tx=0x7f6130004a40 comp rx=0 tx=0).stop 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 shutdown_connections 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f612406b4d0 0x7f612406d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f613815b410 0x7f6138155070 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f613815b030 0x7f6138154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 --2- 172.21.15.99:0/1665254045 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f613806d3b0 0x7f61381510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 >> 172.21.15.99:0/1665254045 conn(0x7f613806acd0 msgr2=0x7f6138071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 shutdown_connections 2024-03-04T10:12:00.261 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.259+0000 7f613e116640 1 -- 172.21.15.99:0/1665254045 wait complete. 2024-03-04T10:12:00.273 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:12:00.273 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:12:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.371+0000 7fbafdf53640 1 Processor -- start 2024-03-04T10:12:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbafdf53640 1 -- start start 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbafdf53640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbaf80b0700 0x7fbaf80b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbafdf53640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fbaf80b47c0 0x7fbaf80b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbafdf53640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaf80ccad0 con 0x7fbaf815b960 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbafdf53640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf80b5190 con 0x7fbaf80b0700 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbafdf53640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaf80cf860 con 0x7fbaf80b47c0 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbaf6ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbaf80b0700 0x7fbaf80b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbaf6ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fbaf80b0700 0x7fbaf80b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47188/0 (socket says 172.21.15.99:47188) 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.372+0000 7fbaf6ffd640 1 -- 172.21.15.99:0/2901786123 learned_addr learned my addr 172.21.15.99:0/2901786123 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf7fff640 1 --2- 172.21.15.99:0/2901786123 >> v2:172.21.15.162:3300/0 conn(0x7fbaf80b47c0 0x7fbaf80b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf6ffd640 1 -- 172.21.15.99:0/2901786123 >> v2:172.21.15.162:3300/0 conn(0x7fbaf80b47c0 msgr2=0x7fbaf80b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf6ffd640 1 --2- 172.21.15.99:0/2901786123 >> v2:172.21.15.162:3300/0 conn(0x7fbaf80b47c0 0x7fbaf80b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf6ffd640 1 -- 172.21.15.99:0/2901786123 >> v1:172.21.15.162:6789/0 conn(0x7fbaf815b960 legacy=0x7fbaf80d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf6ffd640 1 -- 172.21.15.99:0/2901786123 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf80cfab0 con 0x7fbaf80b0700 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf7fff640 1 --2- 172.21.15.99:0/2901786123 >> v2:172.21.15.162:3300/0 conn(0x7fbaf80b47c0 0x7fbaf80b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.373+0000 7fbaf6ffd640 1 --2- 172.21.15.99:0/2901786123 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf80b0700 0x7fbaf80b0b90 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80ce1b0 tx=0x7fbaec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f75e4df6c34f637 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbaf67fc640 1 -- 172.21.15.99:0/2901786123 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec02e070 con 0x7fbaf80b0700 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbaf67fc640 1 -- 172.21.15.99:0/2901786123 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaec037070 con 0x7fbaf80b0700 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbaf67fc640 1 -- 172.21.15.99:0/2901786123 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec031a00 con 0x7fbaf80b0700 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 -- 172.21.15.99:0/2901786123 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf80b0700 msgr2=0x7fbaf80b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 --2- 172.21.15.99:0/2901786123 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf80b0700 0x7fbaf80b0b90 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80ce1b0 tx=0x7fbaec02b880 comp rx=0 tx=0).stop 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 -- 172.21.15.99:0/2901786123 shutdown_connections 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 --2- 172.21.15.99:0/2901786123 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf80b0700 0x7fbaf80b0b90 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 --2- 172.21.15.99:0/2901786123 >> v2:172.21.15.162:3300/0 conn(0x7fbaf80b47c0 0x7fbaf80b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 -- 172.21.15.99:0/2901786123 >> 172.21.15.99:0/2901786123 conn(0x7fbaf806f5f0 msgr2=0x7fbaf806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 -- 172.21.15.99:0/2901786123 shutdown_connections 2024-03-04T10:12:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.374+0000 7fbafdf53640 1 -- 172.21.15.99:0/2901786123 wait complete. 2024-03-04T10:12:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.375+0000 7fbafdf53640 1 Processor -- start 2024-03-04T10:12:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 -- start start 2024-03-04T10:12:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbaf80b0700 0x7fbaf80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaf80b47c0 0x7fbaf81554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 0x7fbaf8155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf815bd40 con 0x7fbaf815b960 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf80d0100 con 0x7fbaf80b47c0 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbafdf53640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf815bec0 con 0x7fbaf80b0700 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbaf7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 0x7fbaf8155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbaf7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 0x7fbaf8155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47202/0 (socket says 172.21.15.99:47202) 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbaf7fff640 1 -- 172.21.15.99:0/308540478 learned_addr learned my addr 172.21.15.99:0/308540478 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbaf6ffd640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaf80b47c0 0x7fbaf81554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.376+0000 7fbaf77fe640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbaf80b0700 0x7fbaf80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbaf7fff640 1 -- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbaf80b0700 msgr2=0x7fbaf80b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbaf7fff640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbaf80b0700 0x7fbaf80b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbaf7fff640 1 -- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaf80b47c0 msgr2=0x7fbaf81554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbaf7fff640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaf80b47c0 0x7fbaf81554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbaf7fff640 1 -- 172.21.15.99:0/308540478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf8153790 con 0x7fbaf815b960 2024-03-04T10:12:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbaf7fff640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 0x7fbaf8155a20 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000cef0 tx=0x7fbaf000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf001a070 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf000d860 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf000dc50 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf8153a70 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.377+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf8154030 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.378+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf8074f60 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.378+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fbaf001d070 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.379+0000 7fbadffff640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbae406b4d0 0x7fbae406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.379+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fbaf008afe0 con 0x7fbaf815b960 2024-03-04T10:12:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.379+0000 7fbaf77fe640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbae406b4d0 0x7fbae406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.383 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.379+0000 7fbaf77fe640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbae406b4d0 0x7fbae406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbae8006fd0 tx=0x7fbae8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.384 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.383+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fbaf0059980 con 0x7fbaf815b960 2024-03-04T10:12:00.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.590+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaf80ce970 con 0x7fbaf815b960 2024-03-04T10:12:00.591 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.590+0000 7fbadffff640 1 -- 172.21.15.99:0/308540478 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+2624 (secure 0 0 0) 0x7fbaf0059280 con 0x7fbaf815b960 2024-03-04T10:12:00.592 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:00.592 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":109,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:54.988200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:11:54.988200+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:12:00.592 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 109 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbae406b4d0 msgr2=0x7fbae406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbae406b4d0 0x7fbae406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fbae8006fd0 tx=0x7fbae8008040 comp rx=0 tx=0).stop 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 msgr2=0x7fbaf8155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 0x7fbaf8155a20 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000cef0 tx=0x7fbaf000a710 comp rx=0 tx=0).stop 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 shutdown_connections 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fbae406b4d0 0x7fbae406d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fbaf815b960 0x7fbaf8155a20 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaf80b47c0 0x7fbaf81554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.594 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 --2- 172.21.15.99:0/308540478 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fbaf80b0700 0x7fbaf80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 >> 172.21.15.99:0/308540478 conn(0x7fbaf806f5f0 msgr2=0x7fbaf80b1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:00.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 shutdown_connections 2024-03-04T10:12:00.595 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.592+0000 7fbafdf53640 1 -- 172.21.15.99:0/308540478 wait complete. 2024-03-04T10:12:00.605 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:12:00.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.703+0000 7fa6de996640 1 Processor -- start 2024-03-04T10:12:00.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.703+0000 7fa6de996640 1 -- start start 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6de996640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6d815dea0 0x7fa6d80cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6de996640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d80cc8d0 0x7fa6d806acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6de996640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d806fd20 con 0x7fa6d80d1200 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6de996640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d806fad0 con 0x7fa6d815dea0 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6de996640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8070000 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6d815dea0 0x7fa6d80cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6dcf33640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa6d80d1200 0x7fa6d80cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42616/0 (socket says 172.21.15.99:42616) 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa6d815dea0 0x7fa6d80cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47206/0 (socket says 172.21.15.99:47206) 2024-03-04T10:12:00.707 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6dcf33640 1 -- 172.21.15.99:0/2827401307 learned_addr learned my addr 172.21.15.99:0/2827401307 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d77fe640 1 --2- 172.21.15.99:0/2827401307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d80cc8d0 0x7fa6d806acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d7fff640 1 -- 172.21.15.99:0/2827401307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d80cc8d0 msgr2=0x7fa6d806acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d7fff640 1 --2- 172.21.15.99:0/2827401307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d80cc8d0 0x7fa6d806acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d7fff640 1 -- 172.21.15.99:0/2827401307 >> v1:172.21.15.99:6789/0 conn(0x7fa6d80d1200 legacy=0x7fa6d80cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d7fff640 1 -- 172.21.15.99:0/2827401307 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80702e0 con 0x7fa6d815dea0 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.704+0000 7fa6d6ffd640 1 -- 172.21.15.99:0/2827401307 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468868543 0 0) 0x7fa6d806fd20 con 0x7fa6d80d1200 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6d7fff640 1 --2- 172.21.15.99:0/2827401307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d815dea0 0x7fa6d80cad60 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80097a0 tx=0x7fa6c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d9c6248f58a975 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6d6ffd640 1 -- 172.21.15.99:0/2827401307 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8035020 con 0x7fa6d815dea0 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6d6ffd640 1 -- 172.21.15.99:0/2827401307 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8031830 con 0x7fa6d815dea0 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6d6ffd640 1 -- 172.21.15.99:0/2827401307 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8031c00 con 0x7fa6d815dea0 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6de996640 1 -- 172.21.15.99:0/2827401307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d815dea0 msgr2=0x7fa6d80cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6de996640 1 --2- 172.21.15.99:0/2827401307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d815dea0 0x7fa6d80cad60 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80097a0 tx=0x7fa6c802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.708 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6de996640 1 -- 172.21.15.99:0/2827401307 shutdown_connections 2024-03-04T10:12:00.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6de996640 1 --2- 172.21.15.99:0/2827401307 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d80cc8d0 0x7fa6d806acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6de996640 1 --2- 172.21.15.99:0/2827401307 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d815dea0 0x7fa6d80cad60 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.705+0000 7fa6de996640 1 -- 172.21.15.99:0/2827401307 >> 172.21.15.99:0/2827401307 conn(0x7fa6d80d0710 msgr2=0x7fa6d80d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:00.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.706+0000 7fa6de996640 1 -- 172.21.15.99:0/2827401307 shutdown_connections 2024-03-04T10:12:00.709 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.706+0000 7fa6de996640 1 -- 172.21.15.99:0/2827401307 wait complete. 2024-03-04T10:12:00.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 Processor -- start 2024-03-04T10:12:00.711 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 -- start start 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 0x7fa6d814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6d80d1200 0x7fa6d80c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d815dea0 0x7fa6d80c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d806b390 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.707+0000 7fa6de996640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d806b210 con 0x7fa6d80d1200 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6de996640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d806b510 con 0x7fa6d815dea0 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 0x7fa6d814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 0x7fa6d814ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47220/0 (socket says 172.21.15.99:47220) 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 -- 172.21.15.99:0/3557469637 learned_addr learned my addr 172.21.15.99:0/3557469637 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 -- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d815dea0 msgr2=0x7fa6d80c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d815dea0 0x7fa6d80c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 -- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6d80d1200 msgr2=0x7fa6d80c92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6d80d1200 0x7fa6d80c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 -- 172.21.15.99:0/3557469637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d80b5be0 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d7fff640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 0x7fa6d814ff10 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8000c00 tx=0x7fa6c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8035070 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8004230 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c803a6f0 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d80b5ec0 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.708+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d80b6480 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.709+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d8074f60 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.710+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa6c8038020 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.710+0000 7fa6d4ff9640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6c406b4d0 0x7fa6c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:00.713 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.710+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fa6c80ac400 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.711+0000 7fa6d77fe640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6c406b4d0 0x7fa6c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:00.714 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.711+0000 7fa6d77fe640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6c406b4d0 0x7fa6c406d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc006fd0 tx=0x7fa6cc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:00.716 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.715+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa6c8040070 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.918+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d80d0c90 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.918+0000 7fa6d4ff9640 1 -- 172.21.15.99:0/3557469637 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+2624 (secure 0 0 0) 0x7fa6c807ae30 con 0x7fa6d80cc8d0 2024-03-04T10:12:00.920 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:00.920 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":109,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T10:10:07.195925+0000","modified":"2024-03-04T10:11:54.988200+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6514":{"gid":6514,"name":"b","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.162:6835/2241177706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":2241177706},{"type":"v1","addr":"172.21.15.162:6835","nonce":2241177706}]},"join_fscid":-1,"laggy_since":"2024-03-04T10:11:54.988200+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-04T10:12:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 109 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6c406b4d0 msgr2=0x7fa6c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6c406b4d0 0x7fa6c406d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc006fd0 tx=0x7fa6cc008040 comp rx=0 tx=0).stop 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 msgr2=0x7fa6d814ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 0x7fa6d814ff10 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8000c00 tx=0x7fa6c80046c0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 shutdown_connections 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa6c406b4d0 0x7fa6c406d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa6d815dea0 0x7fa6d80c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6d80d1200 0x7fa6d80c92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 --2- 172.21.15.99:0/3557469637 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa6d80cc8d0 0x7fa6d814ff10 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 >> 172.21.15.99:0/3557469637 conn(0x7fa6d80d0710 msgr2=0x7fa6d80cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 shutdown_connections 2024-03-04T10:12:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:00.920+0000 7fa6de996640 1 -- 172.21.15.99:0/3557469637 wait complete. 2024-03-04T10:12:00.933 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:12:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.030+0000 7f109524b640 1 Processor -- start 2024-03-04T10:12:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.030+0000 7f109524b640 1 -- start start 2024-03-04T10:12:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f109524b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f109015de00 0x7f10900b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f109524b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f10900b9380 0x7f10900b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f109524b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f109015e1e0 con 0x7f10900b70d0 2024-03-04T10:12:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f109524b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10900d2820 con 0x7f109015de00 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f109524b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d1c10 con 0x7f10900b9380 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f108e59c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f10900b9380 0x7f10900b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f108ed9d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f109015de00 0x7f10900b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f108f59e640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f10900b70d0 0x7f10900b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:53188/0 (socket says 172.21.15.99:53188) 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.031+0000 7f108f59e640 1 -- 172.21.15.99:0/2861945760 learned_addr learned my addr 172.21.15.99:0/2861945760 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.032+0000 7f108e59c640 1 -- 172.21.15.99:0/2861945760 >> v2:172.21.15.162:3300/0 conn(0x7f109015de00 msgr2=0x7f10900b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.032+0000 7f108e59c640 1 --2- 172.21.15.99:0/2861945760 >> v2:172.21.15.162:3300/0 conn(0x7f109015de00 0x7f10900b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.032+0000 7f108e59c640 1 -- 172.21.15.99:0/2861945760 >> v1:172.21.15.162:6789/0 conn(0x7f10900b70d0 legacy=0x7f10900b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.032+0000 7f108dd9b640 1 -- 172.21.15.99:0/2861945760 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3591774137 0 0) 0x7f109015e1e0 con 0x7f10900b70d0 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.032+0000 7f108e59c640 1 -- 172.21.15.99:0/2861945760 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10900d1ef0 con 0x7f10900b9380 2024-03-04T10:12:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.032+0000 7f108ed9d640 1 --2- 172.21.15.99:0/2861945760 >> v2:172.21.15.162:3300/0 conn(0x7f109015de00 0x7f10900b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:12:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f108e59c640 1 --2- 172.21.15.99:0/2861945760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f10900b9380 0x7f10900b6b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1084009d90 tx=0x7f108402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc98158056f9c749 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f108dd9b640 1 -- 172.21.15.99:0/2861945760 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084039070 con 0x7f10900b9380 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f108dd9b640 1 -- 172.21.15.99:0/2861945760 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108402cc50 con 0x7f10900b9380 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f108dd9b640 1 -- 172.21.15.99:0/2861945760 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084034440 con 0x7f10900b9380 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 -- 172.21.15.99:0/2861945760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f10900b9380 msgr2=0x7f10900b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 --2- 172.21.15.99:0/2861945760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f10900b9380 0x7f10900b6b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1084009d90 tx=0x7f108402bea0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 -- 172.21.15.99:0/2861945760 shutdown_connections 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 --2- 172.21.15.99:0/2861945760 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f10900b9380 0x7f10900b6b90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 --2- 172.21.15.99:0/2861945760 >> v2:172.21.15.162:3300/0 conn(0x7f109015de00 0x7f10900b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 -- 172.21.15.99:0/2861945760 >> 172.21.15.99:0/2861945760 conn(0x7f10900d0470 msgr2=0x7f10900d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 -- 172.21.15.99:0/2861945760 shutdown_connections 2024-03-04T10:12:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.033+0000 7f109524b640 1 -- 172.21.15.99:0/2861945760 wait complete. 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.035+0000 7f109524b640 1 Processor -- start 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.035+0000 7f109524b640 1 -- start start 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 0x7f10901f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10900b9380 0x7f10900cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f109015de00 0x7f1090156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900b97e0 con 0x7f10900b70d0 2024-03-04T10:12:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10900d24b0 con 0x7f10900b9380 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10900b5b80 con 0x7f109015de00 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 0x7f10901f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 0x7f10901f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47232/0 (socket says 172.21.15.99:47232) 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 -- 172.21.15.99:0/3817905298 learned_addr learned my addr 172.21.15.99:0/3817905298 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108f59e640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f109015de00 0x7f1090156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108e59c640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10900b9380 0x7f10900cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 -- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f109015de00 msgr2=0x7f1090156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f109015de00 0x7f1090156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 -- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10900b9380 msgr2=0x7f10900cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10900b9380 0x7f10900cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 -- 172.21.15.99:0/3817905298 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10901f5910 con 0x7f10900b70d0 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108e59c640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10900b9380 0x7f10900cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108f59e640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f109015de00 0x7f1090156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f108ed9d640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 0x7f10901f52a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f108800ebe0 tx=0x7f108800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088017070 con 0x7f10900b70d0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108800cc30 con 0x7f10900b70d0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10901f5bf0 con 0x7f10900b70d0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.036+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088013470 con 0x7f10900b70d0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.037+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10901f61b0 con 0x7f10900b70d0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.037+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1090074f60 con 0x7f10900b70d0 2024-03-04T10:12:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.038+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f108801a070 con 0x7f10900b70d0 2024-03-04T10:12:01.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.038+0000 7f10777fe640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f107c06b500 0x7f107c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.038+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7f108808a8c0 con 0x7f10900b70d0 2024-03-04T10:12:01.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.039+0000 7f108e59c640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f107c06b500 0x7f107c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.039+0000 7f108e59c640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f107c06b500 0x7f107c06d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1084036000 tx=0x7f1084030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.043+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1088059370 con 0x7f10900b70d0 2024-03-04T10:12:01.217 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.216+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10900b8010 con 0x7f10900b70d0 2024-03-04T10:12:01.218 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.217+0000 7f10777fe640 1 -- 172.21.15.99:0/3817905298 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+17945 (secure 0 0 0) 0x7f1088058c70 con 0x7f10900b70d0 2024-03-04T10:12:01.219 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:01.219 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":102,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:11:39.985592+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-04T10:10:05.411440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-04T10:10:06.166240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.10","mappings":[{"from":1,"to":2}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.18","mappings":[{"from":1,"to":7}]},{"pgid":"14.31","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6837/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6835/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6832/584503338":"2024-03-05T10:11:24.982726+0000","172.21.15.162:6836/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6835/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6834/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6835/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6834/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6835/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6834/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6835/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6833/584503338":"2024-03-05T10:11:24.982726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:12:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f107c06b500 msgr2=0x7f107c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f107c06b500 0x7f107c06d9c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1084036000 tx=0x7f1084030000 comp rx=0 tx=0).stop 2024-03-04T10:12:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 msgr2=0x7f10901f52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 0x7f10901f52a0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f108800ebe0 tx=0x7f108800c520 comp rx=0 tx=0).stop 2024-03-04T10:12:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 shutdown_connections 2024-03-04T10:12:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f107c06b500 0x7f107c06d9c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f109015de00 0x7f1090156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f10900b9380 0x7f10900cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 --2- 172.21.15.99:0/3817905298 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f10900b70d0 0x7f10901f52a0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 >> 172.21.15.99:0/3817905298 conn(0x7f10900d0470 msgr2=0x7f10900d2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 shutdown_connections 2024-03-04T10:12:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.219+0000 7f109524b640 1 -- 172.21.15.99:0/3817905298 wait complete. 2024-03-04T10:12:01.232 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T10:12:01.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.331+0000 7fe113b7e640 1 Processor -- start 2024-03-04T10:12:01.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.331+0000 7fe113b7e640 1 -- start start 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe113b7e640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe10c0b39c0 0x7fe10c161760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe113b7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fe10c161ca0 0x7fe10c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe113b7e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c06b500 con 0x7fe10c158410 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe113b7e640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c1620f0 con 0x7fe10c0b39c0 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe113b7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10c06b7e0 con 0x7fe10c161ca0 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe111119640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe10c0b39c0 0x7fe10c161760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe111119640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fe10c0b39c0 0x7fe10c161760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47248/0 (socket says 172.21.15.99:47248) 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.332+0000 7fe111119640 1 -- 172.21.15.99:0/1917185894 learned_addr learned my addr 172.21.15.99:0/1917185894 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe11211b640 1 --2- 172.21.15.99:0/1917185894 >> v2:172.21.15.162:3300/0 conn(0x7fe10c161ca0 0x7fe10c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe111119640 1 -- 172.21.15.99:0/1917185894 >> v2:172.21.15.162:3300/0 conn(0x7fe10c161ca0 msgr2=0x7fe10c06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe111119640 1 --2- 172.21.15.99:0/1917185894 >> v2:172.21.15.162:3300/0 conn(0x7fe10c161ca0 0x7fe10c06de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe111119640 1 -- 172.21.15.99:0/1917185894 >> v1:172.21.15.162:6789/0 conn(0x7fe10c158410 legacy=0x7fe10c06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe111119640 1 -- 172.21.15.99:0/1917185894 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c06bac0 con 0x7fe10c0b39c0 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe11211b640 1 --2- 172.21.15.99:0/1917185894 >> v2:172.21.15.162:3300/0 conn(0x7fe10c161ca0 0x7fe10c06de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.333+0000 7fe111119640 1 --2- 172.21.15.99:0/1917185894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c0b39c0 0x7fe10c161760 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0af960 tx=0x7fe0fc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e38e362b8ce78ce server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe110918640 1 -- 172.21.15.99:0/1917185894 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc02f850 con 0x7fe10c0b39c0 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe110918640 1 -- 172.21.15.99:0/1917185894 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc02f9f0 con 0x7fe10c0b39c0 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe110918640 1 -- 172.21.15.99:0/1917185894 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc02fdc0 con 0x7fe10c0b39c0 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 -- 172.21.15.99:0/1917185894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c0b39c0 msgr2=0x7fe10c161760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 --2- 172.21.15.99:0/1917185894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c0b39c0 0x7fe10c161760 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0af960 tx=0x7fe0fc02ba30 comp rx=0 tx=0).stop 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 -- 172.21.15.99:0/1917185894 shutdown_connections 2024-03-04T10:12:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 --2- 172.21.15.99:0/1917185894 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c0b39c0 0x7fe10c161760 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 --2- 172.21.15.99:0/1917185894 >> v2:172.21.15.162:3300/0 conn(0x7fe10c161ca0 0x7fe10c06de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 -- 172.21.15.99:0/1917185894 >> 172.21.15.99:0/1917185894 conn(0x7fe10c0cfef0 msgr2=0x7fe10c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 -- 172.21.15.99:0/1917185894 shutdown_connections 2024-03-04T10:12:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.334+0000 7fe113b7e640 1 -- 172.21.15.99:0/1917185894 wait complete. 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.335+0000 7fe113b7e640 1 Processor -- start 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.335+0000 7fe113b7e640 1 -- start start 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe113b7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe10c0b39c0 0x7fe10c1f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe113b7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe10c158410 0x7fe10c0cee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe113b7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 0x7fe10c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe113b7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c0b3e50 con 0x7fe10c161ca0 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe113b7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c06c080 con 0x7fe10c158410 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe113b7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c06c5a0 con 0x7fe10c0b39c0 2024-03-04T10:12:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 0x7fe10c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 0x7fe10c0ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47254/0 (socket says 172.21.15.99:47254) 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 -- 172.21.15.99:0/3457472054 learned_addr learned my addr 172.21.15.99:0/3457472054 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe111119640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe10c158410 0x7fe10c0cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11191a640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe10c0b39c0 0x7fe10c1f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 -- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe10c0b39c0 msgr2=0x7fe10c1f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe10c0b39c0 0x7fe10c1f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 -- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe10c158410 msgr2=0x7fe10c0cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe10c158410 0x7fe10c0cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.336+0000 7fe11211b640 1 -- 172.21.15.99:0/3457472054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c1538c0 con 0x7fe10c161ca0 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe11191a640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe10c0b39c0 0x7fe10c1f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe11211b640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 0x7fe10c0ccff0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fe10400d7d0 tx=0x7fe10400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104004030 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1040041d0 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104009640 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c153ba0 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.337+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c154440 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.338+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c153d70 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.338+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fe1040097e0 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.339+0000 7fe1027fc640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe0f006b4d0 0x7fe0f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.339+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6819+0+0 (secure 0 0 0) 0x7fe10408a9f0 con 0x7fe10c161ca0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.339+0000 7fe11191a640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe0f006b4d0 0x7fe0f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.339+0000 7fe11191a640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe0f006b4d0 0x7fe0f006d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe108001d70 tx=0x7fe108009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.344 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.343+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fe104059410 con 0x7fe10c161ca0 2024-03-04T10:12:01.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.548+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe10c074f60 con 0x7fe10c161ca0 2024-03-04T10:12:01.550 INFO:tasks.ceph.mon.a.smithi099.stderr:2024-03-04T10:12:01.548+0000 7f5b1c0db640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T10:12:01.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.550+0000 7fe1027fc640 1 -- 172.21.15.99:0/3457472054 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v110) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe104058d10 con 0x7fe10c161ca0 2024-03-04T10:12:01.551 INFO:teuthology.orchestra.run.smithi099.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T10:12:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe0f006b4d0 msgr2=0x7fe0f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe0f006b4d0 0x7fe0f006d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe108001d70 tx=0x7fe108009040 comp rx=0 tx=0).stop 2024-03-04T10:12:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 msgr2=0x7fe10c0ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 0x7fe10c0ccff0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fe10400d7d0 tx=0x7fe10400dc90 comp rx=0 tx=0).stop 2024-03-04T10:12:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 shutdown_connections 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fe0f006b4d0 0x7fe0f006d990 secure :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fe108001d70 tx=0x7fe108009040 comp rx=0 tx=0).stop 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe10c161ca0 0x7fe10c0ccff0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe10c158410 0x7fe10c0cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 --2- 172.21.15.99:0/3457472054 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fe10c0b39c0 0x7fe10c1f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.552+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 >> 172.21.15.99:0/3457472054 conn(0x7fe10c0cfef0 msgr2=0x7fe10c06acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.553+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 shutdown_connections 2024-03-04T10:12:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.553+0000 7fe113b7e640 1 -- 172.21.15.99:0/3457472054 wait complete. 2024-03-04T10:12:01.565 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T10:12:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.667+0000 7f9475e6c640 1 Processor -- start 2024-03-04T10:12:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.667+0000 7f9475e6c640 1 -- start start 2024-03-04T10:12:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f9475e6c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f94700d34a0 0x7f9470161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f9475e6c640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f94700b9410 0x7f9470162120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f9475e6c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94700d2400 con 0x7f94700b5700 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f9475e6c640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94700d26e0 con 0x7f94700b9410 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f9475e6c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94700d29c0 con 0x7f94700d34a0 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f946ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f94700b9410 0x7f9470162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f946ffff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f94700b9410 0x7f9470162120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47256/0 (socket says 172.21.15.99:47256) 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.668+0000 7f946ffff640 1 -- 172.21.15.99:0/2588280379 learned_addr learned my addr 172.21.15.99:0/2588280379 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946effd640 1 --2- 172.21.15.99:0/2588280379 >> v2:172.21.15.162:3300/0 conn(0x7f94700d34a0 0x7f9470161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946ffff640 1 -- 172.21.15.99:0/2588280379 >> v2:172.21.15.162:3300/0 conn(0x7f94700d34a0 msgr2=0x7f9470161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946ffff640 1 --2- 172.21.15.99:0/2588280379 >> v2:172.21.15.162:3300/0 conn(0x7f94700d34a0 0x7f9470161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946ffff640 1 -- 172.21.15.99:0/2588280379 >> v1:172.21.15.162:6789/0 conn(0x7f94700b5700 legacy=0x7f94700d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946ffff640 1 -- 172.21.15.99:0/2588280379 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94700d2ca0 con 0x7f94700b9410 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946effd640 1 --2- 172.21.15.99:0/2588280379 >> v2:172.21.15.162:3300/0 conn(0x7f94700d34a0 0x7f9470161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946ffff640 1 --2- 172.21.15.99:0/2588280379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700b9410 0x7f9470162120 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f94700c7470 tx=0x7f946802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0aac201b2a63f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.669+0000 7f946e7fc640 1 -- 172.21.15.99:0/2588280379 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946802c9f0 con 0x7f94700b9410 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f946e7fc640 1 -- 172.21.15.99:0/2588280379 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946802cb90 con 0x7f94700b9410 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f946e7fc640 1 -- 172.21.15.99:0/2588280379 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94680343f0 con 0x7f94700b9410 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 -- 172.21.15.99:0/2588280379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700b9410 msgr2=0x7f9470162120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 --2- 172.21.15.99:0/2588280379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700b9410 0x7f9470162120 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f94700c7470 tx=0x7f946802be50 comp rx=0 tx=0).stop 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 -- 172.21.15.99:0/2588280379 shutdown_connections 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 --2- 172.21.15.99:0/2588280379 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700b9410 0x7f9470162120 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 --2- 172.21.15.99:0/2588280379 >> v2:172.21.15.162:3300/0 conn(0x7f94700d34a0 0x7f9470161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 -- 172.21.15.99:0/2588280379 >> 172.21.15.99:0/2588280379 conn(0x7f94700d0490 msgr2=0x7f94700d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 -- 172.21.15.99:0/2588280379 shutdown_connections 2024-03-04T10:12:01.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.670+0000 7f9475e6c640 1 -- 172.21.15.99:0/2588280379 wait complete. 2024-03-04T10:12:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.671+0000 7f9475e6c640 1 Processor -- start 2024-03-04T10:12:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.671+0000 7f9475e6c640 1 -- start start 2024-03-04T10:12:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f9475e6c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94700b5700 0x7f94700c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f9475e6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94700b9410 0x7f94700d39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f9475e6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 0x7f94700d3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f9475e6c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9470162660 con 0x7f94700d34a0 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f9475e6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700b5cf0 con 0x7f94700b9410 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f9475e6c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94701627e0 con 0x7f94700b5700 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f946ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 0x7f94700d3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f946ffff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 0x7f94700d3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47266/0 (socket says 172.21.15.99:47266) 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f946ffff640 1 -- 172.21.15.99:0/416970616 learned_addr learned my addr 172.21.15.99:0/416970616 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f946effd640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94700b9410 0x7f94700d39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.672+0000 7f946f7fe640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94700b5700 0x7f94700c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f946ffff640 1 -- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94700b5700 msgr2=0x7f94700c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f946ffff640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94700b5700 0x7f94700c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f946ffff640 1 -- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94700b9410 msgr2=0x7f94700d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f946ffff640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94700b9410 0x7f94700d39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f946ffff640 1 -- 172.21.15.99:0/416970616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94701f7210 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f946ffff640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 0x7f94700d3ee0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f946802c460 tx=0x7f9468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468043020 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946803e450 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468009510 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94701f74f0 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.673+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94701f7ab0 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.674+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9470074f60 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.675+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9468046020 con 0x7f94700d34a0 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.675+0000 7f9457fff640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.675+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6905+0+0 (secure 0 0 0) 0x7f94680acf80 con 0x7f94700d34a0 2024-03-04T10:12:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.675+0000 7f946f7fe640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.676+0000 7f946f7fe640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f945c06b4d0 0x7f945c06d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9460006fd0 tx=0x7f9460008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.679+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f946804d030 con 0x7f94700d34a0 2024-03-04T10:12:01.886 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:01.885+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f94700c9dd0 con 0x7f94700d34a0 2024-03-04T10:12:02.560 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.559+0000 7f9457fff640 1 -- 172.21.15.99:0/416970616 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v111) v1 ==== 104+0+0 (secure 0 0 0) 0x7f946807b950 con 0x7f94700d34a0 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f945c06b4d0 msgr2=0x7f945c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f945c06b4d0 0x7f945c06d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9460006fd0 tx=0x7f9460008040 comp rx=0 tx=0).stop 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 msgr2=0x7f94700d3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 0x7f94700d3ee0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f946802c460 tx=0x7f9468004a40 comp rx=0 tx=0).stop 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 shutdown_connections 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f945c06b4d0 0x7f945c06d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f94700d34a0 0x7f94700d3ee0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94700b9410 0x7f94700d39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 --2- 172.21.15.99:0/416970616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f94700b5700 0x7f94700c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 >> 172.21.15.99:0/416970616 conn(0x7f94700d0490 msgr2=0x7f94700d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 shutdown_connections 2024-03-04T10:12:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.561+0000 7f9475e6c640 1 -- 172.21.15.99:0/416970616 wait complete. 2024-03-04T10:12:02.575 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T10:12:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.681+0000 7f97761e1640 1 Processor -- start 2024-03-04T10:12:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.681+0000 7f97761e1640 1 -- start start 2024-03-04T10:12:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f97761e1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f977006c7f0 0x7f977009a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f97761e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977015efd0 0x7f97700d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f97761e1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9770161dd0 con 0x7f9770161940 2024-03-04T10:12:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f97761e1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97700cfa80 con 0x7f977006c7f0 2024-03-04T10:12:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f97761e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97701628e0 con 0x7f977015efd0 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f977006c7f0 0x7f977009a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f977006c7f0 0x7f977009a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47272/0 (socket says 172.21.15.99:47272) 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 -- 172.21.15.99:0/2342240804 learned_addr learned my addr 172.21.15.99:0/2342240804 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976effd640 1 --1- 172.21.15.99:0/2342240804 >> v1:172.21.15.99:6789/0 conn(0x7f9770161940 0x7f977006cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:42630/0 (socket says 172.21.15.99:42630) 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976ffff640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977015efd0 0x7f97700d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 -- 172.21.15.99:0/2342240804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977015efd0 msgr2=0x7f97700d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977015efd0 0x7f97700d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 -- 172.21.15.99:0/2342240804 >> v1:172.21.15.99:6789/0 conn(0x7f9770161940 legacy=0x7f977006cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976e7fc640 1 -- 172.21.15.99:0/2342240804 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005085655 0 0) 0x7f9770161dd0 con 0x7f9770161940 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976f7fe640 1 -- 172.21.15.99:0/2342240804 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9770162040 con 0x7f977006c7f0 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.682+0000 7f976ffff640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977015efd0 0x7f97700d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.683+0000 7f976f7fe640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977006c7f0 0x7f977009a4f0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f97600098a0 tx=0x7f976002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e209b677cf319f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.683+0000 7f976e7fc640 1 -- 172.21.15.99:0/2342240804 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760034070 con 0x7f977006c7f0 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.683+0000 7f976e7fc640 1 -- 172.21.15.99:0/2342240804 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9760030840 con 0x7f977006c7f0 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.683+0000 7f976e7fc640 1 -- 172.21.15.99:0/2342240804 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9760030c10 con 0x7f977006c7f0 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 -- 172.21.15.99:0/2342240804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977006c7f0 msgr2=0x7f977009a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977006c7f0 0x7f977009a4f0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f97600098a0 tx=0x7f976002b980 comp rx=0 tx=0).stop 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 -- 172.21.15.99:0/2342240804 shutdown_connections 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977015efd0 0x7f97700d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 --2- 172.21.15.99:0/2342240804 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977006c7f0 0x7f977009a4f0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 -- 172.21.15.99:0/2342240804 >> 172.21.15.99:0/2342240804 conn(0x7f97700d1840 msgr2=0x7f97700d1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 -- 172.21.15.99:0/2342240804 shutdown_connections 2024-03-04T10:12:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.684+0000 7f97761e1640 1 -- 172.21.15.99:0/2342240804 wait complete. 2024-03-04T10:12:02.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.685+0000 7f97761e1640 1 Processor -- start 2024-03-04T10:12:02.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.685+0000 7f97761e1640 1 -- start start 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f97761e1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977006c7f0 0x7f97700cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f97761e1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 0x7f97700c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f97761e1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9770161940 0x7f977015be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f97761e1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97700cfcb0 con 0x7f977015efd0 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f97761e1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9770162600 con 0x7f9770161940 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f97761e1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97700d0710 con 0x7f977006c7f0 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 0x7f97700c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 0x7f97700c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:47288/0 (socket says 172.21.15.99:47288) 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 -- 172.21.15.99:0/162213219 learned_addr learned my addr 172.21.15.99:0/162213219 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976f7fe640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977006c7f0 0x7f97700cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976ffff640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9770161940 0x7f977015be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 -- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977006c7f0 msgr2=0x7f97700cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977006c7f0 0x7f97700cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 -- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9770161940 msgr2=0x7f977015be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9770161940 0x7f977015be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 -- 172.21.15.99:0/162213219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977015c940 con 0x7f977015efd0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.686+0000 7f976effd640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 0x7f97700c9200 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f976400c3f0 tx=0x7f976400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.687+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640074e0 con 0x7f977015efd0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.687+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97640047e0 con 0x7f977015efd0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.687+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97700ca600 con 0x7f977015efd0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.687+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764004c10 con 0x7f977015efd0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.687+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97700cabc0 con 0x7f977015efd0 2024-03-04T10:12:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.688+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977006b1c0 con 0x7f977015efd0 2024-03-04T10:12:02.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.688+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9764011070 con 0x7f977015efd0 2024-03-04T10:12:02.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.688+0000 7f9757fff640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f975c06b4d0 0x7f975c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:02.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.689+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 6905+0+0 (secure 0 0 0) 0x7f976408a1c0 con 0x7f977015efd0 2024-03-04T10:12:02.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.689+0000 7f976f7fe640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f975c06b4d0 0x7f975c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:02.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.689+0000 7f976f7fe640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f975c06b4d0 0x7f975c06d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9760036000 tx=0x7f976003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:02.694 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.693+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f977006b1c0 con 0x7f977015efd0 2024-03-04T10:12:02.863 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:02.862+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f97700d2750 con 0x7f977015efd0 2024-03-04T10:12:03.564 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.562+0000 7f9757fff640 1 -- 172.21.15.99:0/162213219 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) v1 ==== 181+0+0 (secure 0 0 0) 0x7f976401a2b0 con 0x7f977015efd0 2024-03-04T10:12:03.564 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_metadata' removed 2024-03-04T10:12:03.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f975c06b4d0 msgr2=0x7f975c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:03.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f975c06b4d0 0x7f975c06d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9760036000 tx=0x7f976003b040 comp rx=0 tx=0).stop 2024-03-04T10:12:03.566 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 msgr2=0x7f97700c9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 0x7f97700c9200 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f976400c3f0 tx=0x7f976400c8b0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 shutdown_connections 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f975c06b4d0 0x7f975c06d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9770161940 0x7f977015be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f977015efd0 0x7f97700c9200 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 --2- 172.21.15.99:0/162213219 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f977006c7f0 0x7f97700cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.564+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 >> 172.21.15.99:0/162213219 conn(0x7f97700d1840 msgr2=0x7f97700cbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.565+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 shutdown_connections 2024-03-04T10:12:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.565+0000 7f97761e1640 1 -- 172.21.15.99:0/162213219 wait complete. 2024-03-04T10:12:03.580 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T10:12:03.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.686+0000 7f17d32a4640 1 Processor -- start 2024-03-04T10:12:03.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.686+0000 7f17d32a4640 1 -- start start 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d32a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f17cc0b3fd0 0x7f17cc09a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d32a4640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17cc0b2830 0x7f17cc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d32a4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17cc09b170 con 0x7f17cc1612a0 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d32a4640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0d0e70 con 0x7f17cc0b2830 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d32a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc09a7f0 con 0x7f17cc0b3fd0 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1841640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17cc0b2830 0x7f17cc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1040640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f17cc1612a0 0x7f17cc099a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56900/0 (socket says 172.21.15.99:56900) 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1040640 1 -- 172.21.15.99:0/490644553 learned_addr learned my addr 172.21.15.99:0/490644553 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1841640 1 -- 172.21.15.99:0/490644553 >> v2:172.21.15.162:3300/0 conn(0x7f17cc0b3fd0 msgr2=0x7f17cc09a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1841640 1 --2- 172.21.15.99:0/490644553 >> v2:172.21.15.162:3300/0 conn(0x7f17cc0b3fd0 0x7f17cc09a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1841640 1 -- 172.21.15.99:0/490644553 >> v1:172.21.15.99:6789/0 conn(0x7f17cc1612a0 legacy=0x7f17cc099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:03.689 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17c3fff640 1 -- 172.21.15.99:0/490644553 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078600532 0 0) 0x7f17cc09b170 con 0x7f17cc1612a0 2024-03-04T10:12:03.690 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.687+0000 7f17d1841640 1 -- 172.21.15.99:0/490644553 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc0d0540 con 0x7f17cc0b2830 2024-03-04T10:12:03.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.688+0000 7f17d1841640 1 --2- 172.21.15.99:0/490644553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc0b2830 0x7f17cc0cbe80 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0cde70 tx=0x7f17c402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a630f6af1865933f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:03.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.688+0000 7f17c3fff640 1 -- 172.21.15.99:0/490644553 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4039070 con 0x7f17cc0b2830 2024-03-04T10:12:03.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.688+0000 7f17c3fff640 1 -- 172.21.15.99:0/490644553 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c402cbf0 con 0x7f17cc0b2830 2024-03-04T10:12:03.691 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.688+0000 7f17c3fff640 1 -- 172.21.15.99:0/490644553 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4034440 con 0x7f17cc0b2830 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.688+0000 7f17d32a4640 1 -- 172.21.15.99:0/490644553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc0b2830 msgr2=0x7f17cc0cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.688+0000 7f17d32a4640 1 --2- 172.21.15.99:0/490644553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc0b2830 0x7f17cc0cbe80 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0cde70 tx=0x7f17c402be40 comp rx=0 tx=0).stop 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.689+0000 7f17d32a4640 1 -- 172.21.15.99:0/490644553 shutdown_connections 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.689+0000 7f17d32a4640 1 --2- 172.21.15.99:0/490644553 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc0b2830 0x7f17cc0cbe80 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.689+0000 7f17d32a4640 1 --2- 172.21.15.99:0/490644553 >> v2:172.21.15.162:3300/0 conn(0x7f17cc0b3fd0 0x7f17cc09a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.689+0000 7f17d32a4640 1 -- 172.21.15.99:0/490644553 >> 172.21.15.99:0/490644553 conn(0x7f17cc0b3320 msgr2=0x7f17cc0b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.689+0000 7f17d32a4640 1 -- 172.21.15.99:0/490644553 shutdown_connections 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.689+0000 7f17d32a4640 1 -- 172.21.15.99:0/490644553 wait complete. 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 Processor -- start 2024-03-04T10:12:03.692 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 -- start start 2024-03-04T10:12:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17cc0b2830 0x7f17cc0b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17cc0b3fd0 0x7f17cc14ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 0x7f17cc14b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc161680 con 0x7f17cc1612a0 2024-03-04T10:12:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0d0b00 con 0x7f17cc0b3fd0 2024-03-04T10:12:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.690+0000 7f17d32a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc0d1300 con 0x7f17cc0b2830 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 0x7f17cc14b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 0x7f17cc14b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39876/0 (socket says 172.21.15.99:39876) 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 -- 172.21.15.99:0/4239838280 learned_addr learned my addr 172.21.15.99:0/4239838280 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d083f640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17cc0b3fd0 0x7f17cc14ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1040640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17cc0b2830 0x7f17cc0b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 -- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17cc0b2830 msgr2=0x7f17cc0b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17cc0b2830 0x7f17cc0b6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 -- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17cc0b3fd0 msgr2=0x7f17cc14ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17cc0b3fd0 0x7f17cc14ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 -- 172.21.15.99:0/4239838280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc0c8e00 con 0x7f17cc1612a0 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d1841640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 0x7f17cc14b240 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f17c4009e80 tx=0x7f17c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4044020 con 0x7f17cc1612a0 2024-03-04T10:12:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c403f470 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc0c70a0 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.691+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4039040 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.692+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc0c7660 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.692+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc074f60 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.693+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f17c4047020 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.693+0000 7f17c1ffb640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.693+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6467+0+0 (secure 0 0 0) 0x7f17c40ad110 con 0x7f17cc1612a0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.693+0000 7f17d1040640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:03.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.694+0000 7f17d1040640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17b006b4d0 0x7f17b006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f17c8007fd0 tx=0x7f17c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:03.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.697+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f17c404e030 con 0x7f17cc1612a0 2024-03-04T10:12:03.868 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:03.866+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17cc15e860 con 0x7f17cc1612a0 2024-03-04T10:12:04.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~51s 2024-03-04T10:12:04.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~71s 2024-03-04T10:12:04.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~19s 2024-03-04T10:12:04.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~38s 2024-03-04T10:12:04.573 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.571+0000 7f17c1ffb640 1 -- 172.21.15.99:0/4239838280 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) v1 ==== 169+0+0 (secure 0 0 0) 0x7f17c407bd20 con 0x7f17cc1612a0 2024-03-04T10:12:04.573 INFO:teuthology.orchestra.run.smithi099.stderr:pool 'cephfs_data' removed 2024-03-04T10:12:04.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17b006b4d0 msgr2=0x7f17b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17b006b4d0 0x7f17b006d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f17c8007fd0 tx=0x7f17c8009040 comp rx=0 tx=0).stop 2024-03-04T10:12:04.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 msgr2=0x7f17cc14b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.575 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 0x7f17cc14b240 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f17c4009e80 tx=0x7f17c4004a40 comp rx=0 tx=0).stop 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 shutdown_connections 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17cc1612a0 0x7f17cc14b240 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17cc0b3fd0 0x7f17cc14ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 --2- 172.21.15.99:0/4239838280 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17cc0b2830 0x7f17cc0b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 >> 172.21.15.99:0/4239838280 conn(0x7f17cc0b3320 msgr2=0x7f17cc06f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 shutdown_connections 2024-03-04T10:12:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.573+0000 7f17d32a4640 1 -- 172.21.15.99:0/4239838280 wait complete. 2024-03-04T10:12:04.590 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-04T10:12:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.692+0000 7fda88280640 1 Processor -- start 2024-03-04T10:12:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.692+0000 7fda88280640 1 -- start start 2024-03-04T10:12:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda88280640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fda800b48a0 0x7fda800b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda88280640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fda8006d3c0 0x7fda800b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda88280640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8006d8a0 con 0x7fda8015a810 2024-03-04T10:12:04.697 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda88280640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8006db80 con 0x7fda8006d3c0 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda88280640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8006de60 con 0x7fda800b48a0 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8681d640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fda8006d3c0 0x7fda800b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8601c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fda8015a810 0x7fda800b4190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56904/0 (socket says 172.21.15.99:56904) 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8601c640 1 -- 172.21.15.99:0/2111362962 learned_addr learned my addr 172.21.15.99:0/2111362962 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8581b640 1 --2- 172.21.15.99:0/2111362962 >> v2:172.21.15.162:3300/0 conn(0x7fda800b48a0 0x7fda800b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8681d640 1 -- 172.21.15.99:0/2111362962 >> v2:172.21.15.162:3300/0 conn(0x7fda800b48a0 msgr2=0x7fda800b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8681d640 1 --2- 172.21.15.99:0/2111362962 >> v2:172.21.15.162:3300/0 conn(0x7fda800b48a0 0x7fda800b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8681d640 1 -- 172.21.15.99:0/2111362962 >> v1:172.21.15.99:6789/0 conn(0x7fda8015a810 legacy=0x7fda800b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8681d640 1 -- 172.21.15.99:0/2111362962 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8006e140 con 0x7fda8006d3c0 2024-03-04T10:12:04.698 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8501a640 1 -- 172.21.15.99:0/2111362962 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254965486 0 0) 0x7fda8006d8a0 con 0x7fda8015a810 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.693+0000 7fda8581b640 1 --2- 172.21.15.99:0/2111362962 >> v2:172.21.15.162:3300/0 conn(0x7fda800b48a0 0x7fda800b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.694+0000 7fda8681d640 1 --2- 172.21.15.99:0/2111362962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda800b2f30 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fda800b4d00 tx=0x7fda7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9ff2f1fd8e720b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.694+0000 7fda8501a640 1 -- 172.21.15.99:0/2111362962 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78035020 con 0x7fda8006d3c0 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.694+0000 7fda8501a640 1 -- 172.21.15.99:0/2111362962 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda78031830 con 0x7fda8006d3c0 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.694+0000 7fda8501a640 1 -- 172.21.15.99:0/2111362962 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78031c00 con 0x7fda8006d3c0 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 -- 172.21.15.99:0/2111362962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 msgr2=0x7fda800b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 --2- 172.21.15.99:0/2111362962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda800b2f30 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fda800b4d00 tx=0x7fda7802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 -- 172.21.15.99:0/2111362962 shutdown_connections 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 --2- 172.21.15.99:0/2111362962 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda800b2f30 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 --2- 172.21.15.99:0/2111362962 >> v2:172.21.15.162:3300/0 conn(0x7fda800b48a0 0x7fda800b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 -- 172.21.15.99:0/2111362962 >> 172.21.15.99:0/2111362962 conn(0x7fda8006acd0 msgr2=0x7fda8006b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:04.699 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 -- 172.21.15.99:0/2111362962 shutdown_connections 2024-03-04T10:12:04.700 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.695+0000 7fda88280640 1 -- 172.21.15.99:0/2111362962 wait complete. 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.696+0000 7fda88280640 1 Processor -- start 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.696+0000 7fda88280640 1 -- start start 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda80071590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda800b48a0 0x7fda8013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8015a810 0x7fda8013d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda800cc130 con 0x7fda8006d3c0 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda800cbfb0 con 0x7fda8015a810 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda80155150 con 0x7fda800b48a0 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda80071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda80071590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39898/0 (socket says 172.21.15.99:39898) 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 -- 172.21.15.99:0/3996491019 learned_addr learned my addr 172.21.15.99:0/3996491019 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8681d640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8015a810 0x7fda8013d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8581b640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda800b48a0 0x7fda8013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 -- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda800b48a0 msgr2=0x7fda8013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda800b48a0 0x7fda8013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 -- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8015a810 msgr2=0x7fda8013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8015a810 0x7fda8013d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 -- 172.21.15.99:0/3996491019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80155420 con 0x7fda8006d3c0 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8581b640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda800b48a0 0x7fda8013cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8681d640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8015a810 0x7fda8013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda8601c640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda80071590 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c7f0 tx=0x7fda7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c018070 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c00a640 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda80155700 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.697+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c00aa30 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.698+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda80155cc0 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.698+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda80067610 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.699+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fda7c01b070 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.699+0000 7fda76ffd640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda6406b4d0 0x7fda6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.700+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fda7c089910 con 0x7fda8006d3c0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.700+0000 7fda8581b640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda6406b4d0 0x7fda6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.700+0000 7fda8581b640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda6406b4d0 0x7fda6406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fda70006fd0 tx=0x7fda70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:04.706 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.704+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fda80067610 con 0x7fda8006d3c0 2024-03-04T10:12:04.874 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.873+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fda80074f60 con 0x7fda8006d3c0 2024-03-04T10:12:04.880 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.878+0000 7fda76ffd640 1 -- 172.21.15.99:0/3996491019 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7fda7c0586a0 con 0x7fda8006d3c0 2024-03-04T10:12:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.880+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda6406b4d0 msgr2=0x7fda6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.880+0000 7fda88280640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda6406b4d0 0x7fda6406d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fda70006fd0 tx=0x7fda70008040 comp rx=0 tx=0).stop 2024-03-04T10:12:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.880+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 msgr2=0x7fda80071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.880+0000 7fda88280640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda80071590 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c7f0 tx=0x7fda7c00ccb0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 shutdown_connections 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fda6406b4d0 0x7fda6406d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda8015a810 0x7fda8013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fda800b48a0 0x7fda8013cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 --2- 172.21.15.99:0/3996491019 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fda8006d3c0 0x7fda80071590 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 >> 172.21.15.99:0/3996491019 conn(0x7fda8006acd0 msgr2=0x7fda800b38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 shutdown_connections 2024-03-04T10:12:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.881+0000 7fda88280640 1 -- 172.21.15.99:0/3996491019 wait complete. 2024-03-04T10:12:04.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion' 2024-03-04T10:12:04.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.991+0000 7f9321b29640 1 Processor -- start 2024-03-04T10:12:04.993 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.991+0000 7f9321b29640 1 -- start start 2024-03-04T10:12:04.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f9321b29640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f931c15d660 0x7f931c15e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f9321b29640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f931c0aeb20 0x7f931c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.995 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f9321b29640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c071890 0x7f931c0cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f9321b29640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c15da40 con 0x7f931c15d660 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f9321b29640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c15ebb0 con 0x7f931c0aeb20 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f9321b29640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0af090 con 0x7f931c071890 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f931c15d660 0x7f931c15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f931c15d660 0x7f931c15e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39910/0 (socket says 172.21.15.99:39910) 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 -- 172.21.15.99:0/3674911642 learned_addr learned my addr 172.21.15.99:0/3674911642 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931affd640 1 --2- 172.21.15.99:0/3674911642 >> v2:172.21.15.162:3300/0 conn(0x7f931c0aeb20 0x7f931c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931bfff640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c071890 0x7f931c0cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 -- 172.21.15.99:0/3674911642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c071890 msgr2=0x7f931c0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c071890 0x7f931c0cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 -- 172.21.15.99:0/3674911642 >> v2:172.21.15.162:3300/0 conn(0x7f931c0aeb20 msgr2=0x7f931c071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:04.996 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 --2- 172.21.15.99:0/3674911642 >> v2:172.21.15.162:3300/0 conn(0x7f931c0aeb20 0x7f931c071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931b7fe640 1 -- 172.21.15.99:0/3674911642 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c15a9f0 con 0x7f931c15d660 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.992+0000 7f931bfff640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c071890 0x7f931c0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.993+0000 7f931b7fe640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c15d660 0x7f931c15e670 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f931c06d200 tx=0x7f930c02dfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1ccad0c61e1087 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.993+0000 7f931a7fc640 1 -- 172.21.15.99:0/3674911642 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c03b070 con 0x7f931c15d660 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.993+0000 7f931a7fc640 1 -- 172.21.15.99:0/3674911642 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c02ed70 con 0x7f931c15d660 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.993+0000 7f931a7fc640 1 -- 172.21.15.99:0/3674911642 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c036660 con 0x7f931c15d660 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 -- 172.21.15.99:0/3674911642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c15d660 msgr2=0x7f931c15e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c15d660 0x7f931c15e670 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f931c06d200 tx=0x7f930c02dfc0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 -- 172.21.15.99:0/3674911642 shutdown_connections 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c071890 0x7f931c0cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 --2- 172.21.15.99:0/3674911642 >> v2:172.21.15.162:3300/0 conn(0x7f931c0aeb20 0x7f931c071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 --2- 172.21.15.99:0/3674911642 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c15d660 0x7f931c15e670 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:04.997 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 -- 172.21.15.99:0/3674911642 >> 172.21.15.99:0/3674911642 conn(0x7f931c06acd0 msgr2=0x7f931c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:04.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 -- 172.21.15.99:0/3674911642 shutdown_connections 2024-03-04T10:12:04.998 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.994+0000 7f9321b29640 1 -- 172.21.15.99:0/3674911642 wait complete. 2024-03-04T10:12:05.000 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.995+0000 7f9321b29640 1 Processor -- start 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.995+0000 7f9321b29640 1 -- start start 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 0x7f931c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f931c0aeb20 0x7f931c0ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c15d660 0x7f931c0b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c15a660 con 0x7f931c071890 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c15afb0 con 0x7f931c0aeb20 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c15b4d0 con 0x7f931c15d660 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 0x7f931c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 0x7f931c0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39926/0 (socket says 172.21.15.99:39926) 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 -- 172.21.15.99:0/2685750563 learned_addr learned my addr 172.21.15.99:0/2685750563 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931bfff640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c15d660 0x7f931c0b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:05.001 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931affd640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f931c0aeb20 0x7f931c0ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 -- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c15d660 msgr2=0x7f931c0b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c15d660 0x7f931c0b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 -- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f931c0aeb20 msgr2=0x7f931c0ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f931c0aeb20 0x7f931c0ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 -- 172.21.15.99:0/2685750563 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0cef40 con 0x7f931c071890 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931bfff640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c15d660 0x7f931c0b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931affd640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f931c0aeb20 0x7f931c0ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f931b7fe640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 0x7f931c0ad930 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f930c02e000 tx=0x7f930c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c03b040 con 0x7f931c071890 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f930c046070 con 0x7f931c071890 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930c03f890 con 0x7f931c071890 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0cf220 con 0x7f931c071890 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.996+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0ade70 con 0x7f931c071890 2024-03-04T10:12:05.002 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.997+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c0cf610 con 0x7f931c071890 2024-03-04T10:12:05.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.998+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f930c02ec10 con 0x7f931c071890 2024-03-04T10:12:05.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.998+0000 7f9303fff640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f930806b4d0 0x7f930806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:05.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.998+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f930c0aeb80 con 0x7f931c071890 2024-03-04T10:12:05.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.998+0000 7f931affd640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f930806b4d0 0x7f930806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:05.003 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:04.999+0000 7f931affd640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f930806b4d0 0x7f930806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9310006fd0 tx=0x7f9310008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:05.004 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.003+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f930c07d9a0 con 0x7f931c071890 2024-03-04T10:12:05.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.237+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]} v 0) v1 -- 0x7f931c0cf3f0 con 0x7f931c071890 2024-03-04T10:12:05.574 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.572+0000 7f9303fff640 1 -- 172.21.15.99:0/2685750563 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_damage.TestDamage.test_object_deletion"]}]=0 v409) v1 ==== 135+0+0 (secure 0 0 0) 0x7f930c07d2a0 con 0x7f931c071890 2024-03-04T10:12:05.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.574+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f930806b4d0 msgr2=0x7f930806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:05.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.574+0000 7f9321b29640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f930806b4d0 0x7f930806d990 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f9310006fd0 tx=0x7f9310008040 comp rx=0 tx=0).stop 2024-03-04T10:12:05.576 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.574+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 msgr2=0x7f931c0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.574+0000 7f9321b29640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 0x7f931c0ad930 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f930c02e000 tx=0x7f930c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 shutdown_connections 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f930806b4d0 0x7f930806d990 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f931c15d660 0x7f931c0b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f931c0aeb20 0x7f931c0ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 --2- 172.21.15.99:0/2685750563 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f931c071890 0x7f931c0ad930 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 >> 172.21.15.99:0/2685750563 conn(0x7f931c06acd0 msgr2=0x7f931c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 shutdown_connections 2024-03-04T10:12:05.577 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:05.575+0000 7f9321b29640 1 -- 172.21.15.99:0/2685750563 wait complete. 2024-03-04T10:12:05.590 INFO:tasks.cephfs_test_runner:That the MDS has a clean 'damaged' response to loss of any single metadata object ... ERROR 2024-03-04T10:12:05.590 INFO:tasks.cephfs_test_runner: 2024-03-04T10:12:05.590 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-04T10:12:05.590 INFO:tasks.cephfs_test_runner:ERROR: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:12:05.590 INFO:tasks.cephfs_test_runner:That the MDS has a clean 'damaged' response to loss of any single metadata object 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs/test_damage.py", line 273, in test_object_deletion 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: self.wait_until_true(lambda: self.is_marked_damaged(0) or self.fs.are_daemons_healthy(), startup_timeout) 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/ceph_test_case.py", line 337, in wait_until_true 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: if condition(): 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs/test_damage.py", line 273, in 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: self.wait_until_true(lambda: self.is_marked_damaged(0) or self.fs.are_daemons_healthy(), startup_timeout) 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs/filesystem.py", line 1037, in are_daemons_healthy 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: mds.check_status() 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 43, in check_status 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: return self.proc.poll() 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 211, in poll 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-03-04T10:12:05.591 INFO:tasks.cephfs_test_runner: 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:Ran 5 tests in 459.576s 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner: 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner: 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:ERROR: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:That the MDS has a clean 'damaged' response to loss of any single metadata object 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs/test_damage.py", line 273, in test_object_deletion 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner: self.wait_until_true(lambda: self.is_marked_damaged(0) or self.fs.are_daemons_healthy(), startup_timeout) 2024-03-04T10:12:05.592 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/ceph_test_case.py", line 337, in wait_until_true 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: if condition(): 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs/test_damage.py", line 273, in 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: self.wait_until_true(lambda: self.is_marked_damaged(0) or self.fs.are_daemons_healthy(), startup_timeout) 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs/filesystem.py", line 1037, in are_daemons_healthy 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: mds.check_status() 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 43, in check_status 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: return self.proc.poll() 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 211, in poll 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-03-04T10:12:05.593 INFO:tasks.cephfs_test_runner: 2024-03-04T10:12:05.594 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:12:06.002 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=54c51a04d1ef4f5785aecd47b768d98d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2024-03-04T10:12:06.004 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-04T10:12:06.013 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-04T10:12:06.031 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-04T10:12:06.032 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:12:06.032 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:12:06.051 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:12:06.052 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-03-04T10:12:06.052 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:12:06.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056356,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=700 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-04T10:12:06.109 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:12:06.109 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:12:06.128 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:12:06.128 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-04T10:12:06.128 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T10:12:06.186 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20040 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-04T10:12:06.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.187 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:12:06.204 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-04T10:12:06.205 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T10:12:06.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.205 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-04T10:12:06.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.276 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T10:12:06.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.347 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link delete ceph-brx 2024-03-04T10:12:06.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.447 DEBUG:teuthology.orchestra.run.smithi099:> ip netns list 2024-03-04T10:12:06.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.463 DEBUG:teuthology.orchestra.run.smithi099:> sudo ip link delete ceph-brx 2024-03-04T10:12:06.534 INFO:teuthology.orchestra.run.smithi099.stderr:Cannot find device "ceph-brx" 2024-03-04T10:12:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:06.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.537 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list 2024-03-04T10:12:06.556 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-04T10:12:06.556 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-04T10:12:06.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-04T10:12:06.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-04T10:12:06.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link delete ceph-brx 2024-03-04T10:12:06.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.812 DEBUG:teuthology.orchestra.run.smithi162:> ip netns list 2024-03-04T10:12:06.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T10:12:06.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo ip link delete ceph-brx 2024-03-04T10:12:06.901 INFO:teuthology.orchestra.run.smithi162.stderr:Cannot find device "ceph-brx" 2024-03-04T10:12:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:06.903 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-04T10:12:06.918 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-04T10:12:06.919 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:12:07.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.025+0000 7f7d197c4640 1 Processor -- start 2024-03-04T10:12:07.027 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.025+0000 7f7d197c4640 1 -- start start 2024-03-04T10:12:07.033 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d197c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 0x7f7d1415e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d197c4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7d140cf650 0x7f7d140cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d197c4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d140cdaa0 con 0x7f7d141616c0 2024-03-04T10:12:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d197c4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d140cfff0 con 0x7f7d140cf650 2024-03-04T10:12:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d197c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14071df0 con 0x7f7d14154a80 2024-03-04T10:12:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d137fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f7d141616c0 0x7f7d14161b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56920/0 (socket says 172.21.15.99:56920) 2024-03-04T10:12:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d137fe640 1 -- 172.21.15.99:0/1190598013 learned_addr learned my addr 172.21.15.99:0/1190598013 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d127fc640 1 --2- 172.21.15.99:0/1190598013 >> v2:172.21.15.162:3300/0 conn(0x7f7d140cf650 0x7f7d140cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416566544 0 0) 0x7f7d140cdaa0 con 0x7f7d141616c0 2024-03-04T10:12:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cfc003520 con 0x7f7d141616c0 2024-03-04T10:12:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.026+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 0x7f7d1415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2396857141 0 0) 0x7f7cfc003520 con 0x7f7d141616c0 2024-03-04T10:12:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 msgr2=0x7f7d1415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 0x7f7d1415e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 >> v2:172.21.15.162:3300/0 conn(0x7f7d140cf650 msgr2=0x7f7d140cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 --2- 172.21.15.99:0/1190598013 >> v2:172.21.15.162:3300/0 conn(0x7f7d140cf650 0x7f7d140cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d140cd010 con 0x7f7d141616c0 2024-03-04T10:12:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f7d0c0019f0 con 0x7f7d141616c0 2024-03-04T10:12:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 >> v1:172.21.15.99:6789/0 conn(0x7f7d141616c0 legacy=0x7f7d14161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7cfc0039e0 0x7f7cfc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 0x7f7cfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc004d40 0x7f7cfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d14071df0 con 0x7f7cfc0039e0 2024-03-04T10:12:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc00b9c0 con 0x7f7cfc004d40 2024-03-04T10:12:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc00bb40 con 0x7f7d14154a80 2024-03-04T10:12:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7cfc0039e0 0x7f7cfc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d127fc640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 0x7f7cfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d137fe640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc004d40 0x7f7cfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d12ffd640 1 -- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 msgr2=0x7f7cfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d14154a80 0x7f7cfc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d12ffd640 1 -- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc004d40 msgr2=0x7f7cfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc004d40 0x7f7cfc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.027+0000 7f7d12ffd640 1 -- 172.21.15.99:0/1190598013 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc003520 con 0x7f7cfc0039e0 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7cfc0039e0 0x7f7cfc003e00 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009af0 tx=0x7f7cfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65209bfaeb6d9b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d04039070 con 0x7f7cfc0039e0 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0402c6b0 con 0x7f7cfc0039e0 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d11ffb640 1 -- 172.21.15.99:0/1190598013 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d04009690 con 0x7f7cfc0039e0 2024-03-04T10:12:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d197c4640 1 -- 172.21.15.99:0/1190598013 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7cfc0039e0 msgr2=0x7f7cfc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d197c4640 1 --2- 172.21.15.99:0/1190598013 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7cfc0039e0 0x7f7cfc003e00 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009af0 tx=0x7f7cfc005830 comp rx=0 tx=0).stop 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d137fe640 1 -- 172.21.15.99:0/1190598013 reap_dead start 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d197c4640 1 -- 172.21.15.99:0/1190598013 shutdown_connections 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d197c4640 1 -- 172.21.15.99:0/1190598013 >> 172.21.15.99:0/1190598013 conn(0x7f7d1406f5f0 msgr2=0x7f7d1406fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d197c4640 1 -- 172.21.15.99:0/1190598013 shutdown_connections 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.028+0000 7f7d197c4640 1 -- 172.21.15.99:0/1190598013 wait complete. 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 Processor -- start 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 -- start start 2024-03-04T10:12:07.041 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc0039e0 0x7f7d14153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d141e7ef0 0x7f7d140b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 0x7f7d140c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140ce340 con 0x7f7d140b49d0 2024-03-04T10:12:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140ce060 con 0x7f7cfc0039e0 2024-03-04T10:12:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.030+0000 7f7d197c4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140ce620 con 0x7f7d141e7ef0 2024-03-04T10:12:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 0x7f7d140c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 0x7f7d140c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39958/0 (socket says 172.21.15.99:39958) 2024-03-04T10:12:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 -- 172.21.15.99:0/2995228958 learned_addr learned my addr 172.21.15.99:0/2995228958 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d127fc640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d141e7ef0 0x7f7d140b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc0039e0 0x7f7d14153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 -- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d141e7ef0 msgr2=0x7f7d140b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d141e7ef0 0x7f7d140b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 -- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc0039e0 msgr2=0x7f7d14153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc0039e0 0x7f7d14153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 -- 172.21.15.99:0/2995228958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d141e76f0 con 0x7f7d140b49d0 2024-03-04T10:12:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d137fe640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 0x7f7d140c7910 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f7d14161aa0 tx=0x7f7d0c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c00ebb0 con 0x7f7d140b49d0 2024-03-04T10:12:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c00ed70 con 0x7f7d140b49d0 2024-03-04T10:12:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d141e79d0 con 0x7f7d140b49d0 2024-03-04T10:12:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c014670 con 0x7f7d140b49d0 2024-03-04T10:12:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.031+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d140ccd90 con 0x7f7d140b49d0 2024-03-04T10:12:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.032+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d14074f60 con 0x7f7d140b49d0 2024-03-04T10:12:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.033+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7d0c014810 con 0x7f7d140b49d0 2024-03-04T10:12:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.033+0000 7f7cfb7fe640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d0006b4d0 0x7f7d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.033+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f7d0c089f80 con 0x7f7d140b49d0 2024-03-04T10:12:07.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.033+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d0006b4d0 0x7f7d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.034+0000 7f7d12ffd640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d0006b4d0 0x7f7d0006d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7d04036000 tx=0x7f7d04030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.037+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f7d0c0586c0 con 0x7f7d140b49d0 2024-03-04T10:12:07.207 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.205+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d141e82d0 con 0x7f7d0006b4d0 2024-03-04T10:12:07.208 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.207+0000 7f7cfb7fe640 1 -- 172.21.15.99:0/2995228958 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13459 (secure 0 0 0) 0x7f7d141e82d0 con 0x7f7d0006b4d0 2024-03-04T10:12:07.208 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:07.209 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:12:07.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d0006b4d0 msgr2=0x7f7d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d0006b4d0 0x7f7d0006d990 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7d04036000 tx=0x7f7d04030000 comp rx=0 tx=0).stop 2024-03-04T10:12:07.210 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 msgr2=0x7f7d140c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 0x7f7d140c7910 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f7d14161aa0 tx=0x7f7d0c00e530 comp rx=0 tx=0).stop 2024-03-04T10:12:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 shutdown_connections 2024-03-04T10:12:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7d0006b4d0 0x7f7d0006d990 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7d140b49d0 0x7f7d140c7910 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.211 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7d141e7ef0 0x7f7d140b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 --2- 172.21.15.99:0/2995228958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7cfc0039e0 0x7f7d14153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 >> 172.21.15.99:0/2995228958 conn(0x7f7d1406f5f0 msgr2=0x7f7d140b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 shutdown_connections 2024-03-04T10:12:07.212 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.208+0000 7f7d197c4640 1 -- 172.21.15.99:0/2995228958 wait complete. 2024-03-04T10:12:07.221 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-04T10:12:05.561515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272896,"kb_used_data":23456,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754701824,"statfs":{"total":773094113280,"available":772814667776,"internally_reserved":0,"allocated":24018944,"data_stored":18625470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6310911,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-260,"num_read_kb":-259,"num_write":-409,"num_write_kb":-6325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001336"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":340,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T10:12:01.558193+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:12:01.558193+0000","last_peered":"2024-03-04T10:12:01.558193+0000","last_clean":"2024-03-04T10:12:01.558193+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:12:01.558193+0000","last_undegraded":"2024-03-04T10:12:01.558193+0000","last_fullsized":"2024-03-04T10:12:01.558193+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1871872,"data_stored":1193288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":622125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36516,"kb_used_data":5336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335324,"statfs":{"total":96636764160,"available":96599371776,"internally_reserved":0,"allocated":5464064,"data_stored":4797168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36524,"kb_used_data":5344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335316,"statfs":{"total":96636764160,"available":96599363584,"internally_reserved":0,"allocated":5472256,"data_stored":4797328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34468,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337372,"statfs":{"total":96636764160,"available":96601468928,"internally_reserved":0,"allocated":3366912,"data_stored":2700016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":602864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1273856,"data_stored":602934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35084,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336756,"statfs":{"total":96636764160,"available":96600838144,"internally_reserved":0,"allocated":3997696,"data_stored":3309747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:12:07.221 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:12:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.319+0000 7f17e935b640 1 Processor -- start 2024-03-04T10:12:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.319+0000 7f17e935b640 1 -- start start 2024-03-04T10:12:07.323 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e935b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e40b2e60 0x7f17e40b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e935b640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17e40b5470 0x7f17e40b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e935b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e415e1e0 con 0x7f17e415de00 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e935b640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e40b98c0 con 0x7f17e40b5470 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e935b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40b32c0 con 0x7f17e40b2e60 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17e40b5470 0x7f17e40b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f17e40b5470 0x7f17e40b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39962/0 (socket says 172.21.15.99:39962) 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 -- 172.21.15.99:0/801104287 learned_addr learned my addr 172.21.15.99:0/801104287 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17dbfff640 1 --2- 172.21.15.99:0/801104287 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e40b2e60 0x7f17e40b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 -- 172.21.15.99:0/801104287 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e40b2e60 msgr2=0x7f17e40b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 --2- 172.21.15.99:0/801104287 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e40b2e60 0x7f17e40b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 -- 172.21.15.99:0/801104287 >> v1:172.21.15.162:6789/0 conn(0x7f17e415de00 legacy=0x7f17e40b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.320+0000 7f17e37fe640 1 -- 172.21.15.99:0/801104287 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40d1c10 con 0x7f17e40b5470 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.321+0000 7f17e37fe640 1 --2- 172.21.15.99:0/801104287 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40b5900 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009e50 tx=0x7f17dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f16a10c70f54ad6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.321+0000 7f17e27fc640 1 -- 172.21.15.99:0/801104287 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc02cb00 con 0x7f17e40b5470 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.321+0000 7f17e27fc640 1 -- 172.21.15.99:0/801104287 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc02cca0 con 0x7f17e40b5470 2024-03-04T10:12:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.321+0000 7f17e27fc640 1 -- 172.21.15.99:0/801104287 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc034610 con 0x7f17e40b5470 2024-03-04T10:12:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 -- 172.21.15.99:0/801104287 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 msgr2=0x7f17e40b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 --2- 172.21.15.99:0/801104287 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40b5900 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009e50 tx=0x7f17dc02bf60 comp rx=0 tx=0).stop 2024-03-04T10:12:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 -- 172.21.15.99:0/801104287 shutdown_connections 2024-03-04T10:12:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 --2- 172.21.15.99:0/801104287 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40b5900 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 --2- 172.21.15.99:0/801104287 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e40b2e60 0x7f17e40b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 -- 172.21.15.99:0/801104287 >> 172.21.15.99:0/801104287 conn(0x7f17e40d0470 msgr2=0x7f17e40d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 -- 172.21.15.99:0/801104287 shutdown_connections 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.322+0000 7f17e935b640 1 -- 172.21.15.99:0/801104287 wait complete. 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.323+0000 7f17e935b640 1 Processor -- start 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.323+0000 7f17e935b640 1 -- start start 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17e40b2e60 0x7f17e4151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e415de00 0x7f17e40b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40d21d0 con 0x7f17e40b5470 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40d2820 con 0x7f17e40b2e60 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e40d24b0 con 0x7f17e415de00 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39972/0 (socket says 172.21.15.99:39972) 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 -- 172.21.15.99:0/2865611089 learned_addr learned my addr 172.21.15.99:0/2865611089 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 -- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e415de00 msgr2=0x7f17e40b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e2ffd640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17e40b2e60 0x7f17e4151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e415de00 0x7f17e40b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 -- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17e40b2e60 msgr2=0x7f17e4151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17e40b2e60 0x7f17e4151110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 -- 172.21.15.99:0/2865611089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e415ee80 con 0x7f17e40b5470 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17dbfff640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40cc640 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00e9b0 tx=0x7f17cc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc017070 con 0x7f17e40b5470 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17cc00c9a0 con 0x7f17e40b5470 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc00cd90 con 0x7f17e40b5470 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e415f120 con 0x7f17e40b5470 2024-03-04T10:12:07.331 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.324+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e415f6e0 con 0x7f17e40b5470 2024-03-04T10:12:07.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.325+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e4074f60 con 0x7f17e40b5470 2024-03-04T10:12:07.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.326+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f17cc01a070 con 0x7f17e40b5470 2024-03-04T10:12:07.332 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.326+0000 7f17db7fe640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c406b500 0x7f17c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.326+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f17cc08a2e0 con 0x7f17e40b5470 2024-03-04T10:12:07.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.326+0000 7f17e2ffd640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c406b500 0x7f17c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.327+0000 7f17e2ffd640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c406b500 0x7f17c406d9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f17e41f6f90 tx=0x7f17d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.333 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.330+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f17cc059070 con 0x7f17e40b5470 2024-03-04T10:12:07.499 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.497+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17e40d0930 con 0x7f17c406b500 2024-03-04T10:12:07.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.498+0000 7f17db7fe640 1 -- 172.21.15.99:0/2865611089 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13459 (secure 0 0 0) 0x7f17e40d0930 con 0x7f17c406b500 2024-03-04T10:12:07.500 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:07.501 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:12:07.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c406b500 msgr2=0x7f17c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c406b500 0x7f17c406d9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f17e41f6f90 tx=0x7f17d4009040 comp rx=0 tx=0).stop 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 msgr2=0x7f17e40cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40cc640 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00e9b0 tx=0x7f17cc00ee70 comp rx=0 tx=0).stop 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 shutdown_connections 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f17c406b500 0x7f17c406d9c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f17e415de00 0x7f17e40b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f17e40b5470 0x7f17e40cc640 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 --2- 172.21.15.99:0/2865611089 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17e40b2e60 0x7f17e4151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 >> 172.21.15.99:0/2865611089 conn(0x7f17e40d0470 msgr2=0x7f17e40b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 shutdown_connections 2024-03-04T10:12:07.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.500+0000 7f17e935b640 1 -- 172.21.15.99:0/2865611089 wait complete. 2024-03-04T10:12:07.513 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-03-04T10:12:05.561515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272896,"kb_used_data":23456,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754701824,"statfs":{"total":773094113280,"available":772814667776,"internally_reserved":0,"allocated":24018944,"data_stored":18625470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6310911,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-260,"num_read_kb":-259,"num_write":-409,"num_write_kb":-6325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001336"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":340,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-04T10:12:01.558193+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:12:01.558193+0000","last_peered":"2024-03-04T10:12:01.558193+0000","last_clean":"2024-03-04T10:12:01.558193+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:12:01.558193+0000","last_undegraded":"2024-03-04T10:12:01.558193+0000","last_fullsized":"2024-03-04T10:12:01.558193+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1871872,"data_stored":1193288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":622125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36516,"kb_used_data":5336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335324,"statfs":{"total":96636764160,"available":96599371776,"internally_reserved":0,"allocated":5464064,"data_stored":4797168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36524,"kb_used_data":5344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335316,"statfs":{"total":96636764160,"available":96599363584,"internally_reserved":0,"allocated":5472256,"data_stored":4797328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34468,"kb_used_data":3288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337372,"statfs":{"total":96636764160,"available":96601468928,"internally_reserved":0,"allocated":3366912,"data_stored":2700016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32420,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339420,"statfs":{"total":96636764160,"available":96603566080,"internally_reserved":0,"allocated":1269760,"data_stored":602864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1273856,"data_stored":602934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35084,"kb_used_data":3904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336756,"statfs":{"total":96636764160,"available":96600838144,"internally_reserved":0,"allocated":3997696,"data_stored":3309747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:12:07.513 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-04T10:12:07.514 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:12:07.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.612+0000 7f465df74640 1 Processor -- start 2024-03-04T10:12:07.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.612+0000 7f465df74640 1 -- start start 2024-03-04T10:12:07.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f465df74640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f465806b7c0 0x7f465815c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f465df74640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46580cf700 con 0x7f465806acd0 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f465df74640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46580cf9a0 con 0x7f46580cc330 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f465df74640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46580d02d0 con 0x7f465806b7c0 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f46577fe640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f465806acd0 0x7f465806b0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56924/0 (socket says 172.21.15.99:56924) 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f46577fe640 1 -- 172.21.15.99:0/989285201 learned_addr learned my addr 172.21.15.99:0/989285201 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158815058 0 0) 0x7f46580cf700 con 0x7f465806acd0 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4644003520 con 0x7f465806acd0 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.613+0000 7f464effd640 1 --2- 172.21.15.99:0/989285201 >> v2:172.21.15.162:3300/0 conn(0x7f465806b7c0 0x7f465815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552685896 0 0) 0x7f4644003520 con 0x7f465806acd0 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 >> v2:172.21.15.162:3300/0 conn(0x7f465806b7c0 msgr2=0x7f465815c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 --2- 172.21.15.99:0/989285201 >> v2:172.21.15.162:3300/0 conn(0x7f465806b7c0 0x7f465815c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 >> v1:172.21.15.162:6789/0 conn(0x7f46580cc330 legacy=0x7f46580cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46580cfc80 con 0x7f465806acd0 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f4648001670 con 0x7f465806acd0 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 >> v1:172.21.15.99:6789/0 conn(0x7f465806acd0 legacy=0x7f465806b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440039b0 0x7f4644003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4644004800 0x7f4644004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46440051c0 0x7f464400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464400c130 con 0x7f46440051c0 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464400c2e0 con 0x7f4644004800 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464400c4e0 con 0x7f46440039b0 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4657fff640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46440051c0 0x7f464400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f464effd640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4644004800 0x7f4644004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f46577fe640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440039b0 0x7f4644003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4657fff640 1 -- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440039b0 msgr2=0x7f4644003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4657fff640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440039b0 0x7f4644003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4657fff640 1 -- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4644004800 msgr2=0x7f4644004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4657fff640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4644004800 0x7f4644004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.614+0000 7f4657fff640 1 -- 172.21.15.99:0/989285201 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644003520 con 0x7f46440051c0 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f4657fff640 1 -- 172.21.15.99:0/989285201 reap_dead start 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f4657fff640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46440051c0 0x7f464400b9c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f4650001e50 tx=0x7f4644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491190191bd1c3c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650001a70 con 0x7f46440051c0 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4650003b90 con 0x7f46440051c0 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f4656ffd640 1 -- 172.21.15.99:0/989285201 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46500050d0 con 0x7f46440051c0 2024-03-04T10:12:07.622 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f465df74640 1 -- 172.21.15.99:0/989285201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46440051c0 msgr2=0x7f464400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f465df74640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46440051c0 0x7f464400b9c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f4650001e50 tx=0x7f4644005830 comp rx=0 tx=0).stop 2024-03-04T10:12:07.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f465df74640 1 -- 172.21.15.99:0/989285201 shutdown_connections 2024-03-04T10:12:07.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.615+0000 7f465df74640 1 --2- 172.21.15.99:0/989285201 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46440051c0 0x7f464400b9c0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.616+0000 7f465df74640 1 -- 172.21.15.99:0/989285201 >> 172.21.15.99:0/989285201 conn(0x7f46580d0770 msgr2=0x7f46580d0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.616+0000 7f465df74640 1 -- 172.21.15.99:0/989285201 shutdown_connections 2024-03-04T10:12:07.623 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.616+0000 7f465df74640 1 -- 172.21.15.99:0/989285201 wait complete. 2024-03-04T10:12:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.617+0000 7f465df74640 1 Processor -- start 2024-03-04T10:12:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.617+0000 7f465df74640 1 -- start start 2024-03-04T10:12:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440051c0 0x7f46580b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46580b4d20 0x7f465814bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 0x7f46580c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4658159850 con 0x7f46580aeab0 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46580d13e0 con 0x7f46580b4d20 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46581599d0 con 0x7f46440051c0 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 0x7f46580c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 0x7f46580c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:39990/0 (socket says 172.21.15.99:39990) 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 -- 172.21.15.99:0/475697743 learned_addr learned my addr 172.21.15.99:0/475697743 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f46577fe640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440051c0 0x7f46580b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f464effd640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46580b4d20 0x7f465814bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 -- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440051c0 msgr2=0x7f46580b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440051c0 0x7f46580b5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 -- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46580b4d20 msgr2=0x7f465814bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46580b4d20 0x7f465814bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 -- 172.21.15.99:0/475697743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46580af390 con 0x7f46580aeab0 2024-03-04T10:12:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4657fff640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 0x7f46580c50f0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f465002c060 tx=0x7f46500014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650007db0 con 0x7f46580aeab0 2024-03-04T10:12:07.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4650006d50 con 0x7f46580aeab0 2024-03-04T10:12:07.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46580b5420 con 0x7f46580aeab0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650039680 con 0x7f46580aeab0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.618+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46580b59e0 con 0x7f46580aeab0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.619+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46580b5100 con 0x7f46580aeab0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.620+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f4650039820 con 0x7f46580aeab0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.620+0000 7f4654ff9640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f463806b500 0x7f463806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.620+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f46500ac170 con 0x7f46580aeab0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.620+0000 7f46577fe640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f463806b500 0x7f463806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.621+0000 7f46577fe640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f463806b500 0x7f463806d9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4648000f60 tx=0x7f4648000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.624+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f4650033030 con 0x7f46580aeab0 2024-03-04T10:12:07.792 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.790+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46580af670 con 0x7f463806b500 2024-03-04T10:12:07.794 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.792+0000 7f4654ff9640 1 -- 172.21.15.99:0/475697743 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13499 (secure 0 0 0) 0x7f46580af670 con 0x7f463806b500 2024-03-04T10:12:07.794 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:07.794 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:12:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f463806b500 msgr2=0x7f463806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f463806b500 0x7f463806d9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4648000f60 tx=0x7f4648000ed0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 msgr2=0x7f46580c50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 0x7f46580c50f0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f465002c060 tx=0x7f46500014a0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 shutdown_connections 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f463806b500 0x7f463806d9c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f46580aeab0 0x7f46580c50f0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f46580b4d20 0x7f465814bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 --2- 172.21.15.99:0/475697743 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f46440051c0 0x7f46580b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 >> 172.21.15.99:0/475697743 conn(0x7f46580d0770 msgr2=0x7f46580b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 shutdown_connections 2024-03-04T10:12:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.793+0000 7f465df74640 1 -- 172.21.15.99:0/475697743 wait complete. 2024-03-04T10:12:07.806 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-03-04T10:12:07.562045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260788,"kb_used_data":11336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713932,"statfs":{"total":773094113280,"available":772827066368,"internally_reserved":0,"allocated":11608064,"data_stored":6113568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6310911,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-260,"num_read_kb":-259,"num_write":-409,"num_write_kb":-6325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.006094"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":344,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-04T10:12:04.581744+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:12:04.581744+0000","last_peered":"2024-03-04T10:12:04.581744+0000","last_clean":"2024-03-04T10:12:04.581744+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:12:04.581744+0000","last_undegraded":"2024-03-04T10:12:04.581744+0000","last_fullsized":"2024-03-04T10:12:04.581744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1896448,"data_stored":1206972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1896448,"data_stored":1206972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:12:07.807 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T10:12:07.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.905+0000 7fc1a0d7e640 1 Processor -- start 2024-03-04T10:12:07.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.905+0000 7fc1a0d7e640 1 -- start start 2024-03-04T10:12:07.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc1a0d7e640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc19c154a80 0x7fc19c15e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc1a0d7e640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c1624e0 con 0x7fc19c0cfae0 2024-03-04T10:12:07.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc1a0d7e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0cd010 con 0x7fc19c0cf650 2024-03-04T10:12:07.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc1a0d7e640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc19c071df0 con 0x7fc19c154a80 2024-03-04T10:12:07.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19ad9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc19c0cfae0 0x7fc19c161dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56938/0 (socket says 172.21.15.99:56938) 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19ad9d640 1 -- 172.21.15.99:0/3048603395 learned_addr learned my addr 172.21.15.99:0/3048603395 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181029268 0 0) 0x7fc19c1624e0 con 0x7fc19c0cfae0 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19a59c640 1 --2- 172.21.15.99:0/3048603395 >> v2:172.21.15.162:3300/0 conn(0x7fc19c154a80 0x7fc19c15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184003520 con 0x7fc19c0cfae0 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204438260 0 0) 0x7fc184003520 con 0x7fc19c0cfae0 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 >> v2:172.21.15.162:3300/0 conn(0x7fc19c154a80 msgr2=0x7fc19c15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 --2- 172.21.15.99:0/3048603395 >> v2:172.21.15.162:3300/0 conn(0x7fc19c154a80 0x7fc19c15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 >> v1:172.21.15.162:6789/0 conn(0x7fc19c0cf650 legacy=0x7fc19c1616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0cff70 con 0x7fc19c0cfae0 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fc1940019f0 con 0x7fc19c0cfae0 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.906+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 >> v1:172.21.15.99:6789/0 conn(0x7fc19c0cfae0 legacy=0x7fc19c161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19959a640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1840039b0 0x7fc184003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19959a640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc184004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19959a640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1840051c0 0x7fc18400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c071df0 con 0x7fc184004800 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18400c060 con 0x7fc1840039b0 2024-03-04T10:12:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18400c260 con 0x7fc1840051c0 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc199d9b640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc184004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19a59c640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1840039b0 0x7fc184003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19ad9d640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1840051c0 0x7fc18400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc199d9b640 1 -- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1840051c0 msgr2=0x7fc18400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc199d9b640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc1840051c0 0x7fc18400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc199d9b640 1 -- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1840039b0 msgr2=0x7fc184003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc199d9b640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1840039b0 0x7fc184003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc199d9b640 1 -- 172.21.15.99:0/3048603395 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184003520 con 0x7fc184004800 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.907+0000 7fc19ad9d640 1 -- 172.21.15.99:0/3048603395 reap_dead start 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc199d9b640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc184004c80 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fc190001750 tx=0x7fc184005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9816a49a0cc8ca10 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190001370 con 0x7fc184004800 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1900052d0 con 0x7fc184004800 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc19959a640 1 -- 172.21.15.99:0/3048603395 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc190003830 con 0x7fc184004800 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/3048603395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 msgr2=0x7fc184004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc184004c80 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fc190001750 tx=0x7fc184005830 comp rx=0 tx=0).stop 2024-03-04T10:12:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/3048603395 shutdown_connections 2024-03-04T10:12:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/3048603395 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc184004c80 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.908+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/3048603395 >> 172.21.15.99:0/3048603395 conn(0x7fc19c06f5f0 msgr2=0x7fc19c06fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.909+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/3048603395 shutdown_connections 2024-03-04T10:12:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.909+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/3048603395 wait complete. 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.910+0000 7fc1a0d7e640 1 Processor -- start 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.910+0000 7fc1a0d7e640 1 -- start start 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc19c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc19c153a70 0x7fc19c160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc19c153200 0x7fc19c1e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0ce060 con 0x7fc184004800 2024-03-04T10:12:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0cdd80 con 0x7fc19c153a70 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19c0ce340 con 0x7fc19c153200 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc19c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc19c153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40008/0 (socket says 172.21.15.99:40008) 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 -- 172.21.15.99:0/2651304571 learned_addr learned my addr 172.21.15.99:0/2651304571 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 -- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc19c153200 msgr2=0x7fc19c1e74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc19c153200 0x7fc19c1e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 -- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc19c153a70 msgr2=0x7fc19c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc19c153a70 0x7fc19c160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 -- 172.21.15.99:0/2651304571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0ca6e0 con 0x7fc184004800 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc19a59c640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc19c153ee0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc18c011910 tx=0x7fc18c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c009590 con 0x7fc184004800 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18c00fa90 con 0x7fc184004800 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c0ca9c0 con 0x7fc184004800 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00fe60 con 0x7fc184004800 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.911+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1602b0 con 0x7fc184004800 2024-03-04T10:12:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.912+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c074f60 con 0x7fc184004800 2024-03-04T10:12:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.913+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc18c012040 con 0x7fc184004800 2024-03-04T10:12:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.913+0000 7fc18affd640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17c06b500 0x7fc17c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.913+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fc18c08a850 con 0x7fc184004800 2024-03-04T10:12:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.913+0000 7fc199d9b640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17c06b500 0x7fc17c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.914+0000 7fc199d9b640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17c06b500 0x7fc17c06d9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc190033000 tx=0x7fc19002f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:07.918+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fc18c019030 con 0x7fc184004800 2024-03-04T10:12:08.091 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.089+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc19c0b71e0 con 0x7fc184004800 2024-03-04T10:12:08.092 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.090+0000 7fc18affd640 1 -- 172.21.15.99:0/2651304571 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+14163 (secure 0 0 0) 0x7fc18c0595e0 con 0x7fc184004800 2024-03-04T10:12:08.092 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:08.093 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":105,"fsid":"8fdf3d83-d8a6-45f1-acf1-cb71ab619f47","created":"2024-03-04T10:03:15.165341+0000","modified":"2024-03-04T10:12:04.563425+0000","last_up_change":"2024-03-04T10:03:26.461194+0000","last_in_change":"2024-03-04T10:03:17.500854+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T10:03:27.478194+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3873f63d-4fe8-4fd9-8e74-98a9d964f0fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6801","nonce":224245114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6803","nonce":224245114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6807","nonce":224245114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":224245114},{"type":"v1","addr":"172.21.15.99:6805","nonce":224245114}]},"public_addr":"172.21.15.99:6801/224245114","cluster_addr":"172.21.15.99:6803/224245114","heartbeat_back_addr":"172.21.15.99:6807/224245114","heartbeat_front_addr":"172.21.15.99:6805/224245114","state":["exists","up"]},{"osd":1,"uuid":"76020627-44f6-4141-93e8-3c45e26c6ee6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6825","nonce":3369025248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6827","nonce":3369025248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6831","nonce":3369025248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":3369025248},{"type":"v1","addr":"172.21.15.99:6829","nonce":3369025248}]},"public_addr":"172.21.15.99:6825/3369025248","cluster_addr":"172.21.15.99:6827/3369025248","heartbeat_back_addr":"172.21.15.99:6831/3369025248","heartbeat_front_addr":"172.21.15.99:6829/3369025248","state":["exists","up"]},{"osd":2,"uuid":"9c243730-6234-466a-8471-62263dda0cc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6817","nonce":739656799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6819","nonce":739656799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6823","nonce":739656799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":739656799},{"type":"v1","addr":"172.21.15.99:6821","nonce":739656799}]},"public_addr":"172.21.15.99:6817/739656799","cluster_addr":"172.21.15.99:6819/739656799","heartbeat_back_addr":"172.21.15.99:6823/739656799","heartbeat_front_addr":"172.21.15.99:6821/739656799","state":["exists","up"]},{"osd":3,"uuid":"c9d1258d-a78b-484c-a559-f05e662abdda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6809","nonce":85197782}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6811","nonce":85197782}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6815","nonce":85197782}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":85197782},{"type":"v1","addr":"172.21.15.99:6813","nonce":85197782}]},"public_addr":"172.21.15.99:6809/85197782","cluster_addr":"172.21.15.99:6811/85197782","heartbeat_back_addr":"172.21.15.99:6815/85197782","heartbeat_front_addr":"172.21.15.99:6813/85197782","state":["exists","up"]},{"osd":4,"uuid":"017c5017-a6a3-4d06-b77d-b1de332c24cc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6801","nonce":3079837853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6803","nonce":3079837853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6807","nonce":3079837853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":3079837853},{"type":"v1","addr":"172.21.15.162:6805","nonce":3079837853}]},"public_addr":"172.21.15.162:6801/3079837853","cluster_addr":"172.21.15.162:6803/3079837853","heartbeat_back_addr":"172.21.15.162:6807/3079837853","heartbeat_front_addr":"172.21.15.162:6805/3079837853","state":["exists","up"]},{"osd":5,"uuid":"92708c04-f7da-4999-b907-96500f70e153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6809","nonce":1764948489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6811","nonce":1764948489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6815","nonce":1764948489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":1764948489},{"type":"v1","addr":"172.21.15.162:6813","nonce":1764948489}]},"public_addr":"172.21.15.162:6809/1764948489","cluster_addr":"172.21.15.162:6811/1764948489","heartbeat_back_addr":"172.21.15.162:6815/1764948489","heartbeat_front_addr":"172.21.15.162:6813/1764948489","state":["exists","up"]},{"osd":6,"uuid":"c506d2a4-2d0c-41d6-8042-91242075b90d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6825","nonce":1448511037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6827","nonce":1448511037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6831","nonce":1448511037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":1448511037},{"type":"v1","addr":"172.21.15.162:6829","nonce":1448511037}]},"public_addr":"172.21.15.162:6825/1448511037","cluster_addr":"172.21.15.162:6827/1448511037","heartbeat_back_addr":"172.21.15.162:6831/1448511037","heartbeat_front_addr":"172.21.15.162:6829/1448511037","state":["exists","up"]},{"osd":7,"uuid":"d68320e0-246d-4455-9926-d79d30c19803","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6817","nonce":2710319333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6819","nonce":2710319333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6823","nonce":2710319333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":2710319333},{"type":"v1","addr":"172.21.15.162:6821","nonce":2710319333}]},"public_addr":"172.21.15.162:6817/2710319333","cluster_addr":"172.21.15.162:6819/2710319333","heartbeat_back_addr":"172.21.15.162:6823/2710319333","heartbeat_front_addr":"172.21.15.162:6821/2710319333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.595480+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.659332+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.635998+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.652719+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.656660+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.709204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.739816+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T10:03:22.611904+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.162:6834/2241177706":"2024-03-05T10:12:01.549168+0000","172.21.15.162:6837/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6835/3671020073":"2024-03-05T10:10:20.734907+0000","172.21.15.99:6832/584503338":"2024-03-05T10:11:24.982726+0000","172.21.15.162:6836/526865753":"2024-03-05T10:11:39.985519+0000","172.21.15.99:6834/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.162:6835/2241177706":"2024-03-05T10:12:01.549168+0000","172.21.15.99:6835/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6834/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6835/932198368":"2024-03-05T10:10:37.745199+0000","172.21.15.99:6834/2735900597":"2024-03-05T10:10:42.995038+0000","172.21.15.99:6835/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6834/334550293":"2024-03-05T10:10:49.033897+0000","172.21.15.99:6835/301817793":"2024-03-05T10:11:09.979220+0000","172.21.15.99:6833/584503338":"2024-03-05T10:11:24.982726+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T10:12:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.091+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17c06b500 msgr2=0x7fc17c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.091+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17c06b500 0x7fc17c06d9c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc190033000 tx=0x7fc19002f000 comp rx=0 tx=0).stop 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.091+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 msgr2=0x7fc19c153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.091+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc19c153ee0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc18c011910 tx=0x7fc18c011dd0 comp rx=0 tx=0).stop 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 shutdown_connections 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc17c06b500 0x7fc17c06d9c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc19c153200 0x7fc19c1e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc19c153a70 0x7fc19c160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 --2- 172.21.15.99:0/2651304571 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc184004800 0x7fc19c153ee0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 >> 172.21.15.99:0/2651304571 conn(0x7fc19c06f5f0 msgr2=0x7fc19c0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 shutdown_connections 2024-03-04T10:12:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:08.092+0000 7fc1a0d7e640 1 -- 172.21.15.99:0/2651304571 wait complete. 2024-03-04T10:12:09.105 INFO:tasks.ceph:Scrubbing osd.0 2024-03-04T10:12:09.106 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.210+0000 7f2f206dc640 1 Processor -- start 2024-03-04T10:12:09.213 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.211+0000 7f2f206dc640 1 -- start start 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.211+0000 7f2f206dc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18143cc0 0x7f2f180ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.211+0000 7f2f206dc640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f180b2aa0 con 0x7f2f180cedc0 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f206dc640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1815e5d0 con 0x7f2f181438e0 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f206dc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1815e8b0 con 0x7f2f18143cc0 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1ec79640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2f180cedc0 0x7f2f180cc2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56946/0 (socket says 172.21.15.99:56946) 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1ec79640 1 -- 172.21.15.99:0/3734775643 learned_addr learned my addr 172.21.15.99:0/3734775643 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:09.215 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1dc77640 1 --2- 172.21.15.99:0/3734775643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18143cc0 0x7f2f180ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029373839 0 0) 0x7f2f180b2aa0 con 0x7f2f180cedc0 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f04003520 con 0x7f2f180cedc0 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051017366 0 0) 0x7f2f1815e5d0 con 0x7f2f181438e0 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f180b2aa0 con 0x7f2f181438e0 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436292143 0 0) 0x7f2f04003520 con 0x7f2f180cedc0 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3734775643 >> v1:172.21.15.162:6789/0 conn(0x7f2f181438e0 legacy=0x7f2f180cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3734775643 >> v1:172.21.15.99:6789/0 conn(0x7f2f180cedc0 legacy=0x7f2f180cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.216 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.212+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3734775643 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1815eb90 con 0x7f2f18143cc0 2024-03-04T10:12:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.213+0000 7f2f1dc77640 1 --2- 172.21.15.99:0/3734775643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18143cc0 0x7f2f180ce880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f080097d0 tx=0x7f2f0802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23f673ef8291b9eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.219 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.213+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08037070 con 0x7f2f18143cc0 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.213+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08030510 con 0x7f2f18143cc0 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.213+0000 7f2f1d476640 1 -- 172.21.15.99:0/3734775643 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f100019f0 con 0x7f2f18143cc0 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 -- 172.21.15.99:0/3734775643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18143cc0 msgr2=0x7f2f180ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 --2- 172.21.15.99:0/3734775643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18143cc0 0x7f2f180ce880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f080097d0 tx=0x7f2f0802b770 comp rx=0 tx=0).stop 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 -- 172.21.15.99:0/3734775643 shutdown_connections 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 --2- 172.21.15.99:0/3734775643 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18143cc0 0x7f2f180ce880 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.220 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 -- 172.21.15.99:0/3734775643 >> 172.21.15.99:0/3734775643 conn(0x7f2f180b1600 msgr2=0x7f2f180b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 -- 172.21.15.99:0/3734775643 shutdown_connections 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.214+0000 7f2f206dc640 1 -- 172.21.15.99:0/3734775643 wait complete. 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.215+0000 7f2f206dc640 1 Processor -- start 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.215+0000 7f2f206dc640 1 -- start start 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f206dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f180cedc0 0x7f2f181f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f206dc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2f181438e0 0x7f2f18142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f206dc640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18142590 0x7f2f18139640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f206dc640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f180cf250 con 0x7f2f181438e0 2024-03-04T10:12:09.221 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f206dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1815f150 con 0x7f2f180cedc0 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f206dc640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f18144220 con 0x7f2f18142590 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2f181438e0 0x7f2f18142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2f181438e0 0x7f2f18142050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40018/0 (socket says 172.21.15.99:40018) 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3897528950 learned_addr learned my addr 172.21.15.99:0/3897528950 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1ec79640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18142590 0x7f2f18139640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1e478640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f180cedc0 0x7f2f181f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18142590 msgr2=0x7f2f18139640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.222 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2f18142590 0x7f2f18139640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f180cedc0 msgr2=0x7f2f181f3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2f180cedc0 0x7f2f181f3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.216+0000 7f2f1dc77640 1 -- 172.21.15.99:0/3897528950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1815b670 con 0x7f2f181438e0 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.217+0000 7f2f1dc77640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2f181438e0 0x7f2f18142050 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009720 tx=0x7f2f080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.217+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08042070 con 0x7f2f181438e0 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.217+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08030ec0 con 0x7f2f181438e0 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.217+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1815b910 con 0x7f2f181438e0 2024-03-04T10:12:09.223 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.217+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08037040 con 0x7f2f181438e0 2024-03-04T10:12:09.224 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.217+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1815bed0 con 0x7f2f181438e0 2024-03-04T10:12:09.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.218+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f18136070 con 0x7f2f181438e0 2024-03-04T10:12:09.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.218+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2f0803a040 con 0x7f2f181438e0 2024-03-04T10:12:09.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.218+0000 7f2f0effd640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2efc06b500 0x7f2efc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.226 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f2f080ab8c0 con 0x7f2f181438e0 2024-03-04T10:12:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f0effd640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f2efc071ef0 0x7f2efc074330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f08008eb0 con 0x7f2efc071ef0 2024-03-04T10:12:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f080b02c0 con 0x7f2f181438e0 2024-03-04T10:12:09.227 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f1e478640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2efc06b500 0x7f2efc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f1ec79640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f2efc071ef0 0x7f2efc074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f1ec79640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f2efc071ef0 0x7f2efc074330 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.219+0000 7f2f1e478640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2efc06b500 0x7f2efc06d9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2f14000f60 tx=0x7f2f14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.229 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.220+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== osd.0 v2:172.21.15.99:6800/224245114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2f08008eb0 con 0x7f2efc071ef0 2024-03-04T10:12:09.247 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.245+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 --> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2f181429a0 con 0x7f2efc071ef0 2024-03-04T10:12:09.248 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.245+0000 7f2f0effd640 1 -- 172.21.15.99:0/3897528950 <== osd.0 v2:172.21.15.99:6800/224245114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2f181429a0 con 0x7f2efc071ef0 2024-03-04T10:12:09.248 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:09.248 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:09.248 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:09.249 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.245+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f2efc071ef0 msgr2=0x7f2efc074330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.245+0000 7f2f206dc640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:6800/224245114,v1:172.21.15.99:6801/224245114] conn(0x7f2efc071ef0 0x7f2efc074330 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2efc06b500 msgr2=0x7f2efc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2efc06b500 0x7f2efc06d9c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2f14000f60 tx=0x7f2f14009040 comp rx=0 tx=0).stop 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2f181438e0 msgr2=0x7f2f18142050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 --2- 172.21.15.99:0/3897528950 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2f181438e0 0x7f2f18142050 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f2f08009720 tx=0x7f2f080046c0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f1ec79640 1 -- 172.21.15.99:0/3897528950 reap_dead start 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 shutdown_connections 2024-03-04T10:12:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 >> 172.21.15.99:0/3897528950 conn(0x7f2f180b1600 msgr2=0x7f2f180cc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 shutdown_connections 2024-03-04T10:12:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.246+0000 7f2f206dc640 1 -- 172.21.15.99:0/3897528950 wait complete. 2024-03-04T10:12:09.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-04T10:12:09.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.360+0000 7f580aacb640 1 Processor -- start 2024-03-04T10:12:09.363 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.361+0000 7f580aacb640 1 -- start start 2024-03-04T10:12:09.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.361+0000 7f580aacb640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.361+0000 7f580aacb640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f58040cc500 0x7f58040cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.361+0000 7f580aacb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58040d0140 con 0x7f58040cfd60 2024-03-04T10:12:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.361+0000 7f580aacb640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58040cf440 con 0x7f58040cc500 2024-03-04T10:12:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.361+0000 7f580aacb640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580415cbb0 con 0x7f580406d3b0 2024-03-04T10:12:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5809068640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f58040cc500 0x7f58040cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f57fbfff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5808867640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f58040cfd60 0x7f58040cbdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:33354/0 (socket says 172.21.15.99:33354) 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5808867640 1 -- 172.21.15.99:0/345922302 learned_addr learned my addr 172.21.15.99:0/345922302 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5809068640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f58040cc500 0x7f58040cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:48906/0 (socket says 172.21.15.99:48906) 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5809068640 1 -- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 msgr2=0x7f58040ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5809068640 1 --2- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5809068640 1 -- 172.21.15.99:0/345922302 >> v1:172.21.15.162:6789/0 conn(0x7f58040cfd60 legacy=0x7f58040cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f5809068640 1 -- 172.21.15.99:0/345922302 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580415c280 con 0x7f58040cc500 2024-03-04T10:12:09.366 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.362+0000 7f57fb7fe640 1 -- 172.21.15.99:0/345922302 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774537129 0 0) 0x7f58040d0140 con 0x7f58040cfd60 2024-03-04T10:12:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.363+0000 7f5809068640 1 --2- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58040cef00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009d90 tx=0x7f57fc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39a23296a8c6a311 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.363+0000 7f57fb7fe640 1 -- 172.21.15.99:0/345922302 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc039070 con 0x7f58040cc500 2024-03-04T10:12:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.363+0000 7f57fb7fe640 1 -- 172.21.15.99:0/345922302 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc02cc50 con 0x7f58040cc500 2024-03-04T10:12:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.363+0000 7f57fb7fe640 1 -- 172.21.15.99:0/345922302 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc034440 con 0x7f58040cc500 2024-03-04T10:12:09.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 -- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 msgr2=0x7f58040cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 --2- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58040cef00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009d90 tx=0x7f57fc02bea0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 -- 172.21.15.99:0/345922302 shutdown_connections 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 --2- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040ce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 --2- 172.21.15.99:0/345922302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58040cef00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 -- 172.21.15.99:0/345922302 >> 172.21.15.99:0/345922302 conn(0x7f580406acd0 msgr2=0x7f580406b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 -- 172.21.15.99:0/345922302 shutdown_connections 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.364+0000 7f580aacb640 1 -- 172.21.15.99:0/345922302 wait complete. 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.365+0000 7f580aacb640 1 Processor -- start 2024-03-04T10:12:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.365+0000 7f580aacb640 1 -- start start 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040aab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58041572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 0x7f5804157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580415bf30 con 0x7f58040cfd60 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580415c840 con 0x7f58040cc500 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580415f520 con 0x7f580406d3b0 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 0x7f5804157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.372 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 0x7f5804157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40030/0 (socket says 172.21.15.99:40030) 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 -- 172.21.15.99:0/3605906326 learned_addr learned my addr 172.21.15.99:0/3605906326 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5808867640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f57fbfff640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58041572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 -- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 msgr2=0x7f58040aab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040aab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 -- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 msgr2=0x7f58041572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58041572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 -- 172.21.15.99:0/3605906326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5804158070 con 0x7f58040cfd60 2024-03-04T10:12:09.373 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f5809068640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 0x7f5804157830 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009d90 tx=0x7f57fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc039050 con 0x7f58040cfd60 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc045070 con 0x7f58040cfd60 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58040a8b10 con 0x7f58040cfd60 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57fc009450 con 0x7f58040cfd60 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.366+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58040a90d0 con 0x7f58040cfd60 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.367+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58040a8930 con 0x7f58040cfd60 2024-03-04T10:12:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.368+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f57fc03e610 con 0x7f58040cfd60 2024-03-04T10:12:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.368+0000 7f57f97fa640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57f006b4d0 0x7f57f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.368+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f57fc0acc80 con 0x7f58040cfd60 2024-03-04T10:12:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.368+0000 7f5808867640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57f006b4d0 0x7f57f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.369+0000 7f5808867640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57f006b4d0 0x7f57f006d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5800007fd0 tx=0x7f5800007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.372+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f58040a8930 con 0x7f58040cfd60 2024-03-04T10:12:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.541+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f5804074f60 con 0x7f57f006b4d0 2024-03-04T10:12:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.541+0000 7f57f97fa640 1 -- 172.21.15.99:0/3605906326 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5804074f60 con 0x7f57f006b4d0 2024-03-04T10:12:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 0 to deep-scrub 2024-03-04T10:12:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57f006b4d0 msgr2=0x7f57f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57f006b4d0 0x7f57f006d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5800007fd0 tx=0x7f5800007de0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 msgr2=0x7f5804157830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 0x7f5804157830 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f57fc009d90 tx=0x7f57fc004a40 comp rx=0 tx=0).stop 2024-03-04T10:12:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 shutdown_connections 2024-03-04T10:12:09.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f57f006b4d0 0x7f57f006d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f58040cfd60 0x7f5804157830 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f58040cc500 0x7f58041572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 --2- 172.21.15.99:0/3605906326 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f580406d3b0 0x7f58040aab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 >> 172.21.15.99:0/3605906326 conn(0x7f580406acd0 msgr2=0x7f58041552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 shutdown_connections 2024-03-04T10:12:09.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.543+0000 7f580aacb640 1 -- 172.21.15.99:0/3605906326 wait complete. 2024-03-04T10:12:09.557 INFO:tasks.ceph:Scrubbing osd.1 2024-03-04T10:12:09.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:09.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.659+0000 7f7a85b59640 1 Processor -- start 2024-03-04T10:12:09.662 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.659+0000 7f7a85b59640 1 -- start start 2024-03-04T10:12:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a85b59640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 0x7f7a800b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a85b59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a800b2fe0 0x7f7a800b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a85b59640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8006af20 con 0x7f7a8015dc80 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a85b59640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8006acd0 con 0x7f7a800671c0 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a85b59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800729b0 con 0x7f7a800b2fe0 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a7effd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 0x7f7a800b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a7ffff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a800b2fe0 0x7f7a800b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a7f7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7a8015dc80 0x7f7a8006bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:33370/0 (socket says 172.21.15.99:33370) 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a7f7fe640 1 -- 172.21.15.99:0/2605884294 learned_addr learned my addr 172.21.15.99:0/2605884294 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.660+0000 7f7a7effd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 0x7f7a800b48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:48910/0 (socket says 172.21.15.99:48910) 2024-03-04T10:12:09.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.661+0000 7f7a7ffff640 1 -- 172.21.15.99:0/2605884294 >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 msgr2=0x7f7a800b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.661+0000 7f7a7ffff640 1 --2- 172.21.15.99:0/2605884294 >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 0x7f7a800b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.661+0000 7f7a7ffff640 1 -- 172.21.15.99:0/2605884294 >> v1:172.21.15.162:6789/0 conn(0x7f7a8015dc80 legacy=0x7f7a8006bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.661+0000 7f7a7ffff640 1 -- 172.21.15.99:0/2605884294 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8006b200 con 0x7f7a800b2fe0 2024-03-04T10:12:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.661+0000 7f7a7effd640 1 --2- 172.21.15.99:0/2605884294 >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 0x7f7a800b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:12:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.661+0000 7f7a7ffff640 1 --2- 172.21.15.99:0/2605884294 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a800b2fe0 0x7f7a800b4e20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009d90 tx=0x7f7a7802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffe4bac303b3dfa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a7e7fc640 1 -- 172.21.15.99:0/2605884294 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78039070 con 0x7f7a800b2fe0 2024-03-04T10:12:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a7e7fc640 1 -- 172.21.15.99:0/2605884294 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7802cc50 con 0x7f7a800b2fe0 2024-03-04T10:12:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a7e7fc640 1 -- 172.21.15.99:0/2605884294 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78034440 con 0x7f7a800b2fe0 2024-03-04T10:12:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 -- 172.21.15.99:0/2605884294 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a800b2fe0 msgr2=0x7f7a800b4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 --2- 172.21.15.99:0/2605884294 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a800b2fe0 0x7f7a800b4e20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009d90 tx=0x7f7a7802bea0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 -- 172.21.15.99:0/2605884294 shutdown_connections 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 --2- 172.21.15.99:0/2605884294 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a800b2fe0 0x7f7a800b4e20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 --2- 172.21.15.99:0/2605884294 >> v2:172.21.15.162:3300/0 conn(0x7f7a800671c0 0x7f7a800b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 -- 172.21.15.99:0/2605884294 >> 172.21.15.99:0/2605884294 conn(0x7f7a80071ca0 msgr2=0x7f7a80072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 -- 172.21.15.99:0/2605884294 shutdown_connections 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.662+0000 7f7a85b59640 1 -- 172.21.15.99:0/2605884294 wait complete. 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.663+0000 7f7a85b59640 1 Processor -- start 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.663+0000 7f7a85b59640 1 -- start start 2024-03-04T10:12:09.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a85b59640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a800671c0 0x7f7a8014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a85b59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a800b2fe0 0x7f7a801570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a85b59640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a8015a500 0x7f7a800af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a85b59640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8006baa0 con 0x7f7a800671c0 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a85b59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8006b7c0 con 0x7f7a800b2fe0 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a85b59640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800ade50 con 0x7f7a8015a500 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a7f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a800671c0 0x7f7a8014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a7f7fe640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a800671c0 0x7f7a8014e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40040/0 (socket says 172.21.15.99:40040) 2024-03-04T10:12:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a7f7fe640 1 -- 172.21.15.99:0/2179252391 learned_addr learned my addr 172.21.15.99:0/2179252391 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a7effd640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a800b2fe0 0x7f7a801570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.664+0000 7f7a7ffff640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a8015a500 0x7f7a800af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7f7fe640 1 -- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a8015a500 msgr2=0x7f7a800af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7f7fe640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f7a8015a500 0x7f7a800af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7f7fe640 1 -- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a800b2fe0 msgr2=0x7f7a801570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7f7fe640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a800b2fe0 0x7f7a801570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7f7fe640 1 -- 172.21.15.99:0/2179252391 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8015cb00 con 0x7f7a800671c0 2024-03-04T10:12:09.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7effd640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7a800b2fe0 0x7f7a801570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a7f7fe640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a800671c0 0x7f7a8014e900 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000c3c0 tx=0x7f7a7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70017070 con 0x7f7a800671c0 2024-03-04T10:12:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a70005ce0 con 0x7f7a800671c0 2024-03-04T10:12:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a700072f0 con 0x7f7a800671c0 2024-03-04T10:12:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8015cde0 con 0x7f7a800671c0 2024-03-04T10:12:09.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.665+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8014ee40 con 0x7f7a800671c0 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.666+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a8015c8b0 con 0x7f7a800671c0 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.666+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f7a7001a070 con 0x7f7a800671c0 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a67fff640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f7a700896b0 con 0x7f7a800671c0 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a67fff640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f7a6c071ec0 0x7f7a6c074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a70009e50 con 0x7f7a6c071ec0 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a70090050 con 0x7f7a800671c0 2024-03-04T10:12:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a7ffff640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f7a6c071ec0 0x7f7a6c074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a7effd640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.667+0000 7f7a7ffff640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f7a6c071ec0 0x7f7a6c074300 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.668+0000 7f7a7effd640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7a801f6c40 tx=0x7f7a74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.675 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.668+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== osd.1 v2:172.21.15.99:6824/3369025248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7a70009e50 con 0x7f7a6c071ec0 2024-03-04T10:12:09.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 --> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7a80074f60 con 0x7f7a6c071ec0 2024-03-04T10:12:09.684 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a67fff640 1 -- 172.21.15.99:0/2179252391 <== osd.1 v2:172.21.15.99:6824/3369025248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7a80074f60 con 0x7f7a6c071ec0 2024-03-04T10:12:09.685 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:09.685 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:09.685 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f7a6c071ec0 msgr2=0x7f7a6c074300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:6824/3369025248,v1:172.21.15.99:6825/3369025248] conn(0x7f7a6c071ec0 0x7f7a6c074300 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a6c06b4d0 msgr2=0x7f7a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f7a6c06b4d0 0x7f7a6c06d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7a801f6c40 tx=0x7f7a74008040 comp rx=0 tx=0).stop 2024-03-04T10:12:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a800671c0 msgr2=0x7f7a8014e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a85b59640 1 --2- 172.21.15.99:0/2179252391 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7a800671c0 0x7f7a8014e900 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000c3c0 tx=0x7f7a7000c880 comp rx=0 tx=0).stop 2024-03-04T10:12:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.681+0000 7f7a7ffff640 1 -- 172.21.15.99:0/2179252391 reap_dead start 2024-03-04T10:12:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.682+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 shutdown_connections 2024-03-04T10:12:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.682+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 >> 172.21.15.99:0/2179252391 conn(0x7f7a80071ca0 msgr2=0x7f7a8006f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.682+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 shutdown_connections 2024-03-04T10:12:09.687 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.682+0000 7f7a85b59640 1 -- 172.21.15.99:0/2179252391 wait complete. 2024-03-04T10:12:09.694 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-04T10:12:09.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.792+0000 7f3f7e20b640 1 Processor -- start 2024-03-04T10:12:09.795 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.792+0000 7f3f7e20b640 1 -- start start 2024-03-04T10:12:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f7e20b640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f3f7806d3b0 0x7f3f7815c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f7e20b640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7806d8b0 con 0x7f3f7815b410 2024-03-04T10:12:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f7e20b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7806db90 con 0x7f3f7815b030 2024-03-04T10:12:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f7e20b640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f780b2720 con 0x7f3f7806d3b0 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f77fff640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f3f7815b410 0x7f3f7815b880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56962/0 (socket says 172.21.15.99:56962) 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f77fff640 1 -- 172.21.15.99:0/3087279258 learned_addr learned my addr 172.21.15.99:0/3087279258 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f6effd640 1 --2- 172.21.15.99:0/3087279258 >> v2:172.21.15.162:3300/0 conn(0x7f3f7806d3b0 0x7f3f7815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697551236 0 0) 0x7f3f7806d8b0 con 0x7f3f7815b410 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.793+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f64003520 con 0x7f3f7815b410 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179654119 0 0) 0x7f3f64003520 con 0x7f3f7815b410 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 >> v2:172.21.15.162:3300/0 conn(0x7f3f7806d3b0 msgr2=0x7f3f7815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 --2- 172.21.15.99:0/3087279258 >> v2:172.21.15.162:3300/0 conn(0x7f3f7806d3b0 0x7f3f7815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 >> v1:172.21.15.162:6789/0 conn(0x7f3f7815b030 legacy=0x7f3f7815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7806de70 con 0x7f3f7815b410 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f3f700019f0 con 0x7f3f7815b410 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 >> v1:172.21.15.99:6789/0 conn(0x7f3f7815b410 legacy=0x7f3f7815b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f640039b0 0x7f3f64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f64004800 0x7f3f64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f640051c0 0x7f3f6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f780b2720 con 0x7f3f640039b0 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6400c160 con 0x7f3f64004800 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6400c360 con 0x7f3f640051c0 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f777fe640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f640039b0 0x7f3f64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f6effd640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f64004800 0x7f3f64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f77fff640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f640051c0 0x7f3f6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f777fe640 1 -- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f640051c0 msgr2=0x7f3f6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f777fe640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f640051c0 0x7f3f6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f777fe640 1 -- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f64004800 msgr2=0x7f3f64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f777fe640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f64004800 0x7f3f64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f777fe640 1 -- 172.21.15.99:0/3087279258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64003520 con 0x7f3f640039b0 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f6effd640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f64004800 0x7f3f64004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.794+0000 7f3f77fff640 1 -- 172.21.15.99:0/3087279258 reap_dead start 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.795+0000 7f3f777fe640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f640039b0 0x7f3f64003dd0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f3f68001750 tx=0x7f3f64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9feb281b921f69 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.795+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68001370 con 0x7f3f640039b0 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.795+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f680052d0 con 0x7f3f640039b0 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.795+0000 7f3f76ffd640 1 -- 172.21.15.99:0/3087279258 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68003830 con 0x7f3f640039b0 2024-03-04T10:12:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 -- 172.21.15.99:0/3087279258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f640039b0 msgr2=0x7f3f64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f640039b0 0x7f3f64003dd0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f3f68001750 tx=0x7f3f64005830 comp rx=0 tx=0).stop 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 -- 172.21.15.99:0/3087279258 shutdown_connections 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/3087279258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f640039b0 0x7f3f64003dd0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 -- 172.21.15.99:0/3087279258 >> 172.21.15.99:0/3087279258 conn(0x7f3f7806acd0 msgr2=0x7f3f7806b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 -- 172.21.15.99:0/3087279258 shutdown_connections 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.796+0000 7f3f7e20b640 1 -- 172.21.15.99:0/3087279258 wait complete. 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.797+0000 7f3f7e20b640 1 Processor -- start 2024-03-04T10:12:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.797+0000 7f3f7e20b640 1 -- start start 2024-03-04T10:12:09.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f640039b0 0x7f3f78149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f78143850 0x7f3f78146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 0x7f3f7814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7806e810 con 0x7f3f781465c0 2024-03-04T10:12:09.807 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7806e690 con 0x7f3f640039b0 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7815c120 con 0x7f3f78143850 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 0x7f3f7814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 0x7f3f7814f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40064/0 (socket says 172.21.15.99:40064) 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 -- 172.21.15.99:0/1865430157 learned_addr learned my addr 172.21.15.99:0/1865430157 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f6effd640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f78143850 0x7f3f78146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f777fe640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f640039b0 0x7f3f78149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 -- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f78143850 msgr2=0x7f3f78146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f78143850 0x7f3f78146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 -- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f640039b0 msgr2=0x7f3f78149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f640039b0 0x7f3f78149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 -- 172.21.15.99:0/1865430157 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78169430 con 0x7f3f781465c0 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f77fff640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 0x7f3f7814f8d0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f3f7815b7f0 tx=0x7f3f70007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7000ea80 con 0x7f3f781465c0 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f70018070 con 0x7f3f781465c0 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f78150c40 con 0x7f3f781465c0 2024-03-04T10:12:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7000ee60 con 0x7f3f781465c0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.798+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f78144200 con 0x7f3f781465c0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.799+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f78074f60 con 0x7f3f781465c0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.800+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f3f7001d460 con 0x7f3f781465c0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.800+0000 7f3f74ff9640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f5806b4d0 0x7f3f5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.800+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f3f7008a4a0 con 0x7f3f781465c0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.800+0000 7f3f777fe640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f5806b4d0 0x7f3f5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.801+0000 7f3f777fe640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f5806b4d0 0x7f3f5806d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3f68033000 tx=0x7f3f68009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.804+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f3f700592c0 con 0x7f3f781465c0 2024-03-04T10:12:09.978 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.976+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f780b3f00 con 0x7f3f5806b4d0 2024-03-04T10:12:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.976+0000 7f3f74ff9640 1 -- 172.21.15.99:0/1865430157 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3f780b3f00 con 0x7f3f5806b4d0 2024-03-04T10:12:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 1 to deep-scrub 2024-03-04T10:12:09.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f5806b4d0 msgr2=0x7f3f5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f5806b4d0 0x7f3f5806d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3f68033000 tx=0x7f3f68009030 comp rx=0 tx=0).stop 2024-03-04T10:12:09.981 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 msgr2=0x7f3f7814f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 0x7f3f7814f8d0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f3f7815b7f0 tx=0x7f3f70007bc0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 shutdown_connections 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f3f5806b4d0 0x7f3f5806d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f3f781465c0 0x7f3f7814f8d0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f3f78143850 0x7f3f78146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 --2- 172.21.15.99:0/1865430157 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f640039b0 0x7f3f78149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 >> 172.21.15.99:0/1865430157 conn(0x7f3f7806acd0 msgr2=0x7f3f781537a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 shutdown_connections 2024-03-04T10:12:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:09.978+0000 7f3f7e20b640 1 -- 172.21.15.99:0/1865430157 wait complete. 2024-03-04T10:12:09.991 INFO:tasks.ceph:Scrubbing osd.2 2024-03-04T10:12:09.992 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:10.095 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.093+0000 7f2c57389640 1 Processor -- start 2024-03-04T10:12:10.096 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.093+0000 7f2c57389640 1 -- start start 2024-03-04T10:12:10.104 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c57389640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f2c5015dc20 0x7f2c500d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c57389640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2c500d0800 0x7f2c500d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c57389640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5015e000 con 0x7f2c500d2ab0 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c57389640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c500d1190 con 0x7f2c500d0800 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c57389640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c500d2f50 con 0x7f2c5015dc20 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c55926640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f2c500d0800 0x7f2c500d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c54924640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f2c500d2ab0 0x7f2c500d3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56970/0 (socket says 172.21.15.99:56970) 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c54924640 1 -- 172.21.15.99:0/4127656182 learned_addr learned my addr 172.21.15.99:0/4127656182 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c55926640 1 --2- 172.21.15.99:0/4127656182 >> v2:172.21.15.99:3300/0 conn(0x7f2c500d0800 0x7f2c500d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40070/0 (socket says 172.21.15.99:40070) 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c55926640 1 -- 172.21.15.99:0/4127656182 >> v2:172.21.15.162:3300/0 conn(0x7f2c5015dc20 msgr2=0x7f2c500d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c55926640 1 --2- 172.21.15.99:0/4127656182 >> v2:172.21.15.162:3300/0 conn(0x7f2c5015dc20 0x7f2c500d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c55926640 1 -- 172.21.15.99:0/4127656182 >> v1:172.21.15.99:6789/0 conn(0x7f2c500d2ab0 legacy=0x7f2c500d3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.094+0000 7f2c55926640 1 -- 172.21.15.99:0/4127656182 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500d3230 con 0x7f2c500d0800 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.095+0000 7f2c55926640 1 --2- 172.21.15.99:0/4127656182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c500d0c50 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2c500b5f80 tx=0x7f2c4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50ffd9745673f55 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.095+0000 7f2c47fff640 1 -- 172.21.15.99:0/4127656182 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48039070 con 0x7f2c500d0800 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.095+0000 7f2c47fff640 1 -- 172.21.15.99:0/4127656182 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4802cc80 con 0x7f2c500d0800 2024-03-04T10:12:10.106 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.095+0000 7f2c47fff640 1 -- 172.21.15.99:0/4127656182 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48034660 con 0x7f2c500d0800 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.095+0000 7f2c57389640 1 -- 172.21.15.99:0/4127656182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 msgr2=0x7f2c500d0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.095+0000 7f2c57389640 1 --2- 172.21.15.99:0/4127656182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c500d0c50 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f2c500b5f80 tx=0x7f2c4802bed0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.096+0000 7f2c57389640 1 -- 172.21.15.99:0/4127656182 shutdown_connections 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.096+0000 7f2c57389640 1 --2- 172.21.15.99:0/4127656182 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c500d0c50 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.096+0000 7f2c57389640 1 --2- 172.21.15.99:0/4127656182 >> v2:172.21.15.162:3300/0 conn(0x7f2c5015dc20 0x7f2c500d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.096+0000 7f2c57389640 1 -- 172.21.15.99:0/4127656182 >> 172.21.15.99:0/4127656182 conn(0x7f2c500725e0 msgr2=0x7f2c500729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.096+0000 7f2c57389640 1 -- 172.21.15.99:0/4127656182 shutdown_connections 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.096+0000 7f2c57389640 1 -- 172.21.15.99:0/4127656182 wait complete. 2024-03-04T10:12:10.107 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.097+0000 7f2c57389640 1 Processor -- start 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.097+0000 7f2c57389640 1 -- start start 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.097+0000 7f2c57389640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c50149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.097+0000 7f2c57389640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c500d2ab0 0x7f2c500cb030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c57389640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c5015dc20 0x7f2c500c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c57389640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500d3ad0 con 0x7f2c500d0800 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c57389640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500d37f0 con 0x7f2c5015dc20 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c57389640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c500d4700 con 0x7f2c500d2ab0 2024-03-04T10:12:10.108 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c50149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c50149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40082/0 (socket says 172.21.15.99:40082) 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 -- 172.21.15.99:0/2669680481 learned_addr learned my addr 172.21.15.99:0/2669680481 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 -- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c500d2ab0 msgr2=0x7f2c500cb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f2c500d2ab0 0x7f2c500cb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 -- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c5015dc20 msgr2=0x7f2c500c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c5015dc20 0x7f2c500c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 -- 172.21.15.99:0/2669680481 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c500c7700 con 0x7f2c500d0800 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c55125640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c50149240 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00b410 tx=0x7f2c4c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.109 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c015020 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c011850 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c011c40 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c500cbb60 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.098+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c500c79a0 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.099+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c50074f60 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f2c4c018020 con 0x7f2c500d0800 2024-03-04T10:12:10.110 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c45ffb640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2c3406b4d0 0x7f2c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f2c4c08a210 con 0x7f2c500d0800 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c45ffb640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f2c34071ec0 0x7f2c34074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c4c004450 con 0x7f2c34071ec0 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c4c091050 con 0x7f2c500d0800 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c54924640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2c3406b4d0 0x7f2c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.100+0000 7f2c55926640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f2c34071ec0 0x7f2c34074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.101+0000 7f2c55926640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f2c34071ec0 0x7f2c34074300 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.101+0000 7f2c54924640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2c3406b4d0 0x7f2c3406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2c5015c700 tx=0x7f2c40007a40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.111 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.101+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== osd.2 v2:172.21.15.99:6816/739656799 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2c4c004450 con 0x7f2c34071ec0 2024-03-04T10:12:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.114+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 --> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2c5015d0a0 con 0x7f2c34071ec0 2024-03-04T10:12:10.117 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.114+0000 7f2c45ffb640 1 -- 172.21.15.99:0/2669680481 <== osd.2 v2:172.21.15.99:6816/739656799 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2c5015d0a0 con 0x7f2c34071ec0 2024-03-04T10:12:10.117 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:10.117 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:10.117 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:10.118 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.114+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f2c34071ec0 msgr2=0x7f2c34074300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.114+0000 7f2c57389640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:6816/739656799,v1:172.21.15.99:6817/739656799] conn(0x7f2c34071ec0 0x7f2c34074300 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2c3406b4d0 msgr2=0x7f2c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f2c3406b4d0 0x7f2c3406d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2c5015c700 tx=0x7f2c40007a40 comp rx=0 tx=0).stop 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 msgr2=0x7f2c50149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 --2- 172.21.15.99:0/2669680481 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2c500d0800 0x7f2c50149240 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00b410 tx=0x7f2c4c00b8d0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c55926640 1 -- 172.21.15.99:0/2669680481 reap_dead start 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 shutdown_connections 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 >> 172.21.15.99:0/2669680481 conn(0x7f2c500725e0 msgr2=0x7f2c50159f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.119 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 shutdown_connections 2024-03-04T10:12:10.120 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.115+0000 7f2c57389640 1 -- 172.21.15.99:0/2669680481 wait complete. 2024-03-04T10:12:10.127 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-04T10:12:10.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.225+0000 7fd0b6085640 1 Processor -- start 2024-03-04T10:12:10.228 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.225+0000 7fd0b6085640 1 -- start start 2024-03-04T10:12:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0b6085640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd0b015c030 0x7fd0b006b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0b6085640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00cc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0b6085640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b015cfe0 con 0x7fd0b01597f0 2024-03-04T10:12:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0b6085640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00d0240 con 0x7fd0b015c030 2024-03-04T10:12:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0b6085640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00cf630 con 0x7fd0b00cc260 2024-03-04T10:12:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0affff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00cc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0aeffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd0b015c030 0x7fd0b006b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0affff640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00cc6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:57944/0 (socket says 172.21.15.99:57944) 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0aeffd640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fd0b015c030 0x7fd0b006b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:48970/0 (socket says 172.21.15.99:48970) 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.226+0000 7fd0affff640 1 -- 172.21.15.99:0/2596882353 learned_addr learned my addr 172.21.15.99:0/2596882353 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.227+0000 7fd0aeffd640 1 -- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 msgr2=0x7fd0b00cc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.227+0000 7fd0aeffd640 1 --2- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00cc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.227+0000 7fd0aeffd640 1 -- 172.21.15.99:0/2596882353 >> v1:172.21.15.162:6789/0 conn(0x7fd0b01597f0 legacy=0x7fd0b006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.227+0000 7fd0ae7fc640 1 -- 172.21.15.99:0/2596882353 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899895786 0 0) 0x7fd0b015cfe0 con 0x7fd0b01597f0 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.227+0000 7fd0aeffd640 1 -- 172.21.15.99:0/2596882353 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00cf910 con 0x7fd0b015c030 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.227+0000 7fd0affff640 1 --2- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0aeffd640 1 --2- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b015c030 0x7fd0b006b3e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40097a0 tx=0x7fd0a402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c73b5fdcf95f109 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0ae7fc640 1 -- 172.21.15.99:0/2596882353 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40315a0 con 0x7fd0b015c030 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0ae7fc640 1 -- 172.21.15.99:0/2596882353 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4037070 con 0x7fd0b015c030 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0ae7fc640 1 -- 172.21.15.99:0/2596882353 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4031960 con 0x7fd0b015c030 2024-03-04T10:12:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 -- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b015c030 msgr2=0x7fd0b006b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b015c030 0x7fd0b006b3e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40097a0 tx=0x7fd0a402b840 comp rx=0 tx=0).stop 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 -- 172.21.15.99:0/2596882353 shutdown_connections 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00cc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2596882353 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b015c030 0x7fd0b006b3e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 -- 172.21.15.99:0/2596882353 >> 172.21.15.99:0/2596882353 conn(0x7fd0b00d0710 msgr2=0x7fd0b00d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 -- 172.21.15.99:0/2596882353 shutdown_connections 2024-03-04T10:12:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.228+0000 7fd0b6085640 1 -- 172.21.15.99:0/2596882353 wait complete. 2024-03-04T10:12:10.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.229+0000 7fd0b6085640 1 Processor -- start 2024-03-04T10:12:10.237 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 -- start start 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01597f0 0x7fd0b014a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 0x7fd0b014ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00d1350 con 0x7fd0b015c030 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00cfed0 con 0x7fd0b01597f0 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0b6085640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b006b920 con 0x7fd0b00cc260 2024-03-04T10:12:10.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0affff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 0x7fd0b014ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0affff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 0x7fd0b014ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40098/0 (socket says 172.21.15.99:40098) 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.230+0000 7fd0affff640 1 -- 172.21.15.99:0/2204798199 learned_addr learned my addr 172.21.15.99:0/2204798199 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0aeffd640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01597f0 0x7fd0b014a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0af7fe640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0affff640 1 -- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 msgr2=0x7fd0b00b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0affff640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0affff640 1 -- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01597f0 msgr2=0x7fd0b014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0affff640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01597f0 0x7fd0b014a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0affff640 1 -- 172.21.15.99:0/2204798199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00b05c0 con 0x7fd0b015c030 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0af7fe640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0aeffd640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01597f0 0x7fd0b014a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0affff640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 0x7fd0b014ab80 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e70 tx=0x7fd0a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800cbe0 con 0x7fd0b015c030 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a800cd80 con 0x7fd0b015c030 2024-03-04T10:12:10.240 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b00b08a0 con 0x7fd0b015c030 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8003bb0 con 0x7fd0b015c030 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.231+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b00b0e60 con 0x7fd0b015c030 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.232+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b006cd70 con 0x7fd0b015c030 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.232+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fd0a801b030 con 0x7fd0b015c030 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.233+0000 7fd097fff640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd09c06b4d0 0x7fd09c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.233+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fd0a808a6b0 con 0x7fd0b015c030 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.233+0000 7fd0af7fe640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd09c06b4d0 0x7fd09c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.241 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.233+0000 7fd0af7fe640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd09c06b4d0 0x7fd09c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0000f60 tx=0x7fd0a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.242 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.237+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fd0a8022020 con 0x7fd0b015c030 2024-03-04T10:12:10.409 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.407+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b00afe00 con 0x7fd09c06b4d0 2024-03-04T10:12:10.410 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.408+0000 7fd097fff640 1 -- 172.21.15.99:0/2204798199 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd0b00afe00 con 0x7fd09c06b4d0 2024-03-04T10:12:10.410 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 2 to deep-scrub 2024-03-04T10:12:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd09c06b4d0 msgr2=0x7fd09c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd09c06b4d0 0x7fd09c06d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0000f60 tx=0x7fd0a0009040 comp rx=0 tx=0).stop 2024-03-04T10:12:10.412 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 msgr2=0x7fd0b014ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 0x7fd0b014ab80 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e70 tx=0x7fd0a800c490 comp rx=0 tx=0).stop 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 shutdown_connections 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fd09c06b4d0 0x7fd09c06d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fd0b015c030 0x7fd0b014ab80 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd0b01597f0 0x7fd0b014a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 --2- 172.21.15.99:0/2204798199 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fd0b00cc260 0x7fd0b00b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.409+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 >> 172.21.15.99:0/2204798199 conn(0x7fd0b00d0710 msgr2=0x7fd0b015d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.410+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 shutdown_connections 2024-03-04T10:12:10.414 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.410+0000 7fd0b6085640 1 -- 172.21.15.99:0/2204798199 wait complete. 2024-03-04T10:12:10.423 INFO:tasks.ceph:Scrubbing osd.3 2024-03-04T10:12:10.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:10.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.524+0000 7fa078481640 1 Processor -- start 2024-03-04T10:12:10.527 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.524+0000 7fa078481640 1 -- start start 2024-03-04T10:12:10.528 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.525+0000 7fa078481640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa07006c7f0 0x7fa07009a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.525+0000 7fa078481640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.525+0000 7fa078481640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa070161dd0 con 0x7fa070161940 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.525+0000 7fa078481640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0700cfa80 con 0x7fa07006c7f0 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.525+0000 7fa078481640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0701628e0 con 0x7fa07015efd0 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa076a1e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa07621d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa07006c7f0 0x7fa07009a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.529 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa075a1c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa070161940 0x7fa07006cbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:33394/0 (socket says 172.21.15.99:33394) 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa076a1e640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3301/0 says I am v2:172.21.15.99:57970/0 (socket says 172.21.15.99:57970) 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa075a1c640 1 -- 172.21.15.99:0/3398524656 learned_addr learned my addr 172.21.15.99:0/3398524656 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa07621d640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fa07006c7f0 0x7fa07009a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.99:48986/0 (socket says 172.21.15.99:48986) 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa07621d640 1 -- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 msgr2=0x7fa0700d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa07621d640 1 --2- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa07621d640 1 -- 172.21.15.99:0/3398524656 >> v1:172.21.15.162:6789/0 conn(0x7fa070161940 legacy=0x7fa07006cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.530 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.526+0000 7fa07621d640 1 -- 172.21.15.99:0/3398524656 --> v2:172.21.15.162:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa070162040 con 0x7fa07006c7f0 2024-03-04T10:12:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.527+0000 7fa07621d640 1 --2- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa07006c7f0 0x7fa07009a4f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0097d0 tx=0x7fa06c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21ce08ac33ff2bb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.527+0000 7fa07521b640 1 -- 172.21.15.99:0/3398524656 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c035020 con 0x7fa07006c7f0 2024-03-04T10:12:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.527+0000 7fa07521b640 1 -- 172.21.15.99:0/3398524656 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c031800 con 0x7fa07006c7f0 2024-03-04T10:12:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.527+0000 7fa07521b640 1 -- 172.21.15.99:0/3398524656 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c031bd0 con 0x7fa07006c7f0 2024-03-04T10:12:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 -- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa07006c7f0 msgr2=0x7fa07009a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.534 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 --2- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa07006c7f0 0x7fa07009a4f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0097d0 tx=0x7fa06c02b8a0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 -- 172.21.15.99:0/3398524656 shutdown_connections 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 --2- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 --2- 172.21.15.99:0/3398524656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa07006c7f0 0x7fa07009a4f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 -- 172.21.15.99:0/3398524656 >> 172.21.15.99:0/3398524656 conn(0x7fa0700d1840 msgr2=0x7fa0700d1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 -- 172.21.15.99:0/3398524656 shutdown_connections 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.528+0000 7fa078481640 1 -- 172.21.15.99:0/3398524656 wait complete. 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 Processor -- start 2024-03-04T10:12:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 -- start start 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa07006c7f0 0x7fa07015c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa070161940 0x7fa0700b67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700cfcb0 con 0x7fa07006c7f0 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070162600 con 0x7fa070161940 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.529+0000 7fa078481640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700d0710 con 0x7fa07015efd0 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa07006c7f0 0x7fa07015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa07006c7f0 0x7fa07015c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40110/0 (socket says 172.21.15.99:40110) 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 -- 172.21.15.99:0/3569597101 learned_addr learned my addr 172.21.15.99:0/3569597101 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa075a1c640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa076a1e640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa070161940 0x7fa0700b67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 -- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 msgr2=0x7fa0700c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 -- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa070161940 msgr2=0x7fa0700b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa070161940 0x7fa0700b67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 -- 172.21.15.99:0/3569597101 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0700c7870 con 0x7fa07006c7f0 2024-03-04T10:12:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa075a1c640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa07015efd0 0x7fa0700c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa076a1e640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa070161940 0x7fa0700b67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa07621d640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa07006c7f0 0x7fa07015c360 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098f0 tx=0x7fa06c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c041070 con 0x7fa07006c7f0 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c03ad00 con 0x7fa07006c7f0 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c035040 con 0x7fa07006c7f0 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0700c7b50 con 0x7fa07006c7f0 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.530+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0701e87f0 con 0x7fa07006c7f0 2024-03-04T10:12:10.538 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.531+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0701e8300 con 0x7fa07006c7f0 2024-03-04T10:12:10.539 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa06c04c060 con 0x7fa07006c7f0 2024-03-04T10:12:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa066ffd640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa05406b4d0 0x7fa05406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fa06c0abe30 con 0x7fa07006c7f0 2024-03-04T10:12:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa066ffd640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7fa054071ec0 0x7fa054074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa06c038ec0 con 0x7fa054071ec0 2024-03-04T10:12:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa06c0ae530 con 0x7fa07006c7f0 2024-03-04T10:12:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa075a1c640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa05406b4d0 0x7fa05406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.532+0000 7fa076a1e640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7fa054071ec0 0x7fa054074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.533+0000 7fa076a1e640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7fa054071ec0 0x7fa054074300 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.533+0000 7fa075a1c640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa05406b4d0 0x7fa05406d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa0700b7740 tx=0x7fa060009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.541 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.533+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== osd.3 v2:172.21.15.99:6808/85197782 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa06c038ec0 con 0x7fa054071ec0 2024-03-04T10:12:10.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.546+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 --> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa070074f60 con 0x7fa054071ec0 2024-03-04T10:12:10.549 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.546+0000 7fa066ffd640 1 -- 172.21.15.99:0/3569597101 <== osd.3 v2:172.21.15.99:6808/85197782 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa070074f60 con 0x7fa054071ec0 2024-03-04T10:12:10.549 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:10.549 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:10.549 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:10.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7fa054071ec0 msgr2=0x7fa054074300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.550 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:6808/85197782,v1:172.21.15.99:6809/85197782] conn(0x7fa054071ec0 0x7fa054074300 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa05406b4d0 msgr2=0x7fa05406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa05406b4d0 0x7fa05406d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa0700b7740 tx=0x7fa060009040 comp rx=0 tx=0).stop 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa07006c7f0 msgr2=0x7fa07015c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 --2- 172.21.15.99:0/3569597101 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa07006c7f0 0x7fa07015c360 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098f0 tx=0x7fa06c0046c0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa076a1e640 1 -- 172.21.15.99:0/3569597101 reap_dead start 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 shutdown_connections 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 >> 172.21.15.99:0/3569597101 conn(0x7fa0700d1840 msgr2=0x7fa0700d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 shutdown_connections 2024-03-04T10:12:10.552 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.547+0000 7fa078481640 1 -- 172.21.15.99:0/3569597101 wait complete. 2024-03-04T10:12:10.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-04T10:12:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.658+0000 7f590faca640 1 Processor -- start 2024-03-04T10:12:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590faca640 1 -- start start 2024-03-04T10:12:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590faca640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f590815de00 0x7f59080b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590faca640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f59080b9380 0x7f59080b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590faca640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590815e1e0 con 0x7f59080b70d0 2024-03-04T10:12:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590faca640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080d2820 con 0x7f59080b9380 2024-03-04T10:12:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590faca640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080d1c10 con 0x7f590815de00 2024-03-04T10:12:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.659+0000 7f590d065640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f59080b9380 0x7f59080b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590e067640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f59080b70d0 0x7f59080b5470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56978/0 (socket says 172.21.15.99:56978) 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590e067640 1 -- 172.21.15.99:0/809409190 learned_addr learned my addr 172.21.15.99:0/809409190 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590d065640 1 --2- 172.21.15.99:0/809409190 >> v2:172.21.15.99:3300/0 conn(0x7f59080b9380 0x7f59080b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40124/0 (socket says 172.21.15.99:40124) 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590d866640 1 --2- 172.21.15.99:0/809409190 >> v2:172.21.15.162:3300/0 conn(0x7f590815de00 0x7f59080b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590d065640 1 -- 172.21.15.99:0/809409190 >> v2:172.21.15.162:3300/0 conn(0x7f590815de00 msgr2=0x7f59080b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590d065640 1 --2- 172.21.15.99:0/809409190 >> v2:172.21.15.162:3300/0 conn(0x7f590815de00 0x7f59080b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590d065640 1 -- 172.21.15.99:0/809409190 >> v1:172.21.15.99:6789/0 conn(0x7f59080b70d0 legacy=0x7f59080b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590d065640 1 -- 172.21.15.99:0/809409190 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080d1ef0 con 0x7f59080b9380 2024-03-04T10:12:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.660+0000 7f590c864640 1 -- 172.21.15.99:0/809409190 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236521230 0 0) 0x7f590815e1e0 con 0x7f59080b70d0 2024-03-04T10:12:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590d065640 1 --2- 172.21.15.99:0/809409190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b9380 0x7f59080b6b90 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f59080ce240 tx=0x7f58f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38dcaf595d890c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590c864640 1 -- 172.21.15.99:0/809409190 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8035020 con 0x7f59080b9380 2024-03-04T10:12:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590c864640 1 -- 172.21.15.99:0/809409190 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f8031830 con 0x7f59080b9380 2024-03-04T10:12:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590c864640 1 -- 172.21.15.99:0/809409190 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8031c00 con 0x7f59080b9380 2024-03-04T10:12:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 -- 172.21.15.99:0/809409190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b9380 msgr2=0x7f59080b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 --2- 172.21.15.99:0/809409190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b9380 0x7f59080b6b90 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f59080ce240 tx=0x7f58f802b8d0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 -- 172.21.15.99:0/809409190 shutdown_connections 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 --2- 172.21.15.99:0/809409190 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b9380 0x7f59080b6b90 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 --2- 172.21.15.99:0/809409190 >> v2:172.21.15.162:3300/0 conn(0x7f590815de00 0x7f59080b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 -- 172.21.15.99:0/809409190 >> 172.21.15.99:0/809409190 conn(0x7f59080d0470 msgr2=0x7f59080d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 -- 172.21.15.99:0/809409190 shutdown_connections 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.661+0000 7f590faca640 1 -- 172.21.15.99:0/809409190 wait complete. 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.662+0000 7f590faca640 1 Processor -- start 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.662+0000 7f590faca640 1 -- start start 2024-03-04T10:12:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590faca640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 0x7f5908151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590faca640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59080b9380 0x7f59080aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590faca640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590815de00 0x7f59080cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590faca640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080b97e0 con 0x7f59080b70d0 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590faca640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080d24b0 con 0x7f59080b9380 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590faca640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080b5b80 con 0x7f590815de00 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 0x7f5908151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 0x7f5908151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40132/0 (socket says 172.21.15.99:40132) 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 -- 172.21.15.99:0/920165392 learned_addr learned my addr 172.21.15.99:0/920165392 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590e067640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590815de00 0x7f59080cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d065640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59080b9380 0x7f59080aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 -- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590815de00 msgr2=0x7f59080cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590815de00 0x7f59080cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 -- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59080b9380 msgr2=0x7f59080aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59080b9380 0x7f59080aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.663+0000 7f590d866640 1 -- 172.21.15.99:0/920165392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080ccec0 con 0x7f59080b70d0 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.664+0000 7f590d866640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 0x7f5908151290 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f5904007e40 tx=0x7f590400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.664+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904014620 con 0x7f59080b70d0 2024-03-04T10:12:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.664+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5904019070 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.664+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59080cd1a0 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.664+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904014a00 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.664+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59080cd760 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.665+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5908074f60 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.665+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5904014ba0 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.666+0000 7f58fe7fc640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58ec06b4d0 0x7f58ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.666+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f590408abb0 con 0x7f59080b70d0 2024-03-04T10:12:10.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.666+0000 7f590d065640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58ec06b4d0 0x7f58ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.666+0000 7f590d065640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58ec06b4d0 0x7f58ec06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f58f8036000 tx=0x7f58f8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.670+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f5904059940 con 0x7f59080b70d0 2024-03-04T10:12:10.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~57s 2024-03-04T10:12:10.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~77s 2024-03-04T10:12:10.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~25s 2024-03-04T10:12:10.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~44s 2024-03-04T10:12:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.839+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f59080ccc60 con 0x7f58ec06b4d0 2024-03-04T10:12:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.840+0000 7f58fe7fc640 1 -- 172.21.15.99:0/920165392 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f58f80318b0 con 0x7f58ec06b4d0 2024-03-04T10:12:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 3 to deep-scrub 2024-03-04T10:12:10.843 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58ec06b4d0 msgr2=0x7f58ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58ec06b4d0 0x7f58ec06d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f58f8036000 tx=0x7f58f8008030 comp rx=0 tx=0).stop 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 msgr2=0x7f5908151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 0x7f5908151290 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f5904007e40 tx=0x7f590400e4b0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 shutdown_connections 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f58ec06b4d0 0x7f58ec06d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f590815de00 0x7f59080cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f59080b9380 0x7f59080aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 --2- 172.21.15.99:0/920165392 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f59080b70d0 0x7f5908151290 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 >> 172.21.15.99:0/920165392 conn(0x7f59080d0470 msgr2=0x7f59080b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 shutdown_connections 2024-03-04T10:12:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:10.842+0000 7f590faca640 1 -- 172.21.15.99:0/920165392 wait complete. 2024-03-04T10:12:10.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:10.855 INFO:tasks.ceph:Scrubbing osd.4 2024-03-04T10:12:10.855 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.032+0000 7f9a64412640 1 Processor -- start 2024-03-04T10:12:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.032+0000 7f9a64412640 1 -- start start 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a64412640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a5c0ceff0 0x7f9a5c1630b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a64412640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c071a70 0x7f9a5c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a64412640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c0b4890 con 0x7f9a5c0b4220 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a64412640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b3670 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a64412640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1625d0 con 0x7f9a5c071a70 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a621ae640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a5c0ceff0 0x7f9a5c1630b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a621ae640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f9a5c0ceff0 0x7f9a5c1630b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40138/0 (socket says 172.21.15.99:40138) 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a621ae640 1 -- 172.21.15.99:0/3726671444 learned_addr learned my addr 172.21.15.99:0/3726671444 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.033+0000 7f9a629af640 1 --2- 172.21.15.99:0/3726671444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c071a70 0x7f9a5c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a621ae640 1 -- 172.21.15.99:0/3726671444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c071a70 msgr2=0x7f9a5c0b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a621ae640 1 --2- 172.21.15.99:0/3726671444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c071a70 0x7f9a5c0b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a621ae640 1 -- 172.21.15.99:0/3726671444 >> v1:172.21.15.162:6789/0 conn(0x7f9a5c0b4220 legacy=0x7f9a5c071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a621ae640 1 -- 172.21.15.99:0/3726671444 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c1628b0 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a621ae640 1 --2- 172.21.15.99:0/3726671444 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c1630b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f9a58009e50 tx=0x7f9a5802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48ab7bf4c23344d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a611ac640 1 -- 172.21.15.99:0/3726671444 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5802ca70 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a611ac640 1 -- 172.21.15.99:0/3726671444 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5802cc10 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.034+0000 7f9a611ac640 1 -- 172.21.15.99:0/3726671444 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580343f0 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 -- 172.21.15.99:0/3726671444 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 msgr2=0x7f9a5c1630b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 --2- 172.21.15.99:0/3726671444 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c1630b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f9a58009e50 tx=0x7f9a5802bed0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 -- 172.21.15.99:0/3726671444 shutdown_connections 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 --2- 172.21.15.99:0/3726671444 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c071a70 0x7f9a5c0b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 --2- 172.21.15.99:0/3726671444 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c1630b0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 -- 172.21.15.99:0/3726671444 >> 172.21.15.99:0/3726671444 conn(0x7f9a5c0b1600 msgr2=0x7f9a5c0b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 -- 172.21.15.99:0/3726671444 shutdown_connections 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.035+0000 7f9a64412640 1 -- 172.21.15.99:0/3726671444 wait complete. 2024-03-04T10:12:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.036+0000 7f9a64412640 1 Processor -- start 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.036+0000 7f9a64412640 1 -- start start 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a64412640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c071a70 0x7f9a5c1f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a64412640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c0b4220 0x7f9a5c0ab510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a64412640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c0aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a64412640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b4b70 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a64412640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0cf3d0 con 0x7f9a5c071a70 2024-03-04T10:12:11.044 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a64412640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0cd7c0 con 0x7f9a5c0b4220 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c0aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c0aba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40146/0 (socket says 172.21.15.99:40146) 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 -- 172.21.15.99:0/1818899715 learned_addr learned my addr 172.21.15.99:0/1818899715 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a621ae640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c071a70 0x7f9a5c1f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a619ad640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c0b4220 0x7f9a5c0ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 -- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c0b4220 msgr2=0x7f9a5c0ab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f9a5c0b4220 0x7f9a5c0ab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 -- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c071a70 msgr2=0x7f9a5c1f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9a5c071a70 0x7f9a5c1f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.037+0000 7f9a629af640 1 -- 172.21.15.99:0/1818899715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c156d60 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a629af640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c0aba50 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400d800 tx=0x7f9a5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400b6c0 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54019070 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c0ac520 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400bac0 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c0acae0 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.038+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9a5c074f60 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.039+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f9a5400bc60 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.039+0000 7f9a52ffd640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4006b500 0x7f9a4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f9a5408a540 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a52ffd640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f9a40071ef0 0x7f9a40074330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9a54001d30 con 0x7f9a40071ef0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a621ae640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4006b500 0x7f9a4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a54091050 con 0x7f9a5c0ceff0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a619ad640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f9a40071ef0 0x7f9a40074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.047 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a621ae640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4006b500 0x7f9a4006d9c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9a58035000 tx=0x7f9a58030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.040+0000 7f9a619ad640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f9a40071ef0 0x7f9a40074330 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.049 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.041+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== osd.4 v2:172.21.15.162:6800/3079837853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9a54001d30 con 0x7f9a40071ef0 2024-03-04T10:12:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.054+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 --> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9a5c0b1bf0 con 0x7f9a40071ef0 2024-03-04T10:12:11.057 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a52ffd640 1 -- 172.21.15.99:0/1818899715 <== osd.4 v2:172.21.15.162:6800/3079837853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9a5c0b1bf0 con 0x7f9a40071ef0 2024-03-04T10:12:11.057 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:11.058 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:11.058 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f9a40071ef0 msgr2=0x7f9a40074330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6800/3079837853,v1:172.21.15.162:6801/3079837853] conn(0x7f9a40071ef0 0x7f9a40074330 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4006b500 msgr2=0x7f9a4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f9a4006b500 0x7f9a4006d9c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9a58035000 tx=0x7f9a58030000 comp rx=0 tx=0).stop 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 msgr2=0x7f9a5c0aba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 --2- 172.21.15.99:0/1818899715 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f9a5c0ceff0 0x7f9a5c0aba50 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400d800 tx=0x7f9a5400dcc0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a629af640 1 -- 172.21.15.99:0/1818899715 reap_dead start 2024-03-04T10:12:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 shutdown_connections 2024-03-04T10:12:11.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 >> 172.21.15.99:0/1818899715 conn(0x7f9a5c0b1600 msgr2=0x7f9a5c165290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 shutdown_connections 2024-03-04T10:12:11.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.055+0000 7f9a64412640 1 -- 172.21.15.99:0/1818899715 wait complete. 2024-03-04T10:12:11.068 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-04T10:12:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 Processor -- start 2024-03-04T10:12:11.170 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 -- start start 2024-03-04T10:12:11.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64e00b2720 0x7f64e0071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f64e0071fc0 0x7f64e006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64e0162de0 con 0x7f64e00aead0 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e01630c0 con 0x7f64e00b2720 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.167+0000 7f64e48b1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64e01633a0 con 0x7f64e0071fc0 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64d7fff640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64e00b2720 0x7f64e0071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64de59c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f64e00aead0 0x7f64e0162590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56990/0 (socket says 172.21.15.99:56990) 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64de59c640 1 -- 172.21.15.99:0/1748681088 learned_addr learned my addr 172.21.15.99:0/1748681088 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64ded9d640 1 --2- 172.21.15.99:0/1748681088 >> v2:172.21.15.162:3300/0 conn(0x7f64e0071fc0 0x7f64e006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64d7fff640 1 -- 172.21.15.99:0/1748681088 >> v2:172.21.15.162:3300/0 conn(0x7f64e0071fc0 msgr2=0x7f64e006d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64d7fff640 1 --2- 172.21.15.99:0/1748681088 >> v2:172.21.15.162:3300/0 conn(0x7f64e0071fc0 0x7f64e006d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64d7fff640 1 -- 172.21.15.99:0/1748681088 >> v1:172.21.15.99:6789/0 conn(0x7f64e00aead0 legacy=0x7f64e0162590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64d7fff640 1 -- 172.21.15.99:0/1748681088 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e00aeeb0 con 0x7f64e00b2720 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.168+0000 7f64ddd9b640 1 -- 172.21.15.99:0/1748681088 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926048619 0 0) 0x7f64e0162de0 con 0x7f64e00aead0 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64d7fff640 1 --2- 172.21.15.99:0/1748681088 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0071a80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f64e00727a0 tx=0x7f64c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14316c1f209d7453 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64ddd9b640 1 -- 172.21.15.99:0/1748681088 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c80315a0 con 0x7f64e00b2720 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64ddd9b640 1 -- 172.21.15.99:0/1748681088 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8037070 con 0x7f64e00b2720 2024-03-04T10:12:11.173 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64ddd9b640 1 -- 172.21.15.99:0/1748681088 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8031960 con 0x7f64e00b2720 2024-03-04T10:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 -- 172.21.15.99:0/1748681088 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 msgr2=0x7f64e0071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1748681088 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0071a80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f64e00727a0 tx=0x7f64c802b840 comp rx=0 tx=0).stop 2024-03-04T10:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 -- 172.21.15.99:0/1748681088 shutdown_connections 2024-03-04T10:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1748681088 >> v2:172.21.15.162:3300/0 conn(0x7f64e0071fc0 0x7f64e006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1748681088 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0071a80 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 -- 172.21.15.99:0/1748681088 >> 172.21.15.99:0/1748681088 conn(0x7f64e006acd0 msgr2=0x7f64e006b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 -- 172.21.15.99:0/1748681088 shutdown_connections 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.169+0000 7f64e48b1640 1 -- 172.21.15.99:0/1748681088 wait complete. 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 Processor -- start 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 -- start start 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e0071fc0 0x7f64e0148120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e00aead0 0x7f64e01fb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e006dde0 con 0x7f64e00b2720 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e006dc60 con 0x7f64e00aead0 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.171+0000 7f64e48b1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e00b31a0 con 0x7f64e0071fc0 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0144c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40156/0 (socket says 172.21.15.99:40156) 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 -- 172.21.15.99:0/1091408670 learned_addr learned my addr 172.21.15.99:0/1091408670 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 -- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e0071fc0 msgr2=0x7f64e0148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e0071fc0 0x7f64e0148120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.179 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 -- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e00aead0 msgr2=0x7f64e01fb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e00aead0 0x7f64e01fb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 -- 172.21.15.99:0/1091408670 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e01fb7e0 con 0x7f64e00b2720 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64ded9d640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0144c30 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f64d8007e40 tx=0x7f64d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8014620 con 0x7f64e00b2720 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64d8019070 con 0x7f64e00b2720 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e01fbac0 con 0x7f64e00b2720 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64d8014a00 con 0x7f64e00b2720 2024-03-04T10:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.172+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64e0145720 con 0x7f64e00b2720 2024-03-04T10:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.173+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64e0074f60 con 0x7f64e00b2720 2024-03-04T10:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.173+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f64d8014ba0 con 0x7f64e00b2720 2024-03-04T10:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.174+0000 7f64d6ffd640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64c006b4d0 0x7f64c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.174+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f64d808ac20 con 0x7f64e00b2720 2024-03-04T10:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.174+0000 7f64de59c640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64c006b4d0 0x7f64c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.175+0000 7f64de59c640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64c006b4d0 0x7f64c006d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f64d0007fd0 tx=0x7f64d0007920 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.182 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.178+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f64d80599b0 con 0x7f64e00b2720 2024-03-04T10:12:11.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.348+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f64e0163680 con 0x7f64c006b4d0 2024-03-04T10:12:11.351 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.349+0000 7f64d6ffd640 1 -- 172.21.15.99:0/1091408670 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f64e0163680 con 0x7f64c006b4d0 2024-03-04T10:12:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 4 to deep-scrub 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64c006b4d0 msgr2=0x7f64c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64c006b4d0 0x7f64c006d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f64d0007fd0 tx=0x7f64d0007920 comp rx=0 tx=0).stop 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 msgr2=0x7f64e0144c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0144c30 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f64d8007e40 tx=0x7f64d800e4b0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 shutdown_connections 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f64c006b4d0 0x7f64c006d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f64e00b2720 0x7f64e0144c30 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f64e00aead0 0x7f64e01fb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 --2- 172.21.15.99:0/1091408670 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f64e0071fc0 0x7f64e0148120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 >> 172.21.15.99:0/1091408670 conn(0x7f64e006acd0 msgr2=0x7f64e00b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 shutdown_connections 2024-03-04T10:12:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.351+0000 7f64e48b1640 1 -- 172.21.15.99:0/1091408670 wait complete. 2024-03-04T10:12:11.364 INFO:tasks.ceph:Scrubbing osd.5 2024-03-04T10:12:11.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:11.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.466+0000 7fc0f1c6c640 1 Processor -- start 2024-03-04T10:12:11.468 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.466+0000 7fc0f1c6c640 1 -- start start 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.466+0000 7fc0f1c6c640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fc0ec15db20 0x7fc0ec0b9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.466+0000 7fc0f1c6c640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec0ce120 con 0x7fc0ec0d3c20 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.466+0000 7fc0f1c6c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ec0ced90 con 0x7fc0ec0b5bc0 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.466+0000 7fc0f1c6c640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0ec0ce730 con 0x7fc0ec15db20 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0eaffd640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fc0ec0d3c20 0x7fc0ec0b54b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:56994/0 (socket says 172.21.15.99:56994) 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0eaffd640 1 -- 172.21.15.99:0/4182707001 learned_addr learned my addr 172.21.15.99:0/4182707001 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122103734 0 0) 0x7fc0ec0ce120 con 0x7fc0ec0d3c20 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d4003520 con 0x7fc0ec0d3c20 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0eb7fe640 1 --2- 172.21.15.99:0/4182707001 >> v2:172.21.15.162:3300/0 conn(0x7fc0ec15db20 0x7fc0ec0b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3073813882 0 0) 0x7fc0d4003520 con 0x7fc0ec0d3c20 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 >> v2:172.21.15.162:3300/0 conn(0x7fc0ec15db20 msgr2=0x7fc0ec0b9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 --2- 172.21.15.99:0/4182707001 >> v2:172.21.15.162:3300/0 conn(0x7fc0ec15db20 0x7fc0ec0b9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 >> v1:172.21.15.162:6789/0 conn(0x7fc0ec0b5bc0 legacy=0x7fc0ec0b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec0ce3c0 con 0x7fc0ec0d3c20 2024-03-04T10:12:11.473 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fc0e0001670 con 0x7fc0ec0d3c20 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 >> v1:172.21.15.99:6789/0 conn(0x7fc0ec0d3c20 legacy=0x7fc0ec0b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d40039b0 0x7fc0d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d4004800 0x7fc0d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d40051c0 0x7fc0d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec0ce730 con 0x7fc0d4004800 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d400c070 con 0x7fc0d40039b0 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d400c270 con 0x7fc0d40051c0 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d4004800 0x7fc0d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0eb7fe640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d40039b0 0x7fc0d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.467+0000 7fc0ebfff640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d40051c0 0x7fc0d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0eaffd640 1 -- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d40051c0 msgr2=0x7fc0d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0d40051c0 0x7fc0d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0eaffd640 1 -- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d40039b0 msgr2=0x7fc0d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d40039b0 0x7fc0d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0ebfff640 1 -- 172.21.15.99:0/4182707001 reap_dead start 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0eaffd640 1 -- 172.21.15.99:0/4182707001 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4003520 con 0x7fc0d4004800 2024-03-04T10:12:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.468+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d4004800 0x7fc0d4004c80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000c240 tx=0x7fc0d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c0fdb833a0237f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0007cf0 con 0x7fc0d4004800 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0030070 con 0x7fc0d4004800 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0ea7fc640 1 -- 172.21.15.99:0/4182707001 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0005d30 con 0x7fc0d4004800 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/4182707001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d4004800 msgr2=0x7fc0d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d4004800 0x7fc0d4004c80 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000c240 tx=0x7fc0d4005830 comp rx=0 tx=0).stop 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/4182707001 shutdown_connections 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 --2- 172.21.15.99:0/4182707001 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0d4004800 0x7fc0d4004c80 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/4182707001 >> 172.21.15.99:0/4182707001 conn(0x7fc0ec072820 msgr2=0x7fc0ec072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.476 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/4182707001 shutdown_connections 2024-03-04T10:12:11.477 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.469+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/4182707001 wait complete. 2024-03-04T10:12:11.480 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.470+0000 7fc0f1c6c640 1 Processor -- start 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.470+0000 7fc0f1c6c640 1 -- start start 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0f1c6c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d4004800 0x7fc0ec152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0f1c6c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0ec15ee80 0x7fc0ec0c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0f1c6c640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0ec15fcc0 0x7fc0ec0cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0f1c6c640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec15df00 con 0x7fc0ec15ee80 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0f1c6c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec0b6300 con 0x7fc0d4004800 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0f1c6c640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec15e080 con 0x7fc0ec15fcc0 2024-03-04T10:12:11.481 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0eaffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0ec15ee80 0x7fc0ec0c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0eaffd640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0ec15ee80 0x7fc0ec0c85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40166/0 (socket says 172.21.15.99:40166) 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0eaffd640 1 -- 172.21.15.99:0/1495164819 learned_addr learned my addr 172.21.15.99:0/1495164819 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.471+0000 7fc0eaffd640 1 -- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0ec15fcc0 msgr2=0x7fc0ec0cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fc0ec15fcc0 0x7fc0ec0cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0eaffd640 1 -- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d4004800 msgr2=0x7fc0ec152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0eb7fe640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d4004800 0x7fc0ec152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0d4004800 0x7fc0ec152c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0eaffd640 1 -- 172.21.15.99:0/1495164819 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ec0c8e40 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0eaffd640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0ec15ee80 0x7fc0ec0c85c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fc0e002c500 tx=0x7fc0ec0cef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0045070 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0004bf0 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec0c9120 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0007cf0 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.472+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec15e680 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.473+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0ec074f60 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.473+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fc0e0007070 con 0x7fc0ec15ee80 2024-03-04T10:12:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0d3fff640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0d806b4d0 0x7fc0d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fc0e00ab8a0 con 0x7fc0ec15ee80 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0d3fff640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7fc0d8071ec0 0x7fc0d8074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc0d80749d0 con 0x7fc0d8071ec0 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0e00abd80 con 0x7fc0ec15ee80 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0eb7fe640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0d806b4d0 0x7fc0d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0ebfff640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7fc0d8071ec0 0x7fc0d8074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0eb7fe640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0d806b4d0 0x7fc0d806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec06d2e0 tx=0x7fc0dc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.474+0000 7fc0ebfff640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7fc0d8071ec0 0x7fc0d8074300 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.475+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== osd.5 v2:172.21.15.162:6808/1764948489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc0d80749d0 con 0x7fc0d8071ec0 2024-03-04T10:12:11.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.488+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 --> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc0ec15d0d0 con 0x7fc0d8071ec0 2024-03-04T10:12:11.491 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0d3fff640 1 -- 172.21.15.99:0/1495164819 <== osd.5 v2:172.21.15.162:6808/1764948489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fc0ec15d0d0 con 0x7fc0d8071ec0 2024-03-04T10:12:11.491 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:11.492 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:11.492 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7fc0d8071ec0 msgr2=0x7fc0d8074300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6808/1764948489,v1:172.21.15.162:6809/1764948489] conn(0x7fc0d8071ec0 0x7fc0d8074300 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0d806b4d0 msgr2=0x7fc0d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fc0d806b4d0 0x7fc0d806d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec06d2e0 tx=0x7fc0dc008040 comp rx=0 tx=0).stop 2024-03-04T10:12:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0ec15ee80 msgr2=0x7fc0ec0c85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 --2- 172.21.15.99:0/1495164819 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc0ec15ee80 0x7fc0ec0c85c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fc0e002c500 tx=0x7fc0ec0cef60 comp rx=0 tx=0).stop 2024-03-04T10:12:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0ebfff640 1 -- 172.21.15.99:0/1495164819 reap_dead start 2024-03-04T10:12:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 shutdown_connections 2024-03-04T10:12:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 >> 172.21.15.99:0/1495164819 conn(0x7fc0ec072820 msgr2=0x7fc0ec0b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 shutdown_connections 2024-03-04T10:12:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.489+0000 7fc0f1c6c640 1 -- 172.21.15.99:0/1495164819 wait complete. 2024-03-04T10:12:11.502 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-04T10:12:11.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.600+0000 7f1e3af80640 1 Processor -- start 2024-03-04T10:12:11.603 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.600+0000 7f1e3af80640 1 -- start start 2024-03-04T10:12:11.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e3af80640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f1e340cc8d0 0x7f1e340cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e3af80640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3406fd20 con 0x7f1e3415dea0 2024-03-04T10:12:11.609 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e3af80640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e34070000 con 0x7f1e3406b3e0 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e3af80640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3406fad0 con 0x7f1e340cc8d0 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e38d1c640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f1e3415dea0 0x7f1e3406acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57002/0 (socket says 172.21.15.99:57002) 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e38d1c640 1 -- 172.21.15.99:0/1114009143 learned_addr learned my addr 172.21.15.99:0/1114009143 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e3951d640 1 --2- 172.21.15.99:0/1114009143 >> v2:172.21.15.162:3300/0 conn(0x7f1e340cc8d0 0x7f1e340cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413001678 0 0) 0x7f1e3406fd20 con 0x7f1e3415dea0 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.601+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e1c003520 con 0x7f1e3415dea0 2024-03-04T10:12:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877567930 0 0) 0x7f1e1c003520 con 0x7f1e3415dea0 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 >> v2:172.21.15.162:3300/0 conn(0x7f1e340cc8d0 msgr2=0x7f1e340cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 --2- 172.21.15.99:0/1114009143 >> v2:172.21.15.162:3300/0 conn(0x7f1e340cc8d0 0x7f1e340cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 >> v1:172.21.15.162:6789/0 conn(0x7f1e3406b3e0 legacy=0x7f1e340cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e340702e0 con 0x7f1e3415dea0 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f1e24001670 con 0x7f1e3415dea0 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e3951d640 1 --2- 172.21.15.99:0/1114009143 >> v2:172.21.15.162:3300/0 conn(0x7f1e340cc8d0 0x7f1e340cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 >> v1:172.21.15.99:6789/0 conn(0x7f1e3415dea0 legacy=0x7f1e3406acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.611 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e1c0039b0 0x7f1e1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e1c0051c0 0x7f1e1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3406fad0 con 0x7f1e1c004800 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c00c060 con 0x7f1e1c0051c0 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c00c260 con 0x7f1e1c0039b0 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e33fff640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e38d1c640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e1c0039b0 0x7f1e1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e3951d640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e1c0051c0 0x7f1e1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e33fff640 1 -- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e1c0039b0 msgr2=0x7f1e1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e33fff640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e1c0039b0 0x7f1e1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e33fff640 1 -- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e1c0051c0 msgr2=0x7f1e1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e33fff640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e1c0051c0 0x7f1e1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e33fff640 1 -- 172.21.15.99:0/1114009143 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c003520 con 0x7f1e1c004800 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e38d1c640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e1c0039b0 0x7f1e1c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e3951d640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e1c0051c0 0x7f1e1c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.602+0000 7f1e3951d640 1 -- 172.21.15.99:0/1114009143 reap_dead start 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e33fff640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e1c004c80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f1e28001750 tx=0x7f1e1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb84b39b1c685fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28001370 con 0x7f1e1c004800 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e280052d0 con 0x7f1e1c004800 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e337fe640 1 -- 172.21.15.99:0/1114009143 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e28003830 con 0x7f1e1c004800 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e3af80640 1 -- 172.21.15.99:0/1114009143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 msgr2=0x7f1e1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e1c004c80 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f1e28001750 tx=0x7f1e1c005830 comp rx=0 tx=0).stop 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e3af80640 1 -- 172.21.15.99:0/1114009143 shutdown_connections 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.603+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1114009143 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e1c004c80 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.604+0000 7f1e3af80640 1 -- 172.21.15.99:0/1114009143 >> 172.21.15.99:0/1114009143 conn(0x7f1e340d0710 msgr2=0x7f1e340d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.604+0000 7f1e3af80640 1 -- 172.21.15.99:0/1114009143 shutdown_connections 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.604+0000 7f1e3af80640 1 -- 172.21.15.99:0/1114009143 wait complete. 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 Processor -- start 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 -- start start 2024-03-04T10:12:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e340c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e340c6200 0x7f1e340c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e340c6660 0x7f1e341505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340d1380 con 0x7f1e1c004800 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340d1200 con 0x7f1e340c6200 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.605+0000 7f1e3af80640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e340d1500 con 0x7f1e340c6660 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e340c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e340c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40192/0 (socket says 172.21.15.99:40192) 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 -- 172.21.15.99:0/1961763258 learned_addr learned my addr 172.21.15.99:0/1961763258 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 -- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e340c6660 msgr2=0x7f1e341505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e340c6660 0x7f1e341505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 -- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e340c6200 msgr2=0x7f1e340c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e340c6200 0x7f1e340c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 -- 172.21.15.99:0/1961763258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e340b3560 con 0x7f1e1c004800 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e38d1c640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e340c8540 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400e690 tx=0x7f1e2400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400c640 con 0x7f1e1c004800 2024-03-04T10:12:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e24018070 con 0x7f1e1c004800 2024-03-04T10:12:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e340c56b0 con 0x7f1e1c004800 2024-03-04T10:12:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2400c9e0 con 0x7f1e1c004800 2024-03-04T10:12:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.606+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e340b3b00 con 0x7f1e1c004800 2024-03-04T10:12:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.607+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e34074f60 con 0x7f1e1c004800 2024-03-04T10:12:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.608+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f1e2400cb80 con 0x7f1e1c004800 2024-03-04T10:12:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.608+0000 7f1e317fa640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e2006b4d0 0x7f1e2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.608+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f1e24089ff0 con 0x7f1e1c004800 2024-03-04T10:12:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.608+0000 7f1e33fff640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e2006b4d0 0x7f1e2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.609+0000 7f1e33fff640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e2006b4d0 0x7f1e2006d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1e28033000 tx=0x7f1e2802f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.612+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f1e24058e10 con 0x7f1e1c004800 2024-03-04T10:12:11.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.782+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e340d0f00 con 0x7f1e2006b4d0 2024-03-04T10:12:11.785 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.784+0000 7f1e317fa640 1 -- 172.21.15.99:0/1961763258 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e340d0f00 con 0x7f1e2006b4d0 2024-03-04T10:12:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 5 to deep-scrub 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e2006b4d0 msgr2=0x7f1e2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e2006b4d0 0x7f1e2006d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1e28033000 tx=0x7f1e2802f000 comp rx=0 tx=0).stop 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 msgr2=0x7f1e340c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e340c8540 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f1e2400e690 tx=0x7f1e2400eb50 comp rx=0 tx=0).stop 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 shutdown_connections 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f1e2006b4d0 0x7f1e2006d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f1e340c6660 0x7f1e341505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1e340c6200 0x7f1e340c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 --2- 172.21.15.99:0/1961763258 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f1e1c004800 0x7f1e340c8540 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 >> 172.21.15.99:0/1961763258 conn(0x7f1e340d0710 msgr2=0x7f1e340cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 shutdown_connections 2024-03-04T10:12:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.785+0000 7f1e3af80640 1 -- 172.21.15.99:0/1961763258 wait complete. 2024-03-04T10:12:11.798 INFO:tasks.ceph:Scrubbing osd.6 2024-03-04T10:12:11.798 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:11.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.900+0000 7f5bd40a4640 1 Processor -- start 2024-03-04T10:12:11.903 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.900+0000 7f5bd40a4640 1 -- start start 2024-03-04T10:12:11.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd40a4640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f5bcc06f1b0 0x7f5bcc0cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd40a4640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc06b790 con 0x7f5bcc0cfef0 2024-03-04T10:12:11.906 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd40a4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc06e6d0 con 0x7f5bcc0700e0 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd40a4640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bcc06e9b0 con 0x7f5bcc06f1b0 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd2641640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f5bcc0cfef0 0x7f5bcc15d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57004/0 (socket says 172.21.15.99:57004) 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd2641640 1 -- 172.21.15.99:0/3671092851 learned_addr learned my addr 172.21.15.99:0/3671092851 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd163f640 1 --2- 172.21.15.99:0/3671092851 >> v2:172.21.15.162:3300/0 conn(0x7f5bcc06f1b0 0x7f5bcc0cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588614631 0 0) 0x7f5bcc06b790 con 0x7f5bcc0cfef0 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.901+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb8003520 con 0x7f5bcc0cfef0 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1861274564 0 0) 0x7f5bb8003520 con 0x7f5bcc0cfef0 2024-03-04T10:12:11.907 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 >> v2:172.21.15.162:3300/0 conn(0x7f5bcc06f1b0 msgr2=0x7f5bcc0cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 --2- 172.21.15.99:0/3671092851 >> v2:172.21.15.162:3300/0 conn(0x7f5bcc06f1b0 0x7f5bcc0cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 >> v1:172.21.15.162:6789/0 conn(0x7f5bcc0700e0 legacy=0x7f5bcc06acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc06ec90 con 0x7f5bcc0cfef0 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f5bc40019f0 con 0x7f5bcc0cfef0 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 >> v1:172.21.15.99:6789/0 conn(0x7f5bcc0cfef0 legacy=0x7f5bcc15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bb80039b0 0x7f5bb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bb8004800 0x7f5bb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.908 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80051c0 0x7f5bb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc06e9b0 con 0x7f5bb80039b0 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb800c060 con 0x7f5bb80051c0 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb800c260 con 0x7f5bb8004800 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bb80039b0 0x7f5bb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd2641640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80051c0 0x7f5bb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd163f640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bb8004800 0x7f5bb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd1e40640 1 -- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bb8004800 msgr2=0x7f5bb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bb8004800 0x7f5bb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.909 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd1e40640 1 -- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80051c0 msgr2=0x7f5bb800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80051c0 0x7f5bb800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd1e40640 1 -- 172.21.15.99:0/3671092851 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8003520 con 0x7f5bb80039b0 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd2641640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80051c0 0x7f5bb800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd2641640 1 -- 172.21.15.99:0/3671092851 reap_dead start 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.902+0000 7f5bd163f640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bb8004800 0x7f5bb8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.903+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bb80039b0 0x7f5bb8003dd0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8001750 tx=0x7f5bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cce1fc4cdd863bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.903+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8001370 con 0x7f5bb80039b0 2024-03-04T10:12:11.910 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.903+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc80052d0 con 0x7f5bb80039b0 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.903+0000 7f5bd0e3e640 1 -- 172.21.15.99:0/3671092851 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8003830 con 0x7f5bb80039b0 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 -- 172.21.15.99:0/3671092851 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bb80039b0 msgr2=0x7f5bb8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bb80039b0 0x7f5bb8003dd0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8001750 tx=0x7f5bb8005830 comp rx=0 tx=0).stop 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 -- 172.21.15.99:0/3671092851 shutdown_connections 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 --2- 172.21.15.99:0/3671092851 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bb80039b0 0x7f5bb8003dd0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 -- 172.21.15.99:0/3671092851 >> 172.21.15.99:0/3671092851 conn(0x7f5bcc099a00 msgr2=0x7f5bcc0b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 -- 172.21.15.99:0/3671092851 shutdown_connections 2024-03-04T10:12:11.911 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.904+0000 7f5bd40a4640 1 -- 172.21.15.99:0/3671092851 wait complete. 2024-03-04T10:12:11.914 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.905+0000 7f5bd40a4640 1 Processor -- start 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.905+0000 7f5bd40a4640 1 -- start start 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80039b0 0x7f5bcc0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bcc09b610 0x7f5bcc0ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bcc160040 0x7f5bcc0b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0704c0 con 0x7f5bcc160040 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0b3000 con 0x7f5bb80039b0 2024-03-04T10:12:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0cf9b0 con 0x7f5bcc09b610 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bcc160040 0x7f5bcc0b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bcc160040 0x7f5bcc0b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40202/0 (socket says 172.21.15.99:40202) 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 -- 172.21.15.99:0/1115243247 learned_addr learned my addr 172.21.15.99:0/1115243247 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd163f640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bcc09b610 0x7f5bcc0ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80039b0 0x7f5bcc0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 -- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bcc09b610 msgr2=0x7f5bcc0ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f5bcc09b610 0x7f5bcc0ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 -- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80039b0 msgr2=0x7f5bcc0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bb80039b0 0x7f5bcc0b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 -- 172.21.15.99:0/1115243247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc0b4330 con 0x7f5bcc160040 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd2641640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bcc160040 0x7f5bcc0b5480 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4001f00 tx=0x7f5bc4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc400ea80 con 0x7f5bcc160040 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc4018070 con 0x7f5bcc160040 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc400ee60 con 0x7f5bcc160040 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc0b4610 con 0x7f5bcc160040 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.906+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc0c7c70 con 0x7f5bcc160040 2024-03-04T10:12:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.907+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5bcc0ca760 con 0x7f5bcc160040 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f5bc401d460 con 0x7f5bcc160040 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bc27fc640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bb006b4d0 0x7f5bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f5bc408af70 con 0x7f5bcc160040 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bc27fc640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f5bb0071ec0 0x7f5bb0074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bc400fe50 con 0x7f5bb0071ec0 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bc408b400 con 0x7f5bcc160040 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bb006b4d0 0x7f5bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.908+0000 7f5bd163f640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f5bb0071ec0 0x7f5bb0074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:11.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.909+0000 7f5bd1e40640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bb006b4d0 0x7f5bb006d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8033000 tx=0x7f5bc8009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.909+0000 7f5bd163f640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f5bb0071ec0 0x7f5bb0074300 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:11.919 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.910+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== osd.6 v2:172.21.15.162:6824/1448511037 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5bc400fe50 con 0x7f5bb0071ec0 2024-03-04T10:12:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 --> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5bcc074f60 con 0x7f5bb0071ec0 2024-03-04T10:12:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bc27fc640 1 -- 172.21.15.99:0/1115243247 <== osd.6 v2:172.21.15.162:6824/1448511037 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5bcc074f60 con 0x7f5bb0071ec0 2024-03-04T10:12:11.926 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:11.926 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:11.926 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f5bb0071ec0 msgr2=0x7f5bb0074300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6824/1448511037,v1:172.21.15.162:6825/1448511037] conn(0x7f5bb0071ec0 0x7f5bb0074300 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bb006b4d0 msgr2=0x7f5bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f5bb006b4d0 0x7f5bb006d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8033000 tx=0x7f5bc8009030 comp rx=0 tx=0).stop 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bcc160040 msgr2=0x7f5bcc0b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.923+0000 7f5bd40a4640 1 --2- 172.21.15.99:0/1115243247 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5bcc160040 0x7f5bcc0b5480 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4001f00 tx=0x7f5bc4007bc0 comp rx=0 tx=0).stop 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.924+0000 7f5bd2641640 1 -- 172.21.15.99:0/1115243247 reap_dead start 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.924+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 shutdown_connections 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.924+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 >> 172.21.15.99:0/1115243247 conn(0x7f5bcc099a00 msgr2=0x7f5bcc0af540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.924+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 shutdown_connections 2024-03-04T10:12:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:11.924+0000 7f5bd40a4640 1 -- 172.21.15.99:0/1115243247 wait complete. 2024-03-04T10:12:11.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-04T10:12:12.038 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.035+0000 7f487b643640 1 Processor -- start 2024-03-04T10:12:12.486 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.035+0000 7f487b643640 1 -- start start 2024-03-04T10:12:12.500 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f487b643640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487406c470 0x7f487406f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f487b643640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f4874066de0 0x7f487415f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f487b643640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48740ad6c0 con 0x7f487406c090 2024-03-04T10:12:12.501 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f487b643640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48740ad960 con 0x7f4874066de0 2024-03-04T10:12:12.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f487b643640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487406ebc0 con 0x7f487406c470 2024-03-04T10:12:12.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f4878bde640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487406c470 0x7f487406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f4879be0640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f4874066de0 0x7f487415f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.502 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f48793df640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f487406c090 0x7f48740af220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.99:33430/0 (socket says 172.21.15.99:33430) 2024-03-04T10:12:12.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f48793df640 1 -- 172.21.15.99:0/479732644 learned_addr learned my addr 172.21.15.99:0/479732644 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:12.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f486bfff640 1 -- 172.21.15.99:0/479732644 <== mon.1 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232049029 0 0) 0x7f48740ad6c0 con 0x7f487406c090 2024-03-04T10:12:12.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f4878bde640 1 -- 172.21.15.99:0/479732644 >> v2:172.21.15.162:3300/0 conn(0x7f4874066de0 msgr2=0x7f487415f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:12.503 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f4878bde640 1 --2- 172.21.15.99:0/479732644 >> v2:172.21.15.162:3300/0 conn(0x7f4874066de0 0x7f487415f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f4878bde640 1 -- 172.21.15.99:0/479732644 >> v1:172.21.15.162:6789/0 conn(0x7f487406c090 legacy=0x7f48740af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.036+0000 7f4878bde640 1 -- 172.21.15.99:0/479732644 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f487406eea0 con 0x7f487406c470 2024-03-04T10:12:12.504 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.037+0000 7f4878bde640 1 --2- 172.21.15.99:0/479732644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487406c470 0x7f487406f680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48640097a0 tx=0x7f486402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9983f828eacae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.037+0000 7f486bfff640 1 -- 172.21.15.99:0/479732644 <== mon.2 v2:172.21.15.162:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48640315a0 con 0x7f487406c470 2024-03-04T10:12:12.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.037+0000 7f486bfff640 1 -- 172.21.15.99:0/479732644 <== mon.2 v2:172.21.15.162:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4864037070 con 0x7f487406c470 2024-03-04T10:12:12.505 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.037+0000 7f486bfff640 1 -- 172.21.15.99:0/479732644 <== mon.2 v2:172.21.15.162:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4864031960 con 0x7f487406c470 2024-03-04T10:12:12.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 -- 172.21.15.99:0/479732644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487406c470 msgr2=0x7f487406f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 --2- 172.21.15.99:0/479732644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487406c470 0x7f487406f680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48640097a0 tx=0x7f486402b840 comp rx=0 tx=0).stop 2024-03-04T10:12:12.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 -- 172.21.15.99:0/479732644 shutdown_connections 2024-03-04T10:12:12.506 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 --2- 172.21.15.99:0/479732644 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f487406c470 0x7f487406f680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 --2- 172.21.15.99:0/479732644 >> v2:172.21.15.162:3300/0 conn(0x7f4874066de0 0x7f487415f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 -- 172.21.15.99:0/479732644 >> 172.21.15.99:0/479732644 conn(0x7f48740cfef0 msgr2=0x7f48740d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:12.507 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 -- 172.21.15.99:0/479732644 shutdown_connections 2024-03-04T10:12:12.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.038+0000 7f487b643640 1 -- 172.21.15.99:0/479732644 wait complete. 2024-03-04T10:12:12.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.039+0000 7f487b643640 1 Processor -- start 2024-03-04T10:12:12.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.039+0000 7f487b643640 1 -- start start 2024-03-04T10:12:12.508 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f487b643640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4874066de0 0x7f487414ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f487b643640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487406c090 0x7f4874145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f487b643640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 0x7f4874150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f487b643640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48740addc0 con 0x7f487414e550 2024-03-04T10:12:12.509 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f487b643640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48740adc40 con 0x7f487406c090 2024-03-04T10:12:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f487b643640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487415faa0 con 0x7f4874066de0 2024-03-04T10:12:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 0x7f4874150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 0x7f4874150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40206/0 (socket says 172.21.15.99:40206) 2024-03-04T10:12:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 -- 172.21.15.99:0/241899616 learned_addr learned my addr 172.21.15.99:0/241899616 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:12.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f48793df640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4874066de0 0x7f487414ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4878bde640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487406c090 0x7f4874145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 -- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4874066de0 msgr2=0x7f487414ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.511 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4874066de0 0x7f487414ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 -- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487406c090 msgr2=0x7f4874145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487406c090 0x7f4874145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.040+0000 7f4879be0640 1 -- 172.21.15.99:0/241899616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48740b3910 con 0x7f487414e550 2024-03-04T10:12:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.041+0000 7f4879be0640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 0x7f4874150c00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f486c009e30 tx=0x7f486c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.041+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c018020 con 0x7f487414e550 2024-03-04T10:12:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.041+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c014480 con 0x7f487414e550 2024-03-04T10:12:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.041+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c014870 con 0x7f487414e550 2024-03-04T10:12:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.041+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487414f3c0 con 0x7f487414e550 2024-03-04T10:12:12.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.041+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48740b3e60 con 0x7f487414e550 2024-03-04T10:12:12.514 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.042+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4874074f60 con 0x7f487414e550 2024-03-04T10:12:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.042+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f486c005d10 con 0x7f487414e550 2024-03-04T10:12:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.043+0000 7f4869ffb640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485806b4d0 0x7f485806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.043+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f486c08b0b0 con 0x7f487414e550 2024-03-04T10:12:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.043+0000 7f48793df640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485806b4d0 0x7f485806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.515 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.043+0000 7f48793df640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485806b4d0 0x7f485806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4870000f60 tx=0x7f4870009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.047+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f486c022030 con 0x7f487414e550 2024-03-04T10:12:12.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.216+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f487406def0 con 0x7f485806b4d0 2024-03-04T10:12:12.516 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.217+0000 7f4869ffb640 1 -- 172.21.15.99:0/241899616 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f487406def0 con 0x7f485806b4d0 2024-03-04T10:12:12.516 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 6 to deep-scrub 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485806b4d0 msgr2=0x7f485806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485806b4d0 0x7f485806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4870000f60 tx=0x7f4870009040 comp rx=0 tx=0).stop 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 msgr2=0x7f4874150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 0x7f4874150c00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f486c009e30 tx=0x7f486c00c490 comp rx=0 tx=0).stop 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 shutdown_connections 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f485806b4d0 0x7f485806d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f487414e550 0x7f4874150c00 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f487406c090 0x7f4874145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 --2- 172.21.15.99:0/241899616 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f4874066de0 0x7f487414ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 >> 172.21.15.99:0/241899616 conn(0x7f48740cfef0 msgr2=0x7f487415fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 shutdown_connections 2024-03-04T10:12:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.219+0000 7f487b643640 1 -- 172.21.15.99:0/241899616 wait complete. 2024-03-04T10:12:12.518 INFO:tasks.ceph:Scrubbing osd.7 2024-03-04T10:12:12.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-04T10:12:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.623+0000 7f6722c73640 1 Processor -- start 2024-03-04T10:12:12.626 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.623+0000 7f6722c73640 1 -- start start 2024-03-04T10:12:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f6722c73640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f671c0671c0 0x7f671c0b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f6722c73640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c06af20 con 0x7f671c15dc80 2024-03-04T10:12:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f6722c73640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c06b200 con 0x7f671c0b2fe0 2024-03-04T10:12:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f6722c73640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f671c06acd0 con 0x7f671c0671c0 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f6720a0f640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f671c15dc80 0x7f671c06bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57008/0 (socket says 172.21.15.99:57008) 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f6720a0f640 1 -- 172.21.15.99:0/2596399938 learned_addr learned my addr 172.21.15.99:0/2596399938 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966537828 0 0) 0x7f671c06af20 con 0x7f671c15dc80 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6704003520 con 0x7f671c15dc80 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671bfff640 1 --2- 172.21.15.99:0/2596399938 >> v2:172.21.15.162:3300/0 conn(0x7f671c0671c0 0x7f671c0b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823112081 0 0) 0x7f6704003520 con 0x7f671c15dc80 2024-03-04T10:12:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 >> v2:172.21.15.162:3300/0 conn(0x7f671c0671c0 msgr2=0x7f671c0b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 --2- 172.21.15.99:0/2596399938 >> v2:172.21.15.162:3300/0 conn(0x7f671c0671c0 0x7f671c0b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 >> v1:172.21.15.162:6789/0 conn(0x7f671c0b2fe0 legacy=0x7f671c0ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c06b4e0 con 0x7f671c15dc80 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7f670c001670 con 0x7f671c15dc80 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 >> v1:172.21.15.99:6789/0 conn(0x7f671c15dc80 legacy=0x7f671c06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67040039b0 0x7f6704003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.632 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6704004800 0x7f6704004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67040051c0 0x7f670400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c06acd0 con 0x7f6704004800 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670400c110 con 0x7f67040051c0 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670400c310 con 0x7f67040039b0 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.624+0000 7f671bfff640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6704004800 0x7f6704004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f6720a0f640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67040039b0 0x7f6704003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f6721210640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67040051c0 0x7f670400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f671bfff640 1 -- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67040039b0 msgr2=0x7f6704003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f671bfff640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67040039b0 0x7f6704003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f671bfff640 1 -- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67040051c0 msgr2=0x7f670400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f671bfff640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f67040051c0 0x7f670400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f6721210640 1 -- 172.21.15.99:0/2596399938 reap_dead start 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f671bfff640 1 -- 172.21.15.99:0/2596399938 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704003520 con 0x7f6704004800 2024-03-04T10:12:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f6720a0f640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f67040039b0 0x7f6704003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.625+0000 7f671bfff640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6704004800 0x7f6704004c80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f671000ba30 tx=0x7f6704005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b17e3bb8e799a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710036020 con 0x7f6704004800 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710032640 con 0x7f6704004800 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f671b7fe640 1 -- 172.21.15.99:0/2596399938 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671000b600 con 0x7f6704004800 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 -- 172.21.15.99:0/2596399938 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6704004800 msgr2=0x7f6704004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6704004800 0x7f6704004c80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f671000ba30 tx=0x7f6704005830 comp rx=0 tx=0).stop 2024-03-04T10:12:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 -- 172.21.15.99:0/2596399938 shutdown_connections 2024-03-04T10:12:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 --2- 172.21.15.99:0/2596399938 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6704004800 0x7f6704004c80 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 -- 172.21.15.99:0/2596399938 >> 172.21.15.99:0/2596399938 conn(0x7f671c071ca0 msgr2=0x7f671c072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 -- 172.21.15.99:0/2596399938 shutdown_connections 2024-03-04T10:12:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.626+0000 7f6722c73640 1 -- 172.21.15.99:0/2596399938 wait complete. 2024-03-04T10:12:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.627+0000 7f6722c73640 1 Processor -- start 2024-03-04T10:12:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.627+0000 7f6722c73640 1 -- start start 2024-03-04T10:12:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6722c73640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6704004800 0x7f671c154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6722c73640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f671c0b0eb0 0x7f671c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6722c73640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f671c1433b0 0x7f671c143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6722c73640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c06baa0 con 0x7f671c0b0eb0 2024-03-04T10:12:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6722c73640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c0729b0 con 0x7f6704004800 2024-03-04T10:12:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6722c73640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671c06bfc0 con 0x7f671c1433b0 2024-03-04T10:12:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f671bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f671c0b0eb0 0x7f671c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f671bfff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f671c0b0eb0 0x7f671c155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40212/0 (socket says 172.21.15.99:40212) 2024-03-04T10:12:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f671bfff640 1 -- 172.21.15.99:0/881363882 learned_addr learned my addr 172.21.15.99:0/881363882 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f6720a0f640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6704004800 0x7f671c154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f671bfff640 1 -- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f671c1433b0 msgr2=0x7f671c143830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.628+0000 7f671bfff640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f671c1433b0 0x7f671c143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f671bfff640 1 -- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6704004800 msgr2=0x7f671c154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f671bfff640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6704004800 0x7f671c154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f671bfff640 1 -- 172.21.15.99:0/881363882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c143d70 con 0x7f671c0b0eb0 2024-03-04T10:12:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f671bfff640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f671c0b0eb0 0x7f671c155c30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f671002bf90 tx=0x7f6710004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710041070 con 0x7f671c0b0eb0 2024-03-04T10:12:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710030a50 con 0x7f671c0b0eb0 2024-03-04T10:12:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c154fc0 con 0x7f671c0b0eb0 2024-03-04T10:12:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710036040 con 0x7f671c0b0eb0 2024-03-04T10:12:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.629+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c155580 con 0x7f671c0b0eb0 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.630+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f671c1557c0 con 0x7f671c0b0eb0 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.630+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f671004c060 con 0x7f671c0b0eb0 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f67197fa640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670806b4d0 0x7f670806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f67100ac510 con 0x7f671c0b0eb0 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f67197fa640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7f6708071ec0 0x7f6708074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6710034e90 con 0x7f6708071ec0 2024-03-04T10:12:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67100ac8c0 con 0x7f671c0b0eb0 2024-03-04T10:12:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f6720a0f640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670806b4d0 0x7f670806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f6721210640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7f6708071ec0 0x7f6708074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.631+0000 7f6720a0f640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670806b4d0 0x7f670806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f670c000fb0 tx=0x7f670c001e00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.632+0000 7f6721210640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7f6708071ec0 0x7f6708074300 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.632+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== osd.7 v2:172.21.15.162:6816/2710319333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6710034e90 con 0x7f6708071ec0 2024-03-04T10:12:12.647 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.645+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 --> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f671c06af20 con 0x7f6708071ec0 2024-03-04T10:12:12.648 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f67197fa640 1 -- 172.21.15.99:0/881363882 <== osd.7 v2:172.21.15.162:6816/2710319333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f671c06af20 con 0x7f6708071ec0 2024-03-04T10:12:12.648 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-03-04T10:12:12.649 INFO:teuthology.orchestra.run.smithi099.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-04T10:12:12.649 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7f6708071ec0 msgr2=0x7f6708074300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6816/2710319333,v1:172.21.15.162:6817/2710319333] conn(0x7f6708071ec0 0x7f6708074300 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670806b4d0 msgr2=0x7f670806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f670806b4d0 0x7f670806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f670c000fb0 tx=0x7f670c001e00 comp rx=0 tx=0).stop 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f671c0b0eb0 msgr2=0x7f671c155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 --2- 172.21.15.99:0/881363882 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f671c0b0eb0 0x7f671c155c30 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f671002bf90 tx=0x7f6710004730 comp rx=0 tx=0).stop 2024-03-04T10:12:12.650 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6721210640 1 -- 172.21.15.99:0/881363882 reap_dead start 2024-03-04T10:12:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 shutdown_connections 2024-03-04T10:12:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 >> 172.21.15.99:0/881363882 conn(0x7f671c071ca0 msgr2=0x7f671c06f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 shutdown_connections 2024-03-04T10:12:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.646+0000 7f6722c73640 1 -- 172.21.15.99:0/881363882 wait complete. 2024-03-04T10:12:12.659 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-04T10:12:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.757+0000 7f8fd9d45640 1 Processor -- start 2024-03-04T10:12:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd9d45640 1 -- start start 2024-03-04T10:12:12.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd9d45640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8fd415de00 0x7f8fd40b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.768 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd9d45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b9380 0x7f8fd40b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd9d45640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd415e1e0 con 0x7f8fd40b70d0 2024-03-04T10:12:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd9d45640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd40d2820 con 0x7f8fd415de00 2024-03-04T10:12:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd9d45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40d1c10 con 0x7f8fd40b9380 2024-03-04T10:12:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8fd415de00 0x7f8fd40b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd37fe640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8fd415de00 0x7f8fd40b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40214/0 (socket says 172.21.15.99:40214) 2024-03-04T10:12:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.758+0000 7f8fd37fe640 1 -- 172.21.15.99:0/1419781297 learned_addr learned my addr 172.21.15.99:0/1419781297 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd2ffd640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b9380 0x7f8fd40b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd37fe640 1 -- 172.21.15.99:0/1419781297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b9380 msgr2=0x7f8fd40b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd37fe640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b9380 0x7f8fd40b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd37fe640 1 -- 172.21.15.99:0/1419781297 >> v1:172.21.15.162:6789/0 conn(0x7f8fd40b70d0 legacy=0x7f8fd40b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd37fe640 1 -- 172.21.15.99:0/1419781297 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40d1ef0 con 0x7f8fd415de00 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd2ffd640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b9380 0x7f8fd40b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.759+0000 7f8fd37fe640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd40b2e60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009870 tx=0x7f8fc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f7bc1f4ce2f043 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd27fc640 1 -- 172.21.15.99:0/1419781297 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4035020 con 0x7f8fd415de00 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd27fc640 1 -- 172.21.15.99:0/1419781297 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4031640 con 0x7f8fd415de00 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd27fc640 1 -- 172.21.15.99:0/1419781297 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc402d040 con 0x7f8fd415de00 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 -- 172.21.15.99:0/1419781297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 msgr2=0x7f8fd40b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd40b2e60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4009870 tx=0x7f8fc402b8f0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 -- 172.21.15.99:0/1419781297 shutdown_connections 2024-03-04T10:12:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b9380 0x7f8fd40b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/1419781297 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd40b2e60 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 -- 172.21.15.99:0/1419781297 >> 172.21.15.99:0/1419781297 conn(0x7f8fd40d0470 msgr2=0x7f8fd40d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 -- 172.21.15.99:0/1419781297 shutdown_connections 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.760+0000 7f8fd9d45640 1 -- 172.21.15.99:0/1419781297 wait complete. 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.761+0000 7f8fd9d45640 1 Processor -- start 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 -- start start 2024-03-04T10:12:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b70d0 0x7f8fd415f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd40b9380 0x7f8fd41608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd414b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b97e0 con 0x7f8fd415de00 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40d24b0 con 0x7f8fd40b9380 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd9d45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b5b80 con 0x7f8fd40b70d0 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd414b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd414b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40224/0 (socket says 172.21.15.99:40224) 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.762+0000 7f8fd3fff640 1 -- 172.21.15.99:0/3935339452 learned_addr learned my addr 172.21.15.99:0/3935339452 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd37fe640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b70d0 0x7f8fd415f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd2ffd640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd40b9380 0x7f8fd41608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd3fff640 1 -- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b70d0 msgr2=0x7f8fd415f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd3fff640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b70d0 0x7f8fd415f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd3fff640 1 -- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd40b9380 msgr2=0x7f8fd41608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd3fff640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd40b9380 0x7f8fd41608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd3fff640 1 -- 172.21.15.99:0/3935339452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd415e740 con 0x7f8fd415de00 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd2ffd640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd40b9380 0x7f8fd41608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd3fff640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd414b970 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c780 tx=0x7f8fcc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc004b60 con 0x7f8fd415de00 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc004d00 con 0x7f8fd415de00 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd415ea20 con 0x7f8fd415de00 2024-03-04T10:12:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc00aa90 con 0x7f8fd415de00 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.763+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd415efe0 con 0x7f8fd415de00 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.764+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4074f60 con 0x7f8fd415de00 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.765+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8fcc00ac30 con 0x7f8fd415de00 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.765+0000 7f8fbbfff640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8fc006b4d0 0x7f8fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.765+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f8fcc08a540 con 0x7f8fd415de00 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.765+0000 7f8fd37fe640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8fc006b4d0 0x7f8fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.766+0000 7f8fd37fe640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8fc006b4d0 0x7f8fc006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4036000 tx=0x7f8fc403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:12.777 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.769+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8fcc020070 con 0x7f8fd415de00 2024-03-04T10:12:12.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.939+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fd40cedf0 con 0x7f8fc006b4d0 2024-03-04T10:12:12.942 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.940+0000 7f8fbbfff640 1 -- 172.21.15.99:0/3935339452 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8fd40cedf0 con 0x7f8fc006b4d0 2024-03-04T10:12:12.942 INFO:teuthology.orchestra.run.smithi099.stderr:instructed osd(s) 7 to deep-scrub 2024-03-04T10:12:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8fc006b4d0 msgr2=0x7f8fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8fc006b4d0 0x7f8fc006d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4036000 tx=0x7f8fc403c040 comp rx=0 tx=0).stop 2024-03-04T10:12:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 msgr2=0x7f8fd414b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd414b970 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c780 tx=0x7f8fcc00cc40 comp rx=0 tx=0).stop 2024-03-04T10:12:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 shutdown_connections 2024-03-04T10:12:12.945 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8fc006b4d0 0x7f8fc006d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8fd415de00 0x7f8fd414b970 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8fd40b9380 0x7f8fd41608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 --2- 172.21.15.99:0/3935339452 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8fd40b70d0 0x7f8fd415f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:12.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 >> 172.21.15.99:0/3935339452 conn(0x7f8fd40d0470 msgr2=0x7f8fd40ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:12.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 shutdown_connections 2024-03-04T10:12:12.946 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:12.942+0000 7f8fd9d45640 1 -- 172.21.15.99:0/3935339452 wait complete. 2024-03-04T10:12:12.955 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:12:13.055 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.053+0000 7fea94d77640 1 Processor -- start 2024-03-04T10:12:13.056 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.053+0000 7fea94d77640 1 -- start start 2024-03-04T10:12:13.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea94d77640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 0x7fea900cad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea94d77640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7fea900cc8d0 0x7fea9006acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.059 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea94d77640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9006fd20 con 0x7fea900d1200 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea94d77640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9006fad0 con 0x7fea900cc8d0 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea94d77640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea90070000 con 0x7fea9015dea0 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8ed9d640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fea900d1200 0x7fea900cd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:57022/0 (socket says 172.21.15.99:57022) 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8ed9d640 1 -- 172.21.15.99:0/4287493711 learned_addr learned my addr 172.21.15.99:0/4287493711 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8dd9b640 1 --2- 172.21.15.99:0/4287493711 >> v2:172.21.15.162:3300/0 conn(0x7fea900cc8d0 0x7fea9006acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8e59c640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 0x7fea900cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.060 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997868696 0 0) 0x7fea9006fd20 con 0x7fea900d1200 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea78003520 con 0x7fea900d1200 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 <== mon.0 v1:172.21.15.99:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 455297401 0 0) 0x7fea78003520 con 0x7fea900d1200 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 msgr2=0x7fea900cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 0x7fea900cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 >> v2:172.21.15.162:3300/0 conn(0x7fea900cc8d0 msgr2=0x7fea9006acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 --2- 172.21.15.99:0/4287493711 >> v2:172.21.15.162:3300/0 conn(0x7fea900cc8d0 0x7fea9006acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 --> v1:172.21.15.99:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900702e0 con 0x7fea900d1200 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 <== mon.0 v1:172.21.15.99:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2842634707 0 0) 0x7fea880019f0 con 0x7fea900d1200 2024-03-04T10:12:13.061 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 >> v1:172.21.15.99:6789/0 conn(0x7fea900d1200 legacy=0x7fea900cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea780039b0 0x7fea78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 0x7fea78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea78004d40 0x7fea780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9006fad0 con 0x7fea78004d40 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea90070000 con 0x7fea780039b0 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.054+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7800b9c0 con 0x7fea9015dea0 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea78004d40 0x7fea780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.062 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8e59c640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea780039b0 0x7fea78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8dd9b640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 0x7fea78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 -- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 msgr2=0x7fea78004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea9015dea0 0x7fea78004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 -- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea780039b0 msgr2=0x7fea78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea780039b0 0x7fea78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 -- 172.21.15.99:0/4287493711 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea78003520 con 0x7fea78004d40 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.055+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea78004d40 0x7fea780051c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fea8800a8a0 tx=0x7fea78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a40340a71397f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8800a4c0 con 0x7fea78004d40 2024-03-04T10:12:13.063 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88034650 con 0x7fea78004d40 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea8d59a640 1 -- 172.21.15.99:0/4287493711 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88034a40 con 0x7fea78004d40 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea94d77640 1 -- 172.21.15.99:0/4287493711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea78004d40 msgr2=0x7fea780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea94d77640 1 --2- 172.21.15.99:0/4287493711 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea78004d40 0x7fea780051c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fea8800a8a0 tx=0x7fea78005830 comp rx=0 tx=0).stop 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea8ed9d640 1 -- 172.21.15.99:0/4287493711 reap_dead start 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea94d77640 1 -- 172.21.15.99:0/4287493711 shutdown_connections 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea94d77640 1 -- 172.21.15.99:0/4287493711 >> 172.21.15.99:0/4287493711 conn(0x7fea900d0710 msgr2=0x7fea900d0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea94d77640 1 -- 172.21.15.99:0/4287493711 shutdown_connections 2024-03-04T10:12:13.064 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.056+0000 7fea94d77640 1 -- 172.21.15.99:0/4287493711 wait complete. 2024-03-04T10:12:13.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 Processor -- start 2024-03-04T10:12:13.067 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 -- start start 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea78004d40 0x7fea90157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea901599c0 0x7fea900b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 0x7fea900aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9006b390 con 0x7fea900b5430 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea9006b210 con 0x7fea901599c0 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea94d77640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9006b510 con 0x7fea78004d40 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea8ed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 0x7fea900aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.068 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea8ed9d640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 0x7fea900aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:40246/0 (socket says 172.21.15.99:40246) 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.058+0000 7fea8ed9d640 1 -- 172.21.15.99:0/1379796290 learned_addr learned my addr 172.21.15.99:0/1379796290 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8e59c640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea78004d40 0x7fea90157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8dd9b640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea901599c0 0x7fea900b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8ed9d640 1 -- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea78004d40 msgr2=0x7fea90157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea78004d40 0x7fea90157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8ed9d640 1 -- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea901599c0 msgr2=0x7fea900b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea901599c0 0x7fea900b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.069 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8ed9d640 1 -- 172.21.15.99:0/1379796290 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea90159e40 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8dd9b640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea901599c0 0x7fea900b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea8ed9d640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 0x7fea900aee70 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fea8802ccb0 tx=0x7fea88001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88002010 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88044070 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9015e320 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88034a00 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.059+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9015e8e0 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.060+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea90074f60 con 0x7fea900b5430 2024-03-04T10:12:13.070 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.060+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fea88034ba0 con 0x7fea900b5430 2024-03-04T10:12:13.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.061+0000 7fea7effd640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:13.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.061+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fea880ac690 con 0x7fea900b5430 2024-03-04T10:12:13.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.061+0000 7fea8e59c640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:13.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.061+0000 7fea8e59c640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea7006b4d0 0x7fea7006d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fea80009980 tx=0x7fea80008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:13.071 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.065+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fea8807b5d0 con 0x7fea900b5430 2024-03-04T10:12:13.235 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.232+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea900ceb10 con 0x7fea7006b4d0 2024-03-04T10:12:13.236 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.233+0000 7fea7effd640 1 -- 172.21.15.99:0/1379796290 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13499 (secure 0 0 0) 0x7fea900ceb10 con 0x7fea7006b4d0 2024-03-04T10:12:13.236 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:13.237 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:12:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea7006b4d0 msgr2=0x7fea7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea7006b4d0 0x7fea7006d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fea80009980 tx=0x7fea80008040 comp rx=0 tx=0).stop 2024-03-04T10:12:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 msgr2=0x7fea900aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 0x7fea900aee70 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fea8802ccb0 tx=0x7fea88001dc0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 shutdown_connections 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fea7006b4d0 0x7fea7006d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fea900b5430 0x7fea900aee70 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fea901599c0 0x7fea900b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 --2- 172.21.15.99:0/1379796290 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fea78004d40 0x7fea90157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 >> 172.21.15.99:0/1379796290 conn(0x7fea900d0710 msgr2=0x7fea9015d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 shutdown_connections 2024-03-04T10:12:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:13.235+0000 7fea94d77640 1 -- 172.21.15.99:0/1379796290 wait complete. 2024-03-04T10:12:13.248 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-03-04T10:12:11.563008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260788,"kb_used_data":11336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713932,"statfs":{"total":773094113280,"available":772827066368,"internally_reserved":0,"allocated":11608064,"data_stored":6113568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6310911,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-260,"num_read_kb":-259,"num_write":-409,"num_write_kb":-6325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002138"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":344,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-04T10:12:04.581744+0000","last_change":"2024-03-04T10:03:28.696832+0000","last_active":"2024-03-04T10:12:04.581744+0000","last_peered":"2024-03-04T10:12:04.581744+0000","last_clean":"2024-03-04T10:12:04.581744+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:12:04.581744+0000","last_undegraded":"2024-03-04T10:12:04.581744+0000","last_fullsized":"2024-03-04T10:12:04.581744+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T10:03:27.478369+0000","last_clean_scrub_stamp":"2024-03-04T10:03:27.478369+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:01:42.856654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574955,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1896448,"data_stored":1206972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574956,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1896448,"data_stored":1206972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:12:13.251 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-04T10:03:27.478369+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=4, tm_hour=10, tm_min=3, tm_sec=27, tm_wday=0, tm_yday=64, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=4, tm_hour=10, tm_min=12, tm_sec=8, tm_wday=0, tm_yday=64, tm_isdst=0) 2024-03-04T10:12:13.251 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-04T10:12:17.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~64s 2024-03-04T10:12:17.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~84s 2024-03-04T10:12:17.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~32s 2024-03-04T10:12:17.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~51s 2024-03-04T10:12:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~70s 2024-03-04T10:12:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~90s 2024-03-04T10:12:23.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~38s 2024-03-04T10:12:23.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~57s 2024-03-04T10:12:29.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~76s 2024-03-04T10:12:29.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~96s 2024-03-04T10:12:29.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~44s 2024-03-04T10:12:29.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~63s 2024-03-04T10:12:33.252 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T10:12:33.364 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.363+0000 7f8c097f1640 1 Processor -- start 2024-03-04T10:12:33.365 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.363+0000 7f8c097f1640 1 -- start start 2024-03-04T10:12:33.367 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c097f1640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8c0415de00 0x7f8c040b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c097f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c040b9380 0x7f8c040b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c097f1640 1 --2- >> v2:172.21.15.162:3300/0 conn(0x7f8c040b70d0 0x7f8c040b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c097f1640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c040d2820 con 0x7f8c0415de00 2024-03-04T10:12:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c097f1640 1 -- --> v2:172.21.15.162:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c0415e1e0 con 0x7f8c040b70d0 2024-03-04T10:12:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c097f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c040d1c10 con 0x7f8c040b9380 2024-03-04T10:12:33.368 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8c0415de00 0x7f8c040b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f8c0415de00 0x7f8c040b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33242/0 (socket says 172.21.15.99:33242) 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 -- 172.21.15.99:0/76923557 learned_addr learned my addr 172.21.15.99:0/76923557 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 -- 172.21.15.99:0/76923557 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c040b9380 msgr2=0x7f8c040b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 --2- 172.21.15.99:0/76923557 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c040b9380 0x7f8c040b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 -- 172.21.15.99:0/76923557 >> v2:172.21.15.162:3300/0 conn(0x7f8c040b70d0 msgr2=0x7f8c040b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c037fe640 1 --2- 172.21.15.99:0/76923557 >> v2:172.21.15.162:3300/0 conn(0x7f8c040b70d0 0x7f8c040b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.369 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 --2- 172.21.15.99:0/76923557 >> v2:172.21.15.162:3300/0 conn(0x7f8c040b70d0 0x7f8c040b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c02ffd640 1 -- 172.21.15.99:0/76923557 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040d1ef0 con 0x7f8c0415de00 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.364+0000 7f8c037fe640 1 --2- 172.21.15.99:0/76923557 >> v2:172.21.15.162:3300/0 conn(0x7f8c040b70d0 0x7f8c040b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.365+0000 7f8c02ffd640 1 --2- 172.21.15.99:0/76923557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c0415de00 0x7f8c040b2e60 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f8c0406dbb0 tx=0x7f8bf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef4490813fdee5e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.365+0000 7f8c01ffb640 1 -- 172.21.15.99:0/76923557 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4035020 con 0x7f8c0415de00 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.365+0000 7f8c01ffb640 1 -- 172.21.15.99:0/76923557 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf4031870 con 0x7f8c0415de00 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.365+0000 7f8c01ffb640 1 -- 172.21.15.99:0/76923557 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4031c40 con 0x7f8c0415de00 2024-03-04T10:12:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.365+0000 7f8c097f1640 1 -- 172.21.15.99:0/76923557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c0415de00 msgr2=0x7f8c040b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.365+0000 7f8c097f1640 1 --2- 172.21.15.99:0/76923557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c0415de00 0x7f8c040b2e60 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f8c0406dbb0 tx=0x7f8bf402b910 comp rx=0 tx=0).stop 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 -- 172.21.15.99:0/76923557 shutdown_connections 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 --2- 172.21.15.99:0/76923557 >> v2:172.21.15.162:3300/0 conn(0x7f8c040b70d0 0x7f8c040b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 --2- 172.21.15.99:0/76923557 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c040b9380 0x7f8c040b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 --2- 172.21.15.99:0/76923557 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c0415de00 0x7f8c040b2e60 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 -- 172.21.15.99:0/76923557 >> 172.21.15.99:0/76923557 conn(0x7f8c040d0470 msgr2=0x7f8c040d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 -- 172.21.15.99:0/76923557 shutdown_connections 2024-03-04T10:12:33.371 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.366+0000 7f8c097f1640 1 -- 172.21.15.99:0/76923557 wait complete. 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.367+0000 7f8c097f1640 1 Processor -- start 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.367+0000 7f8c097f1640 1 -- start start 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c097f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c040b70d0 0x7f8c04153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c097f1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 0x7f8c040b4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c097f1640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c0415de00 0x7f8c040b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c097f1640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c040b97e0 con 0x7f8c040b9380 2024-03-04T10:12:33.375 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c097f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c040d24b0 con 0x7f8c040b70d0 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c097f1640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c040b59b0 con 0x7f8c0415de00 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 0x7f8c040b4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 0x7f8c040b4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33250/0 (socket says 172.21.15.99:33250) 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 -- 172.21.15.99:0/1106640810 learned_addr learned my addr 172.21.15.99:0/1106640810 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c02ffd640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c040b70d0 0x7f8c04153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c037fe640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c0415de00 0x7f8c040b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.376 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 -- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c0415de00 msgr2=0x7f8c040b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c0415de00 0x7f8c040b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 -- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c040b70d0 msgr2=0x7f8c04153130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c040b70d0 0x7f8c04153130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 -- 172.21.15.99:0/1106640810 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c040cd480 con 0x7f8c040b9380 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8c027fc640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 0x7f8c040b4eb0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800b4d0 tx=0x7f8bf800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8014070 con 0x7f8c040b9380 2024-03-04T10:12:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.368+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf8010870 con 0x7f8c040b9380 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.369+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8010c60 con 0x7f8c040b9380 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.369+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c040cc900 con 0x7f8c040b9380 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.369+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c040ccec0 con 0x7f8c040b9380 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.370+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c04074f60 con 0x7f8c040b9380 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.370+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f8bf8017070 con 0x7f8c040b9380 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.370+0000 7f8beb7fe640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bf006b500 0x7f8bf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.371+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f8bf8089fc0 con 0x7f8c040b9380 2024-03-04T10:12:33.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.371+0000 7f8c02ffd640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bf006b500 0x7f8bf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.371+0000 7f8c02ffd640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bf006b500 0x7f8bf006d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4036000 tx=0x7f8bf403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.379 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.375+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f8bf8058700 con 0x7f8c040b9380 2024-03-04T10:12:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.541+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 --> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c040cdd00 con 0x7f8bf006b500 2024-03-04T10:12:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.542+0000 7f8beb7fe640 1 -- 172.21.15.99:0/1106640810 <== mgr.4103 v2:172.21.15.162:6832/1452235113 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7f8bf404b460 con 0x7f8bf006b500 2024-03-04T10:12:33.544 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:33.544 INFO:teuthology.orchestra.run.smithi099.stderr:dumped all 2024-03-04T10:12:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bf006b500 msgr2=0x7f8bf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bf006b500 0x7f8bf006d9c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4036000 tx=0x7f8bf403c040 comp rx=0 tx=0).stop 2024-03-04T10:12:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 msgr2=0x7f8c040b4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 0x7f8c040b4eb0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f8bf800b4d0 tx=0x7f8bf800b990 comp rx=0 tx=0).stop 2024-03-04T10:12:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 shutdown_connections 2024-03-04T10:12:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f8bf006b500 0x7f8bf006d9c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f8c0415de00 0x7f8c040b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8c040b9380 0x7f8c040b4eb0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 --2- 172.21.15.99:0/1106640810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8c040b70d0 0x7f8c04153130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 >> 172.21.15.99:0/1106640810 conn(0x7f8c040d0470 msgr2=0x7f8c040c7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 shutdown_connections 2024-03-04T10:12:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.544+0000 7f8c097f1640 1 -- 172.21.15.99:0/1106640810 wait complete. 2024-03-04T10:12:33.556 INFO:teuthology.orchestra.run.smithi099.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-03-04T10:12:31.567373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":260776,"kb_used_data":11336,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713944,"statfs":{"total":773094113280,"available":772827078656,"internally_reserved":0,"allocated":11608064,"data_stored":6113568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":255315506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002663"},"pg_stats":[{"pgid":"1.0","version":"105'119","reported_seq":355,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-04T10:12:13.498189+0000","last_change":"2024-03-04T10:12:13.498223+0000","last_active":"2024-03-04T10:12:13.498189+0000","last_peered":"2024-03-04T10:12:13.498189+0000","last_clean":"2024-03-04T10:12:13.498189+0000","last_became_active":"2024-03-04T10:03:28.694177+0000","last_became_peered":"2024-03-04T10:03:28.694177+0000","last_unstale":"2024-03-04T10:12:13.498189+0000","last_undegraded":"2024-03-04T10:12:13.498189+0000","last_fullsized":"2024-03-04T10:12:13.498189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'119","last_scrub_stamp":"2024-03-04T10:12:13.498102+0000","last_deep_scrub":"105'119","last_deep_scrub_stamp":"2024-03-04T10:12:13.498102+0000","last_clean_scrub_stamp":"2024-03-04T10:12:13.498102+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:32:23.128534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574959,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1896448,"data_stored":1206972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574960,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1302528,"data_stored":616604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1896448,"data_stored":1206972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T10:12:33.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-04T10:12:33.658 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.656+0000 7fa46af45640 1 Processor -- start 2024-03-04T10:12:33.659 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.656+0000 7fa46af45640 1 -- start start 2024-03-04T10:12:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa46af45640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa46415eb70 0x7fa4640b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa46af45640 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46415ca50 con 0x7fa4640ccbd0 2024-03-04T10:12:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa46af45640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46415cd30 con 0x7fa4640ceec0 2024-03-04T10:12:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa46af45640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa46415d2f0 con 0x7fa46415eb70 2024-03-04T10:12:33.667 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa4694e2640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fa46415eb70 0x7fa4640b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa468ce1640 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7fa4640ccbd0 0x7fa4640ce7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:60620/0 (socket says 172.21.15.99:60620) 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.657+0000 7fa468ce1640 1 -- 172.21.15.99:0/4101677716 learned_addr learned my addr 172.21.15.99:0/4101677716 (peer_addr_for_me v1:172.21.15.99:0/0) 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.658+0000 7fa4694e2640 1 -- 172.21.15.99:0/4101677716 >> v1:172.21.15.162:6789/0 conn(0x7fa4640ceec0 legacy=0x7fa4640b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.658+0000 7fa4694e2640 1 -- 172.21.15.99:0/4101677716 >> v1:172.21.15.99:6789/0 conn(0x7fa4640ccbd0 legacy=0x7fa4640ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.658+0000 7fa4637fe640 1 -- 172.21.15.99:0/4101677716 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924792070 0 0) 0x7fa46415ca50 con 0x7fa4640ccbd0 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.658+0000 7fa4694e2640 1 -- 172.21.15.99:0/4101677716 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46415d010 con 0x7fa46415eb70 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.658+0000 7fa4694e2640 1 --2- 172.21.15.99:0/4101677716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa46415eb70 0x7fa4640b0c70 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009d90 tx=0x7fa45c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d134f4ea950de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.658+0000 7fa4637fe640 1 -- 172.21.15.99:0/4101677716 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c039070 con 0x7fa46415eb70 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa4637fe640 1 -- 172.21.15.99:0/4101677716 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c02cc60 con 0x7fa46415eb70 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa4637fe640 1 -- 172.21.15.99:0/4101677716 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c034660 con 0x7fa46415eb70 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 -- 172.21.15.99:0/4101677716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa46415eb70 msgr2=0x7fa4640b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 --2- 172.21.15.99:0/4101677716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa46415eb70 0x7fa4640b0c70 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009d90 tx=0x7fa45c02beb0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 -- 172.21.15.99:0/4101677716 shutdown_connections 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 --2- 172.21.15.99:0/4101677716 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa46415eb70 0x7fa4640b0c70 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 -- 172.21.15.99:0/4101677716 >> 172.21.15.99:0/4101677716 conn(0x7fa464066dc0 msgr2=0x7fa4640671d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 -- 172.21.15.99:0/4101677716 shutdown_connections 2024-03-04T10:12:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.659+0000 7fa46af45640 1 -- 172.21.15.99:0/4101677716 wait complete. 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.660+0000 7fa46af45640 1 Processor -- start 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.660+0000 7fa46af45640 1 -- start start 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa46af45640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 0x7fa4640c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa46af45640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4640ceec0 0x7fa4640c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa46af45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa46415eb70 0x7fa4640b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa46af45640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4640b7320 con 0x7fa4640ccbd0 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa46af45640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46415d6d0 con 0x7fa46415eb70 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa46af45640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4640b74a0 con 0x7fa4640ceec0 2024-03-04T10:12:33.670 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa468ce1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 0x7fa4640c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa468ce1640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 0x7fa4640c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56748/0 (socket says 172.21.15.99:56748) 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa468ce1640 1 -- 172.21.15.99:0/2300178612 learned_addr learned my addr 172.21.15.99:0/2300178612 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa4694e2640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa46415eb70 0x7fa4640b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.661+0000 7fa463fff640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4640ceec0 0x7fa4640c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa468ce1640 1 -- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4640ceec0 msgr2=0x7fa4640c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa468ce1640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4640ceec0 0x7fa4640c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa468ce1640 1 -- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa46415eb70 msgr2=0x7fa4640b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.671 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa468ce1640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa46415eb70 0x7fa4640b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa468ce1640 1 -- 172.21.15.99:0/2300178612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4640ca3f0 con 0x7fa4640ccbd0 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa463fff640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4640ceec0 0x7fa4640c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa468ce1640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 0x7fa4640c9cf0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c4d0 tx=0x7fa45400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454018070 con 0x7fa4640ccbd0 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa454007500 con 0x7fa4640ccbd0 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4640ca6d0 con 0x7fa4640ccbd0 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454004a20 con 0x7fa4640ccbd0 2024-03-04T10:12:33.672 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.662+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4640cac90 con 0x7fa4640ccbd0 2024-03-04T10:12:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.663+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa464074f60 con 0x7fa4640ccbd0 2024-03-04T10:12:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.663+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7fa45401b070 con 0x7fa4640ccbd0 2024-03-04T10:12:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.664+0000 7fa4617fa640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.664+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7fa454089630 con 0x7fa4640ccbd0 2024-03-04T10:12:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.664+0000 7fa463fff640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.665+0000 7fa463fff640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa45006b4d0 0x7fa45006d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa464155dc0 tx=0x7fa458009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.674 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.668+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7fa4540583c0 con 0x7fa4640ccbd0 2024-03-04T10:12:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.839+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fa46406e660 con 0x7fa4640ccbd0 2024-03-04T10:12:33.843 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.842+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v28)=0 v28) v1 ==== 123+0+0 (secure 0 0 0) 0x7fa454057cc0 con 0x7fa4640ccbd0 2024-03-04T10:12:33.845 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.844+0000 7fa4617fa640 1 -- 172.21.15.99:0/2300178612 <== mon.0 v2:172.21.15.99:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa45405e0d0 con 0x7fa4640ccbd0 2024-03-04T10:12:33.846 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa45006b4d0 msgr2=0x7fa45006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa45006b4d0 0x7fa45006d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa464155dc0 tx=0x7fa458009040 comp rx=0 tx=0).stop 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 msgr2=0x7fa4640c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 0x7fa4640c9cf0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c4d0 tx=0x7fa45400c990 comp rx=0 tx=0).stop 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 shutdown_connections 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7fa45006b4d0 0x7fa45006d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa46415eb70 0x7fa4640b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.847 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7fa4640ceec0 0x7fa4640c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 --2- 172.21.15.99:0/2300178612 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fa4640ccbd0 0x7fa4640c9cf0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 >> 172.21.15.99:0/2300178612 conn(0x7fa464066dc0 msgr2=0x7fa4640cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:33.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 shutdown_connections 2024-03-04T10:12:33.848 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.845+0000 7fa46af45640 1 -- 172.21.15.99:0/2300178612 wait complete. 2024-03-04T10:12:33.857 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T10:12:33.857 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T10:12:33.956 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.955+0000 7f6563f97640 1 Processor -- start 2024-03-04T10:12:33.957 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.955+0000 7f6563f97640 1 -- start start 2024-03-04T10:12:33.958 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6563f97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c0cfd60 0x7f655c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6563f97640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f655c0cf0d0 0x7f655c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6563f97640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655c0d0140 con 0x7f655c06d3b0 2024-03-04T10:12:33.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6563f97640 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655c15cbb0 con 0x7f655c0cf0d0 2024-03-04T10:12:33.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6563f97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c15c280 con 0x7f655c0cfd60 2024-03-04T10:12:33.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f655c0cf0d0 0x7f655c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.959 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f655c0cf0d0 0x7f655c0cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56760/0 (socket says 172.21.15.99:56760) 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 -- 172.21.15.99:0/501544727 learned_addr learned my addr 172.21.15.99:0/501544727 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 -- 172.21.15.99:0/501544727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c0cfd60 msgr2=0x7f655c0cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 --2- 172.21.15.99:0/501544727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c0cfd60 0x7f655c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 -- 172.21.15.99:0/501544727 >> v1:172.21.15.162:6789/0 conn(0x7f655c06d3b0 legacy=0x7f655c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.956+0000 7f6562534640 1 -- 172.21.15.99:0/501544727 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c15c560 con 0x7f655c0cf0d0 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.957+0000 7f6562534640 1 --2- 172.21.15.99:0/501544727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cf0d0 0x7f655c0cc330 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f655c15cd30 tx=0x7f655402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e6e37c6726cd8e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.957+0000 7f6560d31640 1 -- 172.21.15.99:0/501544727 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655402ca50 con 0x7f655c0cf0d0 2024-03-04T10:12:33.960 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.957+0000 7f6560d31640 1 -- 172.21.15.99:0/501544727 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f655402cbf0 con 0x7f655c0cf0d0 2024-03-04T10:12:33.961 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.957+0000 7f6560d31640 1 -- 172.21.15.99:0/501544727 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65540343f0 con 0x7f655c0cf0d0 2024-03-04T10:12:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 -- 172.21.15.99:0/501544727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cf0d0 msgr2=0x7f655c0cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 --2- 172.21.15.99:0/501544727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cf0d0 0x7f655c0cc330 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f655c15cd30 tx=0x7f655402beb0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 -- 172.21.15.99:0/501544727 shutdown_connections 2024-03-04T10:12:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 --2- 172.21.15.99:0/501544727 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cf0d0 0x7f655c0cc330 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 --2- 172.21.15.99:0/501544727 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c0cfd60 0x7f655c0cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 -- 172.21.15.99:0/501544727 >> 172.21.15.99:0/501544727 conn(0x7f655c06acd0 msgr2=0x7f655c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 -- 172.21.15.99:0/501544727 shutdown_connections 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.958+0000 7f6563f97640 1 -- 172.21.15.99:0/501544727 wait complete. 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.959+0000 7f6563f97640 1 Processor -- start 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.959+0000 7f6563f97640 1 -- start start 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6563f97640 1 --2- >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c06d3b0 0x7f655c14bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6563f97640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f655c0cf0d0 0x7f655c14cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6563f97640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 0x7f655c157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6563f97640 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c15f520 con 0x7f655c0cfd60 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6563f97640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c15bf30 con 0x7f655c0cf0d0 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6563f97640 1 -- --> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c15f6a0 con 0x7f655c06d3b0 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 0x7f655c157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 0x7f655c157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:56768/0 (socket says 172.21.15.99:56768) 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 -- 172.21.15.99:0/2758334639 learned_addr learned my addr 172.21.15.99:0/2758334639 (peer_addr_for_me v2:172.21.15.99:0/0) 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6561532640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f655c0cf0d0 0x7f655c14cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6561d33640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c06d3b0 0x7f655c14bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.966 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 -- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c06d3b0 msgr2=0x7f655c14bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c06d3b0 0x7f655c14bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 -- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f655c0cf0d0 msgr2=0x7f655c14cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f655c0cf0d0 0x7f655c14cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 -- 172.21.15.99:0/2758334639 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c14d630 con 0x7f655c0cfd60 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6561d33640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c06d3b0 0x7f655c14bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6561532640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f655c0cf0d0 0x7f655c14cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f6562534640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 0x7f655c157c40 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f655402bef0 tx=0x7f6554004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.960+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554043020 con 0x7f655c0cfd60 2024-03-04T10:12:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.961+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f655403e450 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.961+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655c1e4a80 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.961+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655403e880 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.961+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655c1e5040 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.961+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655c0b22f0 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.962+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 5) v1 ==== 89778+0+0 (secure 0 0 0) 0x7f6554005d10 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.962+0000 7f65527fc640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f654006b4d0 0x7f654006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.962+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5937+0+0 (secure 0 0 0) 0x7f65540ad000 con 0x7f655c0cfd60 2024-03-04T10:12:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.962+0000 7f6561d33640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f654006b4d0 0x7f654006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T10:12:33.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.963+0000 7f6561d33640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f654006b4d0 0x7f654006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f655800a870 tx=0x7f6558008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T10:12:33.969 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:33.967+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (secure 0 0 0) 0x7f655407bd90 con 0x7f655c0cfd60 2024-03-04T10:12:34.171 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.170+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f655c074f60 con 0x7f655c0cfd60 2024-03-04T10:12:34.172 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.171+0000 7f65527fc640 1 -- 172.21.15.99:0/2758334639 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+537 (secure 0 0 0) 0x7f6554033e30 con 0x7f655c0cfd60 2024-03-04T10:12:34.173 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-03-04T10:12:34.173 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":111,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-04T10:12:34.173 INFO:teuthology.orchestra.run.smithi099.stderr:dumped fsmap epoch 111 2024-03-04T10:12:34.174 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f654006b4d0 msgr2=0x7f654006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:34.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f654006b4d0 0x7f654006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f655800a870 tx=0x7f6558008040 comp rx=0 tx=0).stop 2024-03-04T10:12:34.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 msgr2=0x7f655c157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T10:12:34.175 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 0x7f655c157c40 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f655402bef0 tx=0x7f6554004a40 comp rx=0 tx=0).stop 2024-03-04T10:12:34.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 shutdown_connections 2024-03-04T10:12:34.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:6832/1452235113,v1:172.21.15.162:6833/1452235113] conn(0x7f654006b4d0 0x7f654006d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:34.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f655c0cfd60 0x7f655c157c40 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:34.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f655c0cf0d0 0x7f655c14cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:34.176 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 --2- 172.21.15.99:0/2758334639 >> [v2:172.21.15.162:3301/0,v1:172.21.15.162:6790/0] conn(0x7f655c06d3b0 0x7f655c14bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T10:12:34.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.172+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 >> 172.21.15.99:0/2758334639 conn(0x7f655c06acd0 msgr2=0x7f655c06b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T10:12:34.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.173+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 shutdown_connections 2024-03-04T10:12:34.177 INFO:teuthology.orchestra.run.smithi099.stderr:2024-03-04T10:12:34.173+0000 7f6563f97640 1 -- 172.21.15.99:0/2758334639 wait complete. 2024-03-04T10:12:34.185 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-04T10:12:34.185 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-04T10:12:34.185 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-04T10:12:34.186 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:34.186 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a' 2024-03-04T10:12:34.191 INFO:tasks.ceph.mds.a:Stopped 2024-03-04T10:12:34.191 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-04T10:12:34.192 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:34.193 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c' 2024-03-04T10:12:34.198 INFO:tasks.ceph.mds.c:Stopped 2024-03-04T10:12:34.199 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-04T10:12:34.199 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:34.199 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2024-03-04T10:12:34.204 INFO:tasks.ceph.mds.b:Stopped 2024-03-04T10:12:34.204 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-04T10:12:34.204 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:34.204 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2024-03-04T10:12:34.210 INFO:tasks.ceph.mds.d:Stopped 2024-03-04T10:12:34.210 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-04T10:12:34.210 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-04T10:12:34.211 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.244 INFO:tasks.ceph.osd.0:Stopped 2024-03-04T10:12:34.244 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-04T10:12:34.244 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.309 INFO:tasks.ceph.osd.1:Stopped 2024-03-04T10:12:34.310 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-04T10:12:34.310 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.376 INFO:tasks.ceph.osd.2:Stopped 2024-03-04T10:12:34.377 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-04T10:12:34.377 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.441 INFO:tasks.ceph.osd.3:Stopped 2024-03-04T10:12:34.441 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-04T10:12:34.442 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.470 INFO:tasks.ceph.osd.4:Stopped 2024-03-04T10:12:34.470 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-04T10:12:34.470 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.545 INFO:tasks.ceph.osd.5:Stopped 2024-03-04T10:12:34.545 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-04T10:12:34.545 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.614 INFO:tasks.ceph.osd.6:Stopped 2024-03-04T10:12:34.615 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-04T10:12:34.615 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.689 INFO:tasks.ceph.osd.7:Stopped 2024-03-04T10:12:34.690 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-04T10:12:34.690 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-04T10:12:34.690 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.715 INFO:tasks.ceph.mgr.y:Stopped 2024-03-04T10:12:34.715 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-04T10:12:34.716 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.760 INFO:tasks.ceph.mgr.x:Stopped 2024-03-04T10:12:34.761 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-04T10:12:34.761 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-04T10:12:34.761 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.772 INFO:tasks.ceph.mon.a:Stopped 2024-03-04T10:12:34.773 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-04T10:12:34.773 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.814 INFO:tasks.ceph.mon.b:Stopped 2024-03-04T10:12:34.815 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-04T10:12:34.815 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T10:12:34.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-04T10:12:34.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-04T10:12:34.868 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-04T10:12:34.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-04T10:12:34.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-03-04T10:12:34.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-04T10:12:34.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-03-04T10:12:34.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-04T10:12:34.869 INFO:tasks.ceph.mon.c:Stopped 2024-03-04T10:12:34.869 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-04T10:12:34.870 DEBUG:teuthology.orchestra.run.smithi099:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'Error loading MDS rank' | egrep -v 'missing journal object' | egrep -v 'Error recovering journal' | egrep -v 'error decoding table object' | egrep -v 'failed to read JournalPointer' | egrep -v 'Corrupt directory entry' | egrep -v 'Corrupt fnode header' | egrep -v 'corrupt sessionmap header' | egrep -v 'Corrupt dentry' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_READ_ONLY | egrep -v 'force file system read-only' | egrep -v 'with standby daemon mds' | egrep -v 'MDS abort because newly corrupt dentry' | head -n 1 2024-03-04T10:12:35.211 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:12:35.211 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-04T10:12:35.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:12:35.400 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-04T10:12:35.503 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:12:35.503 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-04T10:12:35.572 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:12:35.572 DEBUG:teuthology.orchestra.run.smithi099:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-04T10:12:35.637 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:12:35.637 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-04T10:12:35.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:12:35.862 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-04T10:12:35.948 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:12:35.949 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-04T10:12:36.016 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:12:36.016 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-04T10:12:36.072 INFO:tasks.ceph:Archiving mon data... 2024-03-04T10:12:36.073 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/data/mon.a.tgz 2024-03-04T10:12:36.074 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-03-04T10:12:36.091 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.3LoazHG99D 2024-03-04T10:12:36.091 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3LoazHG99D 2024-03-04T10:12:36.249 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.3LoazHG99D 2024-03-04T10:12:36.334 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.3LoazHG99D is 416KB 2024-03-04T10:12:36.433 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.3LoazHG99D 2024-03-04T10:12:36.448 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/data/mon.b.tgz 2024-03-04T10:12:36.448 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-04T10:12:36.466 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.3xkioTc75H 2024-03-04T10:12:36.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3xkioTc75H 2024-03-04T10:12:36.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.3xkioTc75H 2024-03-04T10:12:36.717 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.3xkioTc75H is 418KB 2024-03-04T10:12:36.816 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.3xkioTc75H 2024-03-04T10:12:36.832 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/data/mon.c.tgz 2024-03-04T10:12:36.832 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-04T10:12:36.885 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.IYZDg5cVaO 2024-03-04T10:12:36.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.IYZDg5cVaO 2024-03-04T10:12:37.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.IYZDg5cVaO 2024-03-04T10:12:37.114 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.IYZDg5cVaO is 418KB 2024-03-04T10:12:37.212 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.IYZDg5cVaO 2024-03-04T10:12:37.226 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-04T10:12:37.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-04T10:12:37.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-04T10:12:37.315 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-04T10:12:37.348 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-04T10:12:37.349 INFO:tasks.ceph:Archiving crash dumps... 2024-03-04T10:12:37.350 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/remote/smithi099/crash 2024-03-04T10:12:37.351 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-04T10:12:37.428 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/crash to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/remote/smithi162/crash 2024-03-04T10:12:37.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-04T10:12:37.465 INFO:tasks.ceph:Shutting down logrotate 2024-03-04T10:12:37.465 DEBUG:teuthology.orchestra.run.smithi099:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:37.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:37.550 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:37.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-04T10:12:37.606 INFO:tasks.ceph:Compressing logs... 2024-03-04T10:12:37.606 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T10:12:37.622 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T10:12:37.704 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-04T10:12:37.704 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44248.log 2024-03-04T10:12:37.704 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.1.44248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37795.log 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.44248.log.gz 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44637.log 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37795.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37955.log 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37795.log.gz 2024-03-04T10:12:37.705 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.3.44637.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.44637.log.gz 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37955.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.audit.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37955.log.gz 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-04T10:12:37.706 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38076.log 2024-03-04T10:12:37.707 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37996.log 2024-03-04T10:12:37.707 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38076.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-04T10:12:37.707 INFO:teuthology.orchestra.run.smithi162.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38076.log.gz 2024-03-04T10:12:37.707 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37996.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37996.log.gz 2024-03-04T10:12:37.707 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37835.log 2024-03-04T10:12:37.707 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37835.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37835.log.gz 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38036.log 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38036.log: 90.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38036.log.gz 2024-03-04T10:12:37.708 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37915.log 2024-03-04T10:12:37.709 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-04T10:12:37.709 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37915.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37915.log.gz 2024-03-04T10:12:37.709 INFO:teuthology.orchestra.run.smithi162.stderr: 92.3%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-04T10:12:37.709 INFO:teuthology.orchestra.run.smithi162.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-04T10:12:37.709 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 90.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-04T10:12:37.814 INFO:teuthology.orchestra.run.smithi162.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-04T10:12:37.854 INFO:teuthology.orchestra.run.smithi162.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-04T10:12:37.966 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59548.log 2024-03-04T10:12:37.967 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69067.log 2024-03-04T10:12:37.967 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50842.log 2024-03-04T10:12:37.967 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69067.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59548.log.gz 2024-03-04T10:12:37.967 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66008.log 2024-03-04T10:12:37.967 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50842.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69067.log.gz 2024-03-04T10:12:37.967 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68335.log 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66008.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50842.log.gz 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59168.log 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67767.log 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.66008.log.gz 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64052.log 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68335.log.gz 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9%/var/log/ceph/ceph-client.admin.67767.log: -- replaced with /var/log/ceph/ceph-client.admin.59168.log.gz 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57469.log 2024-03-04T10:12:37.968 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64052.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67767.log.gz 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57637.log 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64052.log.gz 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53623.log 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46851.log 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57469.log.gz 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75915.log 2024-03-04T10:12:37.969 INFO:teuthology.orchestra.run.smithi099.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57637.log.gz 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53623.log.gz 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70406.log 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75915.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46851.log.gz 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75915.log.gz 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70908.log 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70573.log 2024-03-04T10:12:37.970 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44274.log 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53742.log 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.70908.log.gz 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44274.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.70573.log.gz 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51610.log 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.70406.log.gz 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44274.log.gz 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56693.log 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-03-04T10:12:37.971 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53742.log.gz 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56693.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51610.log.gz 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56693.log.gz 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5/var/log/ceph/ceph-client.admin.46182.log: --verbose -- /var/log/ceph/ceph-client.admin.76563.log 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75333.log 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-03-04T10:12:37.972 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76563.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64752.log 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76563.log.gz 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46858.log 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69109.log 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75333.log.gz 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46858.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64752.log.gz 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56950.log 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46858.log.gz 2024-03-04T10:12:37.973 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50209.log 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69109.log.gz 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56868.log 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50209.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56950.log.gz 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52084.log 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50209.log.gz 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46525.log 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52084.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56868.log.gz 2024-03-04T10:12:37.974 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48399.log 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46525.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52084.log.gz -5 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47706.log 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.48399.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46525.log.gz 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56441.log 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47706.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48399.log.gz 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74380.log 2024-03-04T10:12:37.975 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50716.log 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47706.log.gz 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74380.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56441.log.gz 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71093.log 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50716.log: gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.57076.log -- replaced with /var/log/ceph/ceph-client.admin.74380.log.gz 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72868.log 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50716.log.gz 2024-03-04T10:12:37.976 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51881.log 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.71093.log.gz 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57076.log.gz 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74997.log 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74829.log 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72868.log.gz 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51881.log.gz 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75291.log 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74997.log.gz 2024-03-04T10:12:37.977 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56525.log 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62049.log 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74829.log.gz 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56525.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75291.log.gz 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57385.log 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56525.log.gz 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45330.log 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57385.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.62049.log.gz 2024-03-04T10:12:37.978 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71907.log 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57385.log.gz 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54149.log 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45330.log.gz/var/log/ceph/ceph-client.admin.71907.log: 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67977.log 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62175.log 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.71907.log.gz 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67977.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54149.log.gz 2024-03-04T10:12:37.979 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68164.log 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67977.log.gz 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53066.log 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62175.log: /var/log/ceph/ceph-client.admin.68164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72149.log 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53066.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62175.log.gz 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70834.log 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68164.log.gz 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44064.log 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70834.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.53066.log.gz 2024-03-04T10:12:37.980 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72149.log.gz 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57805.log 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.70834.log.gz 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55918.log 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44064.log.gz 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57299.log 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55918.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.57805.log.gz 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50884.log 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68079.log 89.3% 2024-03-04T10:12:37.981 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55918.log.gz 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50884.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57299.log.gz 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58321.log 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50884.log.gz 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53024.log 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7%/var/log/ceph/ceph-client.admin.58321.log: -- replaced with /var/log/ceph/ceph-client.admin.68079.log.gz 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56567.log 2024-03-04T10:12:37.982 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58321.log.gz 2024-03-04T10:12:37.983 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66985.log 2024-03-04T10:12:37.983 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56567.log: gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.53024.log.gz 2024-03-04T10:12:37.983 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60327.log 2024-03-04T10:12:37.983 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66985.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56567.log.gz 2024-03-04T10:12:37.983 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57034.log 2024-03-04T10:12:37.983 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66985.log.gz 2024-03-04T10:12:37.984 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75249.log 2024-03-04T10:12:37.984 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70797.log 2024-03-04T10:12:37.984 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60327.log.gz 2024-03-04T10:12:37.984 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75249.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57034.log.gz 2024-03-04T10:12:37.984 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63275.log 2024-03-04T10:12:37.984 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75249.log.gz 2024-03-04T10:12:37.985 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49795.log 2024-03-04T10:12:37.985 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70759.log 2024-03-04T10:12:37.985 INFO:teuthology.orchestra.run.smithi099.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.70797.log.gz 2024-03-04T10:12:37.985 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49795.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63275.log.gz 2024-03-04T10:12:37.985 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47058.log 2024-03-04T10:12:37.985 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70759.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49795.log.gz 2024-03-04T10:12:37.986 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65756.log 2024-03-04T10:12:37.986 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47058.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.70759.log.gz 2024-03-04T10:12:37.986 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46185.log 2024-03-04T10:12:37.986 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47058.log.gz 2024-03-04T10:12:37.986 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50584.log 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8%/var/log/ceph/ceph-client.admin.46185.log: -- replaced with /var/log/ceph/ceph-client.admin.65756.log.gz 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60166.log 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45619.log 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46185.log.gz 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60166.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50584.log.gz 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59053.log 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60166.log.gz 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68801.log 2024-03-04T10:12:37.987 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46859.log 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45619.log.gz 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68801.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59053.log.gz 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57511.log 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68801.log.gz 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71537.log 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57511.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46859.log.gz 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65882.log 2024-03-04T10:12:37.988 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58643.log 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57511.log.gz 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37958.log 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71537.log.gz 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58643.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65882.log.gz 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61141.log 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9%/var/log/ceph/ceph-client.admin.37958.log: -- replaced with /var/log/ceph/ceph-client.admin.58643.log.gz 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57595.log 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61141.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.37958.log.gz -5 2024-03-04T10:12:37.989 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68844.log 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59590.log 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.61141.log.gz 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68844.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57595.log.gz 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64145.log 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68844.log.gz 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66339.log 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64145.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59590.log.gz 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45798.log 2024-03-04T10:12:37.990 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64145.log.gz 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60411.log 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45798.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66339.log.gz 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56788.log 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60411.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45798.log.gz 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60889.log 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60411.log.gz 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71685.log 2024-03-04T10:12:37.991 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60889.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.56788.log.gz 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64794.log 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71685.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60889.log.gz 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55615.log 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75375.log 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.71685.log.gz 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55615.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64794.log.gz 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72209.log 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75375.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55615.log.gz 2024-03-04T10:12:37.992 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75081.log 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75375.log.gz 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68759.log 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64355.log 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72209.log.gz 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68759.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75081.log.gz 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52856.log 2024-03-04T10:12:37.993 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68759.log.gz 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66506.log 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64355.log.gz 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61566.log 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64710.log 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52856.log.gz 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47454.log 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66506.log.gz 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61566.log.gz 2024-03-04T10:12:37.994 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72107.log 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47454.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64710.log.gz 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44020.log 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72107.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47454.log.gz 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72318.log 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72107.log.gz 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56399.log 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72318.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44020.log.gz 2024-03-04T10:12:37.995 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58903.log 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5%/var/log/ceph/ceph-client.admin.56399.log: -- replaced with /var/log/ceph/ceph-client.admin.72318.log.gz 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71315.log 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56399.log.gz 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63319.log 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71315.log: gzip 88.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.58903.log.gz --verbose 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.76311.log 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63319.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71315.log.gz 2024-03-04T10:12:37.996 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62450.log 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63319.log.gz 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60584.log 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62450.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76311.log.gz 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56128.log 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62450.log.gz 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74140.log 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75735.log 2024-03-04T10:12:37.997 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60584.log.gz 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74140.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56128.log.gz 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62492.log 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74140.log.gz 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68974.log 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59819.log 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75735.log.gz 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68974.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62492.log.gz 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66585.log 2024-03-04T10:12:37.998 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68974.log.gz 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76605.log 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66585.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59819.log.gz 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66074.log 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.66585.log.gz 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74182.log 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66074.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76605.log.gz 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71167.log 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74182.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66074.log.gz 2024-03-04T10:12:37.999 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54723.log 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71167.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.74182.log.gz 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70244.log 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54723.log: gzip -5 89.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.71167.log.gz 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.49963.log 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54723.log.gz 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46512.log 2024-03-04T10:12:38.000 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49963.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70244.log.gz 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58807.log 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49963.log.gz 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62133.log 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71204.log 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46512.log.gz 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62133.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58807.log.gz 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50500.log 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62133.log.gz 2024-03-04T10:12:38.001 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61440.log 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51839.log 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71204.log.gz 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61440.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50500.log.gz 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59861.log 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61440.log.gz 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54639.log 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50758.log 2024-03-04T10:12:38.002 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51839.log.gz 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54639.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59861.log.gz 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72826.log 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54639.log.gz 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44232.log 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70684.log 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50758.log.gz 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44232.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72826.log.gz 2024-03-04T10:12:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75207.log 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44232.log.gz 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76479.log 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55960.log 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70684.log.gz 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76479.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75207.log.gz 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73565.log 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76479.log.gz 2024-03-04T10:12:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43608.log 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50005.log 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55960.log.gz 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43735.log 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43608.log.gz 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59422.log 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43735.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50005.log.gz 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46722.log 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43735.log.gz 2024-03-04T10:12:38.005 INFO:teuthology.orchestra.run.smithi099.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.73565.log.gz 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69025.log 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59422.log: /var/log/ceph/ceph-client.admin.46722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52549.log 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59422.log.gz 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69025.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46722.log.gz 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60719.log 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52549.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69025.log.gz 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71833.log 2024-03-04T10:12:38.006 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52549.log.gz 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45714.log 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60719.log: /var/log/ceph/ceph-client.admin.71833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75873.log 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60719.log.gz 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45714.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.71833.log.gz 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64625.log 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45714.log.gz 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46860.log 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71796.log 2024-03-04T10:12:38.007 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75873.log.gz 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64625.log.gz 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76083.log 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46860.log.gz 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68019.log 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51923.log 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.71796.log.gz 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68019.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76083.log.gz 2024-03-04T10:12:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74098.log 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68019.log.gz 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75165.log 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46523.log 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51923.log.gz 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75165.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.74098.log.gz 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75830.log 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75165.log.gz 2024-03-04T10:12:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57847.log 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65363.log 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46523.log.gz 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75830.log.gz 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59210.log 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65405.log 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59210.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.57847.log.gz 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65363.log.gz 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56254.log 2024-03-04T10:12:38.010 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59210.log.gz 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76437.log 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65405.log.gz 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65535.log 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56356.log 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56254.log.gz 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76437.log.gz 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45576.log 2024-03-04T10:12:38.011 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65535.log.gz 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53319.log 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61608.log 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56356.log.gz 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.53319.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45576.log.gz 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.53319.log.gz 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70118.log 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:gzip/var/log/ceph/ceph-client.admin.61608.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51796.log 2024-03-04T10:12:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53581.log 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61608.log.gz 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51796.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70118.log.gz 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76143.log 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51796.log.gz 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61075.log 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47954.log 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53581.log.gz 2024-03-04T10:12:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61075.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76143.log.gz 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71056.log 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62397.log 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61075.log.gz 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71056.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47954.log.gz 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45661.log 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73812.log 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62397.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.71056.log.gz 2024-03-04T10:12:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74046.log 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62397.log.gz 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73812.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45661.log.gz 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53827.log 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74046.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73812.log.gz 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64481.log 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74046.log.gz 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53827.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.53827.log.gz 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-04T10:12:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69435.log 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64481.log.gz 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58523.log 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73424.log 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69435.log.gz 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69603.log 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64313.log 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73424.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58523.log.gz 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70321.log 2024-03-04T10:12:38.016 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73424.log.gz 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64313.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69603.log.gz 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71574.log 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70321.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64313.log.gz 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57162.log 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70321.log.gz 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62578.log 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71574.log: /var/log/ceph/ceph-client.admin.57162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46189.log 2024-03-04T10:12:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62578.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.71574.log.gz 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3%gzip 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gz -5 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57162.log.gz --verbose 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.52463.log 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67809.log 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46189.log: /var/log/ceph/ceph-client.admin.52463.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.62578.log.gz 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71722.log 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46189.log.gz 2024-03-04T10:12:38.018 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52463.log.gz 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52593.log 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71722.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.67809.log.gz --verbose 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.74913.log 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66710.log 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71722.log.gz 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74913.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52593.log.gz 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48871.log 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74913.log.gz 2024-03-04T10:12:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57763.log 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.48871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64920.log 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66710.log.gz 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57763.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48871.log.gz 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46186.log 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57763.log.gz 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54902.log 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58849.log 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64920.log.gz 2024-03-04T10:12:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54902.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.46186.log.gz 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72404.log 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72172.log 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54902.log.gz 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72404.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58849.log.gz 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50458.log 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60071.log 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72172.log: /var/log/ceph/ceph-client.admin.50458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63926.log 2024-03-04T10:12:38.021 INFO:teuthology.orchestra.run.smithi099.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.72404.log.gz 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60071.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50458.log.gz 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55118.log 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63926.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60071.log.gz 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58945.log 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63926.log.gz 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55118.log: gzip 92.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.72172.log.gz --verbose 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.53901.log 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55118.log.gz 2024-03-04T10:12:38.022 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47352.log 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68886.log 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53901.log.gz 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73499.log 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.58945.log.gz 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47352.log.gz 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45456.log 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-03-04T10:12:38.023 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68886.log.gz 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45456.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73499.log.gz 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51526.log 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43819.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45456.log.gz 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57257.log 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67091.log 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51526.log.gz 2024-03-04T10:12:38.024 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50338.log 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67091.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.57257.log.gz 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52253.log 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67091.log.gz 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47310.log 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50338.log.gz 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64187.log 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58705.log 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52253.log.gz 2024-03-04T10:12:38.025 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47310.log.gz 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56609.log 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64187.log.gz 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67421.log 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56609.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58705.log.gzgzip 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61033.log 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67421.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56609.log.gz 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76523.log 2024-03-04T10:12:38.026 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67421.log.gz 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49501.log 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61033.log.gz 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74338.log 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76523.log.gz 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74224.log 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5%/var/log/ceph/ceph-client.admin.74338.log: -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52982.log 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74338.log.gz 2024-03-04T10:12:38.027 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69392.log 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52982.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74224.log.gz 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71352.log 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52982.log.gz 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69195.log 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69392.log.gz 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56086.log 2024-03-04T10:12:38.028 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62764.log 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.71352.log.gz 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56086.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69195.log.gz 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56002.log 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56086.log.gz 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56002.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.62764.log.gz 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68377.log 2024-03-04T10:12:38.029 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53132.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56002.log.gz 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59777.log 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53997.log 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59777.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68377.log.gz 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69153.log 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59777.log.gz 2024-03-04T10:12:38.030 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57206.log 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53997.log.gz 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46861.log 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69153.log.gz 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66854.log 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57206.log: /var/log/ceph/ceph-client.admin.46861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68249.log 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57206.log.gz 2024-03-04T10:12:38.031 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46861.log.gz 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62091.log 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66854.log.gz 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62686.log 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62091.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68249.log.gz -5 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52814.log 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62686.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62091.log.gz 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72784.log 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62686.log.gz 2024-03-04T10:12:38.032 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72784.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52814.log.gz 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44190.log 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.72784.log.gz 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73658.log 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44190.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz -5 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.66440.log 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44190.log.gz 2024-03-04T10:12:38.033 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65279.log 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73658.log.gz 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45230.log 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56483.log 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65279.log.gz 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45230.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.66440.log.gz 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54336.log 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45372.log 2024-03-04T10:12:38.034 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56483.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45230.log.gz 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.54336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54336.log.gz 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67599.log 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56483.log.gz 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51298.log 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67599.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.45372.log.gz -5 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51004.log 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51298.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67599.log.gz 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52898.log 2024-03-04T10:12:38.035 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51298.log.gz 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45288.log 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51004.log.gz 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63213.log 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52898.log.gz 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71019.log 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45288.log.gz 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70499.log 2024-03-04T10:12:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55741.log 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71019.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63213.log.gz 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46008.log 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.71019.log.gz 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55741.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.70499.log.gz 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54490.log 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55741.log.gz 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73953.log 2024-03-04T10:12:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54490.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46008.log.gz 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74608.log 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57679.log 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74608.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.54490.log.gz 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61938.log 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57679.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74608.log.gz 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46188.log 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57679.log.gz 2024-03-04T10:12:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.61938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61938.log.gz 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49585.log 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.73953.log.gz 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46190.log 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71500.log 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46188.log.gz 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51088.log 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49585.log.gz 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71500.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46190.log.gz 2024-03-04T10:12:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74650.log 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69350.log 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51088.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.71500.log.gz 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74650.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51088.log.gz 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49249.log 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74650.log.gz 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66668.log 2024-03-04T10:12:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49249.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69350.log.gz 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68122.log 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75651.log 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49249.log.gz 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68122.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66668.log.gz 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65714.log 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68122.log.gz 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45924.log 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65714.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.75651.log.gz 2024-03-04T10:12:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73029.log 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45924.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65714.log.gz -5 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63482.log 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45924.log.gz 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57427.log 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63482.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.73029.log.gz 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54448.log 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57427.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63482.log.gz 2024-03-04T10:12:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65117.log 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57427.log.gz 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47142.log 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65117.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54448.log.gz 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57343.log 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47142.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65117.log.gz 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47142.log.gz 2024-03-04T10:12:38.043 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73206.log 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71648.log 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57343.log.gz 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58565.log 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73206.log.gz 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49417.log 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.71648.log.gz 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52421.log 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58565.log: /var/log/ceph/ceph-client.admin.49417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47412.log 2024-03-04T10:12:38.044 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58565.log.gz 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52421.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49417.log.gz 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63968.log 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47412.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52421.log.gz 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63765.log 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47412.log.gz 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59336.log 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63765.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63968.log.gz 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62294.log 2024-03-04T10:12:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45104.log 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59336.log.gz 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62294.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.63765.log.gz 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51340.log 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62294.log.gz 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53497.log 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45104.log: /var/log/ceph/ceph-client.admin.51340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71463.log 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45104.log.gz 2024-03-04T10:12:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53497.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51340.log.gz 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52337.log 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53497.log.gz 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45966.log 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52337.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71463.log.gz 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65670.log 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45966.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52337.log.gz 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45966.log.gz 2024-03-04T10:12:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52003.log 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65670.log.gz 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53804.log 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.37706.log: /var/log/ceph/ceph-client.admin.52003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71241.log 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53804.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52003.log.gz 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68419.log 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53804.log.gz 2024-03-04T10:12:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75039.log 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71241.log: /var/log/ceph/ceph-client.admin.68419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56212.log 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68419.log.gz 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50253.log 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.71241.log.gz 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43651.log 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56212.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75039.log.gz 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50253.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.56212.log.gz 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.66261.log 2024-03-04T10:12:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72018.log 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50253.log.gz 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43651.log: /var/log/ceph/ceph-client.0.66261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.66261.log.gz 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59650.log 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43651.log.gz 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44148.log 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72018.log: /var/log/ceph/ceph-client.admin.59650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59945.log 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44148.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59650.log.gz 2024-03-04T10:12:38.050 INFO:teuthology.orchestra.run.smithi099.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.72018.log.gz 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64878.log 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59945.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44148.log.gz 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55657.log 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59945.log.gz 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47226.log 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64878.log.gz 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43308.log 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55657.log: /var/log/ceph/ceph-client.admin.47226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50626.log 2024-03-04T10:12:38.051 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55657.log.gz 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43308.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47226.log.gz 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49375.log 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71426.log 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50626.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43308.log.gz 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49375.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50626.log.gz 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70202.log 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64523.log 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71426.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49375.log.gz 2024-03-04T10:12:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76269.log 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71426.log.gz 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70202.log.gz 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65798.log 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54041.log 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64523.log.gz 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65798.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76269.log.gz 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60029.log 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65798.log.gz 2024-03-04T10:12:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56044.log 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45062.log 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60029.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54041.log.gz 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75957.log 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60029.log.gz 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56044.log.gz 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63361.log 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45062.log: /var/log/ceph/ceph-client.admin.75957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72265.log 2024-03-04T10:12:38.054 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45062.log.gz 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63361.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75957.log.gz 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67683.log 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67379.log 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.63361.log.gz 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72265.log.gz 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60496.log 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54968.log 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67683.log.gz 2024-03-04T10:12:38.055 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60496.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67379.log.gz 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61685.log 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60496.log.gz 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76395.log 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67935.log 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61685.log: /var/log/ceph/ceph-client.admin.76395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49543.log 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.54968.log.gz 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76395.log.gz 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52126.log 2024-03-04T10:12:38.056 INFO:teuthology.orchestra.run.smithi099.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.61685.log.gz 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49543.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.64565.log 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67935.log.gz 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51736.log 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8%/var/log/ceph/ceph-client.admin.64565.log: -- replaced with /var/log/ceph/ceph-client.admin.52126.log.gz 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51568.log 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64565.log.gz 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62354.log 2024-03-04T10:12:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51736.log: /var/log/ceph/ceph-client.admin.51568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45020.log 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51736.log.gz 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51568.log.gz 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58481.log 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46509.log 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45020.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62354.log.gz 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47622.log 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45020.log.gz 2024-03-04T10:12:38.058 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58481.log.gz 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61482.log 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70945.log 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46509.log.gz 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61482.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47622.log.gz 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50800.log 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70945.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61482.log.gz 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46855.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70945.log.gz 2024-03-04T10:12:38.059 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48829.log 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.3.log: gzip 89.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50800.log.gz -- 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.64010.log 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.48829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46518.log 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48829.log.gz 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62536.log 2024-03-04T10:12:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75788.log 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64010.log.gz 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62536.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.46518.log.gz -5 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49123.log 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62536.log.gz 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62896.log 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70982.log 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49123.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75788.log.gz 2024-03-04T10:12:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64980.log 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49123.log.gz 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62896.log.gz 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75417.log 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75609.log 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.70982.log.gz 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64980.log.gz 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69477.log 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75609.log: gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.75417.log.gz 2024-03-04T10:12:38.062 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65447.log 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75609.log.gz 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50296.log 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65447.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69477.log.gz 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48955.log 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50296.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65447.log.gz 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58213.log 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50296.log.gz 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.48955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59506.log 2024-03-04T10:12:38.063 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58213.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48955.log.gz 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64271.log 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43389.log 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59506.log.gz 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64271.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68930.log 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58213.log.gz 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43389.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64271.log.gz -5 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.74692.log 2024-03-04T10:12:38.064 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68930.log: gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.43389.log.gz /var/log/ceph/ceph-client.0.55305.log 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49333.log 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68930.log.gz 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9%/var/log/ceph/ceph-client.0.55305.log: -- replaced with /var/log/ceph/ceph-client.admin.74692.log.gz 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46520.log 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55305.log.gz 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51130.log 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49333.log: /var/log/ceph/ceph-client.admin.46520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52728.log 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49333.log.gz 2024-03-04T10:12:38.065 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51130.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46520.log.gz 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55699.log 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51130.log.gz 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52379.log 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52211.log 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52728.log.gz 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55699.log.gz 2024-03-04T10:12:38.066 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48699.log 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52211.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52379.log.gz 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73930.log 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.2.48699.log: 0.0% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52211.log.gz 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67641.log 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.2.48699.log.gz 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60847.log 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67641.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73930.log.gz 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46526.log 2024-03-04T10:12:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60847.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67641.log.gz 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65840.log 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60847.log.gz 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46526.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46526.log.gz 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69519.log 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60931.log 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65840.log.gz 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49627.log 2024-03-04T10:12:38.068 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76353.log 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60931.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69519.log.gz 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75999.log 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60931.log.gz 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76353.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53955.log 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49627.log.gz 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.75999.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.76353.log.gz 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.71611.log 2024-03-04T10:12:38.069 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49039.log 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53955.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75999.log.gz 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65966.log 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53955.log.gz 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49879.log 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.71611.log.gz 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45146.log 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49039.log.gz 2024-03-04T10:12:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65966.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.49879.log: -- /var/log/ceph/ceph-client.admin.70462.log 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65966.log.gz 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70160.log 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49879.log.gz 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45146.log.gz 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67047.log 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:gzip/var/log/ceph/ceph-client.admin.70160.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47100.log 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67047.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70160.log.gz 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.70462.log.gz 2024-03-04T10:12:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73340.log 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67047.log.gz 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51424.log 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75693.log 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47100.log.gz 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73340.log.gz 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44316.log 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:gzip/var/log/ceph/ceph-client.admin.75693.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60805.log 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51424.log.gz 2024-03-04T10:12:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73089.log 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75693.log.gz 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50926.log 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44316.log.gz 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73089.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60805.log.gz 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60246.log 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45840.log 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50926.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73089.log.gz 2024-03-04T10:12:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60246.log: gzip -5 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50926.log.gz 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64439.log 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45840.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60246.log.gz 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44424.log 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45840.log.gz 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.64439.log: -- /var/log/ceph/ceph-client.admin.46187.log 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44424.log: gzip 89.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.73990.log 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64439.log.gz 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44424.log.gz 2024-03-04T10:12:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59903.log 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70383.log 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73990.log: /var/log/ceph/ceph-client.admin.59903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56992.log 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73990.log.gz 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70383.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59903.log.gz 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59987.log 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70383.log.gz 2024-03-04T10:12:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56992.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45498.log 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56992.log.gz 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59987.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53539.log 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59987.log.gz 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68556.log 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53539.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56170.log 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45498.log.gz 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68556.log: 88.9%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.53539.log.gz -- 2024-03-04T10:12:38.076 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.75123.log 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59294.log 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68556.log.gz/var/log/ceph/ceph-client.admin.56170.log: 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5/var/log/ceph/ceph-client.admin.75123.log: --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56170.log.gz 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.72910.log 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59294.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75123.log.gz 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60540.log 2024-03-04T10:12:38.077 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59294.log.gz 2024-03-04T10:12:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47748.log 2024-03-04T10:12:38.078 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72910.log.gz 2024-03-04T10:12:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50416.log 2024-03-04T10:12:38.078 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60540.log.gz 2024-03-04T10:12:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73728.log 2024-03-04T10:12:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50416.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz 2024-03-04T10:12:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49921.log 2024-03-04T10:12:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50416.log.gz -5 2024-03-04T10:12:38.079 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.73602.log 2024-03-04T10:12:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73728.log: /var/log/ceph/ceph-client.admin.49921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55783.log 2024-03-04T10:12:38.079 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73728.log.gz/var/log/ceph/ceph-client.admin.73602.log: 2024-03-04T10:12:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51214.log 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49921.log.gz 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60763.log 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73602.log.gz 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55783.log: /var/log/ceph/ceph-client.admin.51214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45414.log 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51214.log.gz 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60763.log: -- replaced with /var/log/ceph/ceph-client.admin.55783.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55437.log 2024-03-04T10:12:38.080 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60763.log.gz 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr: -5/var/log/ceph/ceph-client.admin.45414.log: --verbose -- /var/log/ceph/ceph-client.admin.59464.log 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55437.log: gzip -5 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45414.log.gz 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.67337.log 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59464.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55437.log.gz 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63082.log 2024-03-04T10:12:38.081 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59464.log.gz 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62747.log 94.1% 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67337.log.gz 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43566.log 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50159.log 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62747.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63082.log.gz 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73770.log 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43566.log.gz 2024-03-04T10:12:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45188.log 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:gzip/var/log/ceph/ceph-client.admin.73770.log: -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50159.log.gz 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.62747.log.gz 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54681.log 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45188.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73770.log.gz 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31676.log 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45188.log.gz 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66626.log 2024-03-04T10:12:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph.tmp-client.admin.31676.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31676.log.gz 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57913.log 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51256.log 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66626.log: /var/log/ceph/ceph-client.admin.57913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59735.log 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66626.log.gz 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57913.log.gz 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47496.log 2024-03-04T10:12:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45882.log 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59735.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51256.log.gz 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72470.log 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59735.log.gz 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47496.log.gz 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.45882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57118.log 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72470.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45882.log.gz 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48913.log 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72470.log.gz/var/log/ceph/ceph-client.admin.57118.log: 2024-03-04T10:12:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49837.log 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57118.log.gz 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.48913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68717.log 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48913.log.gz 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74871.log 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68717.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49837.log.gz 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62217.log 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55052.log 2024-03-04T10:12:38.086 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68717.log.gz 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47268.log 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74871.log.gz 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62217.log.gz 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71389.log 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71389.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47268.log.gz 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60454.log 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55052.log.gz 2024-03-04T10:12:38.087 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47664.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71389.log.gz 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68636.log 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz/var/log/ceph/ceph-client.admin.60454.log: 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38062.log 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56651.log 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60454.log.gz 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68636.log.gz 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.38062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69238.log 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-04T10:12:38.088 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56651.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38062.log.gz 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50117.log 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56651.log.gz 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69238.log.gz 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65491.log 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71944.log 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50117.log: /var/log/ceph/ceph-client.admin.65491.log: gzip 89.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.73150.log 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50117.log.gz 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71944.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65491.log.gz 2024-03-04T10:12:38.089 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51694.log 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73150.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71944.log.gz 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73541.log 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.73150.log.gz 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59252.log 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51694.log: /var/log/ceph/ceph-client.admin.73541.log: gzip -5 89.3% --verbose -- /var/log/ceph/ceph-client.admin.67463.log 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51694.log.gz 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73541.log.gz 2024-03-04T10:12:38.090 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58279.log 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59252.log.gz 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57553.log 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58279.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67463.log.gz 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47538.log 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57553.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58279.log.gz 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57553.log.gz 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.69646.log 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43693.log 2024-03-04T10:12:38.091 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69646.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47538.log.gz 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71981.log 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44358.log 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43693.log.gz 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.69646.log.gz 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61328.log 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49669.log 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44358.log: -- replaced with /var/log/ceph/ceph-client.admin.71981.log.gz 2024-03-04T10:12:38.092 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.61328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72362.log 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.61328.log.gz 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5/var/log/ceph/ceph-client.admin.49669.log: --verbose -- 91.4% -- replaced with /var/log/ceph/ceph-client.admin.44358.log.gz 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr: /var/log/ceph/ceph-client.admin.68206.log 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72362.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49669.log.gz 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65321.log 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72362.log.gz 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67893.log 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68206.log: /var/log/ceph/ceph-client.admin.65321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70722.log 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68206.log.gz 2024-03-04T10:12:38.093 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67893.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65321.log.gz 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67505.log 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67893.log.gz 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49753.log 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70722.log: /var/log/ceph/ceph-client.admin.67505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70536.log 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.70722.log.gz 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49753.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67505.log.gz 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72952.log 2024-03-04T10:12:38.094 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8%/var/log/ceph/ceph-client.admin.70536.log: -- replaced with /var/log/ceph/ceph-client.admin.49753.log.gz 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49207.log 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.72952.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.70536.log.gz 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60677.log 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72952.log.gz 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr:gzip/var/log/ceph/ceph-client.admin.49207.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65628.log 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60677.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49207.log.gz 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.51172.log 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60677.log.gz 2024-03-04T10:12:38.095 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60369.log 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51172.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65628.log.gz 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60635.log 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56296.log 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51172.log.gz/var/log/ceph/ceph-client.admin.60369.log: 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54083.log 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60369.log.gz 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60635.log.gz 2024-03-04T10:12:38.096 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.56296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74422.log 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr:gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56296.log.gz 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54083.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.74524.log 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52295.log 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74422.log.gz 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74524.log: gzip 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54083.log.gz 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74524.log.gz 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62956.log 2024-03-04T10:12:38.097 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52295.log.gz 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52505.log 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43976.log 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58397.log 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62956.log.gz 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43976.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52505.log.gz 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53665.log 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43976.log.gz 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47184.log 2024-03-04T10:12:38.098 INFO:teuthology.orchestra.run.smithi099.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.58397.log.gz 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.53665.log: 86.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53665.log.gz 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48997.log 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47184.log.gz 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.48997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69712.log 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48997.log.gz 2024-03-04T10:12:38.099 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76227.log 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50542.log 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69712.log.gz 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64668.log 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.50542.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76227.log.gz 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.50542.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.66381.log 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51382.log 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64668.log: /var/log/ceph/ceph-client.admin.66381.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64668.log.gz 2024-03-04T10:12:38.100 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66381.log.gz 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49165.log 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51652.log 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51382.log: /var/log/ceph/ceph-client.admin.49165.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51382.log.gz 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49165.log.gz 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74566.log 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70871.log 2024-03-04T10:12:38.101 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74566.log: /var/log/ceph/ceph-client.admin.51652.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51652.log.gz 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74566.log.gz 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63669.log 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54825.log 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70871.log: /var/log/ceph/ceph-client.0.63669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.63669.log.gz 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70871.log.gz 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71759.log 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65197.log 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54825.log.gz 2024-03-04T10:12:38.102 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52686.log 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65197.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.71759.log.gz 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65197.log.gz 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57721.log 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46522.log 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.57721.log: /var/log/ceph/ceph-client.admin.52686.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52686.log.gz 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.57721.log.gz 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73888.log 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67851.log 2024-03-04T10:12:38.103 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46522.log: /var/log/ceph/ceph-client.admin.73888.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46522.log.gz 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.73888.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.70647.log 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51046.log 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70647.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67851.log.gz 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52772.log 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51046.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70647.log.gz 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51046.log.gz 2024-03-04T10:12:38.104 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71278.log 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46862.log 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52772.log: /var/log/ceph/ceph-client.admin.71278.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52772.log.gz 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60973.log 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.71278.log.gz 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.46862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55479.log 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.60973.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46862.log.gz 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60973.log.gz 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74787.log 2024-03-04T10:12:38.105 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74464.log 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55479.log: /var/log/ceph/ceph-client.admin.74787.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74787.log.gz 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.55479.log.gz 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73259.log 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61751.log 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74464.log.gz 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72657.log 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61751.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73259.log.gz 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61751.log.gz 2024-03-04T10:12:38.106 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69899.log 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.72657.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66812.log 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.0.72657.log.gz 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.69899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52940.log 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.69899.log.gz 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76185.log 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66812.log.gz 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.52940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52644.log 2024-03-04T10:12:38.107 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76185.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52940.log.gz 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76185.log.gz 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49459.log 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73113.log 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49459.log: /var/log/ceph/ceph-client.admin.52644.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52644.log.gz 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49459.log.gz 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73382.log 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.73113.log: /var/log/ceph/ceph-client.admin.73382.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.73382.log.gz 2024-03-04T10:12:38.108 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68461.log 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47790.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.73113.log.gz 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58439.log 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63040.log 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58439.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68461.log.gz 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58439.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45756.log 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69301.log 2024-03-04T10:12:38.109 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63040.log: /var/log/ceph/ceph-client.admin.45756.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63040.log.gz 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45756.log.gz 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55381.log 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69301.log: /var/log/ceph/ceph-client.admin.43863.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69301.log.gz 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64836.log 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44829.log 2024-03-04T10:12:38.110 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64836.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.55381.log.gz 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74955.log 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64836.log.gz 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49711.log 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74955.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74955.log.gz 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose 92.5% -- replaced with /var/log/ceph/ceph-client.admin.44829.log.gz 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr: -- /var/log/ceph/ceph-client.admin.64397.log 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71130.log 2024-03-04T10:12:38.111 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49711.log: /var/log/ceph/ceph-client.admin.64397.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49711.log.gz 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64397.log.gz 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70610.log 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66770.log 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.71130.log.gz/var/log/ceph/ceph-client.admin.70610.log: 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65924.log 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.66770.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70610.log.gz 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64229.log 2024-03-04T10:12:38.112 INFO:teuthology.orchestra.run.smithi099.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66770.log.gz 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43907.log 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.64229.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65924.log.gz 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44106.log 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64229.log.gz 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.43907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67133.log 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.44106.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43907.log.gz 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44106.log.gz 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52168.log 2024-03-04T10:12:38.113 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69561.log 2024-03-04T10:12:38.114 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67133.log: /var/log/ceph/ceph-client.admin.52168.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52168.log.gz 2024-03-04T10:12:38.114 INFO:teuthology.orchestra.run.smithi099.stderr: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47580.log 2024-03-04T10:12:38.114 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67133.log.gz 2024-03-04T10:12:38.114 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.69561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68293.log 2024-03-04T10:12:38.114 INFO:teuthology.orchestra.run.smithi099.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69561.log.gz 2024-03-04T10:12:38.114 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.47580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74295.log 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47580.log.gz 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.68293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59011.log 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.74295.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68293.log.gz 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74295.log.gz 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61524.log 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62854.log 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59011.log.gz 2024-03-04T10:12:38.115 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.61524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76041.log 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61524.log.gz 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67217.log 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62854.log.gz 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.76041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43485.log 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67217.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76041.log.gz 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65022.log 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.67217.log.gz/var/log/ceph/ceph-client.admin.43485.log: 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63884.log 2024-03-04T10:12:38.116 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43485.log.gz 2024-03-04T10:12:38.117 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59380.log 2024-03-04T10:12:38.117 INFO:teuthology.orchestra.run.smithi099.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65022.log.gz 2024-03-04T10:12:38.117 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.63884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58765.log 2024-03-04T10:12:38.117 INFO:teuthology.orchestra.run.smithi099.stderr: 89.0%/var/log/ceph/ceph-client.admin.59380.log: -- replaced with /var/log/ceph/ceph-client.admin.63884.log.gz 2024-03-04T10:12:38.117 INFO:teuthology.orchestra.run.smithi099.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49291.log 2024-03-04T10:12:38.117 INFO:teuthology.orchestra.run.smithi099.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59380.log.gz 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.58765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48268.log 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49291.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58765.log.gz 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49291.log.gz 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59693.log 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65586.log 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.48268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.48268.log.gz 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.59693.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-04T10:12:38.118 INFO:teuthology.orchestra.run.smithi099.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59693.log.gz 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.65586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70076.log 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-mds.a.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65586.log.gz 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49081.log 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.70076.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70076.log.gz 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58100.log 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.49081.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49081.log.gz 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55876.log 2024-03-04T10:12:38.119 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.0.58100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.58100.log.gz 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54765.log 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.55876.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55876.log.gz 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67725.log 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.54765.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54765.log.gz 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51466.log 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.67725.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67725.log.gz 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62998.log 2024-03-04T10:12:38.120 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.51466.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51466.log.gz 2024-03-04T10:12:38.121 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62644.log 2024-03-04T10:12:38.121 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62998.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62998.log.gz 2024-03-04T10:12:38.121 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71870.log 2024-03-04T10:12:38.121 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.62644.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62644.log.gz 2024-03-04T10:12:38.121 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/ceph-client.admin.71870.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.71870.log.gz 2024-03-04T10:12:38.122 INFO:teuthology.orchestra.run.smithi162.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-04T10:12:38.122 INFO:teuthology.orchestra.run.smithi099.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-04T10:12:38.122 INFO:teuthology.orchestra.run.smithi099.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-04T10:12:38.259 INFO:teuthology.orchestra.run.smithi099.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-04T10:12:38.709 INFO:teuthology.orchestra.run.smithi162.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-04T10:12:38.728 INFO:teuthology.orchestra.run.smithi099.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-04T10:12:38.730 INFO:teuthology.orchestra.run.smithi162.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-04T10:12:38.731 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-04T10:12:38.802 INFO:teuthology.orchestra.run.smithi099.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-04T10:12:38.858 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-04T10:12:38.860 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-04T10:12:38.860 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m1.194s 2024-03-04T10:12:38.860 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m4.816s 2024-03-04T10:12:38.860 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m0.191s 2024-03-04T10:12:38.936 INFO:teuthology.orchestra.run.smithi099.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-04T10:12:39.031 INFO:teuthology.orchestra.run.smithi099.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-04T10:12:39.033 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-03-04T10:12:39.033 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m1.399s 2024-03-04T10:12:39.034 INFO:teuthology.orchestra.run.smithi099.stderr:user 0m5.455s 2024-03-04T10:12:39.034 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m0.523s 2024-03-04T10:12:39.034 INFO:tasks.ceph:Archiving logs... 2024-03-04T10:12:39.034 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/remote/smithi099/log 2024-03-04T10:12:39.035 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T10:12:39.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-04T10:12:39.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-04T10:12:39.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-04T10:12:39.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-04T10:12:40.355 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/remote/smithi162/log 2024-03-04T10:12:40.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T10:12:41.058 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T10:12:41.067 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T10:12:41.067 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T10:12:41.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T10:12:41.161 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T10:12:41.170 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T10:12:41.171 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T10:12:41.174 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T10:12:41.187 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-03-04T10:12:41.190 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-04T10:12:41.191 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T10:12:41.192 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-03-04T10:12:41.192 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +1314us[+1314us] +/- 106ms 2024-03-04T10:12:41.192 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -1298us[-1270us] +/- 62ms 2024-03-04T10:12:41.192 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +709us[ +737us] +/- 75ms 2024-03-04T10:12:41.192 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:12:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T10:12:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-04T10:12:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +1309us[+1309us] +/- 106ms 2024-03-04T10:12:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -1280us[-1238us] +/- 62ms 2024-03-04T10:12:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +720us[ +720us] +/- 75ms 2024-03-04T10:12:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T10:12:41.195 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T10:12:41.205 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T10:12:41.205 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T10:12:41.214 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T10:12:41.320 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T10:12:41.393 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T10:12:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:41.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T10:12:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:41.457 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T10:12:41.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T10:12:41.475 INFO:teuthology.task.internal:Duration was 964.751715 seconds 2024-03-04T10:12:41.475 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T10:12:41.485 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T10:12:41.485 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T10:12:41.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T10:12:41.529 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T10:12:41.540 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T10:12:41.885 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T10:12:41.885 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-03-04T10:12:41.885 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T10:12:41.904 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-03-04T10:12:41.905 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T10:12:41.928 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T10:12:41.929 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T10:12:41.948 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T10:12:41.999 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T10:12:41.999 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T10:12:42.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T10:12:42.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T10:12:42.152 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T10:12:42.153 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T10:12:42.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T10:12:42.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T10:12:42.218 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:12:42.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:12:42.244 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-03-04T10:12:42.273 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-03-04T10:12:42.295 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:12:42.320 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi099.front.sepia.ceph.com, flagging run as failed 2024-03-04T10:12:42.320 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:12:42.350 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi162.front.sepia.ceph.com, flagging run as failed 2024-03-04T10:12:42.350 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T10:12:42.360 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T10:12:42.361 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/remote/smithi099 2024-03-04T10:12:42.361 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T10:12:57.282 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-03-04T10:12:57.693 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900/remote/smithi099/coredump/1709547069.68637.core.gz: gzip compressed data, was "1709547069.68637.core", last modified: Mon Mar 4 10:12:49 2024, max compression, original size modulo 2^32 240340992 2024-03-04T10:12:57.693 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-03-04T10:12:58.748 INFO:teuthology.task.internal: core looks like: /tmp/tmpwxenjrim: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i a', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/bin/ceph-mds', platform: 'x86_64' 2024-03-04T10:12:58.776 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i a 2024-03-04T10:12:58.777 DEBUG:teuthology.orchestra.run.smithi099:> which 'ceph-mds -f --cluster ceph -i a' 2024-03-04T10:12:58.795 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/which: invalid option -- 'f' 2024-03-04T10:12:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/which: unrecognized option '--cluster' 2024-03-04T10:12:58.796 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph-mds 2024-03-04T10:12:58.796 INFO:teuthology.orchestra.run.smithi099.stdout:/usr/bin/ceph 2024-03-04T10:12:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:58.797 INFO:teuthology.orchestra.run.smithi099.stderr:/usr/bin/which: no a in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2024-03-04T10:12:58.798 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2024-03-04T10:12:58.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T10:12:58.993 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T10:12:58.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T10:12:59.397 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T10:12:59.398 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T10:12:59.404 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T10:12:59.419 INFO:teuthology.orchestra.run.smithi099.stdout: 265758 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest 2024-03-04T10:12:59.419 INFO:teuthology.orchestra.run.smithi099.stdout: 265782 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest/archive 2024-03-04T10:12:59.419 INFO:teuthology.orchestra.run.smithi099.stdout: 265818 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest/archive/syslog 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 265381 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 4 09:56 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 265382 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 4 09:56 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 265399 200 -rw-r--r-- 1 ubuntu ubuntu 204662 Mar 4 10:12 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 403968 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest/archive/audit 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 403970 204 -rw------- 1 ubuntu ubuntu 207286 Mar 4 10:12 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 265817 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 10:11 /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:12:59.420 INFO:teuthology.orchestra.run.smithi099.stdout: 265407 14124 -rw------- 1 root root 240328704 Mar 4 10:10 /home/ubuntu/cephtest/archive/coredump/1709547053.68638.core 2024-03-04T10:12:59.421 INFO:teuthology.orchestra.run.smithi099.stdout: 265410 16956 -rw------- 1 root root 240340992 Mar 4 10:11 /home/ubuntu/cephtest/archive/coredump/1709547069.68637.core 2024-03-04T10:12:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T10:12:59.422 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-04T10:12:59.422 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-04T10:12:59.434 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T10:12:59.478 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T10:12:59.493 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi099 with status 1: "which 'ceph-mds -f --cluster ceph -i a'" 2024-03-04T10:12:59.493 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T10:12:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: 266038 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest 2024-03-04T10:12:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: 266039 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest/archive 2024-03-04T10:12:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: 266041 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest/archive/syslog 2024-03-04T10:12:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: 265382 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 4 09:56 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-04T10:12:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: 265314 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Mar 4 09:56 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-04T10:12:59.501 INFO:teuthology.orchestra.run.smithi162.stdout: 265311 184 -rw-r--r-- 1 ubuntu ubuntu 185894 Mar 4 10:12 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T10:12:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: 406089 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 10:12 /home/ubuntu/cephtest/archive/audit 2024-03-04T10:12:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: 406091 128 -rw------- 1 ubuntu ubuntu 129866 Mar 4 10:12 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-04T10:12:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: 266040 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 10:11 /home/ubuntu/cephtest/archive/coredump 2024-03-04T10:12:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: 265401 17692 -rw------- 1 root root 241401856 Mar 4 10:11 /home/ubuntu/cephtest/archive/coredump/1709547100.45926.core 2024-03-04T10:12:59.502 INFO:teuthology.orchestra.run.smithi162.stdout: 265384 18648 -rw------- 1 root root 241381376 Mar 4 10:11 /home/ubuntu/cephtest/archive/coredump/1709547085.45927.core 2024-03-04T10:12:59.503 INFO:teuthology.orchestra.run.smithi162.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-04T10:12:59.527 DEBUG:teuthology.nuke:shortname: smithi099 2024-03-04T10:12:59.527 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T10:12:59.557 DEBUG:teuthology.nuke:shortname: smithi162 2024-03-04T10:12:59.557 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T10:12:59.561 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 09:48:08.685538', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBA7XaVI3qo8JUjP6Cs8BrbgZKbZvhNnSuf1DkTRSJ1Mvc+9SLi404KEQCk/cU3tb4mMjV3DbVDSXaavcQb4C3k='} 2024-03-04T10:12:59.577 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7580900', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 09:48:08.684168', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDrBdhyWDFx2GvNg6H8LTIqPp5KPcNiv2djc6oiZNzNgrfV3Tc5cSISLGWsTtOPG0rdezwu3vs+c12rnYDTUIG4='} 2024-03-04T10:12:59.609 INFO:teuthology.orchestra.console.smithi099:Power off 2024-03-04T10:12:59.609 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T10:12:59.628 INFO:teuthology.orchestra.console.smithi162:Power off 2024-03-04T10:12:59.628 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T10:12:59.638 DEBUG:teuthology.orchestra.console.smithi099:power off output: Chassis Power Control: Down/Off 2024-03-04T10:12:59.639 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:12:59.651 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-03-04T10:12:59.652 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:12:59.667 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-04T10:12:59.677 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-04T10:13:03.668 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:13:03.678 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:13:03.780 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is on 2024-03-04T10:13:03.790 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-04T10:13:07.783 DEBUG:teuthology.orchestra.console.smithi099:pexpect command: ipmitool -H smithi099.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:13:07.793 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:13:07.895 DEBUG:teuthology.orchestra.console.smithi099:check power output: Chassis Power is off 2024-03-04T10:13:07.902 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-04T10:13:07.997 INFO:teuthology.orchestra.console.smithi099:Power off completed 2024-03-04T10:13:11.904 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T10:13:12.017 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-03-04T10:13:12.117 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-03-04T10:13:12.222 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} duration: 964.7517149448395 failure_reason: 'Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage)' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=54c51a04d1ef4f5785aecd47b768d98d status: fail success: false 2024-03-04T10:13:12.223 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T10:13:12.304 INFO:teuthology.run:FAIL